波谱学杂志, 2023, 40(4): 423-434 doi: 10.11938/cjmr20233076

研究论文

基于CNN-SVM的多维度信息融合半月板撕裂分类方法

赖嘉雯1, 汪宇玲,1,*, 蔡晓宇1, 周丽华2

1.东华理工大学 信息工程学院,江西 南昌 330000

2.东华理工大学 医务所,江西 南昌 330000

Multidimensional Information Fusion Method for Meniscal Tear Classification Based on CNN-SVM

LAI Jiawen1, WANG Yuling,1,*, CAI Xiaoyu1, ZHOU Lihua2

1. School of Information Engineering, East China University of Technology, Nanchang 330000, China

2. School of Clinic, East China University of Technology, Nanchang 330000, China

通讯作者: * Tel: 13870690380, E-mail:wangyuling_119@vip.163.com.

收稿日期: 2023-07-24  

基金资助: 国家自然科学基金(62066003); 国家留学基金项目(CSC202208360143)

Corresponding authors: * Tel: 13870690380, E-mail:wangyuling_119@vip.163.com.

Received: 2023-07-24  

摘要

针对半月板计算机辅助诊断(CAD)系统中半月板撕裂形态各异带来的分类准确率低的问题,提出一种多维度信息融合网络(Multi-Dimensional Information Fusion Network,MDIFNet)模型的半月板撕裂分类方法.首先,使用由四个子网络所构成的卷积神经网络(Convolutional Neural Network,CNN)架构以获取不同视角、不同维度的半月板特征信息;同时,提出了多尺度注意力机制,丰富细粒度特征;最后,构建了基于支持向量机(Support Vector Machines,SVM)的多核模型作为最终的分类器.在MRNet数据集上的实验结果表明,本文提出方法的分类准确率达0.782,较现有先进的基于深度学习的半月板撕裂分类方法有一定提升.

关键词: 半月板撕裂; 多核学习; 多视图学习; 磁共振成像; 深度学习

Abstract

Aiming to address the problem of low classification accuracy caused by the different shapes of meniscus tears in the computer-aided diagnosis (CAD) system for meniscus, a multidimensional information fusion network (MDIFNet) model for menissus tear classification was proposed. Firstly, a convolutional neural network (CNN) architecture consisting of four sub-networks was used to obtain meniscus feature information from different perspectives and dimensions. Simultaneously, multi-scale attention mechanism was proposed to enrich fine-grained features. Finally, a multi kernel model based on support vector machines (SVM) was constructed as the final classifier. The experimental results on the MRNet dataset show that the proposed method has a meniscal tear classification accuracy of 0.782, which has promotion compared to the existing state-of-the-art meniscus tear classification methods based on deep learning.

Keywords: meniscal tear; multi kernel learning; multi-view learning; magnetic resonance imaging; deep learning

PDF (835KB) 元数据 多维度评价 相关文章 导出 EndNote| Ris| Bibtex  收藏本文

本文引用格式

赖嘉雯, 汪宇玲, 蔡晓宇, 周丽华. 基于CNN-SVM的多维度信息融合半月板撕裂分类方法[J]. 波谱学杂志, 2023, 40(4): 423-434 doi:10.11938/cjmr20233076

LAI Jiawen, WANG Yuling, CAI Xiaoyu, ZHOU Lihua. Multidimensional Information Fusion Method for Meniscal Tear Classification Based on CNN-SVM[J]. Chinese Journal of Magnetic Resonance, 2023, 40(4): 423-434 doi:10.11938/cjmr20233076

引言

半月板是位于股骨髁与胫骨平台之间的两个纤维软骨[1],其特定的解剖结构为膝关节提供生物力学功能[2,3].半月板撕裂通常由创伤和退行性病变所引起[4],在长时间得不到治疗的情况下将会增加患有骨性关节炎的风险[5].因此,能够准确且及时地诊断半月板是否存在撕裂,对后续治疗起到至关重要的作用[6].磁共振成像(Magnetic Resonance Imaging,MRI)能够多方位多平面捕捉膝关节细节[7,8],是判断评估半月板损伤的首选成像方式[9].由于MRI数据量大且医生需综合不同视角下的切片进行病理判断,因此基于MRI的半月板撕裂诊断过程非常耗时.

近年,计算机视觉领域不断拓展,计算机辅助诊断(Computer Aided Detection,CAD)技术得到一定的发展.当前针对半月板撕裂分类可以大致划分为两类方法:基于影像组学和基于深度学习的方法.前者根据任务需求,人工定义特征,再使用传统方法实现半月板撕裂分类.Boniatis等[10]在2008年的工作中,利用灰度共生矩阵的方法提取MRI图像的纹理特征,以区分健康和受损半月板.Köse[11]等提出了基于MRI图像的自动诊断系统以实现半月板撕裂分类.该研究利用基于直方图的边缘检测滤波和基于统计分割的方法进行定位,使用模板匹配进行分割,最后分析弯月面面积以对弯月面是否撕裂进行分类.Fu等[12]提出了一种用于半月板撕裂诊断的两阶段计算机系统.使用了带水平集的活动轮廓方法计算半月板区域的位置和形状,通过共生矩阵提取特征,用序列浮动前向选择进行相关特征选择,最后输入到支持向量机(Support Vector Machines,SVM)分类.

相比传统图像分析方法需要人工设计特征,深度学习能够通过数据集自动学习图像有效特征,并在图像中进行定位和识别病变,在医学图像处理与分析方面表现出良好的适用性[13-16].Bien等[17]在2018年率先提出了基于聚合二维卷积神经网络(Convolutional Neural Network,CNN)的全自动深度学习模型MRNet.Tsai等[18]提出了基于残差连接的改进CNN架构ELNet(Efficiently-Layered Network,ELNet),用于对膝关节MRI图像损伤进行分类.Fritz等[19]提出了全自动3D CNN,以关节镜手术报告为标准,使用冠状面和矢状面MRI图像对内外侧半月板撕裂进行分类.同样地,Rizk等[20]也提出一种基于3D CNN的深度学习模型,用于对冠状面和矢状面质子密度脂肪抑制加权图像的内外侧半月板撕裂分类.Lu等[21]提出一种多模态特征融合的深度学习模型,将传统特征与深度特征融合,判断是否存在半月板撕裂.Ma等[22]提出了一种渐进式级联卷积神经网络(Cascaded-Progressive Convolutional Neural Network,C-PCNN),用于诊断膝关节MRI图像中半月板损伤.Hung等[23]提出改进的YOLOv4(You Only Look Once version 4)以对半月板撕裂是否存在进行分类.

由于膝关节结构复杂,诊断需要花费大量的时间,且诊断结果受主观因素影响较大,因此,将深度学习方法应用到半月板MRI影像处理任务中,有助于利用网络快速提取图像特征并识别病变,辅助医生进行决策,提高诊断的准确性和效率.在先前的研究中表明,2D深度学习特征和3D深度学习特征均能有效的对半月板损伤进行表征.然而,基于2D CNN的半月板损伤分类方法实质上是从2D MRI切片中提取相应特征,却丢失了一定有效的3D空间信息;基于3D CNN的半月板损伤分类方法则需要消耗资源过多,占用大量显存.再者,单一视角的特征提取可能会导致信息缺失.此外,半月板在膝关节MRI图像中占比较小,存在大量干扰信息如骨骼、肌肉及其他软组织等,且半月板撕裂形态具有多样性,通常会造成分类模型准确率低等问题.

因此,本文提出了一种半月板撕裂分类方法,从膝关节MRI图像中提取不同维度的半月板特征信息进行融合,以对半月板是否存在撕裂进行分类.首先,通过膝关节MRI三个正交视图(轴向、冠状和矢状)中分别使用2D CNN以提取每个视图的2D特征,从多个视角进行2D切片的提取可以一定程度上弥补单视角2D切片所包含信息的不足.同时,使用3D特征分类效果较好的矢状面作为3D CNN的输入,保留图像中关键的空间信息.此外,本文提出了一种多尺度注意力机制,提升整体模型对半月板的特征标定能力.最后,本文研究了一种多核特征融合方法,以融合三个正交视图提取的深度学习特征以及在核空间中融合不同维度的特征信息,再通过SVM进行分类并进行了一系列实验以评估所提出的方法对半月板撕裂分类的性能.

1 基于CNN-SVM的多维度信息融合网络

本文提出基于CNN-SVM的多维度信息融合网络(Multi-Dimensional Information Fusion Network,MDIFNet),主要由三个2D残差网络(Residual Network,ResNet)分支和一个3D ResNet网络分支组成.2D网络分支负责提取MRI切片所包含的信息,由于单个视角的MRI切片中所包含特征信息量少,因此本文通过三个视角提取的2D信息以整合和丰富图像特征.同时,使用3D分支补充2D特征所丢失的空间信息.整个网络流程如图1所示,主要包括:1)对膝关节MRI图像进行预处理;2)对图像进行多维度特征提取;3)信息融合及判断半月板撕裂与否.

图1

图1   基于CNN-SVM的多维信息融合半月板撕裂分类网络(MDIFNet)

Fig. 1   A multidimensional information fusion meniscal tear classification network (MDIFNet) based on CNN-SVM


1.1 网络结构

由于半月板撕裂在核磁共振图像上呈现异常信号范围小,且与自然图像不同,医学图像一般需要针对某个特定的类别进行更加细粒度的分类,需要选择网络层数适中,且在半月板撕裂分类任务中能保证分类精度与速度的骨干网络.残差网络在各种任务中均能得到出色的结果,能兼顾模型参数量与分类性能,同时保证了推理速度.因此,本文使用了现有的残差网络框架,ResNet18[24]作为2D分支和3D分支的骨干网络进行特征提取.ResNet18结构如图2所示,在Block模块中使用$H(x)=F(x)+x$残差连接方式,将输入x和卷积层的输出$F(x)$直接相加,使特征图进行融合.

图2

图2   ResNet18结构图

Fig. 2   Structure diagram of ResNet18


对于每个2D分支,将一系列的MRI图像作为输入,对MRI图像系列中每个切片均进行深度特征提取,随后进行最大池化(MaxPool)操作将所提取到的特征组合.在3D分支中,将MRI图像系列作为输入,经过一系列卷积操作以提取3D特征.

1.2 多尺度注意力机制

在膝关节MRI图像中半月板只占小部分区域,存在大量骨骼、肌肉及其他软组织等干扰信息,需要抑制大量的无关冗余特性信息,以提升整体模型对半月板的特征提取能力.因此,本文将现有卷积块注意力机制(Convolutional Block Attention Module,CBAM)中的空间注意力模块改进,提出多尺度注意力机制(Multi-scale Convolutional Block Attention Module,MCBAM),如图3所示,以对半月板特征进行重点标定,从而提高网络的特征学习和表达能力.MCBAM添加至3个2D子网络,对骨干网络ResNet18中的残差结构块进行重构,MCBAM添加位置如图4所示.

图3

图3   MCBAM结构图

Fig. 3   Structure diagram of MCBAM


图4

图4   MCBAM所在位置

Fig. 4   Location of MCBAM


1.2.1 通道注意力模块

图5所示,在通道注意力模块中,将输入的的特征图记为$F\in {{R}^{C\times H\times W}}$,其中HW分别记为输入特征图的高和宽,输入特征图的通道数记为C.输入特征图分别经过全局最大池化(MaxPool)和全局平均池化(AvgPool)对全局信息进行提炼与压缩,分别生成${{S}_{1}}\in {{R}^{C\times 1\times 1}}$${{S}_{2}}\in {{R}^{C\times 1\times 1}}$两张特征图.将${{S}_{1}}$${{S}_{2}}$送入共享全连接层FC1、FC2及ReLu激活函数所构成的多层感知机(Multi-Layer Perception,MLP),随后将输出结果按通道进行相加和Sigmoid激活函数操作后得到通道注意力MC,再将输出结果与原图相乘.通道注意力过程公式表示如下:

${F}'=\sigma \{\text{MLP}[\text{AvgPool}(F)]+\text{MLP}[\text{MaxPool}(F)]\}\otimes F$

其中F表示输入特征图,AvgPool表示全局平均池化,MaxPool表示全局最大池化,MLP表示多层感知机,$\sigma $表示Sigmoid激活函数,${F}'$表示经过通道注意力机制所得特征图.

图5

图5   通道注意力模块

Fig. 5   Channel Attention Module


1.2.2 多尺度空间注意力模块

经过通道注意力机制权重标定的输出结果${F}'$作为空间注意力机制的输入,在通道维度上同样进行MaxPool和AvgPool操作,生成两个大小为1×H×W的特征图P1、P2.通过级联操作将得到的两个特征图进行拼接得到特征描述.为了满足MRI图像空间维度上提取更细粒度特征的需求,丰富空间注意力信息,将原本使用单一7×7的卷积核替换成3×3、5×5、7×7卷积核,分别对P3中的信息进行融合,经过1×1卷积对三个不同尺度卷积的输出进行级联后再通道压缩,通过Sigmoid激活函数操作得到多尺度空间注意力MS,最后将${F}'$与空间加权信息MS相乘得到输出结果,如图6所示.多尺度空间注意力过程公式表示如下:

${{M}_{3\times 3}}(F)={{f}^{3\times 3}}[\text{AvgPool}({F}');\text{MaxPool}({F}')]$
${{M}_{5\times 5}}(F)={{f}^{5\times 5}}[\text{AvgPool}({F}');\text{MaxPool}({F}')]$
${{M}_{7\times 7}}(F)={{f}^{7\times 7}}[\text{AvgPool}({F}');\text{MaxPool}({F}')]$
${F}''=\sigma \{{{f}^{1\times 1}}[{{M}_{3\times 3}}(F);{{M}_{5\times 5}}(F);{{M}_{7\times 7}}(F)]\}\otimes {F}'$

其中${F}'$表示经过通道注意力机制所得特征图,AvgPool表示全局平均池化,MaxPool表示全局最大池化,f表示卷积操作,$\sigma $表示Sigmoid激活函数,M表示经过卷积后所得特征权重,${F}''$表示经过多尺度空间注意力机制所得特征图.

图6

图6   多尺度空间注意力模块

Fig. 6   Multi-scale spatial attention module


1.3 SVM分类器

SVM是当前流行的的分类器之一,其基本思想是寻找能够将各类样本分开的最优超平面来实现分类.对输入的非线性训练样本$T\in \{({{x}_{i}},{{y}_{i}})\}$$(i=1,2,\cdots,N)$,其中${{x}_{i}}$为SVM的输入特征,${{y}_{i}}$为输入特征对应标签,N为输入特征数.一个非线性样本分类的超平面优化问题的数学模型为:

$\max \left\{ \frac{1}{2}||\omega |{{|}^{2}} \right\}+C\sum\nolimits_{i=1}^{m}{{{\xi }_{i}}}$

其约束条件为:

${{y}_{i}}(\omega {{x}_{i}}+b)\ge 1-{{\xi }_{i}},\text{ }i=1,2,\cdots,N$
${{\xi }_{i}}\ge 0,\text{ }i=1,2,\cdots,N$

其中,ω为超平面法向量;ξ为松弛变量;b为超平面的偏移量;C为惩罚因子,一般大于0,且取值越大,对分错样本的惩罚程度越大,训练样本的准确率越高,但是模型泛化能力降低.

通过使用拉格朗日乘子法(Lagrange Multiplier Method)得到最优分类决策函数为:

$f(x)=\text{sign}\left[ \sum\nolimits_{i=1}^{N}{a_{i}^{*}{{y}_{i}}K({{x}_{i}},{{y}_{i}})+{{b}^{*}}} \right]$

其中,$a_{i}^{*}$${{b}^{*}}$为拉格朗日乘子,$K({{x}_{i}},{{y}_{i}})$为核函数.

SVM方法中,可以通过核函数将原始样本映射到高维空间,避免样本与样本之间内积运算量增大的问题.核函数的合适与否将对SVM性能造成影响,常用核函数如表1所示.

表1   常用核函数

Table 1  Common kernel function

名称表达式
Linear Kernel$K(x,y)={{x}^{\text{T}}}y$
Polynomial Kernel$K(x,y)={{({{x}^{\text{T}}}y+c)}^{d}}$
Gaussian Kernel$K(x,y)=\exp (-\frac{||x-y|{{|}^{2}}}{2{{\sigma }^{2}}})$
Sigmoid Kernel$K(x,y)=\tanh (\beta {{x}^{\text{T}}}y+\theta )$

新窗口打开| 下载CSV


1.4 多核学习

对于图像分类,所提取特征通常都存在异构性,特别是不同视角、不同维度特征所对应的核函数未必相同,采用单个核函数对所有特征进行处理未必能得到最优映射.多核学习(Multiple Kernel Learning,MKL)[25]旨在利用多个基本核函数的凸组合代替单一核函数,将核函数选择问题转化为对组合系数的选择,通过将异构数据的不同特征分量利用对应的核函数进行映射,使数据在新的特征空间中得到更好的表达,以达到显著提高模型性能的作用.该方法弥补了单一核函数带来的局限性,同时能够很好地处理样本规模过大、数据在高维特征空间分布不平坦及提取到的特征异构性强等问题.

多核学习能够为异构数据源提供对应的核函数,在多视角数据领域被广泛应用[26].因此,本文引入多核学习方法,将MRI图像经过2D子网络和3D子网络后,为提取的不同特征分配核函数,采用线性加权的方式对多个核函数进行融合以形成合成核函数,然后基于合成核函数训练SVM模型,以解决半月板撕裂分类问题.

本文采用的合成核函数$K(x,z)$公式为

$K(x,z)=\sum\nolimits_{j=1}^{M}{{{\beta }_{j}}}{{K}_{j}}(x,z)$

其约束条件为:

${{\beta }_{j}}\ge 0\wedge \sum\nolimits_{j=1}^{M}{{{\beta }_{j}}}=1$

其中,${{K}_{j}}(x,z)$为基础核函数,${{\beta }_{j}}$为基础核函数对应的权重系数,$\wedge $表示且,M为基础核函数的总数.

2 实验部分

2.1 数据集

本文采用的是目前最大公开膝盖MRI数据集MRNet数据集[17],是由斯坦福大学医学中心在12年内收集的1 370例膝关节磁共振影像检查数据所组成.每个样本均包含轴向(质子加权序列)、冠状(T1加权序列)和矢状(T2加权序列)MRI视图.每个MRI切片的大小为256×256,序列中的切片数量为17~61之间(平均值为31.48,标准差为7.97).其中,数据被标记为一般异常、半月板撕裂和前交叉韧带撕裂是否存在(本文仅考虑半月板撕裂情况).在研究中,数据集被随机分为:1 130例训练集,120例验证集,120例测试集.图7展示了不同视角下正常半月板与半月板撕裂在MRI上的表现.

图7

图7   正常半月板与半月板撕裂在MRI上的表现

Fig. 7   The appearance of normal meniscus and meniscus tear on MRI


2.2 实验环境及数据预处理

与其他自然图像分类任务不同,医学图像的病理特征通常位于固定位置,膝关节MR图像包含了大量的骨骼和其他组织等信息,然而半月板的撕裂信号通常出现在股骨髁和胫骨之间的软骨区域,因此,可以通过裁剪图像子区域以减少对半月板撕裂分类无用的特征信息,同时降低输入数据维度.本文使用正方形从图像中心进行裁剪(图像原始尺寸为256×256),将图片尺寸统一为224×224.

为了进一步减少无效信息对模型分类的影响,本文对数据集的部分切片进行清理.由于在样本序列头部和尾部的切片大多为肌肉组织,且骨骼边缘模糊、软骨组织信息较少,因此本文对样本数据进行筛选,分别剔除每例样本头部和尾部10%的切片,通过筛选后的序列切片数量为15~49.在训练期间进行数据增强操作,每个训练样本在-25°到25°之间随机旋转,在-25像素到25像素之间随机移动,并且以50%的概率水平翻转,防止模型过拟合.

本实验的实验环境为NVIDIA RTX3080显卡,编程语言为Python3.9,使用的深度学习框架为PyTorch,使用学习速率自适应算法Adam作为优化器,初始学习率为1e-05,权重衰减率为0.01.批量大小(Batchsize)设置为1,迭代次数为50.

2.3 评价指标

本研究在测试图像上使用四个二分类指标,受试者工作特征(Receiver Operating Characteristic,ROC)曲线下区域面积(Area Under the Curve of ROC,AUC)、准确率(Accuracy)、灵敏度(Sensitivity)以及特异性(Specificity),来评价模型的分类性能.

$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+FP+\text{FN}}$
$\text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{FN}}$
$\text{Specificity}=\frac{\text{TN}}{\text{TN}+\text{FP}}$

其中TP,TN,FN和FP分别表示真阳性、真阴性、假阴性和假阳性半月板撕裂.同时,为了更直观的评估模型性能,引入了ROC曲线及AUC.

3 结果与讨论

3.1 2D特征分类实验结果分析

为了评估本文提出的MDIFNet模型的分类效果,本文利用MRNet数据集进行了一系列对比实验.针对2D三视图,本文分别设计了单核和多核MKL算法,通过MKL-SVM算法分别对单视图、双视图和三视图所提取到的深度特征进行融合后分类,所有视图的深度特征均为512维.表2为2D特征分类实验结果.

表2   2D特征分类实验结果

Table 2  Classification experimental results of two-dimensional feature

MKL-SVM特征ROC-AUCAccuracySensitivitySpecificity
单核2D轴面0.6610.6500.7500.573
2D矢状面0.7280.7160.7190.647
2D冠状面0.6790.6750.7120.648
多核2D轴面+2D矢状面0.7180.7190.7710.649
2D轴面+2D冠状面0.6810.6770.7590.635
2D冠状面+2D矢状面0.7290.7250.7720.659
2D三视图0.7400.7320.7750.664

新窗口打开| 下载CSV


表2实验数据对比来看,在采用单核MKL-SVM算法对单一特征进行半月板分类的结果中,矢状面在单核分类中准确度最高,为0.716,冠状面次之,准确度为0.675,轴面最差,准确度为0.650.从上述结果中看出,半月板撕裂在矢状面所提取的特征最有效.在多核MKL-SVM算法对多特征融合进行分类的结果中,矢状面特征与轴面和冠状面特征分别结合后,准确度均有小幅度上涨.而三个视角的MRI图像所提取的特征全部进行融合后分类,取得实验结果中最高的分类性能,准确度为0.732,灵敏度为0.775,特异度为0.664.从实验结果中可知,矢状面特征对于半月板撕裂分类效果较好,而轴面和冠状面对半月板表征信息视角不同,与矢状面所提取的特征进行融合,可以相互补充,丰富信息,从而取得最优的分类效果.

3.2 3D特征分类实验结果分析

3D特征已被证明在半月板撕裂分类中具有良好的效果,因此本文实验将轴面、矢状面、冠状面的3D深度特征使用单核进行分类,表3为3D特征分类实验结果.由表3可知,使用3D特征进行分类,矢状面提取特征的准确度最佳,为0.743,优于其他两个视角.与表2的单视角2D特征分类结果相比,3D卷积能够更加充分的保留半月板原有的空间信息(如连续切片之间的关系),能够学习到具有强代表性的特征,而这些特征对半月板撕裂大小、位置、方向等敏感性更低,因此基于3D特征的半月板撕裂分类的精度有所提升.通过实验数据分析,在3D子网络中选择分类效果较优的矢状面MRI图像作为输入,以此补充2D特征融合实验所丢失的关键空间信息.

表3   3D特征分类实验结果

Table 3  Classification experimental results of three-dimensional feature

特征ROC-AUCAccuracySensitivitySpecificity
3D轴面0.7040.7000.7310.676
3D矢状面0.7760.7430.7100.765
3D冠状面0.7330.7200.6780.723

新窗口打开| 下载CSV


3.3 不同分类器实验结果分析

表4为不同分类器实验结果.综合表4中二分类评价指标分析,在使用MKL-SVM作为分类器,各项指标均优于使用Softmax作为分类器的情况.由于多核学习能够为异构数据提供对应的核函数,将提取到的多特征映射到合成核函数空间,因此本文使用的MKL-SVM分类器更有利于对多视角多维度的特征进行分类.

表4   不同分类器实验结果

Table 4  Experimental results of different classifiers

特征分类器ROC-AUCAccuracySensitivitySpecificity
3D矢状面+2D三视图Softmax0.8310.7750.7620.795
MKL-SVM0.8440.7820.7690.808

新窗口打开| 下载CSV


3.4 不同注意力机制分类实验结果分析

为了验证本文提出的多尺度注意力机制(MCBAM)的有效性,分别评估了无注意力机制、现有的注意力机制及本文提出的多尺度注意力机制对半月板撕裂判断的性能.表5展示了不同注意力机制分类实验结果.

表5   不同注意力机制分类实验结果

Table 5  Experimental results of different attention mechanisms

特征注意力机制ROC-AUCAccuracySensitivitySpecificity
3D矢状面+2D三视图无注意力机制0.8260.7650.7520.790
SE0.8350.7720.7590.791
ECA0.8310.7730.7560.807
CBAM0.8330.7790.7650.798
MCBAM0.8440.7820.7690.808

新窗口打开| 下载CSV


表5可得,融入了注意力机制后,各项指标都有所提升,这说明采用注意力机制能够更加有效地标定到半月板撕裂的关键信息,提高模型的分类性能.由于SE(Squeeze-and-Excitation)和ECA(Efficient Channel Attention)注意力机制只关注图像通道方向的特征信息,而CBAM和MCBAM的方法则关注了通道与空间两个维度的特征信息,从而后者方法准确率更高.此外,本文使用的MCBAM在空间维度融合了多尺度信息,从而提取图片的显著特征更全面,因此,获得了最佳的分类结果.

3.5 不同方法的实验结果对比分析

为了证明本文提出算法的有效性,将本文提出的模型和一些现有相关研究工作进行对比,表6为不同方法在MRNet数据集上的分类实验结果对比.

表6   不同方法在MRNet数据集上的实验结果对比

Table 6  Comparison of experimental results using different methods on the MRNet dataset

研究模型ROC-AUCAccuracySensitivitySpecificity
Bien等[15]MRNet0.8430.7780.7500.799
Tsai等[16]ELNet0.8690.7750.8140.745
Dunnhofer等[27]ELNet0.8950.7610.8720.676
Shin等[28]AlexNet0.7920.7660.8540.673
本文MDIFNet0.8450.7820.7690.808

新窗口打开| 下载CSV


由于半月板主要由纤维软骨构成,然而MRI图像对软骨成像具有局限性.在这种情况下,与其他模型相比,本文提出的MDIFNet方法对膝关节MRI图像半月板撕裂的分类准确率更高,达到0.782.同时,MDIFNet的灵敏度与特异度达到较好的平衡,与本文研究目的相符.其余方法仅采用的2D切片进行特征提取,忽略了3D空间特征信息的重要性.本文提出的方法以更深的ResNet18模型为基础,通过对多维度、多视角特征进行融合,充分利用数据的差异性与互补性,同时引入了多尺度注意力机制,取得了较好的分类效果.

4 结论

针对膝关节MRI图像半月板撕裂的分类任务,本文提出了一种基于CNN-SVM的多维度信息融合方法.在特征提取阶段,选择了多视角MRI图像作为输入,在保证分类图像对象一致性的情况下,利用不同视角数据之间的差异性进行信息互补,从而得到更加全面的特征描述.在2D分支上嵌入了多尺度注意力机制,以对半月板撕裂特性进行重点标定及丰富细粒度特性信息.同时引入了3D分支获取半月板的3D空间特征,补充2D特征丢失的空间信息.在特征融合阶段,利用多核学习将4个特征集映射到合成核空间中进行分类,取得了较好的实验结果.此外,针对数据集较小的不足,采用了一系列数据增强方法在一定程度缓解模型过拟合的情况,提高了模型的泛化性.实验结果表明,本文提出的网络模型的准确率达到0.782,相比于现有的其他方法获得了更好的分类效果,适合用于MRI图像半月板撕裂研究.

本次研究中,仅在MRNet数据集进行实验和改进,后续将在其他的数据集上进行实验,验证模型的泛化能力.其次,当前模型进行的是半月板有无撕裂分类,半月板撕裂位置及严重程度等其他病理特征将是后续的研究方向.此外,由于年龄、职业、关节炎等人口统计学以及临床症状和体征对半月板损伤具有关联性,因此由单模态特征转化为多模态特征,或许有助于增加模型的临床意义.

利益冲突

参考文献

GHOSH P, TAYLOR T K F.

The Knee joint meniscus: a fibrocartilage of some distinction

[J]. Clin Orthop Relat R, 1987, 224: 52-63.

[本文引用: 1]

MARKES A R, HODAX J D, MA C B.

Meniscus form and function

[J]. Clin Sport Med, 2020, 39(1): 1-12.

DOI:S0278-5919(19)30068-7      PMID:31767101      [本文引用: 1]

The menisci are 2 fibrocartilaginous crescents anchored via bony and ligamentous attachments to surrounding structures. Their biochemical composition and multilayered structure make them ideal for converting compressive forces to tensile forces in addition to improving joint congruity and providing shock absorption to weight bearing. The medial meniscus maintains more attachments at both the horns and the midbody than the lateral meniscus, making it more susceptible to injury. Understanding of the gross anatomy, vascular anatomy, biochemical composition, and microstructure is key to understanding causes of meniscal pathology as well as treatment options for restoring its primary functions.Copyright © 2019 Elsevier Inc. All rights reserved.

HUTCHINSON I D, MORAN C J, POTTER H G, et al.

Restoration of the meniscus: form and function

[J]. Am J Sport Med, 2014, 42(4): 987-998.

DOI:10.1177/0363546513498503      URL     [本文引用: 1]

Over the past 2 decades there has been a profound shift in our perception of the role of the meniscus in the knee joint. Orthopaedic opinion now favors salvaging and restoring the damaged meniscus where possible. Basic science is characterizing its form (anatomy) and functionality (biological and biomechanical) in an attempt to understand the effect of meniscal injury and repair on the knee joint as a whole. The meniscus is a complex tissue and has warranted extensive basic science, translational, and clinical research to identify techniques to augment healing and even replace the meniscus. The application of quantitative magnetic resonance image sequencing to the meniscus and articular cartilage of the affected compartment promises to add a quantifiable outcome measure to the body of clinical evidence that supports restoration of the meniscus. This article discusses the recent advances and outcomes in the pursuit of meniscal restoration with particular focus on the use of augmentation strategies in meniscal repair, meniscal imaging, and translational strategies.

BEAUFILS P, PUJOL N.

Management of traumatic meniscal tear and degenerative meniscal lesions. Save the meniscus

[J]. Orthop Traumatol-Sur, 2017, 103(8): S237-S244.

[本文引用: 1]

OZEKI N, KOGA H, SEKIYA I.

Degenerative meniscus in knee osteoarthritis: from pathology to treatment

[J]. Life, 2022, 12(4): 603.

DOI:10.3390/life12040603      URL     [本文引用: 1]

Knee osteoarthritis is a common degenerative joint disease characterized by chronic knee pain and disability in daily living. The lesion can involve the cartilage as well as the synovium, bone, ligaments, and meniscus, indicating a complicated pathology for knee osteoarthritis. The association with the meniscus has recently attracted much attention. Meniscal tears can initiate and progress knee osteoarthritis, with deleterious effects on the important roles of the meniscus in load distribution, shock absorption, and stability of the knee joint. Degenerative meniscus lesions are commonly observed in elderly people, but they have less impact on the prognosis of osteoarthritis. However, they are often accompanied by meniscal extrusion, which substantially decreases the hoop function of the meniscus and increases the risk of knee osteoarthritis. When surgical treatment is necessary, meniscal tears should be repaired to the greatest extent possible to preserve meniscus function. Long-term studies show better clinical outcomes and less degenerative osteoarthritis changes following meniscal repair than following partial meniscectomy. For meniscal extrusion repair, centralization techniques have been proposed that involve suturing the meniscus-capsule complex to the edge of the tibial plateau. Advancements in orthobiologics, such as platelet-rich plasma or stem cell therapy, have the potential to prevent the initiation or progression of osteoarthritis.

OAKDEN-RAYNER L, CARNEIRO G, BESSEN T, et al.

Precision radiology: predicting longevity using feature engineering and deep learning methods in a radiomics framework

[J]. Sci Rep, 2017, 7(1): 1648.

DOI:10.1038/s41598-017-01931-w      [本文引用: 1]

Precision medicine approaches rely on obtaining precise knowledge of the true state of health of an individual patient, which results from a combination of their genetic risks and environmental exposures. This approach is currently limited by the lack of effective and efficient non-invasive medical tests to define the full range of phenotypic variation associated with individual health. Such knowledge is critical for improved early intervention, for better treatment decisions, and for ameliorating the steadily worsening epidemic of chronic disease. We present proof-of-concept experiments to demonstrate how routinely acquired cross-sectional CT imaging may be used to predict patient longevity as a proxy for overall individual health and disease status using computer image analysis techniques. Despite the limitations of a modest dataset and the use of off-the-shelf machine learning methods, our results are comparable to previous ‘manual’ clinical methods for longevity prediction. This work demonstrates that radiomics techniques can be used to extract biomarkers relevant to one of the most widely used outcomes in epidemiological and clinical research – mortality, and that deep learning with convolutional neural networks can be usefully applied to radiomics research. Computer image analysis applied to routinely collected medical images offers substantial potential to enhance precision medicine initiatives.

JIN W, LI X, FATEHI M, et al.

Guidelines and evaluation of clinical explainable AI in medical image analysis

[J]. Med Image Anal, 2023, 84: 102684.

DOI:10.1016/j.media.2022.102684      URL     [本文引用: 1]

FARUCH-BILFELD M, LAPÈGUE F, CHIAVASSA H, et al.

Imaging of meniscus and ligament injuries of the knee

[J]. Diagn Interv Imag, 2016, 97(7-8): 749-765.

[本文引用: 1]

NGUYEN J C, DE SMET A A, GRAF B K, et al.

MR imaging-based diagnosis and classification of meniscal tears

[J]. Radiographics, 2014, 34(4): 981-999.

DOI:10.1148/rg.344125202      PMID:25019436      [本文引用: 1]

Magnetic resonance (MR) imaging is currently the modality of choice for detecting meniscal injuries and planning subsequent treatment. A thorough understanding of the imaging protocols, normal meniscal anatomy, surrounding anatomic structures, and anatomic variants and pitfalls is critical to ensure diagnostic accuracy and prevent unnecessary surgery. High-spatial-resolution imaging of the meniscus can be performed using fast spin-echo and three-dimensional MR imaging sequences. Normal anatomic structures that can mimic a tear include the meniscal ligament, meniscofemoral ligaments, popliteomeniscal fascicles, and meniscomeniscal ligament. Anatomic variants and pitfalls that can mimic a tear include discoid meniscus, meniscal flounce, a meniscal ossicle, and chondrocalcinosis. When a meniscal tear is identified, accurate description and classification of the tear pattern can guide the referring clinician in patient education and surgical planning. For example, longitudinal tears are often amenable to repair, whereas horizontal and radial tears may require partial meniscectomy. Tear patterns include horizontal, longitudinal, radial, root, complex, displaced, and bucket-handle tears. Occasionally, meniscal tears can be difficult to detect at imaging; however, secondary indirect signs, such as a parameniscal cyst, meniscal extrusion, or linear subchondral bone marrow edema, should increase the radiologist's suspicion for an underlying tear. Awareness of common diagnostic errors can ensure accurate diagnosis of meniscal tears. Online supplemental material is available for this article.©RSNA, 2014.

BONIATIS I, PANAYIOTAKIS G, PANAGIOTOPOULOS E.

A computer-based system for the discrimination between normal and degenerated menisci from magnetic resonance images

[C]// 2008 IEEE International Workshop on Imaging Systems and Techniques, Chania, Greece: IEEE, 2008: 335-339.

[本文引用: 1]

KÖSE C, GENÇALIOĞLU O, ŞEVIK U.

An automatic diagnosis method for the knee meniscus tears in MR images

[J]. Expert Syst Appl, 2009, 36(2): 1208-1216.

DOI:10.1016/j.eswa.2007.11.036      URL     [本文引用: 1]

FU J C, LIN C C, WANG C N, et al.

Computer-aided diagnosis for knee meniscus tears in magnetic resonance imaging

[J]. J Ind Prod Eng, 2013, 30(2): 67-77.

[本文引用: 1]

MA Y, XING C J, XIAO L.

Knee joint image segmentation and model construction based on cascaded network

[J]. Chinese J Magn Reson, 2022, 39(2): 184-195.

[本文引用: 1]

马岩, 邢藏菊, 肖亮.

基于级联网络的膝关节图像分割与模型构建

[J]. 波谱学杂志, 2022, 39(02): 184-195.

[本文引用: 1]

LECUN Y, BENGIO Y, HINTON G.

Deep learning

[J]. Nature, 2015, 521(7553): 436-444.

DOI:10.1038/nature14539      [本文引用: 1]

ZEILER M D, FERGUS R.

Visualizing and understanding convolutional networks

[C]// Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland: Springer International Publishing, 2014: 818-833.

[本文引用: 2]

HU Y W, SU X Y, KE X T, et al.

The research progress of diagnosing meniscus injury in MRI based on deep learning

[J]. Chin J Magn Reson Imaging, 2022, 13(05):167-170.

[本文引用: 2]

胡伟艺, 苏娴彦, 柯晓婷, .

基于深度学习的MRI诊断半月板损伤的研究进展

[J]. 磁共振成像, 2022, 13(05): 167-170.

[本文引用: 2]

BIEN N, RAJPURKAR P, BALL R L, et al.

Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet

[J]. PLoS Med, 2018, 15(11): e1002699.

DOI:10.1371/journal.pmed.1002699      URL     [本文引用: 2]

TSAI C H, KIRYATI N, KONEN E, et al.

Knee injury detection using MRI with efficiently-layered network (ELNet)

[C]// Medical Imaging with Deep Learning. Montréel, Canada: PMLR, 2020: 784-794.

[本文引用: 1]

FRITZ B, MARBACH G, CIVARDI F, et al.

Deep convolutional neural network-based detection of meniscus tears: comparison with radiologists and surgery as standard of reference

[J]. Skeletal Radiol, 2020, 49: 1207-1217.

DOI:10.1007/s00256-020-03410-2      PMID:32170334      [本文引用: 1]

To clinically validate a fully automated deep convolutional neural network (DCNN) for detection of surgically proven meniscus tears.One hundred consecutive patients were retrospectively included, who underwent knee MRI and knee arthroscopy in our institution. All MRI were evaluated for medial and lateral meniscus tears by two musculoskeletal radiologists independently and by DCNN. Included patients were not part of the training set of the DCNN. Surgical reports served as the standard of reference. Statistics included sensitivity, specificity, accuracy, ROC curve analysis, and kappa statistics.Fifty-seven percent (57/100) of patients had a tear of the medial and 24% (24/100) of the lateral meniscus, including 12% (12/100) with a tear of both menisci. For medial meniscus tear detection, sensitivity, specificity, and accuracy were for reader 1: 93%, 91%, and 92%, for reader 2: 96%, 86%, and 92%, and for the DCNN: 84%, 88%, and 86%. For lateral meniscus tear detection, sensitivity, specificity, and accuracy were for reader 1: 71%, 95%, and 89%, for reader 2: 67%, 99%, and 91%, and for the DCNN: 58%, 92%, and 84%. Sensitivity for medial meniscus tears was significantly different between reader 2 and the DCNN (p = 0.039), and no significant differences existed for all other comparisons (all p ≥ 0.092). The AUC-ROC of the DCNN was 0.882, 0.781, and 0.961 for detection of medial, lateral, and overall meniscus tear. Inter-reader agreement was very good for the medial (kappa = 0.876) and good for the lateral meniscus (kappa = 0.741).DCNN-based meniscus tear detection can be performed in a fully automated manner with a similar specificity but a lower sensitivity in comparison with musculoskeletal radiologists.

RIZK B, BRAT H, ZILLE P, et al.

Meniscal lesion detection and characterization in adult knee MRI: a deep learning model approach with external validation

[J]. Phys Medica, 2021, 83: 64-71.

DOI:10.1016/j.ejmp.2021.02.010      URL     [本文引用: 1]

LU L X, ZHOU J Z, GUO Y C, et al.

Prediction of knee injury based on multimodal fusion

[J]. Comput Eng Appl, 2021, 57(09): 225-232.

[本文引用: 1]

陆莉霞, 邹俊忠, 郭玉成, .

多模态融合的膝关节损伤预测

[J]. 计算机工程与应用, 2021, 57(09): 225-232.

[本文引用: 1]

MA Y, QIN Y, LIANG C, et al.

Visual cascaded-progressive convolutional neural network (C-PCNN) for diagnosis of meniscus injury

[J]. Diagnostics, 2023, 13(12): 2049.

DOI:10.3390/diagnostics13122049      URL     [本文引用: 1]

Objective: The objective of this study is to develop a novel automatic convolutional neural network (CNN) that aids in the diagnosis of meniscus injury, while enabling the visualization of lesion characteristics. This will improve the accuracy and reduce diagnosis times. Methods: We presented a cascaded-progressive convolutional neural network (C-PCNN) method for diagnosing meniscus injuries using magnetic resonance imaging (MRI). A total of 1396 images collected in the hospital were used for training and testing. The method used for training and testing was 5-fold cross validation. Using intraoperative arthroscopic diagnosis and MRI diagnosis as criteria, the C-PCNN was evaluated based on accuracy, sensitivity, specificity, receiver operating characteristic (ROC), and evaluation performance. At the same time, the diagnostic accuracy of doctors with the assistance of cascade- progressive convolutional neural networks was evaluated. The diagnostic accuracy of a C-PCNN assistant with an attending doctor and chief doctor was compared to evaluate the clinical significance. Results: C-PCNN showed 85.6% accuracy in diagnosing and identifying anterior horn injury, and 92% accuracy in diagnosing and identifying posterior horn injury. The average accuracy of C-PCNN was 89.8%, AUC = 0.86. The diagnosis accuracy of the attending physician with the aid of the C-PCNN was comparable to that of the chief physician. Conclusion: The C-PCNN-based MRI technique for diagnosing knee meniscus injuries has significant practical value in clinical practice. With a high rate of accuracy, clinical auxiliary physicians can increase the speed and accuracy of diagnosis and decrease the number of incorrect diagnoses.

HUNG T N K, VY V P T, TRI N M, et al.

Automatic detection of meniscus tears using backbone convolutional neural networks on knee MRI

[J]. J Magn Reson Imaging, 2023, 57(3): 740-749.

DOI:10.1002/jmri.v57.3      URL     [本文引用: 1]

HE K, ZHANG X, REN S, et al.

Deep residual learning for image recognition

[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA: IEEE, 2016: 770-778.

[本文引用: 1]

BACH F R, LANCKRIET G R G, JORDAN M I.

Multiple kernel learning, conic duality, and the SMO algorithm

[C]// Proceedings of the twenty-first international conference on Machine learning. Banff, Alberta, Canada: ICML, 2004: 6.

[本文引用: 1]

SUBRAHMANYA N, SHIN Y C.

Sparse multiple kernel learning for signal processing applications

[J]. IEEE T Pattern Anal, 2009, 32(5): 788-798.

DOI:10.1109/TPAMI.2009.98      URL     [本文引用: 1]

DUNNHOFER M, MARTINEL N, MICHELONI C.

Improving MRI-based knee disorder diagnosis with pyramidal feature details

[C]// Medical Imaging with Deep Learning. Lübeck, Germany: PMLR, 2021: 131-147.

[本文引用: 1]

SHIN H, CHOI G S, SHON O J, et al.

Development of convolutional neural network model for diagnosing meniscus tear using magnetic resonance image

[J]. BMC Musculoskel Dis, 2022, 23(1): 1-9.

DOI:10.1186/s12891-021-04954-7      [本文引用: 1]

The aim of this study was to discuss the treatment of extra-articular distal humeral shaft fractures using ultrasound-guided preoperative localization of radial nerve.

/