Automatic Segmentation of Knee Joint Synovial Magnetic Resonance Images Based on 3D VNetTrans

doi:10.11938/cjmr20222988

Abstract

Abstract:

Knee joint is commonly hurt by rheumatoid arthritis (RA). Accurate segmentation of synovium is essential for the diagnosis and treatment of RA. This paper proposes an algorithm based on improved VNet for automatically segmenting knee joint synovial magnetic resonance images. Firstly, the knee joint magnetic resonance images of 39 patients with synovitis were preprocessed. VNetTrans was constructed by embedding Transformer at the bottom of VNet. The MemSwish activation function was used for training. The average Dice score of the final model is 0.758 5 and the HD is 24.6 mm. Compared with VNet, the proposed model increased Dice score by 0.083 6 and decreased HD by 10 mm. Experimental results demonstrated that the proposed algorithm achieved satisfying 3D segmentation of the synovial hyperplasia area in the knee magnetic resonance images. It can be utilized to facilitate the diagnosis and monitoring of RA.

Key words: magnetic resonance image, medical image segmentation, deep learning, synovitis

CLC Number:

O482.53

Ying-shan WANG, Ao-qi DENG, Jin-ling MAO, Zhong-qi ZHU, Jie SHI, Guang YANG, Wei-wei MA, Qing LU, Hong-zhi WANG. Automatic Segmentation of Knee Joint Synovial Magnetic Resonance Images Based on 3D VNetTrans[J]. Chinese Journal of Magnetic Resonance, 2022, 39(3): 303-315.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Table 3

Fig.6

Fig.7

Table 4

References 28

1	SMOLEN J S, ALETAHA D, BARTON A, et alRheumatoid arthritis[J]. Nat Rev Dis Primers, 2018,4 (1): 18001. doi: 10.1038/nrdp.2018.1
2	JIN S Y, LI M T, FANG Y F, et alChinese registry of rheumatoid arthritis (CREDIT): II. prevalence and risk factors of major comorbidities in Chinese patients with rheumatoid arthritis[J]. Arthritis ResTher, 2017,19 (1): 251.
3	中华医学会风湿病学分会2018中国类风湿关节炎诊疗指南[J]. 中华内科杂志, 2018,57 (4): 242- 251. doi: 10.3760/cma.j.issn.0578-1426.2018.04.004
	ASSOCIATION C R2018 Chinese guideline for the diagnosis and treatment of rheumatoid arthritis[J]. Chin J Intern Med, 2018,57 (4): 242- 251. doi: 10.3760/cma.j.issn.0578-1426.2018.04.004
4	SUGIMOTO H, TAKEDA A, KANO SAssessment of disease activity in rheumatoid arthritis using magnetic resonance imaging: quantification of pannus volume in the hands[J]. Bri J Rheumatol, 1998,37 (8): 854- 861. doi: 10.1093/rheumatology/37.8.854
5	OSTERGAARD MDifferent approaches to synovial membrane volume determination by magnetic resonance imaging: manual versus automated segmentation[J]. Rheumatology, 1997,36 (11): 1166- 1177. doi: 10.1093/rheumatology/36.11.1166
6	SAKASHITA T, KAMISHIMA T, KOBAYASHI Y, et alAccurate quantitative assessment of synovitis in rheumatoid arthritis using pixel-by-pixel, time-intensity curve shape analysis[J]. Br J Radiol, 2016,89 (1061): 20151000. doi: 10.1259/bjr.20151000
7	FOTINOS-HOYER A K, GUERMAZI A, JARA H, et alAssessment of synovitis in the osteoarthritic knee: comparison between manual segmentation, semiautomated segmentation, and semiquantitative assessment using contrast-enhanced fat-suppressed T1-weighted MRI[J]. Magn Reson Med, 2010,64 (2): 604- 609. doi: 10.1002/mrm.22401
8	PERRY T A, GAIT A, O’NEILL T W, et alMeasurement of synovial tissue volume in knee osteoarthritis using a semiautomated MRI-based quantitative approach[J]. Magn Reson Med, 2019,81 (5): 3056- 3064. doi: 10.1002/mrm.27633
9	WANG A, FRANKE A, WESARG S. Semi-automatic segmentation of JIA-induced inflammation in MRI images of ankle joints[C]// Medical Imaging 2019: Image Processing, SPIE, 2019, 10949: 875-881.
10	ANDERSEN J K H, PEDERSEN J S, LAURSEN M S, et alNeural networks for automatic scoring of arthritis disease activity on ultrasound images[J]. RMD open, 2019,5 (1): e000891. doi: 10.1136/rmdopen-2018-000891
11	CHRISTENSEN A B H, JUST S A, ANDERSEN J K H, et alApplying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients[J]. Ann Rheum Dise, 2020,79 (9): 1189- 1193. doi: 10.1136/annrheumdis-2019-216636
12	IQBAL I, SHAHZAD G, RAFIQ N, et alDeep learning-based automated detection of human knee joint's synovial fluid from magnetic resonance images with transfer learning[J]. IET Image Processing, 2020,14 (10): 1990- 1998. doi: 10.1049/iet-ipr.2019.1646
13	WONG L M, SHI L, XIAO F, et alFully automated segmentation of wrist bones on T2-weighted fat-suppressed MR images in early rheumatoid arthritis[J]. Quant Imag Med Surg, 2019,9 (4): 579. doi: 10.21037/qims.2019.04.03
14	魏小娜, 邢嘉祺, 王振宇, 等基于改进U-Net的关节滑膜磁共振图像的分割[J]. 计算机应用, 2020,40 (11): 3340- 3345.
	WEI X N, XIN J Q, WANG Z Y, et alMagnetic resonance image segmentation of articular synovium based on improved U-Net[J]. Journal of Computer Applications, 2020,40 (11): 3340- 3345.
15	王振宇, 王颖珊, 毛瑾玲, 等基于Dense-UNet++的关节滑膜磁共振图像分割[J]. 波谱学杂志, 2022,39 (2): 208- 219.
	WANG Z Y, WANG Y S, MAO J L, et alMagnetic resonance images segmentation of synovium based on Dense-UNet++[J]. Chinese J Magn Reson, 2022,39 (2): 208- 219.
16	WANG W X, CHEN C, DING M, et al. TransBTS: Multimodal brain tumor segmentation using transformer[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2021: 109-119.
17	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2015: 3431-3440.
18	RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional Networks for Biomedical Image Segmentation[C]// International Conference on Medical image computing and computer-assisted intervention, Cham: Springer, 2015: 234-241.
19	ÇIçEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]// International conference on medical image computing and computer-assisted intervention. Springer, 2016: 424-432.
20	MILLETARI F, NAVAB N, AHMADI S-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation[C]// 2016 fourth international conference on 3D vision (3DV), IEEE, 2016: 565-571.
21	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proceedings of the Advances in Neural Information Processing Systems, 2017: 6000-6010.
22	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[OL]. arXiv preprint arXiv: 2010.11929, 2020.
23	NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]// Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel: 2010: 807-814.
24	RAMACHANDRAN P, ZOPH B, LE Q V. Searching for activation functions[J]. arXiv preprint arXiv: 1710.05941, 2017.
25	TAN M, LE Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C]// International conference on machine learning, PMLR, 2019: 6105-6114.
26	YUAN L, CHEN Y, WANG T, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 558-567.
27	YUSHKEVICH P A, GAO Y, GERIG G. ITK-SNAP: An interactive tool for semi-automatic segmentation of multi-modality biomedical images[C]// 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2016: 3342-3345.
28	HATAMIZADEH A, TANG Y, NATH V, et al. Unetr: Transformers for 3d medical image segmentation[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 574-584.

模型	卷积核维度	Dice	HD/mm
UNet	2D	0.6489	89.8
VNet	2D	0.6640	52.5
UNet	3D	0.6528	72.2
VNet	3D	0.6749	34.6

模型	激活函数	Dice	HD/mm	训练耗时
UNet	ReLU	0.6528	72.2	23min
	Swish	0.6613	50.8	34min
	MemSwish	0.6670	37.4	30min
VNetTrans	ReLU	0.7140	33.6	5h58min
	Swish	0.7505	29.0	6h36min
	MemSwish	0.7585	24.6	6h21min

模型	Dice	Sensitivity	Specificity	RVD/%	HD/mm
UNet	0.6528	0.7828	0.9864	33.84	72.2
VNet	0.6749	0.7262	0.9908	9.21	34.6
TransBTS	0.6996	0.7186	0.9917	4.87	36.0
UNETR	0.6355	0.6560	0.9920	?3.31	68.8
VNetTrans	0.7585	0.7633	0.9943	?1.34	24.6

网络模型	激活函数	是否加入Transformer模块	Dice	Sensitivity	Specificity	RVD/%	HD/mm
VNet	ReLU	否	0.6749	0.7262	0.9908	9.21	34.6
VNet + MemSwish	MemSwish	否	0.6962	0.7580	0.9904	14.92	35.8
VNet + Transformer	ReLU	是	0.7140	0.7527	0.9913	2.52	33.6
This work	MemSwish	是	0.7585	0.7633	0.9943	?1.34	24.6

[1]	LIU Kewen, JIANG Yukang, CHEN Fang, CHEN Junfei, LU Yuan, CHEN Li, LIU Chaoyang. A Deep Learning-based Method for LF-NMR Relaxation Time Spectrum Inversion [J]. Chinese Journal of Magnetic Resonance, 2026, 43(3): 307-320.
[2]	MA Suchao, WANG Yuanjun. Research Progress on Diffusion Magnetic Resonance Imaging Noise Reduction Methods Based on Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2026, 43(3): 367-388.
[3]	XIE Xinyi, WANG Yuanjun. Research Progress on Super-resolution Reconstruction of Brain Diffusion Magnetic Resonance Images [J]. Chinese Journal of Magnetic Resonance, 2026, 43(2): 223-240.
[4]	LI Yinghao, WANG Lihui, WANG Sucheng, ZHU Zhongqi, HUANG Changdong, LI Renfeng, CAO Kaiming, HU Haiyang, JIA Yiming, LIANG Songtao, YANG Guang, LU Qing, WANG Hongzhi. Study on Pancreas Automatic Segmentation, Regional Quantification, and Diabetes Assessment [J]. Chinese Journal of Magnetic Resonance, 2025, 42(4): 378-389.
[5]	GAO Zhaoyao, ZHANG Zhan, HU Liangliang, XU Guangyu, ZHOU Sheng, HU Yuxin, LIN Zijie, ZHOU Chao. PMRI Image Reconstruction Method Based on Virtual Coils and GRAPPA-enhanced Network [J]. Chinese Journal of Magnetic Resonance, 2025, 42(4): 390-401.
[6]	ZHANG Mingyu, XIAO Sa, SHI Shengjie, ZHANG Xuecheng, ZHOU Xin. Research on a Multi-modal Enhanced Denoising Diffusion Model for Hyperpolarized ¹²⁹Xe MRI [J]. Chinese Journal of Magnetic Resonance, 2025, 42(4): 364-377.
[7]	CAO Fei, XU Qianqian, CHEN Hao, ZU Jie, LI Xiaowen, TIAN Jin, BAO Lei. An Intelligent Diagnosis Method for NIID Based on Cross Self-supervision and DWI [J]. Chinese Journal of Magnetic Resonance, 2025, 42(2): 154-163.
[8]	XUE Peiyang, GENG Chen, LI Yuxin, BAO Yifang, LU Yucheng, DAI Yakang. A Classification Method for Cerebral Aneurysms in TOF-MRA Based on Improved 3D ResNet50 Model [J]. Chinese Journal of Magnetic Resonance, 2025, 42(1): 56-66.
[9]	NING Xinzhou, HUANG Zhen, CHEN Xiqu, LIU Xinjie, CHEN Gang, ZHANG Zhi, BAO Qingjia, LIU Chaoyang. Research on Transformer Super-Resolution Reconstruction Algorithm for Ultrafast Spatiotemporal Encoding Magnetic Resonance Imaging [J]. Chinese Journal of Magnetic Resonance, 2024, 41(4): 454-468.
[10]	YANG Liming, WANG Yuanjun. Research Progress of Denoising Algorithms for Diffusion Tensor Images [J]. Chinese Journal of Magnetic Resonance, 2024, 41(3): 341-361.
[11]	Dai Junlong, He Cong, Wu Jie, Bian Yun. Pancreatic Cystic Neoplasms Segmentation Network Combining Dual Decoding and Global Attention Upsampling Modules [J]. Chinese Journal of Magnetic Resonance, 2024, 41(2): 151-161.
[12]	YANG Yu, CHEN Bo, WU Liubin, LIN Enping, HUANG Yuqing, CHEN Zhong. Spectrum Reconstruction for Laplace NMR: From Handcraft Regularization to Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2024, 41(2): 191-208.
[13]	CHANG Bo, SUN Haoyun, GAO Qingyu, WANG Lijia. Research Progress on Cardiac Segmentation in Different Modal Medical Images by Traditional Methods and Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2024, 41(2): 224-244.
[14]	XU Zhenshun, YUAN Xiaohan, HUANG Ziheng, SHAO Chengwei, WU Jie, BIAN Yun. Multi-source Feature Classification Model of Pancreatic Mucinous and Serous Cystic Neoplasms Based on Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2024, 41(1): 19-29.
[15]	LAI Jiawen, WANG Yuling, CAI Xiaoyu, ZHOU Lihua. Multidimensional Information Fusion Method for Meniscal Tear Classification Based on CNN-SVM [J]. Chinese Journal of Magnetic Resonance, 2023, 40(4): 423-434.