Research on Transformer Super-Resolution Reconstruction Algorithm for Ultrafast Spatiotemporal Encoding Magnetic Resonance Imaging

doi:10.11938/cjmr20243110

Abstract

Abstract:

Spatio-temporal encoding (SPEN) magnetic resonance imaging (MRI) is an ultrafast MRI technique. However, resolution of the original image acquired with SPEN is relatively low, requiring super-resolution reconstruction based on sequence physics principles to improve spatial resolution. As the existing SPEN super-resolution reconstruction algorithms based on deep learning have confined abilities to capture long-range dependencies, this paper proposes a transformer-based SPEN MRI super-resolution reconstruction algorithm. An encoder-decoder structure is adopted, and a transformer module is introduced to extract local context information and long-range dependencies of feature maps. Experimental results show that the proposed reconstruction method can reconstruct a super-resolution image with high spatial resolution and no aliasing artifacts from the low-resolution SPEN image without adding additional sampling points. Compared to the existing super-resolution methods, the proposed method achieves better results on both clinical and preclinical datasets.

Key words: ultrafast MRI, spatio-temporal encoding(SPEN), deep learning, super-resolution, image reconstruction

CLC Number:

O482.53

NING Xinzhou, HUANG Zhen, CHEN Xiqu, LIU Xinjie, CHEN Gang, ZHANG Zhi, BAO Qingjia, LIU Chaoyang. Research on Transformer Super-Resolution Reconstruction Algorithm for Ultrafast Spatiotemporal Encoding Magnetic Resonance Imaging[J]. Chinese Journal of Magnetic Resonance, 2024, 41(4): 454-468.

Figures/Tables 9

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig.6

Fig. 7

Fig. 8

Table 1

References 38

[1]	EMAM M M, HOUSSEIN E H, GHONIEM R M. A modified reptile search algorithm for global optimization and image segmentation: Case study brain MRI images[J]. Comput Biol Med, 2023, 152: 106404.
[2]	WALTER S S, FRITZ B, KIJOWSKI R, et al. 2D versus 3D MRI of osteoarthritis in clinical practice and research[J]. Skeletal Radiol, 2023, 52(11): 2211-2224. doi: 10.1007/s00256-023-04309-4 pmid: 36907953
[3]	TSAO J. Ultrafast imaging: principles, pitfalls, solutions, and applications[J]. J Magn Reson Imaging, 2010, 32(2): 252-266. doi: 10.1002/jmri.22239 pmid: 20677249
[4]	GIL R, FERNANDES F F, SHEMESH N. Neuroplasticity-driven timing modulations revealed by ultrafast functional magnetic resonance imaging[J]. NeuroImage, 2021, 225: 117446.
[5]	HIRSCH F W, FRAHM J, SORGE I, et al. Real-time magnetic resonance imaging in pediatric radiology—new approach to movement and moving children[J]. Pediatr Radiol, 2021, 51: 840-846.
[6]	MILON A, FLAMENT V, GUENICHE Y, et al. How to optimize breast MRI protocol? The value of combined analysis of ultrafast and diffusion-weighted MRI sequences[J]. Diagn Interv Imaging, 2023, 104(6): 284-291.
[7]	LIAO C, CAO X, CHO J, et al. Highly efficient MRI through multi-shot echo planar imaging[C]// Wavelets and Sparsity XVIII, 2019, 11138: 353-365.
[8]	DUONG S T, PHUNG S L, BOUZERDOUM A, et al. An unsupervised deep learning technique for susceptibility artifact correction in reversed phase-encoding EPI images[J]. Magn Reson Imaging, 2020, 71: 1-10. doi: S0730-725X(19)30732-5 pmid: 32407764
[9]	SHROT Y, FRYDMAN L. Spatially encoded NMR and the acquisition of 2D magnetic resonance images within a single scan[J]. J Magn Reson, 2005, 172(2): 179-190. pmid: 15649744
[10]	CAI S H, CAI C B, CHEN Z. Spatiotemporally encoded nuclear magnetic resonance and its applications[J]. J Xiamen University(Natural Science), 2021, 60(3): 542-554.
	蔡淑惠, 蔡聪波, 陈忠. 时空编码核磁共振方法及其应用[J]. 厦门大学学报(自然科学版), 2021, 60(3): 542-554.
[11]	TAL A, FRYDMAN L. Spatial encoding and the single-scan acquisition of high definition MR images in inhomogeneous fields[J]. J Magn Reson, 2006, 182(2): 179-194. pmid: 16843690
[12]	TAL A, FRYDMAN L. Single-scan multidimensional magnetic resonance[J]. Prog Nucl Magn Reson Spectrosc, 2010, 57(3): 241-292.
[13]	BEN-ELIEZER N, IRANI M, FRYDMAN L. Super-resolved spatially encoded single-scan 2D MRI[J]. Magn Reson Med, 2010, 63(6): 1594-1600.
[14]	CHEN Y, LI J, QU X, et al. Partial Fourier transform reconstruction for single-shot MRI with linear frequency-swept excitation[J]. Magn Reson Med, 2013, 69(5): 1326-1336. doi: 10.1002/mrm.24366 pmid: 22706702
[15]	CAI C, DONG J, CAI S, et al. An efficient de-convolution reconstruction method for spatiotemporal-encoding single-scan 2D MRI[J]. J Magn Reson, 2013, 228: 136-147. doi: 10.1016/j.jmr.2012.12.020 pmid: 23433507
[16]	CHEN L, LI J, ZHANG M, et al. Super-resolved enhancing and edge deghosting (SEED) for spatiotemporally encoded single-shot MRI[J]. Med Image Anal, 2015, 23(1): 1-14. doi: 10.1016/j.media.2015.03.004 pmid: 25910683
[17]	ZHONG S, CHEN M, WEI X, et al. Understanding aliasing effects and their removal in SPEN MRI: A k-space perspective[J]. Magn Reson Med, 2023, 90(1): 166-176. doi: 10.1002/mrm.29638 pmid: 36961093
[18]	LI Y J, YANG X Y, YANG X M. Magnetic resonance image reconstruction of multi-scale residual unet fused with attention mechanism[J]. Chinese J Magn Reson, 2023, 40(3):307-319.
	李奕洁, 杨馨雨, 杨晓梅. 融合注意力机制的多尺度残差Unet的磁共振图像重建[J]. 波谱学杂志, 2023, 40(3): 307-319.
[19]	LU Q Q, LIAN Z F, LI J L, et al. Magnetic resonance R₂^* parameter mapping of liver based on self-supervised deep neural network[J]. Chinese J Magn Reson, 2023, 40(3):258-269.
	陆琪琪, 连梓锋, 李嘉龙. 等. 基于自监督网络的肝脏磁共振R₂^*参数图像重建[J]. 波谱学杂志, 2023, 40(3): 258-269.
[20]	DE LEEUW DEN BOUTER M L, IPPOLITO G, O’REILLY T P, et al. Deep learning-based single image super-resolution for low-field MR brain images[J]. Sci Rep, 2022, 12(1): 6362. doi: 10.1038/s41598-022-10298-6 pmid: 35430586
[21]	SONG L, WANG Q, LIU T, et al. Deep robust residual network for super-resolution of 2D fetal brain MRI[J]. Sci Rep, 2022, 12(1): 406. doi: 10.1038/s41598-021-03979-1 pmid: 35013383
[22]	XIANG P C, CAI C B, WANG J C, et al. Super-resolved reconstruction method for spatiotemporally encoded magnetic resonance imaging based on deep neural network>[J]. Acta Phys Sin, 2022, 71(5):363-371.
	向鹏程, 蔡聪波, 王杰超. 等. 基于深度神经网络的时空编码磁共振成像超分辨率重建方法[J]. 物理学报, 2022, 71(5): 363-371.
[23]	CHEN X, WANG W, HUANG J, et al. Ultrafast water-fat separation using deep learning-based single-shot MRI[J]. Magn Reson Med, 2022, 87(6): 2811-2825.
[24]	CHOI S R, LEE M. Transformer architecture and attention mechanisms in genome data analysis: a comprehensive review[J]. Biology, 2023, 12(7): 1033.
[25]	RAHALI A, AKHLOUFI M A. End-to-end transformer-based models in textual-based NLP[J]. AI, 2023, 4(1): 54-110.
[26]	YIN Y, TANG Z, WENG H. Application of visual transformer in renal image analysis[J]. BioMed Eng OnLine, 2024, 23(1): 27. doi: 10.1186/s12938-024-01209-z pmid: 38439100
[27]	ZU B, CAO T, LI Y, et al. SwinT-SRNet: Swin transformer with image super-resolution reconstruction network for pollen images classification[J]. Eng Appl Artif Intell, 2024, 133: 108041.
[28]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]// International Conference on Learning Representations, 2021.
[29]	LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// IEEE/CVF International Conference on Computer Vision, 2021: 9992-10002.
[30]	CHEN H, WANG Y, GUO T, et al. Pre-trained image processing transformer[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 12294-12305.
[31]	LIANG J, CAO J, SUN G, et al. SwinIR: Image restoration using swin transformer[C]// IEEE/CVF International Conference on Computer Vision Workshops, 2021: 833-1844.
[32]	ZAMIR S W, ARORA A, KHAN S, et al. Restormer: Efficient transformer for high-resolution image restoration[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5718-5729.
[33]	WANG Z, CUN X, BAO J, et al. Uformer: A general u-shaped transformer for image restoration[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 17662-17672.
[34]	SCHMIDT R, SEGINER A, FRYDMAN L. Interleaved multishot imaging by spatiotemporal encoding: A fast, self-referenced method for high-definition diffusion and functional MRI[J]. Magn Reson Med, 2016, 75(5): 1935-1948. doi: 10.1002/mrm.25742 pmid: 26108165
[35]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Neural Information Processing Systems, 2017: 6000-6010.
[36]	WU H, XIAO B, CODELLA N, et al. Cvt: Introducing convolutions to vision transformers[C]// IEEE/CVF International Conference on Computer Vision, 2021: 22-31.
[37]	YUAN K, GUO S, LIU Z, et al. Incorporating convolution designs into visual transformers[C]// IEEE/CVF International Conference on Computer Vision, 2021: 559-568.
[38]	VAN ESSEN D C, SMITH S M, BARCH D M, et al. The WU-Minn human connectome project: an overview[J]. NeuroImage, 2013, 80: 62-79. doi: 10.1016/j.neuroimage.2013.05.041 pmid: 23684880

方法	PSNR/dB	SSIM	FLOPS	模型参数/MB
传统迭代重建	31.08	0.8928	-	-
U-Net重建	38.58	0.9812	5.48	5.3
Transformer (C=12)	38.85	0.9801	4.42	3.0
Transformer (C=24)	45.22	0.9928	16.14	11.6
Transformer (C=48)	53.31	0.9983	61.48	46.2
Transformer (T=1)	42.34	0.9877	8.71	6.8
Transformer (T=2)	45.22	0.9928	16.14	11.6
Transformer (T=3)	45.41	0.9928	23.57	16.4
Transformer (Without GDL)	42.83	0.9901	16.14	11.6
Transformer (With GDL)	45.22	0.9928	16.14	11.6