Learning Cross-Modality Representations from Multi-Modal Images
Research output: Contribution to journal › Journal article › Research › peer-review
Standard
Learning Cross-Modality Representations from Multi-Modal Images. / van Tulder, Gijs; de Bruijne, Marleen.
In: IEEE Transactions on Medical Imaging, Vol. 38, No. 2, 2019, p. 638-648.Research output: Contribution to journal › Journal article › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - JOUR
T1 - Learning Cross-Modality Representations from Multi-Modal Images
AU - van Tulder, Gijs
AU - de Bruijne, Marleen
PY - 2019
Y1 - 2019
N2 - Machine learning algorithms can have difficulties adapting to data from different sources, for example from different imaging modalities. We present and analyze three techniques for unsupervised cross-modality feature learning, using a shared autoencoder-like convolutional network that learns a common representation from multi-modal data. We investigate a form of feature normalization, a learning objective that minimizes crossmodality differences, and modality dropout, in which the network is trained with varying subsets of modalities. We measure the same-modality and cross-modality classification accuracies and explore whether the models learn modality-specific or shared features. This paper presents experiments on two public datasets, with knee images from two MRI modalities, provided by the Osteoarthritis Initiative, and brain tumor segmentation on four MRI modalities from the BRATS challenge. All three approaches improved the cross-modality classification accuracy, with modality dropout and per-feature normalization giving the largest improvement. We observed that the networks tend to learn a combination of cross-modality and modality-specific features. Overall, a combination of all three methods produced the most cross-modality features and the highest cross-modality classification accuracy, while maintaining most of the samemodality accuracy.
AB - Machine learning algorithms can have difficulties adapting to data from different sources, for example from different imaging modalities. We present and analyze three techniques for unsupervised cross-modality feature learning, using a shared autoencoder-like convolutional network that learns a common representation from multi-modal data. We investigate a form of feature normalization, a learning objective that minimizes crossmodality differences, and modality dropout, in which the network is trained with varying subsets of modalities. We measure the same-modality and cross-modality classification accuracies and explore whether the models learn modality-specific or shared features. This paper presents experiments on two public datasets, with knee images from two MRI modalities, provided by the Osteoarthritis Initiative, and brain tumor segmentation on four MRI modalities from the BRATS challenge. All three approaches improved the cross-modality classification accuracy, with modality dropout and per-feature normalization giving the largest improvement. We observed that the networks tend to learn a combination of cross-modality and modality-specific features. Overall, a combination of all three methods produced the most cross-modality features and the highest cross-modality classification accuracy, while maintaining most of the samemodality accuracy.
KW - Autoencoders
KW - Deep learning
KW - Representation learning
KW - Transfer learning
UR - http://www.scopus.com/inward/record.url?scp=85052900021&partnerID=8YFLogxK
U2 - 10.1109/TMI.2018.2868977
DO - 10.1109/TMI.2018.2868977
M3 - Journal article
C2 - 30188817
VL - 38
SP - 638
EP - 648
JO - I E E E Transactions on Medical Imaging
JF - I E E E Transactions on Medical Imaging
SN - 0278-0062
IS - 2
ER -
ID: 203054362