Signal, Image and Video Processing Обработка сигналов

Multi-scale class token fusion of vision transformers for brain tumor classification

Manalı, Doğu, Demirel, Hasan
Springer Nature · 2026-11-13 · DOI 10.1007/s11760-026-05886-2

Аннотация

Accurate classification of brain tumors in magnetic resonance images is essential for diagnosis and treatment planning. Single-network approaches are sensitive to model capacity and data partitioning, whereas heterogeneous ensembles are computationally demanding. This study proposes a multi-scale class token fusion method that combines the Tiny, Small, and Base variants of the Vision Transformer. After fine-tuning, the backbones are frozen, and their class tokens are projected into the embedding space of a reference model by learnable linear mappings. The projected tokens are averaged and passed to the fixed classifier of the reference model; consequently, only 0.11 to 1.03 million parameters are trained. An analytical derivation shows that this formulation is equivalent to a jointly learned logit-level combination of linear classifiers with a convex objective and that its attainable decision functions are independent of the reference model up to a fixed bias. All pairwise and triple combinations were evaluated with three reference models by stratified five-fold cross-validation on a public four-class dataset of 7,023 magnetic resonance images and compared with soft voting by McNemar’s test with Holm correction. The individual backbones achieved mean accuracies between 95.83% and 97.65%, whereas all fusion configurations achieved between 98.49% and 98.82%. The fusion of the Tiny and Small variants attained the highest mean accuracy of 98.82%, an area under the receiver operating characteristic curve of 99.96%, and a weighted F1-score of 98.82%. Fusion significantly outperformed every individual backbone and was more accurate than soft voting in all twelve configurations. These findings demonstrate that lightweight fusion effectively exploits the complementary representations of Vision Transformers at different scales.

Читать у издателя →