Does Your ViT Still Need U-Net for Segmentation?
2026-06-30 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors looked at how computers identify parts of medical images, which usually uses a special kind of model called U-Net. They explored newer models called Vision Transformers (ViTs) that can understand the whole image better without needing the usual U-Net shape. They created a new method called EoSeg that only uses the encoder part of ViTs, and tested it on many types of medical images with good results. Their work shows that the older U-Net decoder isn't always needed with these modern models.
Medical image segmentationU-NetVision Transformer (ViT)Encoder-only architectureSelf-attentionDense predictionQuery-based modelingMulti-level queryCTMRI
Authors
Xin Li, Wenhui Zhu, Xuanzhao Dong, Xiwen Chen, Yanxi Chen, Yujian Xiong, Hao Wang, Oana M. Dumitrascu, Yalin Wang
Abstract
Medical image segmentation is dominated by U-Net-style encoder-decoder architectures. Vision Transformers (ViTs) overcome the limited receptive field of convolutional networks through self-attention, enabling modeling of long-range dependencies. Early ViT-based segmentation methods typically retained U-Net-style decoders because pretrained ViT representations were insufficient to support accurate dense prediction. Recent advances in large-scale pretraining have redefined the representation capability of ViTs, reducing the reliance on U-Net-style decoder architectures in modern vision models. This prompts two questions: Is the U-Net paradigm still necessary for medical image segmentation? If not, how should an encoder-only segmentation framework be designed? Motivated by these questions, we explore key architectural choices for encoder-only medical image segmentation based on modern ViT backbones and establish a query-based encoder-only design with multi-level query modeling and learnable block fusion, realized in Encoder-only Segmentation (EoSeg). Extensive experiments across seven benchmark datasets spanning CT, MRI, histopathology, endoscopy, and dermoscopy validate the effectiveness of the proposed design across diverse medical imaging modalities, including mDice scores of 85.50% on Synapse, 91.73% on ACDC, and 93.27% on GlaS. The results demonstrate that a U-Net-style decoder is no longer necessary for medical image segmentation with modern ViT backbones and further show that EoSeg provides an effective encoder-only design. Code is available at: https://github.com/Retinal-Research/EoSeg