HERA

Abstract

Vision foundation models (VFMs) have achieved strong performance across various vision tasks. However, it still remains challenging to apply VFMs for cross-domain few-shot segmentation (CD-FSS), which segments objects of novel classes under domain shifts using only a few labeled exemplars. The challenge is mainly driven by two factors: (1) limited labeled exemplars per novel class relative to the scale of VFM pre-training, making the model prone to overfitting during retraining, and (2) target-domain shifts underrepresented during pre-training, inducing cross-domain inconsistency and layer-wise sensitivity.

To address these issues, we propose Hierarchical Exemplar Representation Adaptation (HERA), a three-stage select–regularize–calibrate VFM-based segmentation framework that learns effectively from limited labels and adapts to novel domains without source-data retraining. We first design Hierarchical Layer Selection (HLS) to adaptively identify the most informative VFM layer using a data-dependent Exemplar Transfer Risk (ETR) computed for each candidate layer. Then, Prior-Guided Regularization (PGR) regularizes interactions on the selected representation, yielding well-structured local signals for the subsequent stage. Furthermore, Pixel-wise Adaptive Calibration (PAC) combines the selected representation with the refined interaction maps to calibrate pixel-wise predictions, producing consistent masks.

Together, these stages form a hierarchical select–regularize–calibrate pipeline that guides frozen VFM features in new domains while fine-tuning less than 2.7% of parameters at test time. Extensive experiments show that HERA surpasses the state of the art by more than 4.1 mIoU across multiple CD-FSS benchmarks.

Method

Overview of HERA. Hierarchical Layer Selection (HLS) estimates the leave-one-out layer risk $\mathcal{R}_{\mathrm{layer}}$ and routes each episode to $\ell^\star$, selected from either a single-layer candidate $\ell_{\mathrm{single}}$ or a local-fusion candidate $\ell_{\mathrm{fusion}}$. At $\ell^\star$, Prior-Guided Regularization (PGR) calibrates self-attention with entropy-gated Gaussian priors, while Pixel-wise Adaptive Calibration (PAC) introduces residual logits $\ell_{\mathrm{sim}}$, $\ell_{\mathrm{attn}}$, and $\ell_{\mathrm{img}}$, which are fused with the base logit in a select–regularize–calibrate hierarchy. Here, $\mathcal{L}$, $\mathcal{U}$, and $\mathcal{H}$ denote the numbers of layers, fusion candidates, and attention heads, respectively.

BibTeX

@inproceedings{ma2026selective,
  title     = {Selective, Regularized, and Calibrated: Harnessing Vision Foundation Models for Cross-Domain Few-Shot Semantic Segmentation},
  author    = {Ma, Junyuan and Xiang, Xunzhi and Li, Wenbin and Fan, Qi and Gao, Yang},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages     = {12385--12395},
  year      = {2026}
}

Selective, Regularized, and Calibrated: Harnessing Vision Foundation Models for Cross-Domain Few-Shot Semantic Segmentation

Abstract

Method

Poster

BibTeX