No time to train!

The performance of image segmentation models has historically been constrained by the high cost of collecting large-scale annotated data. The Segment Anything Model (SAM) alleviates this original problem through a promptable, semantics-agnostic, segmentation paradigm and yet still requires manual visual-prompts or complex domain-dependent prompt-generation rules to process a new image. Towards reducing this new burden, our work investigates the task of object segmentation when provided with, alternatively, only a small set of reference images. Our key insight is to leverage strong semantic priors, as learned by foundation models, to identify corresponding regions between a reference and a target image. We find that correspondences enable automatic generation of instance-level segmentation masks for downstream tasks and instantiate our ideas via a multi-stage, training-free method incorporating (1) memory bank construction; (2) representation aggregation and (3) semantic-aware feature matching. Our experiments show significant improvements on segmentation metrics, leading to state-of-the-art performance on COCO FSOD (36.8% nAP), PASCAL VOC Few-Shot (71.2% nAP50) and outperforming existing training-free approaches on the Cross-Domain FSOD benchmark (22.4% nAP).

We employ a memory-based approach to store discriminative representations for object categories.

1. Constructing a memory bank from reference images:
   Details

   Given a set of reference images \( \{ I_r^j \}_{j=1}^{N_r} \) and their corresponding instance masks \( \{ M_r^{j,i} \}_{j=1}^{N_r} \) for category \( i \), we extract dense feature maps \( F_r^j \in \mathbb{R}^{H' \times W' \times d} \) using a pretrained frozen encoder \( \mathcal{E} \)^[link], where \( d \) is the feature dimension and \( H', W' \) denote the spatial resolution of the feature map. The corresponding instance masks \( M_r^{j,i} \in \{0,1\}^{H' \times W'} \) are resized to match this resolution. For each category \( i \), we store the masked features:

   \[ \mathcal{F}_r^{j,i} = F_r^j \odot M_r^{j,i} \]

   where \( \odot \) denotes element-wise multiplication. These category-wise feature sets are stored in a memory bank \( \mathcal{M}_i \), which is synchronised across GPUs to ensure consistency in distributed settings.

2. Refine representations via two-stage feature aggregation:
   Details

   To construct category prototypes, we first compute instance-wise feature representations, and then aggregate them into class-wise prototypes:

   1. Instance-wise prototypes:
      Each instance \( k \) in reference image \( I_r^j \) has its own prototype, computed by averaging the feature embeddings within its corresponding mask:

      \[ P_r^{j,k} = \frac{1}{\| M_r^{j,k} \|_1} \sum_{(u,v)} M_r^{j,k}(u,v) F_r^j(u,v) \]

      where \( P_r^{j,k} \in \mathbb{R}^d \) represents the mean feature representation of the \( k \)-th instance in image \( I_r^j \).

   2. Class-wise prototype:
      We compute the category prototype \( P_i \) by averaging all instance-wise prototypes belonging to the same category \( i \):

      \[ P_i = \frac{1}{N_i} \sum_{j=1}^{N_r} \sum_{k \in \mathcal{K}_i^j} P_r^{j,k}, \]

      where \( \mathcal{K}_i^j \) is the set of instances in image \( I_r^j \) that belong to category \( i \), and \( N_i = \sum_{j=1}^{N_r} |\mathcal{K}_i^j| \) is the total number of instances belonging to category \( i \). These class-wise prototypes \( P_i \) are stored in the memory bank.

3. Performing inference on the target images through feature matching and semantic-aware soft merging:
   Details

   For a target image \( I_t \), we extract dense features \( F_t \in \mathbb{R}^{H' \times W' \times d} \) using the same encoder \( \mathcal{E} \). We use the frozen SAM model to generate \( N_m \) candidate instance masks \( \{ M_t^m \}_{m=1}^{N_m} \), where \( M_t^m \in \{0,1\}^{H' \times W'} \). Each mask \( M_t^m \) is used to compute a feature representation via average pooling and L2 normalisation:

   \[ P_t^m = \frac{1}{\| M_t^m \|_1} \sum_{(u,v)} M_t^m(u,v) F_t(u,v), \quad \hat{P}_t^m = \frac{P_t^m}{\| P_t^m \|_2} \]

   where \( \hat{P}_t^m \in \mathbb{R}^d \) is the normalised mask feature.

   To classify each candidate mask, we compute:

   1. Feature Matching:
      We compute the cosine similarity between \( \hat{P}_t^m \) and category prototypes \( P_i \):

      \[ S_t^m = \max_i \left( \frac{\hat{P}_t^m \cdot P_i}{\| P_i \|_2} \right) \]

      which provides the classification score \( S_t^m \) for mask \( M_t^m \).

   2. Semantic-Aware Soft Merging:
      To handle overlapping predictions, we introduce a novel soft merging strategy. Given two masks \( M_t^m \) and \( M_t^{m'} \) of the same category, we compute their intersection-over-self (IoS):

      \[ \text{IoS}(M_t^m, M_t^{m'}) = \frac{\sum (M_t^m \cap M_t^{m'})}{\sum M_t^m} \]

      and weight it by feature similarity:

      \[ w_{m,m'} = \frac{\hat{P}_t^m \cdot \hat{P}_t^{m'}}{\|\hat{P}_t^{m'}\|_2} \]

      The final score for each mask is adjusted using a decay factor:

      \[ S_t^m \leftarrow S_t^m \cdot \sqrt{(1 - \text{IoS}(M_t^m, M_t^{m'}) w_{m,m'})} \]

      reducing redundant detections while preserving distinct instances that may partially overlap. Finally, we rank masks by their adjusted scores, and select the top-\( K \) predictions as the final output.

Performance varies with the choice of reference images, as results depend on their quality. To measure this, the method was evaluated on COCO-20i using different random seeds. Figure 4 shows that variance decreases with more reference images—higher n-shot settings yield lower standard deviations. While 1-3 shots show greater sensitivity to reference selection, 5+ shots offer stable results. This suggests that certain reference images are more effective, and identifying optimal reference sets is a promising direction for future work.

The proposed training-free pipeline is lightweight and efficient:

• memory bank construction is done once at 0.1 seconds per image
• semantic matching is fast at 0.0003 seconds per image using parallel cosine similarity
• soft merging runs at 0.006 seconds per image via a parallel NMS implementation

As shown in Table 4, the method outperforms Matcher and accelerates SAM's automatic mask generation by 3x through optimized point sampling, faster mask filtering, and reduced post-processing.

(Hide / Show) Efficiency Comparison Table

Method	Time (sec/img)
Matcher [45]	120.014
Training-free (ours) with SAM AMG	3.5092
Training-free (ours)	0.9292

Table 4: Time to process an image on 20 reference classes with A100.

Table 5 highlights the effectiveness of the proposed semantic-aware soft-merging strategy, which outperforms other aggregation variants like covariance similarity, instance softmax, score decay, iterative mask refinement, and attention-guided global averaging.

(Hide / Show) Aggregation Strategy Table

Aggregation strategy	10-shot nAP
Hard-merging (hard threshold of 1 IoS)	31.2
Soft-merging (without semantics)	35.7
Soft-merging (with semantics)	36.6

Table 5: Ablation on aggregation strategies.

This work presents a novel training-free few-shot instance segmentation method that combines SAM's mask proposals with DINOv2's semantic features. The approach builds a reference-based memory bank, refines features through aggregation, and classifies new instances using cosine similarity and semantic-aware soft merging. It achieves state-of-the-art results on COCO-FSOD, PASCAL VOC, and CD-FSOD without any fine-tuning, and shows potential for semantic segmentation as well. Future directions include learning to select optimal reference images, improving DINOv2's feature localization, and exploring lightweight fine-tuning to enhance the memory bank.