Iowa State University, Ames, Iowa, USA
Foundation vision–language models (VLMs) excel on natural images, but their utility for biomedical microscopy remains underexplored. In this paper, we investigate how in-context learning enables state-of-the-art VLMs to perform few-shot object detection when large annotated datasets are unavailable, as is often the case with microscopic images. We introduce the Micro-OD benchmark, a curated collection of 252 images specifically curated for in-context learning, with bounding-box annotations spanning 11 cell types across four sources, including two in-lab expert-annotated sets. We systematically evaluate eight VLMs under few-shot conditions and compare variants with and without implicit test-time reasoning tokens. We further implement a hybrid Few-Shot Object Detection (FSOD) pipeline that combines a detection head with a VLM-based few-shot classifier, which enhances the few-shot performance of recent VLMs on our benchmark. Across datasets, we observe that zero-shot performance is weak due to the domain gap; however, few-shot support consistently improves detection, with marginal gains achieved after six shots. We observe that some reasoning variant models show task-specific gains, but the effect varies across models and settings. Our results highlight in-context adaptation as a practical path for microscopy, and our benchmark provides a reproducible testbed for advancing open-vocabulary detection in biomedical imaging.
Improvement in mF1 score over the best zero-shot baseline via the cascaded FSOD pipeline
State-of-the-art VLMs evaluated under zero-shot and few-shot (K=1,3,6) conditions
Top mF1 score achieved by the GPT-4o cascaded pipeline at K=6
Annotated cells across 11 types — a few-shot eval suite for microscopy VLMs
We evaluate four prompt configurations — from pure text zero-shot to a cascaded SAM + VLM classifier — across eight models on the Micro-OD benchmark.
Zero-Shot-T uses text only. Few-Shot-V adds visual crops. Few-Shot-MMD combines both for end-to-end detection. Few-Shot-MMC decouples localization (SAM) from classification (VLM) — the strongest overall configuration.
A curated benchmark for few-shot object detection specifically designed for microscopy. Unlike prior benchmarks that sample broadly from dissimilar classes, Micro-OD targets fine-grained visual localization where cell types share high semantic and morphological similarity.
Platelets, Red Blood Cells, and White Blood Cells. A well-established benchmark with bounding-box annotations for all three categories.
Thin blood smears annotated across parasitic stages: Gametocyte, Ring, Schizont, Trophozoite, RBCs, and WBCs. Tests fine-grained class separation.
Re-annotated into morphological categories: Polygonal, Round, and Spindle cells. Irregular and heterogeneous morphologies test shape-based generalization.
Expert-annotated in-lab collection from Iowa State's BACM lab. Imaged via AFM-mounted cameras at 20× zoom with two resolutions.
Sample detections
Splits are generated via a two-phase Integer Linear Programme (ILP) over 1,000 shuffled seeds, optimizing for class diversity (CPC) and balance (CBE). Final split achieves an SSS of 0.74.
mF1 is averaged over IoU thresholds 0.05–0.70; Mean IoU is computed over true positives at IoU=0.5.
Zero-Shot Baselines
| Model | mF1 ↑ | Mean IoU ↑ |
|---|---|---|
| Grounding DINO | 0.04 | 0.53 |
| OWL-ViT | 0.03 | 0.40 |
| Gemini-2.5-Flash | 0.05 | 0.57 |
| Gemini-2.5-Flash-Thinking | 0.06 | 0.59 |
| Claude-3.7-Sonnet | 0.03 | 0.34 |
| Claude-3.7-Sonnet-Thinking | 0.03 | 0.31 |
Few-Shot Results (mF1)
| Model | K=1 | K=3 | K=6 | Mean IoU (K=6) |
|---|---|---|---|---|
| Few-Shot-V (Visual only) | ||||
| OWL-ViT | 0.08 | 0.12 | 0.17 | 0.81 |
| Few-Shot-MMD (Multi-modal detection) | ||||
| GPT-4o | 0.05 | 0.06 | 0.07 | 0.53 |
| GPT-5 | 0.07 | 0.08 | 0.09 | 0.57 |
| Gemini-2.5-Flash | 0.05 | 0.07 | 0.07 | 0.55 |
| Gemini-2.5-Flash-Thinking | 0.19 | 0.22 | 0.23 | 0.77 |
| Claude-3.7-Sonnet | 0.04 | 0.04 | 0.05 | 0.44 |
| Claude-3.7-Sonnet-Thinking | 0.05 | 0.06 | 0.07 | 0.47 |
| Few-Shot-MMC (Cascaded: SAM + VLM classifier) ★ Best overall | ||||
| GPT-4o | 0.24 | 0.27 | 0.29 | 0.82 |
| GPT-5 | 0.26 | 0.27 | 0.28 | 0.78 |
| Gemini-2.5-Flash | 0.21 | 0.21 | 0.22 | 0.79 |
| Gemini-2.5-Flash-Thinking | 0.16 | 0.18 | 0.17 | 0.56 |
| Claude-3.7-Sonnet | 0.10 | 0.14 | 0.13 | 0.79 |
| Claude-3.7-Sonnet-Thinking | 0.09 | 0.15 | 0.16 | 0.79 |
Cascaded pipeline is useful. Decoupling localization (SAM) from classification (VLM) substantially outperforms end-to-end detection across all models — peak mF1 of 0.30 vs 0.23 for the best end-to-end method.
Zero-shot is insufficient. All models show weak zero-shot performance (mF1 ≤ 0.06) due to the significant domain gap between web-scale pretraining and microscopy imagery.
"Thinking" tokens are task-dependent. Gemini-2.5-Flash-Thinking excels at end-to-end detection, but non-thinking models are often stronger for cascaded classification — reasoning helps complex localization, not fine-grained categorization.
If you find this work useful, please cite:
Ganguly S, Biswas A, Rade J, Hasib MHH, Masud N, Singla N, Dash A, Bhattacharjee U, Sarkar S, Balu A, Sarkar A and Krishnamurthy A (2026) In-context adaptation of VLMs for few-shot cell detection in optical microscopy. Front. Artif. Intell. 9:1761903. doi: 10.3389/frai.2026.1761903
