CoCo-IR: Contextual Composed Image Retrieval

Shengcao Cao1,2, Tanmaya Shekhar Dabral2, Zhongli Ding2, Madhuri Shanbhogue2, Kaifeng Chen3,
Zhe Li2, Mojtaba Seyedhosseini2, Yu-Xiong Wang1, Liang-Yan Gui1
1University of Illinois Urbana-Champaign, 2Google DeepMind, 3OpenAI. Work done at Google DeepMind.
arXiv

Visual search is iterative — CoCo-IR lets users refine retrieval across turns, exactly where single-turn models break down.

CoCo-IR teaser: single-turn CIR vs. multi-turn contextual retrieval

A standard single-turn CIR model handles an isolated instruction, but cannot follow an evolving dialogue. Even when a powerful external model (e.g., Gemini-2.5-Pro) summarizes the entire history into one prompt, context is inevitably compressed and retrieval fails. Our TIE model natively processes the full multi-turn context, resolving instructions that refer back to earlier images (e.g., “the original viewpoint”, “the second image”) within a single unified framework.

📍 New Task: Contextual Composed Image Retrieval

Real-world visual search is rarely a single command: users explore, refine, and change their minds. We introduce Contextual Composed Image Retrieval (CoCo-IR), which reformulates instruction-based retrieval as an interactive, multi-turn dialogue. At each turn the model must interpret a new instruction against the entire interaction history (the initial image, every prior instruction, and every intermediate result), so a complex search goal can be decomposed into a sequence of simple, context-aware steps.

🚀 New Model: Transformable Image Embedding (TIE)

Transformable Image Embedding (TIE) model architecture

TIE unifies multimodal comprehension and embedding generation in a single Large Multimodal Model. A dedicated <EMB> token acts as a global information bottleneck that aggregates the full dialogue into a compact, query-aware embedding. A hybrid attention mask lets the embeddings evolve as the conversation unfolds: full attention within each turn for deep multimodal fusion, and causal attention across turns to respect temporal flow. The model is trained end-to-end with a contrastive objective.

💎 Scalable, Autonomous Data Engine

LMM self-reflection scores instruction and image-pair quality

No large-scale multi-turn retrieval dataset exists, and manual annotation is prohibitively expensive. Our fully autonomous, LMM-powered data engine generates high-quality data through self-reflection: an LMM proposes a transformation instruction and then scores its own quality and ambiguity, letting us filter for only the cleanest examples.

LMM verifier mines hard negatives that look similar but fail the instruction

To learn fine-grained distinctions, the engine uses LMMs as verifiers to mine challenging hard negatives: images that look visually similar to the target but fail the instruction. The result is a high-quality dataset that reaches state-of-the-art performance with roughly 14× fewer training samples than prior work.

🏆 New SOTA: Single-Turn and Multi-Turn

TIE first sets a new state of the art on standard single-turn CIR benchmarks, reaching 39.4 mAP@5 on CIRCO and 38.7 R@1 on CIRR, proving its representations are strong even before any multi-turn context comes into play.

Single-turn CIR comparison across FIQ, CIRR, and CIRCO

Single-turn CIR across three benchmarks. TIE is the most consistent method, leading on CIRR and CIRCO and staying competitive on FIQ, whereas prior methods excel on one benchmark but fall behind on others.

The real story, however, is multi-turn. On the new CoCo-IR benchmark, prior CIR models must be adapted with proxy inputs, and they collapse as the dialogue deepens, even when given an oracle Gemini-2.5-Pro summary of the full history. TIE instead ingests the entire context natively and stays robust, keeping the gap wide all the way to the fourth turn.

Recall@1 across turns: TIE stays robust while adapted baselines drop off

Recall@1 across turns. The best baseline is shown under three adaptation strategies; all degrade sharply with depth, while TIE’s native full-context modeling holds up. Multi-turn visual dialogue cannot be losslessly compressed into a single text prompt.

📚 BibTeX

@inproceedings{cao2026cocoir,
  title={CoCo-IR: Contextual Composed Image Retrieval},
  author={Shengcao Cao and Tanmaya Shekhar Dabral and Zhongli Ding and Madhuri Shanbhogue and Kaifeng Chen and Zhe Li and Mojtaba Seyedhosseini and Liang-Yan Gui and Yu-Xiong Wang},
  booktitle={ECCV},
  year={2026}
}