Context
Open-ended change search needs language. A useful representation should distinguish a new residential block from soil disturbance while remaining grounded in the two input dates.
Approach
I built a captioning pipeline around Qwen3-VL with explicit temporal prompts and structured outputs. The pipeline supports batched vLLM inference, resumable shards, validation, and deterministic manifests. Caption quality is inspected alongside the image pairs rather than scored only as text.
For retrieval, I combined DINOv3 visual features with Qwen3 Embedding in a shared space. The model learns to connect ordered image pairs with their change descriptions.
Learning the transition
Construction and demolition can involve the same objects but describe opposite transitions. I combined features from both dates with their differences and used reversed changed pairs during training to make temporal order matter.
Each pair can have several valid descriptions. Training retains those associations, while evaluation checks retrieval from text to image pair and from image pair to text. The multimodal training article explains the design and recorded validation results.
Technical decisions
- Keep raw captions, normalized captions, and embeddings as separate artifacts.
- Record model, prompt, crop, and checkpoint metadata with every run.
- Evaluate retrieval using held-out identities, not screenshots selected after the fact.
- Use qualitative contact sheets to expose temporal mistakes hidden by aggregate metrics.
Limitations
Caption evaluation remains difficult because multiple descriptions can be correct. Exact-pair retrieval penalizes valid semantic neighbours, while language similarity can reward vague captions. Generated-caption errors can also propagate into retrieval training.