A query such as new residential construction asks for something that happened between two dates. Recognizing buildings in an aerial image is only part of the task. The model also needs to distinguish a new development from an existing neighbourhood or a demolition site.
In FOTBCD-CC, I trained a multimodal retrieval model to connect descriptions of change with before-and-after image pairs. The goal was to let someone search an archive in language, then inspect the imagery behind each match.
Combining visual detail with language
I chose DINOv3 SAT-493M for the visual representation and Qwen3-Embedding-0.6B for text. DINO supplies satellite-pretrained visual features, while Qwen represents the meaning of the query. Training connects those two representations so they can be compared directly.
This gives captioning and search separate roles. A captioning model writes a description after inspecting a pair. The retrieval model turns image pairs and text into compact vectors that can be indexed and ranked. Captions provide the connection between the imagery and language during training; search does not require generating a fresh caption for every candidate.
Learning a shared embedding space
The visual branch learns the transition between dates, while the text branch represents its description. Both produce vectors in the same embedding space.
The encoders start with different representations. I trained projection layers to map both into a shared embedding space: image pairs and text descriptions each become normalized 512-dimensional vectors, with matching changes represented by similar directions.
Searching by change, appearance, and context
The search can go beyond broad categories such as construction. A query can combine what changed, what the buildings look like, and the surrounding landscape.
Searching 420,175 temporal pairs for new red-roofed houses constructed beside agricultural fields returns these three locations. The results bring together the new housing, red roofs, and neighbouring fields. Each row connects the same place before and after.
Dark-roofed houses among trees
The query new dark-roofed houses constructed among trees combines a different roof colour with wooded surroundings.
Trees removed along a road
A new roundabout at a road intersection
A new sports field on open land
Woodland cleared to bare soil
See results for no durable physical change
Query: No durable physical change occurred between the observations. This is the shared no-change sentence used during training.
Representing the transition between dates
I used the same visual encoder for both dates and combined their features at corresponding image patches. The representation retains what was present before, what is present after, and the difference between them.
Keeping both observations matters because a difference alone loses context. A new paved surface beside a road and a change inside a cultivated field may produce visual differences, but they describe different processes. The original scene features give the model context for interpreting the transition.
I included both signed and absolute feature differences. The signed difference changes when the dates are swapped, preserving temporal direction. The absolute difference describes the magnitude of the feature change. Spatial fusion then combines local evidence into one representation of the pair.
These features do not automatically distinguish permanent change from season or lighting. That distinction has to be learned from the paired descriptions and unchanged examples.
Teaching the model what counts as a match
One location can contain several events: construction, a new access road, and vegetation removal can occur together. I therefore trained each changed pair against all of its captions, rather than choosing a single description and discarding the others.
I used a multi-positive contrastive objective. It brings matching image pairs and descriptions closer together while separating them from other candidates. Training works in both directions, from image pair to text and from text to image pair. I balanced caption contributions so locations with more descriptions would not dominate learning.
Unchanged pairs needed different treatment. Their captions often describe the landscape without stating whether it changed. I trained those pairs against a shared no-change description, giving the model an explicit target for stable scenes.
The supervision comes from generated captions, including the teacher data used for caption-model distillation. That makes caption quality a retrieval concern too: an incorrect description can teach the model an incorrect association.
Teaching temporal direction with reversed pairs
I used reversed image pairs as negative examples during training. For every changed pair, I swapped the before and after images while keeping the original captions. This applied across change types; unchanged pairs were excluded.
The comparison keeps the scene and objects the same while reversing the transition. It encourages the model to learn what happened between dates and which direction the change took. Reversed pairs reuse the visual features already computed, adding negative examples without another pass through the visual encoder.
Aligning the encoders before adapting them
I structured training around an initial alignment stage with the pretrained encoders frozen, followed by joint fine-tuning. The first stage lets the new temporal fusion and projection layers learn to connect the existing visual and language representations. Joint training then allows both encoders to adapt to the retrieval task.
I used smaller learning rates for the pretrained encoders than for the new layers. This balances task adaptation with preserving the representations the encoders already provide. I also avoided geometric augmentations that could contradict directional wording, while applying mild shared brightness and contrast changes to both dates.
Evaluating whether the right evidence is retrieved
I evaluated retrieval in both directions. Recall@10 measures how often a known matching result appears among the first ten candidates.
| Direction | Recall@1 | Recall@10 | Median rank |
|---|---|---|---|
| Text → changed pair | 13.33% | 42.99% | 16 |
| Changed pair → text | 18.02% | 58.02% | 8 |
These validation results cover 1,210 changed pairs and 2,168 event-caption strings. Including unchanged scenes as distractors gives text-to-pair Recall@10 of 41.84%. The split separates pair identities but does not establish performance in unseen geographic regions.
The results also expose the remaining ranking challenge: the known match frequently falls outside the first few results. Exact-match metrics do not capture every useful answer, since another location may truthfully show the requested event. Inspecting the returned images remains necessary to distinguish valid alternatives from incorrect matches.
In Search Any Change, this representation connects the training work to an analyst’s workflow: describe a transition or provide an example, retrieve candidate locations, and compare the two dates to assess the evidence.