Captioning change and learning to retrieve it
Change captioning asks a model to describe what happened between two observations of the same place. In aerial imagery, that means comparing the dates, identifying a physical transition, and explaining it in the context of the surrounding scene.
FOTBCD provides the before-and-after aerial imagery; FOTBCD-CC adds captions and models for describing and searching change. The work has three stages. First, I asked a large vision-language model, Qwen3.8-Max, to annotate roughly 20,000 selected image pairs, producing a teacher set with 29,447 captions. Second, I fine-tuned the smaller Qwen3-VL-8B on those annotations and used it to annotate all 223,675 pairs, producing the full caption set of 321,200 captions. Third, I used those captions to train a separate retrieval model that embeds image pairs and text into the same vector space, so a description or an example pair can find similar changes.
The large captioning model is the teacher and the smaller fine-tuned captioner is the student. The retrieval model is a separate model trained on the caption sets. Distilling the captioning task into a self-hosted model kept annotation costs manageable across hundreds of thousands of pairs.
Open dataset and captioning model
The complete FOTBCD-CC caption dataset and the Qwen3-VL-8B LoRA adapter used to generate it are now openly available on Hugging Face.
| Release | Contents | License |
|---|---|---|
| FOTBCD-CC | 321,200 English captions for 223,675 temporal pairs | CC BY 4.0 |
| FOTBCD-CC Qwen3-VL-8B LoRA | 21,823,488-parameter captioning adapter | Apache 2.0 |
| FOTBCD | 223,675 paired aerial crops and their building annotations | CC BY 4.0 |
FOTBCD-CC is distributed as Parquet with train, validation, and test splits. Each row contains the change decision, one to five captions, acquisition and spatial metadata, and stable references to the corresponding images in FOTBCD. The imagery is not duplicated: every row records the pinned FOTBCD revision, WebDataset shard, and sample key needed to retrieve the exact before-and-after pair.
The annotations can be loaded directly with Hugging Face Datasets:
from datasets import load_dataset
dataset = load_dataset("abdevd/FOTBCD-CC")
example = dataset["train"][0]
print(example["durable_change"])
print(example["captions"])
print(example["image_shard"], example["sample_key"])The released adapter applies the project’s durable-change definition to two ordered aerial images and returns structured JSON containing a boolean decision and a caption list. The base Qwen3-VL-8B weights are not duplicated in the adapter repository.
What an example contains
Each example contains two aligned aerial crops of the same location, ordered from earlier to later, and a structured response with a durable_change decision and a captions list. The complete crops provide context for describing buildings, roads, vegetation, and land use.
A durable change is a lasting physical alteration: construction, demolition, a new paved road, an expanded building, or woodland removed for a different land use. A visible difference alone is insufficient. Shadows, parked vehicles, seasonal vegetation, and crop cycles can change appearance without changing the place’s lasting physical features. A cultivated field becoming bare soil does not establish land conversion; a new road or development occupying it does.
When durable change is visible, captions describe the events. One pair can contain several events, so I kept a list of descriptions rather than forcing everything into one sentence. This generated Qwen3.8-Max response illustrates the format:
{
"durable_change": true,
"captions": [
"A residential subdivision with detached houses and paved streets now occupies the grassy area adjacent to the cultivated field, replacing the unpaved track and open vegetation visible in the earlier view.",
"Paved roads with marked crosswalks have been constructed through the former open land, connecting the new housing plots along the field boundary."
]
}When durable_change is false, the response still contains a caption describing the scene. Here is another generated teacher response:
{
"durable_change": false,
"captions": [
"A large cultivated field bordered by a diagonal hedgerow and scattered trees retains its boundaries and surrounding vegetation across both views, with only the crop cover and surface texture differing between the dates."
]
}These scene descriptions preserve useful information about land cover, objects, and spatial relationships. They remain in the caption sets for scene understanding and other image–language tasks. For retrieval training, however, I replace them with a shared no-change sentence: the search model learns whether a transition occurred, while the stored annotation still describes the scene.
Stage 1: building the teacher set
Distillation was a cost decision. Calling Qwen3.8-Max for every one of the 223,675 image pairs would mean paying for large-model inference across the entire collection. Instead, I paid for teacher annotations on a smaller sample, used them to train a smaller captioner, and ran that captioner on self-hosted GPUs for the full collection.
I first used Qwen3.8-Max to caption a selected sample of 20,000 pairs. Existing building annotations from FOTBCD helped balance the selection, but were not disclosed to the captioning model. The teacher had to inspect the full scene and decide what changed from the imagery itself.
The teacher set contains:
| Teacher set | Count |
|---|---|
| Temporal pairs | 19,997 |
| Generated captions | 29,447 |
| Pairs classified as durable change | 12,104 |
| Pairs classified as no durable change | 7,893 |
I used these outputs for response distillation into Qwen3-VL-8B through supervised LoRA fine-tuning. The smaller captioner learned both the structured decision and the descriptions from the larger teacher, allowing me to generate the full caption set without a paid teacher call for each pair.
Generating the supervision also needed to be affordable. I structured requests with reusable instructions in a shared system prompt and the changing imagery in a separate user message. An explicit ephemeral cache marker let repeated calls reuse the shared instructions. Approximately 61.2% of input usage came from cached reads. At the rates used for these calls, cached input cost $0.137 per million tokens compared with $1.65 for ordinary input, a 91.7% saving on reused input tokens.
I also disabled thinking for this structured task, capped response length, and preserved successful outputs when processing resumed. These choices reduced unnecessary generation and repeated work.
The fine-tuning split kept pair identities separate across 15,997 training pairs, 2,000 validation pairs, and 2,000 test pairs. On the test set, the adapted model improved over the base Qwen3-VL-8B:
| Metric | Base model | Fine-tuned model |
|---|---|---|
| Durable-change accuracy | 62.35% | 89.60% |
| Durable-change F1 | 62.44% | 91.33% |
| Caption-count accuracy | 11.95% | 79.20% |
| Token F1 | 0.142 | 0.496 |
| ROUGE-L | 0.112 | 0.432 |
These results measure agreement with teacher-generated references. They show that the student learned the task, while visual correctness remains a separate concern from matching a reference sentence.
Learn more about distillation, cost-saving choices, and evaluation →
Stage 2: generating the full caption set
With the adapted Qwen3-VL-8B, I captioned the full collection of 223,675 temporal pairs, comprising 447,350 images.
I served the fine-tuned captioner through vLLM and made processing restartable. Distillation moved this bulk workload to a self-hosted model, giving me control over GPU throughput and runtime.
| Full caption set | Count |
|---|---|
| Temporal pairs | 223,675 |
| Generated captions | 321,200 |
| Pairs classified as durable change | 134,116 (59.96%) |
| Pairs classified as no durable change | 89,559 (40.04%) |
| Captions attached to changed pairs | 231,640 |
| Scene captions attached to no-change pairs | 89,560 |
The export contains a successful, valid JSON annotation for every pair. Captions average 26.13 words. There are 134,160 pairs with one caption, 81,683 with two, and 7,832 with three to five, reflecting the need to describe several events within one location.
The teacher set teaches the smaller captioner the task. The full caption set supplies annotations at scale and becomes supervision for the separate retrieval model.
Explore the full dataset’s composition, caption statistics, and language analysis →
Stage 3: training the retrieval model
Captions turn the imagery into image–text training examples: each changed pair has one or more descriptions of what happened. I trained a separate model to bring matching pairs and descriptions close together in a shared vector space, while separating nonmatching examples. This makes it possible to search imagery with a new description without generating captions at query time.
The retrieval model uses DINOv3 SAT-493M for imagery and Qwen3-Embedding-0.6B for text. Both branches project into a shared embedding space of normalized 512-dimensional vectors. A description and an image pair showing that description can then be compared directly.
The visual representation needs to capture a transition, not just recognize objects. I processed both dates with the same visual encoder and combined their features with signed and absolute differences. Keeping the original features retains the scene context; the signed difference preserves the order of the observations.
I trained changed pairs against all their event captions using a multi-positive contrastive objective. This lets one location match several descriptions without choosing just one event. I balanced their contributions so pairs with more captions would not dominate training.
For this change-retrieval objective, unchanged pairs match a shared no-change sentence. Their scene descriptions remain in the dataset for other uses. I also used reversed versions of every changed pair as negatives, encouraging the model to distinguish the original transition from its reverse. Reusing the encoded visual features makes those negatives inexpensive to add.
Training began by aligning the new temporal and projection layers with frozen encoders, followed by joint fine-tuning with smaller learning rates for the pretrained branches. I developed the approach using the teacher set, then extended training to the full caption set.
The resulting model supports queries combining an event with appearance and context, such as new dark-roofed houses constructed among trees. The technical article shows the architecture and actual retrieval results, including roads, roundabouts, sports fields, and woodland clearing.
Learn more about multimodal training and explore the retrieval examples →
Using the models to explore change
Search Any Change brings the retrieval model into an interface for searching 420,175 before-and-after patch pairs from the Rhône department through text, points, and multiple examples. Rhône was excluded from the project’s training data. This search index is a separate imagery collection from the 223,675-pair full caption set, showing how the trained model can embed an unseen region for search without first assigning its pairs captions. A description leads to candidate locations whose earlier and later imagery can be inspected together.
Semantic Change Mapping uses example-based similarity to score and export spatial fields. Both applications build on the captioning and retrieval work: descriptions make the imagery interpretable, and learned embeddings make it searchable.