FOTBCD-CC started with a practical question: how could I describe change across a large aerial-image archive without paying for a large-model API call on every pair?
The archive contains 223,675 before-and-after pairs. Each description requires comparing two dates, deciding whether a lasting physical change occurred, and explaining what happened. A useful caption might describe a housing development replacing part of a field. A poor one might mistake seasonal crop differences for permanent land conversion.
I chose distillation because of the cost of large-model API calls. Using Qwen3.8-Max on all 223,675 pairs would mean paying for teacher inference across the entire archive. Instead, I used it to annotate a selected sample, trained the much smaller Qwen3-VL-8B on those outputs, and ran the smaller captioner on self-hosted GPUs for bulk annotation.
Using the larger model as a teacher
I began with small captioning pilots and manual review before selecting 20,000 pairs for teacher-generated supervision. Existing building annotations helped balance the sample, but I kept those labels out of the prompt. The teacher (Qwen3.8-Max) had to compare the imagery itself, including roads, vegetation, and land-use changes beyond the original building task.
I also included unchanged scenes and allowed several captions for pairs containing several events. Those choices define what the student (Qwen3-VL-8B) should learn: when to report change, when to describe a stable scene, and when one sentence misses part of the transition.
The resulting dataset contained 19,997 pairs and 29,447 captions. It gave me a focused source of supervision without requiring the teacher to caption the entire archive first.
Reducing the cost of teacher calls
The same detailed instructions were needed for every pair. I kept them in a shared system text block at the beginning of each request and placed the changing images in the user message after it. To enable explicit prompt caching, I added "cache_control": {"type": "ephemeral"} to that system text block. This let repeated calls reuse the shared instructions.
Cached tokens accounted for approximately 61.2% of reported input usage. At the rates used for these calls, ordinary input cost $1.65 per million tokens, while cached reads cost $0.137 per million. That is a 91.7% saving on reused input tokens. Cache creation cost $2.063 per million tokens, and output cost $4.951 per million.
I disabled thinking and bounded the output because the task called for concise descriptions and a change decision. I tracked token usage to assess spending before expanding the runs. Processing also retained successful results, so an interruption did not require paying again for work already completed.
These decisions addressed both sources of API cost: how much each request consumes and how often it has to be repeated.
Transferring the task to a smaller student
The purpose of this transfer was to reduce annotation costs: once trained, the smaller model could caption the full archive without a paid Qwen3.8-Max call for every pair. Fine-tuning and self-hosted inference still consume GPU time, but they move the bulk workload away from the teacher API.
I used response distillation: the student learned from the teacher’s descriptions and change decisions. This does not require access to the larger model’s weights. LoRA fine-tuning provided the means to adapt Qwen3-VL-8B while training a limited set of additional parameters.
The training approach kept the vision tower and visual aligner frozen and adapted the language model. That constrained the training footprint while teaching the student the expected aerial-change outputs. Distillation describes the transfer from teacher to student; fine-tuning describes how I trained the student.
The trade-off is the quality of the supervision. Teacher mistakes can become student training targets, so successful distillation alone does not establish that the descriptions are visually correct.
Checking what the student actually learned
I compared the adapted student with the base model on a separate 2,000-pair test set. The comparison covered whether a durable change was reported, how many events were described, and agreement with the reference wording.
| Metric | Base Qwen3-VL-8B | Distilled student |
|---|---|---|
| Durable-change accuracy | 62.35% | 89.60% |
| Durable-change F1 | 62.44% | 91.33% |
| Caption-count accuracy | 11.95% | 79.20% |
| Token F1 | 0.142 | 0.496 |
| ROUGE-L | 0.112 | 0.432 |
The student improved on both change decisions and reference-caption agreement. Caption-count accuracy was particularly relevant because a plausible description of one event can still omit others.
I kept the interpretation narrow. The references came from the teacher, so matching them is a measure of transfer rather than independent visual truth. Visual review remains necessary.
Where the base and fine-tuned models disagree
Across the 2,000 test pairs, the fine-tuned model agrees with the teacher’s durable-change decision on 646 pairs where the base model disagrees. The base model agrees on 101 pairs where the fine-tuned model disagrees. Both agree on 1,146 pairs, and both disagree on 107. The net gain is 545 correct decisions against the teacher references, or 27.25 percentage points in accuracy.
The nine comparisons below show selected improvements in change decisions and event descriptions. They are illustrative examples, selected from saved test predictions rather than a random sample. Each image shows the earlier crop on the left and the later crop on the right. Both models’ captions are reproduced verbatim; the teacher’s decision and caption count provide the reference.
1. A new house and swimming pool
Teacher reference: durable_change: true · 1 caption.
Base Qwen3-VL-8B: durable_change: false
- The grassy field shows signs of drying or seasonal change, appearing more brownish in the after image.
- The trees and shrubs remain largely unchanged in position and density between the two images.
Fine-tuned Qwen3-VL-8B: durable_change: true
- A residential house with a dark roof and an adjacent rectangular swimming pool now occupies the grassy area along the western boundary, replacing the open field visible in the earlier view.
The base model attributes the difference to seasonal drying. The fine-tuned model identifies the new house and pool replacing open ground.
2. Tree clearing beside an industrial building
Teacher reference: durable_change: true · 1 caption.
Base Qwen3-VL-8B: durable_change: false
- Two white cars are now visible on the highway in the second image, which were absent in the first image.
Fine-tuned Qwen3-VL-8B: durable_change: true
- Dense tree cover adjacent to the industrial building has been cleared, exposing bare earth and low vegetation where a continuous forest canopy previously stood.
The base model focuses on cars on the highway. The fine-tuned model describes the lasting removal of tree cover beside the building.
3. A new building beside a football pitch
Teacher reference: durable_change: true · 1 caption.
Base Qwen3-VL-8B: durable_change: false
- A red-roofed building has been added to the lower-left corner of the field.
- A white vehicle is present on the road in the first image but absent in the second.
Fine-tuned Qwen3-VL-8B: durable_change: true
- A rectangular building with a reddish-brown roof now occupies the grassy area between the road and the soccer field, replacing the open lawn and scattered trees visible in the earlier view.
The base caption mentions a new building but its change decision is false. Fine-tuning makes the decision consistent with the construction described in the caption.
4. Crop appearance is not land conversion
Teacher reference: durable_change: false · 1 caption.
Base Qwen3-VL-8B: durable_change: true
- The large, neatly planted green field in the first image has been replaced by a dry, brown field in the second image.
- The field now shows visible tracks from vehicles or machinery, indicating recent activity.
- The surrounding trees and buildings remain unchanged, confirming the transformation is localized to the field.
Fine-tuned Qwen3-VL-8B: durable_change: false
- A large cultivated field with distinct parallel rows lies adjacent to a dense tree line and residential structures with reddish-brown roofs.
The base model treats a green-to-brown field and machinery tracks as durable change. The fine-tuned model keeps the no-change decision and describes the cultivated scene.
5. Ploughing is not a lasting physical transition
Teacher reference: durable_change: false · 1 caption.
Base Qwen3-VL-8B: durable_change: true
- The field has been plowed, creating distinct parallel furrows.
- The pattern of the plowed field is clearly visible, indicating recent agricultural activity.
Fine-tuned Qwen3-VL-8B: durable_change: false
- A paved road curves along the boundary of a large agricultural field, bordered by a line of trees and a building with a reddish roof.
The base model reports ploughing as durable change. The fine-tuned model distinguishes agricultural surface differences from construction or permanent land conversion.
6. Recognizing a seasonal difference without calling it durable
Teacher reference: durable_change: false · 1 caption.
Base Qwen3-VL-8B: durable_change: true
- The lush green crop field has been harvested, leaving behind a brown, tilled field with visible furrows.
- The change indicates a seasonal transition from growing season to post-harvest period.
Fine-tuned Qwen3-VL-8B: durable_change: false
- A cultivated field with visible crop rows lies adjacent to a dense tree line and a building with a dark roof.
The base captions explicitly describe a seasonal transition, yet the change decision is true. The fine-tuned model returns false and a scene caption.
7. Two new structures, two event descriptions
Teacher reference: durable_change: true · 2 captions.
Base Qwen3-VL-8B: durable_change: true
- The large field has changed from green to dry, harvested brown.
- New tire tracks are visible across the field, indicating recent vehicle activity.
- The buildings and surrounding trees remain unchanged, confirming the location.
Fine-tuned Qwen3-VL-8B: durable_change: true
- A large rectangular building with a dark grey roof now occupies the bare earth area adjacent to the tree line and dirt road, replacing the open ground and scattered materials visible in the earlier view.
- A second long structure with a grey roof featuring a grid of lighter panels has been constructed along the southern edge of the field, bordering the existing dirt track.
Both models return true, but the base captions discuss harvesting and tire tracks. The fine-tuned model describes the two new structures separately.
8. Housing construction and a new road
Teacher reference: durable_change: true · 2 captions.
Base Qwen3-VL-8B: durable_change: true
- The orderly rows of a vineyard have been replaced by a cluster of modern villas with swimming pools.
- The natural, undeveloped terrain surrounding the vineyard is now interspersed with landscaped gardens and paved roads.
- The dense forest bordering the area remains, but the development has significantly altered the land use and landscape character.
Fine-tuned Qwen3-VL-8B: durable_change: true
- A residential subdivision with multiple houses featuring red-tiled roofs and private swimming pools now occupies the cultivated field that previously lay between the forest edge and the scrubland.
- A paved road with parked vehicles runs along the northern boundary of the new development, replacing the open agricultural land visible in the earlier view.
Both models recognize development. The fine-tuned model separates the houses and pools from the road construction, matching the two-event structure of the teacher reference.
9. A housing development and its street network
Teacher reference: durable_change: true · 2 captions.
Base Qwen3-VL-8B: durable_change: true
- The large grassy field in the foreground has been developed into a residential neighborhood with multiple houses and paved roads.
- Several cars are now parked in driveways and along the streets, indicating new residents or activity.
- The area previously dominated by open land is now densely built with houses and landscaping.
Fine-tuned Qwen3-VL-8B: durable_change: true
- New residential houses with dark roofs and paved driveways now occupy the open grassy field that previously lay south of the existing row of homes.
- A network of paved streets has been constructed through the former field, connecting the new housing development to the pre-existing road infrastructure.
The base model repeats the development description and adds parked cars. The fine-tuned model gives separate captions for the new houses and the connecting streets.
These examples make the measured improvement concrete: fewer agricultural false alarms, better recognition of lasting changes, and more focused event descriptions. The 101 decisions that regress also matter; fine-tuning improves aggregate agreement with the teacher but does not make every prediction correct.
Conclusion
The cost of calling a large model across the entire archive motivated this approach. I used Qwen3.8-Max to annotate roughly 20,000 pairs, reduced the cost of those calls through prompt caching, and distilled its responses into Qwen3-VL-8B. The smaller captioner then generated the full caption set of 321,200 descriptions across 223,675 pairs, providing supervision for multimodal retrieval.
Fine-tuning improved durable-change accuracy from 62.35% to 89.60% against held-out teacher references. The comparisons show what that gain looks like: recognizing construction and clearing, avoiding agricultural false alarms, and describing separate events more clearly. These results support using the adapted model for bulk captioning, while agreement with the teacher remains distinct from independently verified visual correctness.