École Nationale Supérieure de
Techniques Avancées

Résumé

This report presents the fine-tuning of the Large Language-and-Vision Assistant (LLaVA) for the task of satellite image captioning. The project aimed to adapt a general-purpose multimodal model to a specialized domain. The methodology followed a structured pipeline: (i) preparation of a dataset of approximately 2100 annotated satellite images, (ii) establishment of a baseline by evaluating the pretrained LLaVA model with BLEU and METEOR metrics, (iii) fine-tuning with parameter-efficient LoRA adapters, and (iv) quantitative and qualitative evaluation of the adapted model. Results showed that fine-tuning consistently improved captions compared to the baseline, with hallucinations reduced and outputs more aligned with reference descriptions. However, absolute performance remained modest, with BLEU-4 and METEOR scores significantly lower than those achieved on large-scale benchmarks such as MS-COCO. Training and validation losses confirmed that most improvements occurred during the first epochs, with performance plateauing afterwards. This indicates that the main limitation lies in dataset size and domain complexity rather than model capacity.