École Nationale Supérieure de
Techniques Avancées

Abstract

Recent advances in Deep Learning have seen the rise of Transformers, with models containing theattention mechanism achieving new state-of-the-art performances in a variety of fields, from NaturalLanguage Processing to Computer Vision. As new Transformers based architectures are constantlyproposed, it becomes important to understand how these models differ from classical Neural Networksthat have been used as the standard solution for each domain, such as Recurrent Neural Networksfor sequence-to-sequence applications and Convolutional Neural Networks in Computer Vision. Thiswork aims to analyse the state-of-the-art in semantic segmentation with Transformers, and studyhow these models behave in the context of aerial imagery processing.