Academic Journal
Self-supervised audiovisual representation learning for remote sensing data
| Title: | Self-supervised audiovisual representation learning for remote sensing data |
|---|---|
| Authors: | Konrad Heidler, Lichao Mou, Di Hu, Pu Jin, Guangyao Li, Chuang Gan, Ji-Rong Wen, Xiao Xiang Zhu |
| Source: | International Journal of Applied Earth Observations and Geoinformation, Vol 116, Iss , Pp 103130- (2023) |
| Publisher Information: | Elsevier, 2023. |
| Publication Year: | 2023 |
| Collection: | LCC:Physical geography LCC:Environmental sciences |
| Subject Terms: | Self-supervised learning, Multi-modal learning, Representation learning, Audiovisual dataset, Physical geography, GB3-5030, Environmental sciences, GE1-350 |
| More Details: | Many deep learning approaches make extensive use of backbone networks pretrained on large datasets like ImageNet, which are then fine-tuned. In remote sensing, the lack of comparable large annotated datasets and the diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pretrained backbone networks in remote sensing, we devise a self-supervised approach for pretraining deep neural networks. By exploiting the correspondence between co-located imagery and audio recordings, this is done completely label-free, without the need for manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and crowd-sourced audio samples all around the world. Using this dataset, we then pretrain ResNet models to map samples from both modalities into a common embedding space, encouraging the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pretrained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pretraining strategies for remote sensing imagery. The dataset, code and pretrained model weights are available at https://github.com/khdlr/SoundingEarth. |
| Document Type: | article |
| File Description: | electronic resource |
| Language: | English |
| ISSN: | 1569-8432 |
| Relation: | http://www.sciencedirect.com/science/article/pii/S1569843222003181; https://doaj.org/toc/1569-8432 |
| DOI: | 10.1016/j.jag.2022.103130 |
| Access URL: | https://doaj.org/article/7e59733d574f49a7a91a10630abec717 |
| Accession Number: | edsdoj.7e59733d574f49a7a91a10630abec717 |
| Database: | Directory of Open Access Journals |
| ISSN: | 15698432 |
|---|---|
| DOI: | 10.1016/j.jag.2022.103130 |
| Published in: | International Journal of Applied Earth Observations and Geoinformation |
| Language: | English |