Bibliographic Details
| Title: |
Towards LLM Agents for Earth Observation |
| Authors: |
Kao, Chia Hsiang, Zhao, Wenting, Revankar, Shreelekha, Speas, Samuel, Bhagat, Snehal, Datta, Rajeev, Phoo, Cheng Perng, Mall, Utkarsh, Vondrick, Carl, Bala, Kavita, Hariharan, Bharath |
| Publication Year: |
2025 |
| Collection: |
Computer Science |
| Subject Terms: |
Computer Science - Artificial Intelligence |
| More Details: |
Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. Here we ask: Are AI systems ready for reliable Earth Observation? We introduce \datasetnamenospace, a benchmark of 140 yes/no questions from NASA Earth Observatory articles across 13 topics and 17 satellite sensors. Using Google Earth Engine API as a tool, LLM agents can only achieve an accuracy of 33% because the code fails to run over 58% of the time. We improve the failure rate for open models by fine-tuning synthetic data, allowing much smaller models (Llama-3.1-8B) to achieve comparable accuracy to much larger ones (e.g., DeepSeek-R1). Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward. The project page is available at https://iandrover.github.io/UnivEarth.
Comment: 36 pages |
| Document Type: |
Working Paper |
| Access URL: |
http://arxiv.org/abs/2504.12110 |
| Accession Number: |
edsarx.2504.12110 |
| Database: |
arXiv |
