PhilEO MajorTOM: Scaling-up the pretraining of Geospatial Foundation Models

Table Of Contents

1. Introduction
2. Datasets
3. New Models
4. Usage
5. Models and Data
6. To run the model

Introduction

This repository is an extension of the previously introduced PhilEO Bench, and is linked to Paper. The latter paper is a product of a collaboration between ESA's Phi-lab and Leonardo Labs.

Due to protocol constraints, we have open-sourced a selection of files.

Main paper: PhilEO MajorTOM

Also: Paper: PhilEO Scaling-Up

---

The PhilEO Bench serves as a framework that allows users to benchmark various Geospatial Foundation Models (GFMs) against each other on three downstream tasks: road density estimation, building density estimation, and land cover classification. The first two tasks are pixel-wise regression, while the last downstream task is semantic segmentation using 11 classes from ESA WorldCover.

In Paper, we expand on PhilEO Bench, by scaling up the pretraining of the Geo-Aware U-Net to subsets extracted from MajorTOM. Moreover, we demonstrate that the PhilEO ViT UPerNet outperforms its CNN decoder-based counterpart across all three downstream tasks.

This repo can be considered a cleaned-up version of the previously mentioned PhilEO Bench repo, with additional files related to pretraining and fine-tuning the aforemetioned models.

Datasets

The datasets used for pretraining are extracted from the MajorTOM repo. In particular, we pretrained the Geo-Aware U-Net on the MajorTOM 23TB Sentinel-2 dataset, and its smaller 2TB subset, called FastTOM. This yields increased performance w.r.t. the previously used 0.5TB PhilEO Globe dataset.

For fine-tuning, we use the labelled 0.4TB PhilEO Bench downstream dataset.

The file majortom.py, found in the data folder, contains a PyTorch implementation for formatting the extracted data from MajorTOM.

Data: S-2: http://huggingface.co/datasets/NikolaosDionelis2023/s2-phileobench/tree/main

Also: Data: S-1: http://huggingface.co/datasets/NikolaosDionelis2023/s1-phileobench/tree/main

New Models

In addition to the already published models from the PhilEO Bench, which can be found in the folder phileo-bench, we also added the following files to the aforementioned folder:

• decoder_UperNet.py: contains the standard UPerNet implementation.

• model_PhiViTUperNet.py: contains the implementation for the PhilEO ViT UPerNet.

Also, the folder model holds 2 model files:

• phileo_cnn.py: the GeoDINO architecture based on a U-Net design (i.e. the PhilEO CNN).

• phileo_vit.py: an adaptation to the GeoDINO architecture, using a ViT instead of a U-Net (i.e. the PhilEO ViT).

Usage

This repo offers a better use of computational resources, by leveraging the power of Distributed Data Parallel (DDP) training in PyTorch, and thus effectively allowing you to utilize all your available GPUs.

In particular, the following 2 scripts can be used for fine-tuning, using a DDP paradigm:

• train_model_ddp.py: fine-tune the PhilEO CNN.

• train_model_vit_ddp.py: fine-tune the PhilEO ViT.

Models and Data

Model weights: Models

Data: Data S-2

Also: Data: Data S-1

Paper: PhilEO MajorTOM

Also: Paper: PhilEO Scaling-Up

---

Important documents: ./docs/PaperPhilEO18092025.pdf

Also: PhilEO Bench: IGARSS Paper

GitHub: Code

PhilEO Bench

---

To run the model

Usage: To run the model:

git clone https://github.com/ESA-PhiLab/PhilEO-MajorTOM.git

cd PhilEO-MajorTOM

(pip install -r requirements.txt)

(or: pip install -r requirementsalternative.txt)

python train_model_ddp.py

(or python train_model_vit_ddp.py)

---

Also:

python ./further_experiments/n_shot_experiment2.py --read_yaml=./further_experiments/default_args2.yml

For FLOPs: python ./further_experiments/n_shot_experiment.py --read_yaml=./further_experiments/default_args.yml

---

Additional main files: mamba_foundation.py in the folder 'model'

vit_upernet_pretraining.py in the folder 'model'