feat: Complete multi-satellite API integration and optimize usage

This commit implements the integration of three satellite data sources:
- Sentinel-5P for air pollutants and trace gases
- MODIS for aerosol optical depth measurements
- ERA5 for meteorological reanalysis data

Key improvements:
- Unified API design across all satellite platforms
- Consistent data processing pipeline (fetch → download → process)
- Optimized parallel downloading with progress tracking
- Added automatic file retention management
- Enhanced documentation with usage examples

The implementation includes quality control filters and standardized
visualization outputs for each data source, making satellite data
more accessible for environmental analysis.

Author: Alex870521

.env.example

@@ -0,0 +1,11 @@
+# Copernicus Open Access Hub API credentials
+COPERNICUS_USERNAME=your_username
+COPERNICUS_PASSWORD=your_password
+
+# NASA EARTHDATA credentials
+EARTHDATA_USERNAME=your_username
+EARTHDATA_PASSWORD=your_password
+
+# CDS API credentials
+CDSAPI_URL=https://cds.climate.copernicus.eu/api/v2
+CDSAPI_KEY=your_key

README.md

@@ -1,4 +1,4 @@
-## <div align="center">Satellite_S5P</div>
+## <div align="center">Satellite Data Processing Toolkit</div>
 
 <div align="center">
 
@@ -8,37 +8,67 @@
 </div>
 
 <div align="center">
-<a href="https://github.com/Alex870521"><img src="https://cdn.simpleicons.org/github/0A66C2" width="3%" alt="GitHub"></a>
+<a href="https://github.com/Alex870521"><img src="https://cdn.simpleicons.org/github/0A66C2" width="3%" alt="LinkedIn"></a>
 <span style="margin: 0 1%"></span>
 <a href="https://www.linkedin.com/in/Alex870521/"><img src="https://cdn.simpleicons.org/linkedin/0A66C2" width="3%" alt="LinkedIn"></a>
 <span style="margin: 0 1%"></span>
 <a href="https://medium.com/@alex870521"><img src="https://cdn.simpleicons.org/medium/0A66C2" width="3%" alt="Medium"></a></div>
 
 ---
 
-A Python toolkit for retrieving, processing, and visualizing Sentinel-5P satellite data, with a focus on air pollutants like NO₂, CO, SO₂, O₃, and HCHO.
+A comprehensive Python toolkit for retrieving, processing, and visualizing satellite data from multiple sources: Sentinel-5P, MODIS, and ERA5. This toolkit focuses on atmospheric data including air pollutants (NO₂, CO, SO₂, O₃, HCHO), aerosol optical depth, and meteorological parameters.
 
 ## <div align="center">Features</div>
 
-- Automated data retrieval from Copernicus Open Access Hub
-- Advanced data processing with interpolation and quality control
-- High-quality visualization of pollutant concentration maps
-- Time series analysis capabilities
-- Built-in file retention management to prevent storage overflow
+- **Multi-platform Support**:
+  - **Sentinel-5P**: Trace gases and air pollutants
+  - **MODIS**: Aerosol optical depth (AOD) measurements
+  - **ERA5**: Reanalysis of atmospheric, land, and oceanic climate variables
+
+- **Unified Data Access**:
+  - Automated data retrieval from Copernicus Open Access Hub, NASA Earthdata, and Climate Data Store
+  - Consistent API across different data sources
+
+- **Advanced Processing**:
+  - Quality control and filtering
+  - Spatial interpolation and regridding
+  - Temporal aggregation
+  - Station-based data extraction
+
+- **Visualization**:
+  - High-quality concentration and parameter maps
+  - Customizable geographic boundaries
+  - Time series analysis capabilities
+
+- **Resource Management**:
+  - Efficient download handling with caching
+  - Built-in file retention management
 
 ## <div align="center">Prerequisites</div>
 
 Before using this toolkit, you need to complete the following steps:
 
-1. **Copernicus Account Setup**:
+1. **Copernicus Account** (for Sentinel-5P and ERA5):
    - Register for a free account at [Copernicus Open Access Hub](https://scihub.copernicus.eu/dhus/#/home)
-   - Save your username and password for API access
+   - For ERA5, also register at [Climate Data Store](https://cds.climate.copernicus.eu/)
 
-2. **Environment Configuration**:
+2. **NASA Earthdata Account** (for MODIS):
+   - Register at [NASA Earthdata](https://urs.earthdata.nasa.gov/)
+
+3. **Environment Configuration**:
    - Create a `.env` file in the project root directory with your credentials:
      ```
+     # Sentinel-5P credentials
      COPERNICUS_USERNAME=your_username
      COPERNICUS_PASSWORD=your_password
+     
+     # ERA5 credentials
+     CDSAPI_URL=https://cds.climate.copernicus.eu/api/v2
+     CDSAPI_KEY=your_key
+     
+     # NASA Earthdata credentials
+     EARTHDATA_USERNAME=your_username
+     EARTHDATA_PASSWORD=your_password
      ```
 
 ## <div align="center">Installation</div>
@@ -56,81 +86,164 @@ pip install -r requirements.txt
 
 ## <div align="center">Usage</div>
 
-### Fetching Satellite Data
+### Sentinel-5P Example
+
 ```python
-from src.api.sentinel_api import S5PFetcher
-
-# Initialize fetcher
-fetcher = S5PFetcher(max_workers=3)
-
-# Fetch products
-products = await fetcher.fetch_data(
-    file_class='OFFL',  # 'OFFL' for offline data, 'NRTI' for near-real-time
-    file_type='NO2___',  # Available types: 'NO2___', 'CO____', 'SO2___', 'O3__PR', 'HCHO__'
-    start_date='2025-02-20',
-    end_date='2025-03-06',
-    boundary=(120, 122, 22, 25),
-    limit=None
+"""SENTINEL-5P Data Processing Example"""
+from datetime import datetime
+from src.api import SENTINEL5PHub
+
+# 1. Set parameters
+start_date, end_date = datetime(2025, 3, 1), datetime(2025, 3, 13)
+
+# File class: 'NRTI' (Near Real-Time) or 'OFFL' (Offline processed)
+file_class = 'NRTI'
+
+# Available file types: 'NO2___', 'O3____', 'CO____', 'SO2___', 'CH4___', 'CLOUD_', 'AER_AI'
+file_type = 'NO2___'
+
+# Define region boundary (min_lon, max_lon, min_lat, max_lat)
+boundary = (120, 122, 22, 25)
+
+# 2. Create data hub instance
+sentinel_hub = SENTINEL5PHub(max_workers=3)
+
+# 3. Fetch data
+products = sentinel_hub.fetch_data(
+    file_class=file_class,
+    file_type=file_type,
+    start_date=start_date,
+    end_date=end_date,
+    boundary=boundary
 )
 
-# Download products
-fetcher.parallel_download(products)
-```
+# 4. Download data
+sentinel_hub.download_data(products)
 
-Example output:
-```
-╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
-│              Fetching sentinel-5p products (NO₂) from 2025-02-20 to 2025-03-06 ...               │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
-╭──────────────────────────────────────── Found 7 Products ────────────────────────────────────────╮
-│                                                                                                  │
-│                                       Product Information                                        │
-│┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ │
-│┃ No. ┃ Time                ┃ Name                                                  ┃      Size ┃ │
-│┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │
-││   1 │ 2025-02-25T04:54:18 │ S5P_OFFL_L2__NO2____20250225T043243...50226T204825.nc │ 589.88 MB │ │
-││   2 │ 2025-02-24T05:13:14 │ S5P_OFFL_L2__NO2____20250224T045140...50225T210805.nc │ 591.67 MB │ │
-││   3 │ 2025-02-24T03:31:44 │ S5P_OFFL_L2__NO2____20250224T031009...50226T055814.nc │ 593.46 MB │ │
-││   4 │ 2025-02-23T03:50:40 │ S5P_OFFL_L2__NO2____20250223T032906...50224T235016.nc │ 593.48 MB │ │
-││   5 │ 2025-02-22T04:09:37 │ S5P_OFFL_L2__NO2____20250222T034802...50223T200228.nc │ 590.79 MB │ │
-││   6 │ 2025-02-21T04:28:32 │ S5P_OFFL_L2__NO2____20250221T040659...50222T202822.nc │ 589.21 MB │ │
-││   7 │ 2025-02-20T04:47:30 │ S5P_OFFL_L2__NO2____20250220T042555...50221T204529.nc │ 564.02 MB │ │
-│└─────┴─────────────────────┴───────────────────────────────────────────────────────┴───────────┘ │
-│                                                                                                  │
-╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
+# 5. Process data
+sentinel_hub.process_data()
 ```
 
-### Processing Data
+### MODIS Example
+
 ```python
-from src.processing.data_processor import S5Processor
+"""MODIS Data Processing Example"""
+from datetime import datetime
+from src.api import MODISHub
+
+# 1. Set parameters
+start_date, end_date = datetime(2025, 3, 1), datetime(2025, 3, 12)
 
-# Initialize processor
-processor = S5Processor(
-    interpolation_method='rbf',
-    resolution=(5.5, 3.5),  # Resolution in km
-    mask_qc_value=0.5       # Quality control threshold
+# Product types: 'MOD04' (Terra) or 'MYD04' (Aqua)
+modis_product_type = "MYD04"
+
+# 2. Create data hub instance
+modis_hub = MODISHub()
+
+# 3. Fetch data
+products = modis_hub.fetch_data(
+    file_type=modis_product_type,
+    start_date=start_date,
+    end_date=end_date
 )
 
-# Process data
-processor.process_each_data(
-    file_class='OFFL',
-    file_type='NO2___',
-    start_date='2025-02-20',
-    end_date='2025-03-06',
+# 4. Download data
+modis_hub.download_data(products)
+
+# 5. Process data
+modis_hub.process_data()
+```
+
+### ERA5 Example
+
+```python
+"""ERA5 Data Processing Example"""
+from datetime import datetime
+from src.api import ERA5Hub
+
+# 1. Set parameters
+start_date, end_date = datetime(2025, 3, 1), datetime(2025, 3, 19)
+
+# Variables to retrieve (more options available)
+variables = ['boundary_layer_height']
+
+# Pressure levels in hPa (set to None for surface data only)
+pressure_levels = None
+
+# Region boundary (min_lon, max_lon, min_lat, max_lat)
+boundary = (119, 123, 21, 26)
+
+# Define observation stations
+STATIONS = [
+    {"name": "FS", "lat": 22.6294, "lon": 120.3461},  # Kaohsiung Fengshan
+    {"name": "NZ", "lat": 22.7422, "lon": 120.3339},  # Kaohsiung Nanzi
+    {"name": "TH", "lat": 24.1817, "lon": 120.5956},  # Taichung
+    {"name": "TP", "lat": 25.0330, "lon": 121.5654}   # Taipei
+]
+
+# 2. Create data hub instance
+era5_hub = ERA5Hub(timezone='Asia/Taipei')
+
+# 3. Fetch data
+era5_hub.fetch_data(
+    start_date=start_date,
+    end_date=end_date,
+    boundary=boundary,
+    variables=variables,
+    pressure_levels=pressure_levels,
+    stations=STATIONS,
 )
+
+# 4. Download data
+era5_hub.download_data()
+
+# 5. Process data
+era5_hub.process_data()
 ```
 
-## <div align="center">Data Description</div>
+## <div align="center">Data Sources</div>
+
+### Sentinel-5P
+- **Provider**: European Space Agency (ESA)
+- **Products**: NO₂, O₃, CO, SO₂, HCHO, Cloud, Aerosol Index
+- **Resolution**: 7 km x 3.5 km (at nadir)
+- **Frequency**: Daily global coverage
+
+### MODIS
+- **Provider**: NASA
+- **Products**: Aerosol Optical Depth (AOD)
+- **Platforms**: Terra (MOD04) and Aqua (MYD04) satellites
+- **Resolution**: 10 km at nadir
+- **Frequency**: 1-2 days global coverage
+
+### ERA5
+- **Provider**: European Centre for Medium-Range Weather Forecasts (ECMWF)
+- **Products**: Reanalysis dataset with 100+ atmospheric, land and oceanic parameters
+- **Resolution**: 0.25° x 0.25° global grid (about 31 km)
+- **Frequency**: Hourly data, monthly updates
+
+## <div align="center">Processing Pipeline</div>
+
+All data sources follow a consistent workflow:
+
+1. **Data Discovery**: Query available products based on date range and region
+2. **Download Management**: Efficient parallel downloading with error handling
+3. **Quality Control**: Filtering based on data quality flags
+4. **Spatial Processing**: 
+   - Sentinel-5P: RBF interpolation of sparse satellite data
+   - MODIS: Processing of gridded AOD values
+   - ERA5: Extraction of point values for weather stations
+5. **Visualization**: Generation of standardized maps and plots
+6. **Export**: Structured data storage in NetCDF and CSV formats
+
+## <div align="center">Automatic Data Management</div>
 
-### NO2 Data Processing
-This toolkit processes nitrogen dioxide (NO2) and other pollutant data from satellite observations with these key features:
+The toolkit includes built-in data retention management to prevent disk space issues:
 
-- **Quality Control**: Filters out low-quality measurements
-- **Spatial Interpolation**: Converts sparse satellite readings to regular grid
-- **Temporal Aggregation**: Daily, weekly, and monthly averages
-- **Visualization**: Generates high-quality maps and time series plots
-- **Data Export**: Creates NetCDF files compatible with other analysis tools
+- Automatically cleans files older than the configured retention period
+- Maintains directory structure while removing outdated files
+- Can be scheduled for periodic execution or triggered manually
 
 ## <div align="center">Contact</div>
 
-For bug reports and feature requests please visit [GitHub Issues](https://github.com/Alex870521/Satellite_S5P/issues).
+For bug reports and feature requests please visit [GitHub Issues](https://github.com/Alex870521/Satellite_DataKit/issues).