docs: improve README.md

Author: Alex870521

README.md

@@ -1,31 +1,136 @@
-# Satellite_S5P
-### NO2 Data Processing Script
+## <div align="center">Satellite_S5P</div>
 
-This Python script is designed to process and analyze nitrogen dioxide (NO2) data from satellite observations. Here's a breakdown of its main components:
+<div align="center">
 
-### Data Extraction:
-The script uses the extract_data function to filter and extract relevant NO2 data from a given dataset based on predefined geographical conditions (latitude and longitude ranges) and a quality threshold (qa_value).
+![Python](https://img.shields.io/badge/Python-3.12-blue.svg)
+![GitHub last commit](https://img.shields.io/github/last-commit/Alex870521/aeroviz?logo=github)
 
-### Data Interpolation:
-The interp_data function utilizes a KD-tree algorithm to find the nearest neighbors in a 2D array of latitude and longitude. It then performs linear interpolation using the map_coordinates function to estimate NO2 values at specified target coordinates.
+</div>
 
-### Main Loop:
+<div align="center">
+<a href="https://github.com/Alex870521"><img src="https://cdn.simpleicons.org/github/0A66C2" width="3%" alt="GitHub"></a>
+<span style="margin: 0 1%"></span>
+<a href="https://www.linkedin.com/in/Alex870521/"><img src="https://cdn.simpleicons.org/linkedin/0A66C2" width="3%" alt="LinkedIn"></a>
+<span style="margin: 0 1%"></span>
+<a href="https://medium.com/@alex870521"><img src="https://cdn.simpleicons.org/medium/0A66C2" width="3%" alt="Medium"></a></div>
 
-The script iterates through folders corresponding to different months of a specified year, each containing NetCDF files of satellite observations.
-For each NetCDF file, it extracts and interpolates NO2 data and stores the results.
+---
 
-### Data Aggregation:
-The script aggregates the processed NO2 data from multiple files within a month, calculating the average.
+A Python toolkit for retrieving, processing, and visualizing Sentinel-5P satellite data, with a focus on air pollutants like NO₂, CO, SO₂, O₃, and HCHO.
 
-### Output:
-The processed NO2 data is stored in NetCDF format, with each file representing a month of observations.
+## <div align="center">Features</div>
 
-### Folder Skipping:
-The script checks if the output NetCDF file already exists. If it does, it skips processing that particular folder to avoid unnecessary computations.
+- Automated data retrieval from Copernicus Open Access Hub
+- Advanced data processing with interpolation and quality control
+- High-quality visualization of pollutant concentration maps
+- Time series analysis capabilities
+- Built-in file retention management to prevent storage overflow
 
-### Usage:
-The script is designed to be run for a specific year, processing monthly satellite observations of NO2 data and generating NetCDF files for further analysis or visualization.
+## <div align="center">Prerequisites</div>
 
-### Dependencies:
-The script relies on various Python libraries, including NumPy, xarray, scipy, and netCDF4, for efficient data manipulation and processing.
-This script is particularly useful for researchers or analysts working with satellite-derived NO2 data, allowing them to extract, interpolate, and aggregate information for further study.
+Before using this toolkit, you need to complete the following steps:
+
+1. **Copernicus Account Setup**:
+   - Register for a free account at [Copernicus Open Access Hub](https://scihub.copernicus.eu/dhus/#/home)
+   - Save your username and password for API access
+
+2. **Environment Configuration**:
+   - Create a `.env` file in the project root directory with your credentials:
+     ```
+     COPERNICUS_USERNAME=your_username
+     COPERNICUS_PASSWORD=your_password
+     ```
+
+## <div align="center">Installation</div>
+
+```bash
+# Clone the repository
+git clone https://github.com/Alex870521/Satellite_S5P.git
+
+# Navigate to the directory
+cd Satellite_S5P
+
+# Install required packages
+pip install -r requirements.txt
+```
+
+## <div align="center">Usage</div>
+
+### Fetching Satellite Data
+```python
+from src.api.sentinel_api import S5PFetcher
+
+# Initialize fetcher
+fetcher = S5PFetcher(max_workers=3)
+
+# Fetch products
+products = await fetcher.fetch_data(
+    file_class='OFFL',  # 'OFFL' for offline data, 'NRTI' for near-real-time
+    file_type='NO2___',  # Available types: 'NO2___', 'CO____', 'SO2___', 'O3__PR', 'HCHO__'
+    start_date='2025-02-20',
+    end_date='2025-03-06',
+    boundary=(120, 122, 22, 25),
+    limit=None
+)
+
+# Download products
+fetcher.parallel_download(products)
+```
+
+Example output:
+```
+╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
+│              Fetching sentinel-5p products (NO₂) from 2025-02-20 to 2025-03-06 ...               │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭──────────────────────────────────────── Found 7 Products ────────────────────────────────────────╮
+│                                                                                                  │
+│                                       Product Information                                        │
+│┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ │
+│┃ No. ┃ Time                ┃ Name                                                  ┃      Size ┃ │
+│┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │
+││   1 │ 2025-02-25T04:54:18 │ S5P_OFFL_L2__NO2____20250225T043243...50226T204825.nc │ 589.88 MB │ │
+││   2 │ 2025-02-24T05:13:14 │ S5P_OFFL_L2__NO2____20250224T045140...50225T210805.nc │ 591.67 MB │ │
+││   3 │ 2025-02-24T03:31:44 │ S5P_OFFL_L2__NO2____20250224T031009...50226T055814.nc │ 593.46 MB │ │
+││   4 │ 2025-02-23T03:50:40 │ S5P_OFFL_L2__NO2____20250223T032906...50224T235016.nc │ 593.48 MB │ │
+││   5 │ 2025-02-22T04:09:37 │ S5P_OFFL_L2__NO2____20250222T034802...50223T200228.nc │ 590.79 MB │ │
+││   6 │ 2025-02-21T04:28:32 │ S5P_OFFL_L2__NO2____20250221T040659...50222T202822.nc │ 589.21 MB │ │
+││   7 │ 2025-02-20T04:47:30 │ S5P_OFFL_L2__NO2____20250220T042555...50221T204529.nc │ 564.02 MB │ │
+│└─────┴─────────────────────┴───────────────────────────────────────────────────────┴───────────┘ │
+│                                                                                                  │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+### Processing Data
+```python
+from src.processing.data_processor import S5Processor
+
+# Initialize processor
+processor = S5Processor(
+    interpolation_method='rbf',
+    resolution=(5.5, 3.5),  # Resolution in km
+    mask_qc_value=0.5       # Quality control threshold
+)
+
+# Process data
+processor.process_each_data(
+    file_class='OFFL',
+    file_type='NO2___',
+    start_date='2025-02-20',
+    end_date='2025-03-06',
+)
+```
+
+## <div align="center">Data Description</div>
+
+### NO2 Data Processing
+This toolkit processes nitrogen dioxide (NO2) and other pollutant data from satellite observations with these key features:
+
+- **Quality Control**: Filters out low-quality measurements
+- **Spatial Interpolation**: Converts sparse satellite readings to regular grid
+- **Temporal Aggregation**: Daily, weekly, and monthly averages
+- **Visualization**: Generates high-quality maps and time series plots
+- **Data Export**: Creates NetCDF files compatible with other analysis tools
+
+## <div align="center">Contact</div>
+
+For bug reports and feature requests please visit [GitHub Issues](https://github.com/Alex870521/Satellite_S5P/issues).

main.py

@@ -22,7 +22,7 @@ async def fetch_data(file_class: ClassInput,
     """下載數據的工作流程"""
     try:
         rich_print(
-            f"正在獲取 sentinel-5p 衛星數據 ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
+            f"Fetching sentinel-5p products ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
 
         fetcher = S5PFetcher(max_workers=3)
 
@@ -36,18 +36,18 @@ async def fetch_data(file_class: ClassInput,
         )
 
         if products:
-            if rich_print("是否要下載數據？", confirm=True):
+            if rich_print("Do you want to download data？", confirm=True):
                 rich_print(
-                    f"開始下載 sentinel-5p 衛星數據 ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
+                    f"Start download sentinel-5p products ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
                 fetcher.parallel_download(products)
-                rich_print("數據下載完成！")
+                rich_print("Data download completed！")
             else:
-                rich_print("已取消下載操作")
+                rich_print("Download canceled")
         else:
-            rich_print("找不到符合條件的數據")
+            rich_print("No data matching the criteria was found")
 
     except Exception as e:
-        error_message = f"下載數據失敗: {str(e)}"
+        error_message = f"Failed to download data: {str(e)}"
         rich_print(error_message)
         logger.error(error_message)
 
@@ -58,9 +58,9 @@ def process_data(file_class: ClassInput,
                  end_date: str | datetime):
     """處理數據的工作流程"""
     try:
-        if rich_print("是否要處理數據？", confirm=True):
+        if rich_print("Do you want to process the data？", confirm=True):
             rich_print(
-                f"正在處理 sentinel-5p 衛星數據 ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
+                f"Processing sentinel-5p products ({PRODUCT_CONFIGS[file_type].display_name}) from {start_date} to {end_date} ...")
 
             processor = S5Processor(
                 interpolation_method='rbf',
@@ -75,12 +75,12 @@ def process_data(file_class: ClassInput,
                 end_date=end_date,
             )
 
-            rich_print("數據完成處理")
+            rich_print("Data processing completed")
         else:
-            rich_print("已取消處理操作")
+            rich_print("Processing canceled")
 
     except Exception as e:
-        error_message = f"處理數據失敗: {str(e)}"
+        error_message = f"Failed to process data: {str(e)}"
         rich_print(error_message)
         logger.error(error_message)
 
@@ -89,15 +89,15 @@ def main():
     # 步驟：
     # 1. 前往src.config.settings中更改輸出路徑（硬碟路徑）
     # 2. 設定參數
-    start, end = '2022-01-01', '2024-12-31'
+    start, end = '2025-02-20', '2025-03-06'
     file_class: ClassInput = 'OFFL'
     file_type: TypeInput = 'NO2___'
 
     # 3. 設定輸入輸出配置
     setup(file_type=file_type, start_date=start, end_date=end)
 
     # 4. 下載數據 (需要有.env 內含 COPERNICUS 帳號密碼才能用)
-    # asyncio.run(fetch_data(file_class=file_class, file_type=file_type, start_date=start, end_date=end))
+    asyncio.run(fetch_data(file_class=file_class, file_type=file_type, start_date=start, end_date=end))
 
     # 5. 處理與繪製數據
     process_data(file_class=file_class, file_type=file_type, start_date=start, end_date=end)

requirements.txt

@@ -19,4 +19,6 @@ rioxarray~=0.18.2
 imageio~=2.36.0
 pillow~=10.4.0
 pytest~=8.3.4
-tkcalendar~=1.6.1
+tkcalendar~=1.6.1
+certifi~=2024.8.30
+schedule~=1.2.2

run_pipeline.py

@@ -25,15 +25,7 @@
 # 設定輸入輸出配置
 setup(file_type='NO2___', start_date=today, end_date=today)
 
-# 設定日誌
-logging.basicConfig(
-    level=logging.INFO,
-    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
-    handlers=[
-        logging.FileHandler(LOGS_DIR / 'satellite_pipeline.log'),
-        logging.StreamHandler()
-    ]
-)
+
 logger = logging.getLogger(__name__)
 
 
@@ -117,7 +109,7 @@ def clean_old_files():
         retention_manager = FileRetentionManager(retention_days=DATA_RETENTION_DAYS)
 
         # 定義要清理的檔案類型（這裡可以根據實際情況調整）
-        file_types = ['NO2___', 'SO2___', 'O3__PR', 'HCHO__', 'CO____']
+        file_types = ['NO2___', 'HCHO__', 'CO____']
 
         # 執行清理，只針對FIGURE_DIR中的檔案
         results = retention_manager.clean_satellite_figure_data(BASE_DIR, file_types)
@@ -143,7 +135,7 @@ async def daily_task():
     # 設定參數 - 只處理當天的數據
     today = datetime.now().strftime('%Y-%m-%d')
     file_class: ClassInput = 'NRTI'
-    file_type: list[TypeInput] = ['NO2___', 'SO2___', 'O3__PR', 'HCHO__', 'CO____']
+    file_type: list[TypeInput] = ['NO2___', 'HCHO__', 'CO____']
 
     for file_tp in file_type:
         # 設定輸入輸出配置

src/api/sentinel_api.py

@@ -66,13 +66,13 @@ def __init__(self, max_workers: int = 5):
         }
 
     async def fetch_data(self,
-                   file_class: ClassInput,
-                   file_type: TypeInput,
-                   start_date: str,
-                   end_date: str,
-                   boundary: tuple[float, float, float, float] | None = None,
-                   limit: int | None = None,
-                   ) -> list[dict]:
+                        file_class: ClassInput,
+                        file_type: TypeInput,
+                        start_date: str,
+                        end_date: str,
+                        boundary: tuple[float, float, float, float] | None = None,
+                        limit: int | None = None,
+                        ) -> list[dict]:
         """
         擷取數據
 

src/config/settings.py

@@ -1,5 +1,8 @@
 """API 設定和常數"""
 from pathlib import Path
+import os
+import certifi
+os.environ['SSL_CERT_FILE'] = certifi.where()
 
 # API URLs
 COPERNICUS_TOKEN_URL = "https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token"
@@ -18,7 +21,7 @@
 DOWNLOAD_TIMEOUT = 180
 
 # 存儲路徑
-BASE_DIR = Path("/Volumes/Transcend/Sentinel-5P")
+BASE_DIR = Path("/Users/chanchihyu/DataCenter/Satellite")
 RAW_DATA_DIR = BASE_DIR / "raw"
 PROCESSED_DATA_DIR = BASE_DIR / "processed"
 GEOTIFF_DIR = BASE_DIR / "geotiff"