|
2 | 2 |
|
3 | 3 | Wiki statistics tracker and Archive.org backup status checker. |
4 | 4 |
|
5 | | -## Features |
6 | | - |
7 | | -- 📊 Track MediaWiki site statistics over time |
8 | | -- 📦 Check Archive.org for existing backups |
9 | | -- 🚀 Fast API built with FastAPI + httpx |
10 | | -- 💾 MongoDB + Beanie ODM |
11 | | -- 📝 Comprehensive logging with loguru |
12 | | -- ⚡ Managed with uv for lightning-fast dependency management |
13 | | - |
14 | | -## Quick Start |
15 | | - |
16 | | -### Development |
17 | | - |
18 | | -```bash |
19 | | -# Install uv (if not already installed) |
20 | | -curl -LsSf https://astral.sh/uv/install.sh | sh |
21 | | - |
22 | | -# Install dependencies |
23 | | -uv sync |
24 | | - |
25 | | -# Copy environment file |
26 | | -cp .env.example .env |
27 | | - |
28 | | -# Start MongoDB (Docker) |
29 | | -docker-compose up -d mongodb |
30 | | - |
31 | | -# Run development server |
32 | | -uv run python -m wikikeeper.app.main |
33 | | - |
34 | | -# API will be available at http://localhost:8000 |
35 | | -# API docs at http://localhost:8000/docs |
36 | | -``` |
37 | | - |
38 | | -### Production (Docker) |
39 | | - |
40 | | -```bash |
41 | | -# Build and start all services |
42 | | -docker-compose up -d |
43 | | - |
44 | | -# View logs |
45 | | -docker-compose logs -f |
46 | | - |
47 | | -# Stop services |
48 | | -docker-compose down |
49 | | -``` |
50 | | - |
51 | | -## Requirements |
52 | | - |
53 | | -- Python 3.13+ |
54 | | -- MongoDB 8+ |
55 | | -- uv (package manager) |
56 | | - |
57 | | -## API Endpoints |
58 | | - |
59 | | -- `GET /` - API info |
60 | | -- `GET /health` - Health check |
61 | | -- `GET /api/wikis` - List wikis |
62 | | -- `POST /api/wikis` - Add new wiki |
63 | | -- `GET /api/wikis/{id}` - Get wiki details |
64 | | -- `POST /api/wikis/{id}/check` - Trigger stats collection |
65 | | -- `GET /api/wikis/{id}/stats` - Get historical stats |
66 | | -- `GET /api/wikis/{id}/archives` - Get archive info |
67 | | -- `POST /api/wikis/{id}/check-archive` - Check Archive.org |
68 | | -- `GET /api/stats/summary` - Overall statistics |
69 | | - |
70 | | -## Architecture |
71 | | - |
72 | | -``` |
73 | | -wikikeeper/ |
74 | | -├── src/wikikeeper/ |
75 | | -│ ├── app/ # FastAPI application |
76 | | -│ ├── core/ # Config, exceptions, logging |
77 | | -│ ├── db/ # Database connection |
78 | | -│ ├── models/ # Beanie ODM models |
79 | | -│ └── services/ # Business logic |
80 | | -│ ├── mediawiki.py # MediaWiki API client (httpx) |
81 | | -│ ├── archive_checker.py # Archive.org checker |
82 | | -│ └── collector.py # Data collector |
83 | | -├── tests/ # Unit tests |
84 | | -├── logs/ # Application logs |
85 | | -└── docker-compose.yml |
86 | | -``` |
87 | | - |
88 | | -## Implementation Notes |
89 | | - |
90 | | -### MediaWiki API Client |
91 | | -- Uses **httpx** (async) instead of requests |
92 | | -- Fetches siteinfo (statistics + general info) |
93 | | -- Reference: wikiteam3 implementation (not used as dependency) |
94 | | -- API: https://www.mediawiki.org/wiki/API:Siteinfo |
95 | | - |
96 | | -### Archive.org Checker |
97 | | -- Uses **internetarchive** library for search |
98 | | -- Uses **httpx** for metadata fetching |
99 | | -- Reference: wikiapiary-wikiteam-bot (not used as dependency) |
100 | | -- Docs: https://archive.org/help/aboutsearch.htm |
101 | | - |
102 | | -### Database |
103 | | -- MongoDB with Beanie ODM |
104 | | -- Time-series data for statistics |
105 | | -- Separate collection for Archive.org metadata |
106 | | - |
107 | | -## Development |
108 | | - |
109 | | -```bash |
110 | | -# Install dependencies |
111 | | -uv sync |
112 | | - |
113 | | -# Run with dev dependencies |
114 | | -uv sync --dev |
115 | | - |
116 | | -# Run tests |
117 | | -uv run pytest |
118 | | - |
119 | | -# Run with coverage |
120 | | -uv run pytest --cov=wikikeeper --cov-report=html |
121 | | - |
122 | | -# Type checking |
123 | | -uv run mypy src/ |
124 | | - |
125 | | -# Linting |
126 | | -uv run ruff check src/ |
127 | | - |
128 | | -# Format code |
129 | | -uv run ruff format src/ |
130 | | -``` |
131 | | - |
132 | | -## Migration from pdm |
133 | | - |
134 | | -This project was migrated from pdm to uv for better performance and compatibility: |
135 | | - |
136 | | -- Removed pdm.lock, .venv, and pdm-specific files |
137 | | -- Updated build backend from pdm to hatchling |
138 | | -- Updated Python version to 3.13 |
139 | | -- All commands now use `uv` instead of `pdm` |
140 | | - |
141 | | -## License |
142 | | - |
143 | | -AGPL-3.0-or-later |
0 commit comments