Skip to content

Commit 496cbf9

Browse files
committed
docs: document shared utilities and parallel worker architecture
- Add shared/ module documentation to ARCHITECTURE.md - Document parallel processing with worker pool - Update data flow diagram for parallel operations
1 parent c05b4af commit 496cbf9

File tree

1 file changed

+25
-5
lines changed

1 file changed

+25
-5
lines changed

ARCHITECTURE.md

Lines changed: 25 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,15 @@ src/
6363
│ ├── vimeoDownloader.ts # Vimeo-specific download
6464
│ └── hlsDownloader.ts # Generic HLS download (ffmpeg-based)
6565
66+
├── shared/ # Shared utilities
67+
│ ├── index.ts # Exports
68+
│ ├── auth.ts # Session management
69+
│ ├── http.ts # HTTP utilities
70+
│ ├── url.ts # URL parsing utilities
71+
│ ├── slug.ts # Slug generation
72+
│ ├── fs.ts # File system utilities
73+
│ └── parallelWorker.ts # Parallel processing with browser tabs
74+
6675
├── state/ # State management
6776
│ ├── index.ts # State exports
6877
│ └── database.ts # SQLite database for sync state
@@ -155,18 +164,29 @@ Centralized configuration with Zod validation.
155164
156165
4. Authenticate ─────────► Browser session (cached or interactive)
157166
158-
5. Navigate ────────────► Extract course structure (modules, lessons)
167+
5. Navigate ────────────► Extract course structure (parallel browser tabs)
159168
160-
6. For each lesson: │
161-
├─► Extract ─────────► Get video URL + text content
162-
├─► Save content ────► Write Markdown to disk
163-
└─► Queue video ─────► Add to download queue
169+
6. Extract content ────► Parallel workers extract lessons simultaneously
170+
├─► Get video URL + text content
171+
├─► Write Markdown to disk
172+
└─► Queue video for download
164173
165174
7. Process queue ───────► Download videos with concurrency control
166175
167176
8. Done ────────────────► Summary output
168177
```
169178

179+
### Parallel Processing
180+
181+
The `parallelWorker` module provides a shared worker pool for parallel operations:
182+
183+
- **Worker Pool**: Multiple browser tabs share the same authenticated session
184+
- **Task Queue**: Tasks are distributed across workers automatically
185+
- **Progress Tracking**: Real-time aggregated progress across all workers
186+
- **Error Isolation**: Failed tasks don't crash other workers
187+
188+
Used by: Course scanning, content extraction, video downloads.
189+
170190
## Platform-Specific Details
171191

172192
### Skool

0 commit comments

Comments
 (0)