|
63 | 63 | │ ├── vimeoDownloader.ts # Vimeo-specific download |
64 | 64 | │ └── hlsDownloader.ts # Generic HLS download (ffmpeg-based) |
65 | 65 | │ |
| 66 | +├── shared/ # Shared utilities |
| 67 | +│ ├── index.ts # Exports |
| 68 | +│ ├── auth.ts # Session management |
| 69 | +│ ├── http.ts # HTTP utilities |
| 70 | +│ ├── url.ts # URL parsing utilities |
| 71 | +│ ├── slug.ts # Slug generation |
| 72 | +│ ├── fs.ts # File system utilities |
| 73 | +│ └── parallelWorker.ts # Parallel processing with browser tabs |
| 74 | +│ |
66 | 75 | ├── state/ # State management |
67 | 76 | │ ├── index.ts # State exports |
68 | 77 | │ └── database.ts # SQLite database for sync state |
@@ -155,18 +164,29 @@ Centralized configuration with Zod validation. |
155 | 164 | ▼ |
156 | 165 | 4. Authenticate ─────────► Browser session (cached or interactive) |
157 | 166 | │ |
158 | | -5. Navigate ────────────► Extract course structure (modules, lessons) |
| 167 | +5. Navigate ────────────► Extract course structure (parallel browser tabs) |
159 | 168 | │ |
160 | | -6. For each lesson: │ |
161 | | - ├─► Extract ─────────► Get video URL + text content |
162 | | - ├─► Save content ────► Write Markdown to disk |
163 | | - └─► Queue video ─────► Add to download queue |
| 169 | +6. Extract content ────► Parallel workers extract lessons simultaneously |
| 170 | + ├─► Get video URL + text content |
| 171 | + ├─► Write Markdown to disk |
| 172 | + └─► Queue video for download |
164 | 173 | │ |
165 | 174 | 7. Process queue ───────► Download videos with concurrency control |
166 | 175 | │ |
167 | 176 | 8. Done ────────────────► Summary output |
168 | 177 | ``` |
169 | 178 |
|
| 179 | +### Parallel Processing |
| 180 | + |
| 181 | +The `parallelWorker` module provides a shared worker pool for parallel operations: |
| 182 | + |
| 183 | +- **Worker Pool**: Multiple browser tabs share the same authenticated session |
| 184 | +- **Task Queue**: Tasks are distributed across workers automatically |
| 185 | +- **Progress Tracking**: Real-time aggregated progress across all workers |
| 186 | +- **Error Isolation**: Failed tasks don't crash other workers |
| 187 | + |
| 188 | +Used by: Course scanning, content extraction, video downloads. |
| 189 | + |
170 | 190 | ## Platform-Specific Details |
171 | 191 |
|
172 | 192 | ### Skool |
|
0 commit comments