refactor(scraper): replace Crawlee with hand-rolled fetch+cheerio

[ThatXliner]

Summary

• Drop Crawlee + Playwright — replaced with fetch + cheerio directly, removing ~2100 lines of lockfile and heavy dependencies
• Unified upsert pipeline — merged upsertBill, upsertGovernmentContent, upsertCourtCase into a single upsertContent(type, data) using a discriminated union
• Shared utilities — added fetchWithRetry() (retry + backoff + Retry-After + timeout) and log() (timestamped, scraper-prefixed logging)
• Simplified runner — main.ts is now a loop over Scraper[] objects with yargs for CLI parsing
• Fixed pre-existing TS errors in google-images.ts

Net result: -2100 lines, fewer deps, unified patterns, same behavior.

Test plan

• cd apps/scraper && npx tsc --noEmit — compiles with only pre-existing @acme/db errors
• pnpm run start:dev govtrack — fetches bills, logs with [HH:MM:SS] [GovTrack] prefix
• pnpm run start:dev whitehouse — fetches articles with pagination
• pnpm run start:dev congress — fetches from Congress.gov API
• pnpm run start:dev scotus — fetches from CourtListener API
• pnpm run start:dev --help — shows yargs help output
• grep -r "crawlee" apps/scraper/src/ — no matches
• Docker build still works

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
billion-nextjs	Ready	Preview, Comment	Apr 3, 2026 5:19pm

[lcai000]

removing crawlee and playwright dependencies is good, more simple
reduced LOC is good

TODO: make more descriptive logging, the current logging prints out a bunch of random stuff and a lot is unnecesarry, change it to be better