feat: CLI and Agent Interface for OpenScreen

[lupuletic]

Motivation

OpenScreen is a powerful desktop app for screen recording and video editing, but it currently requires a GUI for all operations. As AI agents become the primary consumers of developer tools, OpenScreen needs a programmatic interface that agents can call directly — without a display or mouse.

Projects like Remotion have proven this model: a CLI as the primary agent interface, agent skills for guidance, and lightweight MCP for discovery. This issue proposes bringing the same approach to OpenScreen.

What we're building

1. CLI (openscreen) — 20+ commands

Project management (pure Node.js, instant):

• project create — Create .openscreen project from video recording
• project info — Inspect project metadata, regions, settings
• project validate — Validate project file integrity
• project edit — Modify settings (wallpaper, padding, aspect ratio, etc.)

Region management (pure Node.js, instant):

• zoom add/list/remove — Manage zoom effects (depth 1-6, focus coordinates)
• trim add/list/remove — Cut sections from the video
• speed add/list/remove — Change playback speed (0.25x - 2x)
• annotate add/list/remove — Add text, image, or arrow overlays

Rendering (Electron headless, seconds-minutes):

• render — Export as MP4 with all effects applied (WebCodecs + PixiJS)
• gif — Export as animated GIF
• still — Capture single frame as PNG/JPEG (planned)
• frames — Export frame sequence (planned)

All commands support --json for machine-readable output.

2. Shared code extraction (src/shared/)

Pure TypeScript types and logic extracted from the Electron app so both CLI and GUI share the same validation, normalization, and project schema code. Zero breaking changes — original files become thin re-exports.

3. Electron headless bridge

A --cli-export mode in electron/main.ts that creates a hidden BrowserWindow, runs the full export pipeline (WebCodecs + PixiJS), and streams progress to stdout. This reuses 100% of the existing rendering code — no FFmpeg dependency, no reimplementation.

4. Agent skills (SKILL.md + rules)

Following the agentskills.io protocol and Remotion's pattern: a master SKILL.md with 7 rule files covering project setup, zoom regions, trim/speed, annotations, export, frames, and troubleshooting.

5. Minimal MCP server

Single openscreen_help tool for documentation search. Agents with shell access call the CLI directly; the MCP server is just for discovery.

Architecture

CLI (Node.js)                    Electron (headless)
┌─────────────────────┐         ┌──────────────────────┐
│ project create/edit │         │ --cli-export mode     │
│ zoom/trim/speed add │         │ CliExportRenderer     │
│ annotate add/remove │         │ VideoExporter (WebCodecs) │
│ shortcuts get/set   │         │ GifExporter (PixiJS)  │
│ --json output       │ ──────▶ │ Progress via IPC      │
└─────────────────────┘ spawn   └──────────────────────┘
        │                               │
        ▼                               ▼
  src/shared/                     Valid MP4/GIF files
  (types, validation,             with all effects applied
   project schema)

Current status

🚧 Work in progress — Core implementation complete and tested E2E. Draft PR linked below.

What's done

• Shared code extraction (7 shared modules, zero breaking changes)
• CLI core (20+ commands, --json dual output, input validation)
• Electron headless bridge (MP4 + GIF export with progress)
• Agent skills (SKILL.md + 7 rule files)
• Minimal MCP server
• Security review + code quality review applied
• All 33 existing unit tests pass

What's next

• still command — capture single frame as PNG/JPEG
• frames command — export frame sequence
• CLI unit tests
• Contribution guide for CLI extensions

[TrendpilotAI]

@lupuletic this is supercool ! Programmatic is the way forward and I'm really excited to give this a try out. Happy to contribute more in this direction.

[siddharthvaddem]

can someone actually validate the need for this by testing out the #350

[lupuletic]

@siddharthvaddem , @TrendpilotAI Some screenshots below from using the CLI - the idea was for users to record via the app and then have an agent turn it into a real demo

going to fix the conflicts and clean it up and test more, I think the skill is key for having agent edited videos that look good with appropriately placed annotations, zooms, speed up, trims, etc.

[lupuletic]

Status update — PR #350 is ready for review

@siddharthvaddem @TrendpilotAI — following up on the validation ask, here's where things stand:

✅ Shipped in the PR

• Full CLI with --json dual output (project create/info/validate/edit, zoom/trim/speed/annotate add/list/remove, render/gif, shortcuts get/set)
• Shared code extraction into src/shared/* (pure TypeScript, zero GUI breaking changes — the original files are thin re-exports)
• Electron headless bridge (--cli-export mode) that reuses 100% of the existing WebCodecs + PixiJS export pipeline
• Agent skills — SKILL.md + 6 rule files covering project setup, zoom, trim/speed, annotations, export/render, and troubleshooting
• Minimal MCP server with a single openscreen_help doc-search tool

❌ Cut from the original scope (and why)

The still (single-frame capture) and frames (sequence export) commands from the original issue are not in this PR. They were stub commands that errored with "not yet implemented" at runtime, which misleads agents reading the skill docs. I removed the stubs, deleted the corresponding frames-stills.md rule file, and updated SKILL.md + the MCP keyword map so the command surface matches what actually works. These can come in a follow-up PR once there's an implementation — the skill docs shouldn't promise features the CLI doesn't deliver.

🎬 End-to-end validation on a real recording

I ran the CLI end-to-end on a real 47.96s AV1 recording (the test-yc.openscreen project you shared — navigating Y Combinator's Co-Founder Matching) with an agent-driven edit:

• 3 zoom regions (depth 3 on the YC landing hero, depth 3 on the profile header, depth 4 on the Accept CTA)
• 1 speed region (1.5× through the inbox navigation)
• 1 trim region (cut the outro tail)
• 2 text annotations ("Browse candidate profiles", "One click to accept")
• 1 red arrow figure annotation pointing at the Accept button

Output: 40.30s MP4, 1920×1080, 25.7 MB. Duration math checks to 0.01s (47.96 − 5 − 8/1.5 = 40.29s), confirming the renderer composes speed + trim + zoom + annotation layers without drift. All 7 skill rules walked cleanly as documented, sample frames visually verified that every effect landed at the right timestamp.

The demo mirrors the exact workflow you imagined: record in the OpenScreen app → hand the .openscreen file to an agent → get back a polished MP4. Full command log is in the PR description.

🔧 Rebased + conflicts resolved

The branch was ~80 commits behind main when I picked it up (dual-frame webcam layout #347, zoom focus fix, blur annotations, i18n rounds, etc.). Rebased cleanly onto current main and forward-ported upstream's additions into src/shared/*:

• "dual-frame" layout (with portrait-aspect restriction via isPortraitAspectRatio) in project-schema.ts
• BlurShape / BlurData / blur intensity constants + blurData? field on AnnotationRegion in types.ts
• PlaybackSpeed = number with clampPlaybackSpeed + extended SPEED_OPTIONS (3×/4×/5×)
• WebcamSizePreset + DEFAULT_WEBCAM_SIZE_PRESET
• addBlur action in shortcuts.ts
• zoomInDurationMs/zoomOutDurationMs on ZoomRegion

Also fixed a post-rebase Electron regression: upstream PR #300 added a path guard on the read-binary-file IPC handler that rejects reads outside RECORDINGS_DIR. Our CLI writes its export config to os.tmpdir(), so post-rebase, headless renders would fail with Failed to read CLI export config. Addressed by exposing a small approveReadablePath() helper from electron/ipc/handlers.ts and pre-approving the CLI config path (plus the video paths it references) in runCliExport() before the hidden renderer loads. Minimal, documented extension of the existing trust model — not a bypass. Fixed TDZ bug on safetyTimer and an unhandled loadURL() rejection in the same function while I was there.

🧪 Current test plan

• tsc --noEmit (root + CLI) — 0 errors
• biome check . — 0 errors
• vitest --run --exclude '**/*.browser.test.*' — 40/40 pass (includes 9 new upstream tests that all pass against the forward-ported src/shared/* modules)
• Real-world headless render on test-yc.openscreen — verified end-to-end post-rebase
• PR feat: add CLI and agent interface for OpenScreen #350 mergeable: MERGEABLE (no conflicts)
• CodeRabbit + Codex review threads addressed or resolved as stale

The PR is ready for review and merge. Happy to cut still / frames as a follow-up issue if that's useful.