Skip to content

Conversation

@fank
Copy link
Member

@fank fank commented Jan 30, 2026

Summary

This PR implements chunked streaming playback for large recordings that would otherwise crash browsers due to memory constraints. Recordings are converted from JSON to binary formats (Protobuf/FlatBuffers) and split into chunks that are loaded on-demand during playback.

Key Features

  • Chunked binary formats: Convert large JSON recordings to Protobuf or FlatBuffers
  • On-demand streaming: Load only the chunks needed for current playback position
  • Event-driven conversion: Conversion starts immediately after upload (no waiting for cron)
  • Background worker: Retries failed conversions and handles batch processing
  • Browser LRU cache: Keep 3 chunks in memory for smooth playback
  • Optional persistent cache: Enable with ?cache=1 URL parameter

Architecture

Upload → Save JSON.gz → Trigger Conversion → Create Chunks
                              ↓
                    manifest.pb (metadata, entities, events)
                    chunks/0000.pb, 0001.pb, ... (frame data)
                              ↓
Playback → Load manifest → Stream chunks on-demand

New Endpoints

Endpoint Description
GET /api/v1/operations/:id/format Get storage format info
GET /api/v1/operations/:id/manifest Stream manifest (binary)
GET /api/v1/operations/:id/chunk/:index Stream chunk (binary)

Configuration

{
  "conversion": {
    "enabled": true,
    "interval": "5m",
    "storageEngine": "protobuf"
  }
}

Files Changed

  • Backend: Storage engine interface, Protobuf/FlatBuffers engines, conversion worker
  • Frontend: ChunkManager, StorageManager, streaming playback in ocap.js
  • Schemas: Protobuf and FlatBuffers definitions

Test plan

  • Upload a small recording, verify immediate conversion
  • Upload a large recording (500MB+), verify chunked playback works
  • Verify playback controls (play, pause, seek) work in streaming mode
  • Verify legacy JSON recordings still work
  • Test with ?cache=1 to enable browser caching

claude and others added 19 commits January 29, 2026 23:27
This design document outlines the architecture for handling gigabyte-scale
mission recordings without browser memory issues. Key features:

- Storage engine abstraction (JSON legacy, Protobuf, FlatBuffers)
- Database tracks storage format per recording
- Background conversion queue for JSON → binary format
- Chunked loading with OPFS/IndexedDB caching
- Maximum memory budget of ~22MB regardless of recording size
- New API endpoints for manifest and chunk streaming

Addresses recurring memory limit issues reported by users.

https://claude.ai/code/session_01X5htkP9AzhbWxEbVjtUoZ5
Adds database migration v3 to support multiple storage formats
(json, protobuf, flatbuffers) and track conversion progress.
Extends Operation struct and database queries to track which
storage format each recording uses and its conversion status.
Defines Engine interface for pluggable storage formats with
Manifest, Chunk, and Frame types for chunked playback.
Provides backward-compatible reading of existing gzipped JSON
recordings. Does not support chunked loading or conversion.
Defines Manifest, Chunk, Frame, EntityState, Event, and Marker
messages for efficient binary serialization.
Reads manifest and chunks from protobuf files. Supports streaming
via GetChunkReader for efficient chunk delivery.
Converts legacy JSON recordings to chunked protobuf format.
Parses entities, events, markers, and times into manifest,
then writes frame data into separate chunk files.
- Add FlatBuffers schema and generated Go code for zero-copy reads
- Implement FlatBuffersEngine with Convert, GetManifest, GetChunk methods
- Add --format flag to CLI convert command (protobuf/flatbuffers)
- Update conversion worker to support configurable storage format
- Add StorageFormat field to worker Config
- Implement protobuf engine Convert method (was stub)
- Fix operation Select query parameter order and date filtering
- Add frontend streaming playback support:
  - ProtobufDecoder for parsing binary chunks
  - StorageManager for OPFS/IndexedDB caching
  - ChunkManager for LRU chunk management
  - loadOperation() helper to auto-detect format
- Update index.html to load streaming scripts
- Add loading indicators and Safari ITP warning
FlatBuffers:
- Add comprehensive tests for FlatBuffers engine
- Create JavaScript FlatBuffers decoder for frontend
- Fix handler to return correct content type (x-flatbuffers)
- Update ChunkManager to support format parameter
- Update processOpStreaming to pass format to decoder

Entity state updates (streaming mode):
- Add Unit.updateFromState() for isInVehicle, isPlayer, name
- Add Vehicle.updateFromState() for crewIds
- Fix chunk loading race condition - don't remove markers during load

The FlatBuffers implementation is now complete with end-to-end support
for both protobuf and flatbuffers storage formats.
Move scattered schema files into a clean structure:

Before (messy):
- flatbuffers/ocap.fbs
- ocap/fb/*.go
- proto/ocap.proto
- proto/ocap.pb.go
- static/scripts/proto/ocap.js
- static/scripts/flatbuffers/decoder.js

After (clean):
- schemas/protobuf/ocap.proto + ocap.pb.go
- schemas/flatbuffers/ocap.fbs + generated/*.go
- static/scripts/decoders/protobuf.js + flatbuffers.js

Update all Go import paths accordingly.
Server-side:
- Add GetManifestReader() to Engine interface for raw binary streaming
- Implement GetManifestReader in JSON, Protobuf, and FlatBuffers engines
- Update handler to stream raw manifest for binary formats
- Add --set-format CLI command for testing format switching

Client-side:
- Fix FlatBuffers decoder field order (times=8, events=9 per schema)
- Add markers array to manifest initialization
- Make StorageManager cache format-aware (prevents cross-format cache hits)
- Update ChunkManager to pass format when accessing storage
- Add overview of chunked streaming feature
- Document configuration settings for conversion
- Add storage formats comparison table
- Include ASCII workflow diagram
- Add Mermaid flowcharts for playback and conversion flows
- Document CLI convert commands
- Update Docker environment variables
- Fix build commands (./src/web → ./cmd)
- Create docs/streaming-architecture.md with Mermaid flowcharts
- Add browser caching explanation
- Simplify README with link to detailed docs
- Add event-driven conversion: trigger conversion immediately after upload
  instead of waiting for the background worker interval
- Use structured JSON logging (slog) for consistent log format
- Make browser caching opt-in via ?cache=1 URL parameter to avoid
  stale cache issues during development
- Fix marker position array order in streaming mode
- Calculate and store mission duration from recording metadata
- Return auto-generated ID from Store() for immediate use
The --all flag now converts all recordings in the database, including
re-converting already converted ones. This is useful for:
- Converting existing JSON recordings after upgrade
- Re-converting when changing formats (protobuf → flatbuffers)

The background worker still uses SelectPending for incremental processing.
@fank fank merged commit ca03246 into main Jan 30, 2026
3 checks passed
@fank fank deleted the claude/large-recording-playback-bNyF9 branch January 30, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants