[M4-P15] CI Validation & Documentation for Jupytext Migration

[Gorkowski]

Dependencies:

#996 [M4-P14] ──► #THIS [M4-P15]

Dependencies:

• M4-P14: [M4-P14] Pre-commit Hook Implementation for Jupytext Sync #996

---

Description

Add CI validation for notebook sync status, clean up any stale files, and update documentation. This is Phase 15 (final phase) of the M4 Jupytext Full Migration maintenance plan.

This phase completes the M4 migration by ensuring CI catches out-of-sync notebooks and documentation reflects the new workflow.

Context

This is the final phase of the M4 maintenance plan to complete Jupytext paired sync migration. After all notebooks are converted (M4-P1 through M4-P13) and pre-commit hooks are implemented (M4-P14), this phase adds CI validation and updates documentation.

Value:

• CI fails if notebooks are out of sync (prevents broken notebooks in main)
• CI executes notebooks to ensure they work (including slow simulation notebooks)
• Documentation updated with final workflow
• Maintenance plan marked as Shipped

Scope

Estimated Lines of Code: ~100 lines
Complexity: Medium

Files to Modify:

• .github/workflows/test.yml or new workflow file (+50 LOC)
• adw-docs/documentation_guide.md (+30 LOC)
• adw-docs/dev-plans/maintenance/M4-jupytext-full-migration.md (status update)
• adw-docs/dev-plans/maintenance/index.md (status update)

Files to Delete:

• Any remaining .bak files from notebook conversions

Acceptance Criteria

CI Validation

• Add CI job to check notebook sync status (validate_notebook --check-sync)
• CI fails if any notebook is out of sync
• Add CI job to execute all notebooks (with appropriate timeout handling)
• Configure timeout handling for slow notebooks (1200s for Simulations)
• CI executes on PR and main branch pushes

Cleanup

• Remove any .ipynb.bak files created during conversion
• Verify no stale .py files without matching notebooks
• Verify no orphaned notebooks without .py pairs

Documentation Updates

• Update adw-docs/documentation_guide.md with final Jupytext workflow
• Document the pre-commit hook behavior
• Document CI validation behavior
• Document how to handle slow notebooks
• Update M4 maintenance plan status to "Shipped"
• Update maintenance index with completion status

Technical Notes

CI Workflow Addition

# Add to .github/workflows/test.yml or create new workflow

jobs:
  notebook-validation:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      
      - name: Install dependencies
        run: |
          pip install -e .[dev]
          pip install jupytext nbconvert nbclient
      
      - name: Check notebook sync status
        run: |
          python3 .opencode/tool/validate_notebook.py docs/Examples --recursive --check-sync
      
      - name: Execute notebooks (fast)
        run: |
          python3 .opencode/tool/run_notebook.py docs/Examples --recursive --timeout 600 \
            --exclude "docs/Examples/Simulations/*"
      
      - name: Execute slow notebooks
        run: |
          python3 .opencode/tool/run_notebook.py docs/Examples/Simulations --recursive --timeout 1200

Documentation Guide Update

Add section to adw-docs/documentation_guide.md:

## Jupytext Paired Sync Workflow

All notebooks in `docs/Examples/` use Jupytext paired sync:

### Editing Notebooks

1. Edit the `.py` file (not the `.ipynb`)
2. Pre-commit hook automatically:
   - Lints the `.py` file
   - Syncs to `.ipynb`
   - Executes the notebook
   - Stages both files

### Slow Notebooks (Simulations)

Notebooks in `docs/Examples/Simulations/` are excluded from pre-commit
(too slow) but validated in CI.

### CI Validation

CI checks:
- All notebooks are in sync (`--check-sync`)
- All notebooks execute without errors

Maintenance Plan Status Update

Update adw-docs/dev-plans/maintenance/M4-jupytext-full-migration.md:

• Change Status: Ready to Status: Shipped
• Add completion date to Change Log
• Update Phase Summary Table with all phases completed

Cleanup Commands

# Find and remove .bak files
find docs/Examples -name "*.ipynb.bak" -delete

# Verify all notebooks have .py pairs
find docs/Examples -name "*.ipynb" | while read nb; do
    py="${nb%.ipynb}.py"
    if [ ! -f "$py" ]; then
        echo "Missing: $py"
    fi
done

References

Maintenance Plan:

• adw-docs/dev-plans/maintenance/M4-jupytext-full-migration.md - Full migration plan

Related:

• [M4-P14] Pre-commit Hook Implementation for Jupytext Sync #996 - M4-P14 (dependency - pre-commit hooks)
• [M4-P1] Jupytext Migration: Aerosol + Equilibria (3 notebooks) #983-[M4-P13] Jupytext Migration: Simulations - Organic + Biomass (2 long-running notebooks) #995 - All conversion phases
• adw-docs/documentation_guide.md - To be updated