🚀 PromptKit Self‑Improvement Roadmap

[Alan-Jowett]

🚀 PromptKit Self‑Improvement Roadmap

A staged, verifiable plan for PromptKit to introspect, audit, and improve its own prompt components.

This issue tracks the long‑term evolution of PromptKit into a self‑auditing, self‑optimizing semantic system.
Each step is independent, adds value on its own, and moves PromptKit toward full introspection.

---

✅ 1. Prompt Graph Integrity Checks

Goal: Ensure all prompt components reference each other correctly.

Tasks

• Detect broken references
• Detect orphaned prompt components
• Detect missing required companion components (e.g., persona without protocol)

Verification

• CI job produces a structural integrity report
• Zero structural issues is the target state

---

✅ 2. Prompt Metadata Completeness Auditor

Goal: Ensure every prompt component includes required metadata.

Tasks

• Define required metadata fields
• Build a metadata completeness checker
• Add CI enforcement

Verification

• CI fails when metadata is missing
• Report lists incomplete components

---

🚧 3. Prompt Quality Profiler

Goal: Score each prompt using a consistent quality rubric.

Tasks

• Define rubric (clarity, determinism, constraints, ambiguity, failure modes)
• Implement profiler persona
• Add scoring thresholds

Verification

• CI outputs a score per prompt
• Prompts below threshold are flagged

---

🚧 4. Prompt Behavioral Auditor

Goal: Validate prompt behavior using controlled test cases.

Tasks

• Define test harness prompt
• Define expected‑behavior prompts
• Implement output comparison prompt

Verification

• Pass/fail per test case
• Behavioral drift reports generated

---

🚧 5. Prompt Drift Detector

Goal: Detect divergence between related prompt components.

Tasks

• Identify drift‑sensitive relationships (persona ↔ protocol, protocol ↔ format, etc.)
• Implement drift comparison prompt
• Add drift severity scoring

Verification

• CI reports drift levels
• Zero drift is the target state

---

🚧 6. Prompt Cross‑Audit

Goal: Ensure consistency across persona, protocol, format, validator, and improver components.

Tasks

• Implement cross‑audit prompt
• Define invariants for each relationship
• Add CI enforcement

Verification

• CI outputs mismatch report
• Zero mismatches is the target state

---

🔮 7. Prompt Improver (Human‑in‑the‑Loop)

Goal: Automatically propose improvements when audits fail.

Tasks

• Implement improver persona
• Generate patch proposals
• Generate PR descriptions

Verification

• Automated PRs created
• Human review required

---

🔮 8. Full Self‑Improvement Loop (Future)

Goal: PromptKit continuously audits and improves itself.

Tasks

• Nightly audit pipeline
• Automatic PR creation
• Semantic convergence metrics

Verification

• Automated PRs appear regularly
• Drift decreases over time
• Quality scores trend upward

---

📌 Notes

This roadmap is intentionally incremental.
Each step stands alone and improves PromptKit even if later steps are never implemented.

[Alan-Jowett]

Step 1: Prompt Graph Integrity Checks — Done ✅

Merged in PR #114. Added \ ests/validate-graph-integrity.py\ with 6 checks:

1. Broken paths — manifest \path\ fields → actual files on disk
2. Broken references — template persona/protocol/format/taxonomy refs → manifest entries
3. Orphaned files — component files on disk not listed in manifest
4. Missing companions — templates without required persona or protocols
5. Pipeline integrity — pipeline stage templates exist in manifest
6. Frontmatter refs — template file frontmatter cross-checked against manifest

CI workflow updated to run both validators on all component directories.

---

Remaining steps

• Step 2: Prompt Metadata Completeness Auditor — define required fields per component type, build checker, add CI enforcement
• Steps 3–8 — not yet started