Skip to content

Add vibe-vs-structured demo with 3 scenarios#163

Draft
abeltrano wants to merge 2 commits intomainfrom
demo/vibe-vs-structured
Draft

Add vibe-vs-structured demo with 3 scenarios#163
abeltrano wants to merge 2 commits intomainfrom
demo/vibe-vs-structured

Conversation

@abeltrano
Copy link
Copy Markdown
Collaborator

Summary

Adds docs/demo/ with complete presenter materials for a live demo comparing plain "vibe" prompts against PromptKit-assembled structured prompts on the same task.

Scenarios

# Scenario Sample File Key Contrast
1 Code Review demo_server.c (5 planted bugs) Detection rate, false positives, severity classification
2 Requirements Authoring rate_limiter_description.md Testability, atomicity, implicit requirements surfaced
3 Bug Investigation demo_queue.c (TOCTOU race + red herring) Hypothesis rigor, red herring rejection, confidence labeling

What's Included

  • Code samples with carefully planted issues (answers hidden in source comments)
  • Presenter script (demo-script.md) with per-scenario timing, talking points, and scorecards
  • README explaining how to run the demo

Demo Format

Each scenario follows the same structure:

  1. Run the task with a plain vibe prompt ("Review this C code for bugs.")
  2. Run the same task with a PromptKit-assembled prompt (e.g., review-cpp-code template)
  3. Score both outputs on a 6-factor rubric (detection rate, specificity, epistemic honesty, etc.)

Designed for a 15-20 min demo (2 scenarios) or 25-30 min (all 3).

abeltrano and others added 2 commits April 1, 2026 15:24
Adds docs/demo/ with complete presenter materials for a live demo
comparing plain 'vibe' prompts against PromptKit-assembled structured
prompts on the same task.

Three scenarios with pre-built code samples and scorecards:
1. Code review — C echo server with 5 planted bugs (demo_server.c)
2. Requirements authoring — rate limiter from 3-sentence description
3. Bug investigation — TOCTOU race condition with red herring (demo_queue.c)

Includes README, full presenter script with timing notes, talking
points, and appendix with planted bug reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Strip all planted-bug comments, hints, and annotations from
demo_server.c and demo_queue.c so the LLM cannot see the answers
when the code is provided as context during the demo.

All answer details (bug descriptions, interleavings, red herrings,
and the implicit-requirements checklist) are now in answer-key.md,
which the presenter references but never includes in LLM context.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant