docs: document preflight sentinel, evaluator interfaces, and timeouts

[ASRagab]

Summary

• Evaluator interface selection guide — new "2b. Choose Your Evaluator Interface" section in optimization-guide with decision table (Python API vs CLI command vs HTTP), code examples, and guidance to prefer the Python API for in-process evaluators
• Preflight sentinel documentation — documents the undocumented __optimize_anything_preflight__ sentinel sent by the CLI with a 10-second timeout, and how to handle it in evaluators
• Timeout documentation — documents the 30-second command_evaluator timeout (not configurable via CLI)
• Preflight guards in evaluator patterns — adds fast-return sentinel detection to the shared I/O contract and all 4 evaluator templates (prompt, code, docs, agent)

Motivation

While integrating optimize-anything with a production evaluator that makes real Gemini API calls (~30-60s per eval), the CLI preflight timed out at 10s. The evaluator parsed the preflight payload as a real candidate because the sentinel value (__optimize_anything_preflight__) is undocumented. Additionally, the 30s command_evaluator timeout would have been the next blocker even after fixing preflight.

These docs prevent future users from hitting the same wall and guide them toward the Python API when their evaluator is in-process Python code.

Test plan

• Verify all three skill files render correctly when loaded by Claude Code
• Verify preflight guard examples work: echo '{"candidate":"__optimize_anything_preflight__"}' | python3 eval_prompt.py returns {"score":0.5} instantly
• Verify no existing skill content was removed or altered (additions only)

🤖 Generated with Claude Code

---

[devin-ai-integration]

Devin Review found 1 potential issue.

View 3 additional findings in Devin Review.

[devin-ai-integration]

🟡 Pattern 2 (bash evaluator) missing preflight guard despite commit claiming all patterns updated

The commit message states "add preflight guard to all evaluator pattern templates" and the I/O contract section (skills/evaluator-patterns/SKILL.md:14) recommends a preflight guard for all evaluators, yet Pattern 2 (evaluator.sh, lines 138–208) is the only template that was not updated with one. Ironically, this is the bash evaluator that runs pytest — arguably the slowest pattern and the most likely to exceed the 10-second preflight timeout (src/optimize_anything/preflight.py:99). Users who copy this template verbatim will have their evaluator fail the preflight check, blocking optimization entirely (src/optimize_anything/cli.py:490-491).

(Refers to lines 138-209)

Prompt for agents

In skills/evaluator-patterns/SKILL.md, add a preflight guard to the Pattern 2 bash evaluator template (evaluator.sh). After the line that extracts the candidate (line 143, candidate="$(printf '%s' "$payload" | python3 -c ...)"), add a preflight check before the workdir/pytest logic, similar to:

# Preflight guard for optimize-anything CLI
if [ "$candidate" = "__optimize_anything_preflight__" ]; then
  printf '{"score":0.5}\n'
  exit 0
fi

This should be inserted around line 144, before the mktemp/workdir creation on line 145.

---

Was this helpful? React with 👍 or 👎 to provide feedback.