C · ProductionAdvanced45 min read
10. Evals, golden sets & failure modes
Prompt Engineering · 48 pages source format
Ship prompts like code: version them, test on golden cases, track regressions. Know the failure modes: hallucination, sycophancy, prompt injection.
What you'll learn
- Eval harnesses
- Golden sets
- Failure modes
Build a 10-case eval set
Mix easy, medium, adversarial. Run before/after prompt changes. No vibes-only shipping.
Do this now
10 cases for one prompt. Baseline score. Improve prompt. Re-score.
Clear?
Failure mode checklist
Hallucination (invents facts), sycophancy (agrees with wrong premise), injection (ignores instructions), drift (forgets format).
Do this now
Deliberately trigger each failure mode once. Document the trigger.
Clear?