Shipping without evals is gambling
LLM demos look strong in a slide deck. Production fails on edge cases, prompt drift, stale retrieval, and silent quality regressions. An evaluation harness is how you catch that before customers do.
Evals will not make a weak product idea succeed. They do stop you from shipping a strong demo that quietly degrades every week.
Build a golden set first
Collect 30–100 real tasks from your domain — not synthetic toy prompts. Include easy wins, ambiguous asks, and hostile cases (injection attempts, missing context, contradictory instructions).
Label expected outcomes in a form you can score:
- Exact or near-exact strings for short answers
- JSON schemas for structured outputs
- Human rubrics (accuracy, tone, safety) for open-ended replies
Refresh the set when product language or policies change. A stale golden set creates false confidence.
Offline metrics that matter
Track a small set of numbers you can explain to product and finance:
- Task success rate against the golden set
- Factual grounding / citation hit rate for RAG features
- Format validity for structured outputs
- Latency and token cost per successful task
Online signals (thumbs, edits, deflection) matter later. First you need offline detection of regressions when someone edits a prompt or swaps a model.
Prompt and model changes need CI
Treat prompts like code. Store them in version control. When a system prompt, tool schema, or model ID changes, re-run the golden set in CI and block merges on large quality or cost drops.
Separate “experiment” runs from “release gates.” Experiments can be noisy. Release gates should be boring and stable.
Human-in-the-loop and cost budgets
For high-risk domains, route low-confidence or high-impact answers to humans. Combine model signals with business rules — confidence alone is not enough.
Cap tokens per request, cache retrieval where safe, and prefer smaller models for classification and routing. Quality and margin are both product requirements.
Next step
If you need a production LLM feature with evals and guardrails, see generative AI development, AI development, and ChatGPT integration.