Workflow
Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
1 min readupdated 2026-08-01
/ quick answer
Create a scored dataset first, then treat every change as an experiment against it. Stop guessing whether a change improved anything.
Stop guessing whether a change improved anything. The problem it solves: Teams tweak prompts based on a handful of eyeball tests and ship regressions they never see. Create a scored dataset first, then treat every change as an experiment against it. It runs in 6 steps, starting with collect 30–50 real inputs, weighted toward known failures. This workflow node is part of the Onexial knowledge graph and links to related concepts, workflows and tools below.
Problem
Teams tweak prompts based on a handful of eyeball tests and ship regressions they never see.
Solution
Create a scored dataset first, then treat every change as an experiment against it.
Steps
- 01Collect 30–50 real inputs, weighted toward known failures.
- 02Define the expected outcome per case: required facts, structure, forbidden content.
- 03Choose a scorer per case type: assertion for structure, rubric for quality.
- 04Record a baseline score before changing anything.
- 05Run the suite on every prompt, model or retrieval change; compare against baseline.
- 06Add every new production failure to the dataset the day it happens.
Tools Used
Prompts Used
Related Dictionary
/ frequently asked
What does the Build an Eval Suite Before Optimising Prompts workflow do?
Create a scored dataset first, then treat every change as an experiment against it.
What problem does Build an Eval Suite Before Optimising Prompts solve?
Teams tweak prompts based on a handful of eyeball tests and ship regressions they never see.
How many steps does Build an Eval Suite Before Optimising Prompts take?
6 steps. It starts with collect 30–50 real inputs, weighted toward known failures. and ends with add every new production failure to the dataset the day it happens..
Which tools does Build an Eval Suite Before Optimising Prompts need?
It uses ai-observability-stack, ai-testing-stack — each linked below with its own node.
↳ connected nodes
Dictionary↳ linked
AI Evaluation
AI evaluation is the measurement layer of an AI system: a fixed set of cases, a scoring method and a tracked pass rate you can regress against.
Dictionary↳ linked
AI Testing
AI testing covers two things: using AI to generate and maintain tests, and testing AI systems whose output is non-deterministic.
Tool Stack↳ linked
AI Observability Stack
Traces, cost, evals and quality drift for AI systems in production.
Tool Stack↳ linked
AI Testing Stack
Test deterministic code and probabilistic AI output in one pipeline.
Prompt↳ linked
Eval Rubric Prompt
Builds a scoring rubric a grader model can apply consistently.
Dictionary↳ linked
AI Monitoring
AI monitoring is production observability for model-driven systems: traces, cost, latency, tool failures and output-quality drift.
Comparison↳ linked
Model-Graded Evals vs Assertion Evals
Assertions are cheap, fast and objective; model grading captures quality you cannot express as a rule.
Use Case↳ linked
Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.