456
Tool Stack

AI Testing Stack

Test deterministic code and probabilistic AI output in one pipeline.

1 min readupdated 2026-08-01

/ quick answer

Keep quality measurable when part of the system is non-deterministic. Test deterministic code and probabilistic AI output in one pipeline.

Test deterministic code and probabilistic AI output in one pipeline. Keep quality measurable when part of the system is non-deterministic. The stack combines Vitest or Pytest (unit and integration), Playwright (end-to-end), Langfuse or Braintrust (eval runs and scoring), GitHub Actions (gates on every commit), A stored regression set of real production failures. This tool stack node is part of the Onexial knowledge graph and links to related concepts, workflows and tools below.
Purpose
Keep quality measurable when part of the system is non-deterministic.
Tools Included
  • Vitest or Pytest (unit and integration)
  • Playwright (end-to-end)
  • Langfuse or Braintrust (eval runs and scoring)
  • GitHub Actions (gates on every commit)
  • A stored regression set of real production failures
Workflow Supported
Alternatives
  • Promptfoo
  • DeepEval
  • Custom eval harness
/ frequently asked

What is the AI Testing Stack stack for?

Keep quality measurable when part of the system is non-deterministic.

Which tools are in this stack?

Vitest or Pytest (unit and integration), Playwright (end-to-end), Langfuse or Braintrust (eval runs and scoring), GitHub Actions (gates on every commit), A stored regression set of real production failures.

Are there alternatives to this stack?

Yes — Promptfoo, DeepEval, Custom eval harness.

/ topics#ai-development#stack#testing
↳ connected nodes
Workflow↳ linked
Build a Test Suite for a Non-Deterministic AI Feature
Grade probabilistic output without brittle snapshot tests.
Workflow↳ linked
Build an AI Code Review Loop
Catch what agents get wrong before a human reads the PR.
Dictionary↳ linked
AI Software Engineering
AI software engineering is the practice of building software where agents write most of the code and humans own architecture, review and verification.
Dictionary↳ linked
AI Testing
AI testing covers two things: using AI to generate and maintain tests, and testing AI systems whose output is non-deterministic.
Prompt↳ linked
AI Test Generation Prompt
Generates behavioural tests and edge cases, not tautologies.
Use Case↳ linked
Two-Founder Startup Ships Like a Team of Six
Coding agents plus a hard verification gate replaced three planned hires.
Dictionary↳ linked
AI Evaluation
AI evaluation is the measurement layer of an AI system: a fixed set of cases, a scoring method and a tracked pass rate you can regress against.
Workflow↳ linked
Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
Comparison↳ linked
Model-Graded Evals vs Assertion Evals
Assertions are cheap, fast and objective; model grading captures quality you cannot express as a rule.
Use Case↳ linked
Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.