Tool Stack
AI Observability Stack
Traces, cost, evals and quality drift for AI systems in production.
1 min readupdated 2026-08-01
/ quick answer
Know what your AI system actually did, what it cost and whether it got worse. Traces, cost, evals and quality drift for AI systems in production.
Traces, cost, evals and quality drift for AI systems in production. Know what your AI system actually did, what it cost and whether it got worse. The stack combines Langfuse (traces, evals, datasets), OpenTelemetry (spans across services), Sentry (errors and regressions), Postgres (run log and metrics), Grafana or Metabase (dashboards and alerts). This tool stack node is part of the Onexial knowledge graph and links to related concepts, workflows and tools below.
Purpose
Know what your AI system actually did, what it cost and whether it got worse.
Tools Included
- Langfuse (traces, evals, datasets)
- OpenTelemetry (spans across services)
- Sentry (errors and regressions)
- Postgres (run log and metrics)
- Grafana or Metabase (dashboards and alerts)
Workflow Supported
Alternatives
- Braintrust
- LangSmith
- Helicone
Use Cases
/ frequently asked
What is the AI Observability Stack stack for?
Know what your AI system actually did, what it cost and whether it got worse.
Which tools are in this stack?
Langfuse (traces, evals, datasets), OpenTelemetry (spans across services), Sentry (errors and regressions), Postgres (run log and metrics), Grafana or Metabase (dashboards and alerts).
Are there alternatives to this stack?
Yes — Braintrust, LangSmith, Helicone.
↳ connected nodes
Workflow↳ linked
Monitor an AI System in Production
See quality, cost and failure drift before your users report it.
Workflow↳ linked
Build an Eval Suite Before Optimising Prompts
Stop guessing whether a change improved anything.
Workflow↳ linked
Cut Agent Costs by 60% Without Losing Quality
A measurable cost-reduction pass for any agent already in production.
Use Case↳ linked
Platform Catches a 19% Quality Drop Before Users Did
Continuous sampling and evals caught silent degradation after a model update.
Dictionary↳ linked
Agent Cost Control
Agent cost control is the practice of budgeting tokens, steps and model tiers per task so autonomous systems stay economically viable at scale.
Workflow↳ linked
Ship an Autonomous Workflow Safely
Move an automation from human-triggered to autonomous without losing control.
Dictionary↳ linked
Context Engineering
Context engineering is the discipline of deciding exactly what information enters a model's context window, in what order and at what cost.
Dictionary↳ linked
AI Evaluation
AI evaluation is the measurement layer of an AI system: a fixed set of cases, a scoring method and a tracked pass rate you can regress against.
Dictionary↳ linked
AI Monitoring
AI monitoring is production observability for model-driven systems: traces, cost, latency, tool failures and output-quality drift.
Comparison↳ linked
Model-Graded Evals vs Assertion Evals
Assertions are cheap, fast and objective; model grading captures quality you cannot express as a rule.
Prompt↳ linked
Eval Rubric Prompt
Builds a scoring rubric a grader model can apply consistently.