AI Glossary · Applied AI Engineering
Evals
Automated tests that measure model or agent quality on task-specific criteria — accuracy, safety, groundedness, latency, cost — usually run in CI.
Definition
What is Evals?
Evals is automated tests that measure model or agent quality on task-specific criteria — accuracy, safety, groundedness, latency, cost — usually run in CI.
- Category
- Applied AI Engineering
- Glossary set
- 10 related terms
- Audience
- Enterprise AI leaders
Why does Evals matter in enterprise AI?
Evals matters in enterprise AI programs because it helps business and technology leaders align vocabulary, scope, ownership, and measurable outcomes.
Related terms in Applied AI Engineering
- AI Engineer
- A software engineer who ships production LLM systems — prompt design, RAG, tool integration, evals, cost/latency tuning. Distinct from ML engineer.
- Copilot
- An AI assistant embedded in a workflow (email, CRM, IDE, contact center desktop) that suggests actions in-line while a human remains in control.
- Cost per Task
- End-to-end token, retrieval and infrastructure cost to complete one unit of work — the correct unit economics for LLM systems, not price-per-token.
- Guardrails
- Runtime filters and policies that block unsafe inputs or outputs — PII, jailbreaks, toxicity, off-topic responses, unauthorized tool calls.
- IDP (Intelligent Document Processing)
- Extracting structured data from unstructured documents (claims, invoices, contracts) using OCR, layout models and LLMs.
- Latency Budget
- Allocation of end-to-end response time across retrieval, model calls and tool hops. Voice AI budgets are typically <1.2s per turn.