LLM evals
AI systemsDefined term
An eval is a repeatable measurement of model behavior: a dataset of cases, a task definition, and a scoring method — exact match, rubric, or a judging model. For agents, evals extend to whole trajectories: did the loop reach the goal, at what cost, with what failure modes.
Evals are the difference between a demo and a system. Because model updates shift behavior invisibly, teams pin expectations in eval suites the way software teams pin regressions in tests — and judge-model evals inherit their own calibration problem, which is why humans audit the judges.
Browse the glossary
AI systems5 of 28 terms
- Tool-calling loop⦿ liveThe cycle at the core of every AI agent: the model emits a structured tool call, the runtime executes it, the result re-enters context.AI systems
- Agent orchestration⦿ liveThe control layer above tool-calling loops: routing work across steps or agents, managing state, retries, and stopping rules.AI systems
- Answer engineA system that responds to queries with a synthesized, cited answer rather than a list of links — the AI search layer.AI systems
- Citation selectionHow an answer engine chooses which retrieved sources to cite — the mechanism that decides visibility in AI search.AI systems
- LLM evalsSystematic tests that measure model or agent behavior against defined expectations — the unit tests of AI systems.AI systems