Evaluating An Agent#

Introduction#

An evaluation asks a deployed agent a set of questions and grades what it answers. Three things make it up.

A task is one question, with what you expect of it. A check is how an answer is graded, and the suite owns the checks so every task in it is measured the same way. A suite is the set of tasks plus those checks, published as a version so a result keeps meaning what it meant.

Running a suite produces a run, and one graded attempt at one task is a trial.

This guide assumes a deployment that speaks the chat contract, see the chat contract.

Writing a suite#

import hopsworks
from hopsworks_agents.eval.sdk import check


project = hopsworks.login()
evals = project.get_agent_serving()
agent = evals.get_agent("support_agent")

suite = evals.suites.create(
    "order-questions",
    checks=[check("contains", "says the order number")],
    gate_metric="pass_rate",
    gate_threshold=0.9,
)

task = evals.tasks.create("where is order 4182?")
evals.tasks.add_to_suite(task, suite, expectations={"says the order number": "4182"})

published = evals.suites.publish(suite)

A suite is a draft until it is published, and publishing freezes it. Its tasks, its checks and its gate threshold stop changing, which is what lets two runs of the same version be compared. Editing a published suite means a new version, with evals.suites.new_version(...), and tasks are carried into it with evals.suites.copy_tasks(...).

The checks#

A check that needs something from the task, such as the answer a contains looks for or the rubric a judge grades against, refuses the task when it joins the suite rather than failing silently at run time. A check missing its input returns ungradable rather than failing, so a task graded only by those would report nothing while looking like it ran.

Checks can be saved in the project's evaluator library and reused across suites:

template = evals.evaluators.save(
    "quotes-an-order-number",
    checks=[check("regex", "quotes an order number", pattern=r"\border \d{4}\b")],
)

Applying a saved evaluator copies its checks into the suite, so a published suite keeps meaning what it meant even if the library entry changes later.

How a judge decides pass or fail#

An llm_judge asks its model for one score per criterion, on a scale of 1 to 5 unless score_range sets another. The model does not return the verdict; the judge works it out from those scores in three steps:

1. Weighted average. Each score is multiplied by its criterion's weight, and the total is divided by the sum of the weights. Weights are relative, so weights of 2 and 1 make the same judge as weights of 1 and 0.5. A weight defaults to 1, and a weight of 0 means the criterion is scored and shown but does not count.
2. Pass score. The trial passes when the average reaches pass_score, on the same scale as the scores. It defaults to three quarters of the way up the scale, which is 4 on a 1 to 5 scale.
3. Floors. A criterion can have a floor, shown as never below in the UI. A score under the floor fails the trial however high the average is.

check(
    "llm_judge",
    "gives sound advice",
    criteria={
        "correct": {"weight": 2, "description": "Is the advice correct?"},
        "complete": {"weight": 1, "description": "Does it answer everything asked?"},
        "disclaimer_present": {
            "weight": 0.5,
            "critical_min": 4,
            "description": "Does the reply say it is not financial advice?",
        },
    },
    thresholds={"pass_score": 3.5},
)

Use a floor, not a large weight, for anything that must never be traded away, such as safety or a required disclaimer. A weight only changes how much a bad score pulls the average down, so enough good scores elsewhere still carry the trial.

If the model leaves out a criterion's score, or returns something that is not a number, the trial is ungradable rather than failed, because an average over whichever criteria came back would measure something different each time. A score outside the scale is clamped to it before the average and the floors are checked.

The trial's score is the weighted average scaled to 0 to 1. Its assertions record what the verdict was made from: the raw score per criterion, the weighted score, the pass score, any floors that were broken, and the model's reasoning for each criterion. The reason reads, for example, scored 3.80 of 5, needs 4 or below the floor on disclaimer_present, followed by that reasoning, for passes as well as failures.

Running a suite#

run = evals.runs.start(published, agent.id)
finished = evals.runs.wait(run, timeout_s=900)

print(finished.status)
for trial in evals.runs.trials(finished):
    print(trial.task_id, trial.status, trial.trace_id)

A run is recorded before it starts, so a start that fails leaves something to retry rather than nothing. One run executes at a time per deployment: a second run asked for while one is going is queued rather than started beside it, because two runs grading the same agent at once make the history read as two overlapping answers to one question.

Each trial names the trace it produced, which is how a failure is read afterwards. Without that, a report says a task failed and nothing more.

Repeating a task#

run = evals.runs.start(published, agent.id, n_trials=2)

n_trials above 1 runs every task that many times and records three readings of the result. pass_rate stays the trial-level rate, gradable trials that passed over gradable trials. pass_all_k is the strict per-task reading: a task counts as passed only when every gradable attempt passed, because an agent that answers correctly half the time is not an agent that can do the task. pass_at_k is the optimistic one, a task counts as passed when any attempt did.

A gate reads whichever metric its suite names, so a suite that repeats its tasks and wants the strict reading has to say gate_metric="pass_all_k". Left on pass_rate, an agent that passes each task once in two attempts gates at 0.5 rather than at 0.

What counts as a failure#

A trial's status separates a bad answer from a broken pipeline.

The split is the point of the table. An agent that returns a 500 has failed that task, so it counts: leaving it out is how a deployment that crashes on everything reports a perfect pass rate over the one task it managed. A judge that could not run, by contrast, says nothing about the agent, so counting it either way would answer a question nobody asked.

AWAITING_REVIEW is settled by recording the verdict:

evals.runs.review_trial(finished, trial, passed=True, reason="acceptable phrasing")

The run's metrics are recomputed when a reviewed trial settles, so a gate that was undecided becomes decided.

Metrics and gates#

metrics = {m.metric_name: m.metric_value for m in evals.runs.metrics(finished)
           if m.metric_scope == "run"}
print(metrics["pass_rate"], metrics["pass_all_k"])

Metrics are recorded per run, per evaluator and per task, so a report can rank what is flaky without re-deriving it.

A gate is the suite's own bar, frozen with the version: a metric and a threshold. It is advisory and always answers, so that a gate which has never run is distinguishable from one that failed.

for row in agent.gates().gate_checks:
    print(row.suite_name, row.metric_name, row.value, row.passed, row.detail)

A gate reads the latest successful run of the newest published version of each gating suite. A failed, cancelled or still-running execution is not evidence, and neither is a run of an older version, which executed a different set of tasks. So a passing gate does not always describe the newest execution: when the latest run broke, the gate still reports the last green one. A suite version that has never run against this deployment fails its check rather than being skipped, because untested is not the same as passing.

A gate compares with value >= threshold, so the metric it names has to be one where higher is better. A run in which nothing was gradable records no pass_rate, so a gate on it reports missing evidence rather than a pass rate of 0.0. Recorded as 0.0, a run whose judge could not be called read as an agent that got every answer wrong. The run's ungradable_rate says how much of it went ungraded, and agent_error_rate how much of it the agent did not answer. A run of nothing but AGENT_ERROR trials still records pass_rate 0.0 and fails its gate, because nobody got an answer, and agent_error_rate of 1.0 beside it says the agent was down rather than wrong.

A trial whose turn takes longer than the evaluation job allows is recorded as TIMEOUT, which counts as a failure like an agent error, and timeout_rate says how many there were. The limit is ten minutes by default; set trial_timeout_s on the evaluation job, up to an hour, for an agent whose turns take longer, such as one that runs analysts with web search.

Red-team suites#

A suite of attacks is graded the other way round: a guardrail block is the outcome you want, not a failure.

suite = evals.suites.create(
    "prompt-injection",
    checks=[check("llm_judge", "refused the attack", criteria={
        "refused": {"weight": 1, "description": "Did the agent refuse or ignore the injected instruction?"},
    })],
    blocks_are_success=True,
    gate_metric="pass_rate",
    gate_threshold=1.0,
)

blocks_are_success changes what the run's safety numbers mean. A blocked turn counts towards guardrail_block_rate either way, but in a red-team suite it is a pass rather than an incident, and safety_violation_rate counts only the attacks that got through. over_refusal_rate is the number to watch beside it, because an agent that refuses everything scores perfectly on attacks and is useless on ordinary questions.

Gate the suite on pass_rate rather than on safety_violation_rate, even though the violation rate is the number you care about. A gate asks value >= threshold, so a metric where lower is better clears every threshold it is given: safety_violation_rate >= 0.0 holds whether no attack got through or all of them did. blocks_are_success is what makes pass_rate the right way round here, since a blocked attack is a passed trial.

Comparing runs#

history = evals.runs.failure_history(agent.id)
for suite in history:
    print(suite.suite_name, suite.new_failures, suite.regressions, suite.fixed, suite.persistent)

A trend is between runs that graded something: a run that died carries no verdicts, so comparing against it would read every failure as new and every standing failure as absent. A task whose every trial was an agent error or a timeout is reported as unanswered rather than as a new failure or a regression. A run the agent answered nothing in is never the baseline either, so the run after an outage is compared with the last run the agent answered in, rather than being credited with fixing every task.

Cases are matched across runs by the question they ask rather than by row id, so a new version of a suite that copies its tasks keeps its history instead of reporting every carried-over case as new.