# Evals

Evals in the current Provon implementation are workflow evaluator runs, not the canonical
diagnostic Rule system. Diagnostic Rules publish persisted Findings through the diagnostics
pipeline; workflow evals produce evaluation logs, score artifacts, and downstream data records.

The implemented evaluator operation is:

```text
eval.evaluator.run
```

It judges an upstream trace or span subject with either an inline deterministic rule or an LLM
judge.

## Inputs

The evaluator requires a trace or span subject from upstream workflow context. If no `traceId` or
`spanId` is available, the step fails.

Common parameters:

| Parameter   | Required | Purpose                                                |
| ----------- | -------- | ------------------------------------------------------ |
| `judgeName` | Yes      | Stable identifier written as `gen_ai.evaluation.name`  |
| `mode`      | Yes      | `llm` or `rule`; defaults to `llm` in node definitions |
| `passLabel` | No       | Label for passing output; default `pass`               |
| `failLabel` | No       | Label for failing output; default `fail`               |
| `severity`  | No       | Default severity for failing output; default `medium`  |

## LLM Judge Mode

LLM mode requires `prompt`. `model` can override the Gateway model used for the judge when the
runtime does not provide a default model.

The prompt should instruct the judge to return strict JSON with score, label, and rationale.

## Rule Judge Mode

Rule mode requires `ruleField`. `ruleOperator` defaults to `contains`.

Supported operators:

```text
equals           not_equals       contains
not_contains     matches_pattern  gt
gte              lt               lte
```

`matches_pattern` uses wildcard matching: `*` matches any text and `?` matches one character. It is
not a regular expression operator.

## Outputs

Evaluator runs produce:

- workflow context including the evaluator step and score IDs;
- score artifacts;
- judgement metadata persisted as `gen_ai.evaluation.result`;
- a data record that downstream workflow nodes can route to notifications, issue systems, or data
  destinations.

## Relationship To Findings

Workflow evals can create or feed Finding-shaped records in automation flows, but diagnostic Rules
and server-side Rule Runs remain the canonical source of persisted Findings in the Findings inbox.

## Related Pages

- [Datasets](./datasets.md)
- [Findings](./findings.md)
- [Trace diagnosis with CLI](./trace-diagnosis.md)