# Diagnostic Rules

Provon ships five built-in Diagnostic Rules. Each Rule detects a family of failure modes, applies
thresholds, and decides which signals require model adjudication.

Rules are configured per project. You can enable or disable each Rule, change its thresholds, and
override individual signal categories.

## Built-In Rules

| Rule                | Key                   | Detects                                                                                       |
| ------------------- | --------------------- | --------------------------------------------------------------------------------------------- |
| Runtime reliability | `runtime-reliability` | Runtime errors, incomplete termination, and false success after execution failures            |
| Tool correctness    | `tool-correctness`    | Failed tool loops, ignored negative results, and incorrect tool-result interpretation         |
| Task fulfillment    | `task-fulfillment`    | Missing required actions, skipped verification, constraint failures, and premature completion |
| Answer grounding    | `answer-grounding`    | Unsupported final answers and unsupported external claims                                     |
| Conversation health | `conversation-health` | User corrections, dissatisfaction, runaway context growth, and failure to converge            |

## Configuration Options

| Option                    | Default | Description                                                                                                                                                        |
| ------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `keywords`                | `[]`    | One per line. When non-empty, the Rule only evaluates conversations containing at least one keyword. Matching is case-insensitive over captured conversation text. |
| `minSignalScore`          | `0.25`  | Minimum signal score for a signal to enter the diagnosis pipeline. Signals below this are discarded before adjudication and causal grouping.                       |
| `minDiagnosticConfidence` | `0.25`  | Minimum diagnostic confidence required to publish a Finding candidate. Candidates below this are filtered out.                                                     |
| `adjudicationMode`        | `auto`  | `auto` confirms deterministic signals directly and escalates judgment signals to a model. `deterministic-only` makes no model calls.                               |
| `adjudicationModel`       | `null`  | Override the deployment default model. Only used when `adjudicationMode` is `auto`.                                                                                |
| `signalOverrides`         | `{}`    | Per-category overrides keyed by category key. Each override can set `enabled: false` or change `severity`.                                                         |

## Signal Categories By Rule

A category marked **requires adjudication** is only published in `auto` mode. In
`deterministic-only` mode it is suppressed.

### Runtime reliability

| Category                  | Family       | Default severity | Requires adjudication |
| ------------------------- | ------------ | ---------------- | --------------------- |
| `runtime_error`           | runtime      | high             | no                    |
| `explicit_non_completion` | conversation | high             | no                    |
| `no_terminal_answer`      | runtime      | medium           | no                    |
| `unsupported_final_claim` | answer       | high             | yes                   |

### Tool correctness

| Category                       | Family       | Default severity | Requires adjudication |
| ------------------------------ | ------------ | ---------------- | --------------------- |
| `repeated_failed_tool_use`     | tool_use     | high             | no                    |
| `tool_result_misread`          | tool_use     | high             | no                    |
| `negative_tool_result_ignored` | tool_use     | medium           | no                    |
| `policy_precondition_skipped`  | planning     | high             | no                    |
| `non_convergence`              | conversation | medium           | no                    |

### Task fulfillment

| Category                       | Family       | Default severity | Requires adjudication |
| ------------------------------ | ------------ | ---------------- | --------------------- |
| `required_action_missing`      | planning     | medium           | no                    |
| `goal_not_verified`            | planning     | medium           | yes                   |
| `policy_precondition_skipped`  | planning     | high             | no                    |
| `subagent_handoff_unfulfilled` | coordination | high             | no                    |
| `plan_step_not_executed`       | planning     | high             | no                    |
| `premature_finalization`       | planning     | medium           | yes                   |

### Answer grounding

| Category                     | Family | Default severity | Requires adjudication |
| ---------------------------- | ------ | ---------------- | --------------------- |
| `unsupported_final_claim`    | answer | high             | yes                   |
| `unsupported_external_claim` | answer | medium           | no                    |

### Conversation health

| Category                        | Family       | Default severity | Requires adjudication |
| ------------------------------- | ------------ | ---------------- | --------------------- |
| `user_explicit_correction`      | conversation | medium           | no                    |
| `implicit_user_dissatisfaction` | conversation | medium           | yes                   |
| `context_growth`                | conversation | low              | no                    |
| `non_convergence`               | conversation | medium           | no                    |

## How To Tune

### Reduce false positives

1. Increase `minDiagnosticConfidence` for the Rule.
2. Disable noisy categories with `signalOverrides`.
3. Add `keywords` so the Rule only runs on relevant conversations.

Example API patch:

```json
{
  "configuration": {
    "minDiagnosticConfidence": 0.7,
    "keywords": ["payment", "refund"],
    "signalOverrides": {
      "implicit_user_dissatisfaction": { "enabled": false },
      "context_growth": { "severity": "info" }
    }
  }
}
```

### Reduce cost in high-volume pipelines

Switch to `deterministic-only`. This skips all model adjudication and publishes only signals that
can be confirmed from structured evidence. Expect lower recall for categories like
`unsupported_final_claim` and `implicit_user_dissatisfaction`.

### CI screening

Use `provon diagnose` with `--deterministic-only` for local or CI checks. Server-side Rules can stay
in `auto` mode for persisted Findings.

## Severity Levels

| Severity   | When to use                                                     |
| ---------- | --------------------------------------------------------------- |
| `critical` | Agent produced a wrong or harmful outcome with high confidence. |
| `high`     | Clear failure mechanism that likely affects user value.         |
| `medium`   | Detected problem that may be recoverable or context-dependent.  |
| `low`      | Minor degradation, noise, or best-practice violation.           |
| `info`     | Observation that does not require immediate action.             |