Diagnostic Rules

Provon ships five built-in Diagnostic Rules. Each Rule detects a family of failure modes, applies thresholds, and decides which signals require model adjudication.

View as Markdown Open the plain-text version of this page.

Rules are configured per project. You can enable or disable each Rule, change its thresholds, and override individual signal categories.

Built-In Rules#

Rule	Key	Detects
Runtime reliability	`runtime-reliability`	Runtime errors, incomplete termination, and false success after execution failures
Tool correctness	`tool-correctness`	Failed tool loops, ignored negative results, and incorrect tool-result interpretation
Task fulfillment	`task-fulfillment`	Missing required actions, skipped verification, constraint failures, and premature completion
Answer grounding	`answer-grounding`	Unsupported final answers and unsupported external claims
Conversation health	`conversation-health`	User corrections, dissatisfaction, runaway context growth, and failure to converge

Configuration Options#

Option	Default	Description
`keywords`	`[]`	One per line. When non-empty, the Rule only evaluates conversations containing at least one keyword. Matching is case-insensitive over captured conversation text.
`minSignalScore`	`0.25`	Minimum signal score for a signal to enter the diagnosis pipeline. Signals below this are discarded before adjudication and causal grouping.
`minDiagnosticConfidence`	`0.25`	Minimum diagnostic confidence required to publish a Finding candidate. Candidates below this are filtered out.
`adjudicationMode`	`auto`	`auto` confirms deterministic signals directly and escalates judgment signals to a model. `deterministic-only` makes no model calls.
`adjudicationModel`	`null`	Override the deployment default model. Only used when `adjudicationMode` is `auto`.
`signalOverrides`	`{}`	Per-category overrides keyed by category key. Each override can set `enabled: false` or change `severity`.

Signal Categories By Rule#

A category marked requires adjudication is only published in auto mode. In deterministic-only mode it is suppressed.

Runtime reliability#

Category	Family	Default severity	Requires adjudication
`runtime_error`	runtime	high	no
`explicit_non_completion`	conversation	high	no
`no_terminal_answer`	runtime	medium	no
`unsupported_final_claim`	answer	high	yes

Tool correctness#

Category	Family	Default severity	Requires adjudication
`repeated_failed_tool_use`	tool_use	high	no
`tool_result_misread`	tool_use	high	no
`negative_tool_result_ignored`	tool_use	medium	no
`policy_precondition_skipped`	planning	high	no
`non_convergence`	conversation	medium	no

Task fulfillment#

Category	Family	Default severity	Requires adjudication
`required_action_missing`	planning	medium	no
`goal_not_verified`	planning	medium	yes
`policy_precondition_skipped`	planning	high	no
`subagent_handoff_unfulfilled`	coordination	high	no
`plan_step_not_executed`	planning	high	no
`premature_finalization`	planning	medium	yes

Answer grounding#

Category	Family	Default severity	Requires adjudication
`unsupported_final_claim`	answer	high	yes
`unsupported_external_claim`	answer	medium	no

Conversation health#

Category	Family	Default severity	Requires adjudication
`user_explicit_correction`	conversation	medium	no
`implicit_user_dissatisfaction`	conversation	medium	yes
`context_growth`	conversation	low	no
`non_convergence`	conversation	medium	no

How To Tune#

Reduce false positives#

Increase minDiagnosticConfidence for the Rule.
Disable noisy categories with signalOverrides.
Add keywords so the Rule only runs on relevant conversations.

Example API patch:

{
  "configuration": {
    "minDiagnosticConfidence": 0.7,
    "keywords": ["payment", "refund"],
    "signalOverrides": {
      "implicit_user_dissatisfaction": { "enabled": false },
      "context_growth": { "severity": "info" }
    }
  }
}

Reduce cost in high-volume pipelines#

Switch to deterministic-only. This skips all model adjudication and publishes only signals that can be confirmed from structured evidence. Expect lower recall for categories like unsupported_final_claim and implicit_user_dissatisfaction.

CI screening#

Use provon diagnose with --deterministic-only for local or CI checks. Server-side Rules can stay in auto mode for persisted Findings.

Severity Levels#

Severity	When to use
`critical`	Agent produced a wrong or harmful outcome with high confidence.
`high`	Clear failure mechanism that likely affects user value.
`medium`	Detected problem that may be recoverable or context-dependent.
`low`	Minor degradation, noise, or best-practice violation.
`info`	Observation that does not require immediate action.