Skip to content

Findings

A Finding is Provon's core artifact: a supported, actionable problem distilled from agent conversation evidence. It is more specific than a trace error and more durable than an ale

View as Markdown Open the plain-text version of this page.
On this page Browse sections

Rule, Run, Finding#

Provon keeps diagnosis in three layers:

Layer Responsibility Persistent result
Rule Defines which signals to detect and the publication thresholds Project configuration
Run Applies one Rule to one conversation activity snapshot Queued, running, completed, skipped, or failed execution
Finding Publishes a supported problem for review and repair Evidence-backed review item

The execution path is:

text
OTel spans -> conversation facts -> diagnostic signals -> adjudication
           -> Finding candidates -> published Findings

A Rule Run can complete without publishing a Finding. That is the expected result for healthy conversations or candidates below the configured thresholds.

Built-In Rules#

Every project receives five fixed Rule definitions. Projects can enable, disable, and configure them, but cannot redefine their ownership boundaries.

Rule Detects
Runtime reliability Runtime errors, incomplete termination, and false success after execution failures
Tool correctness Failed tool loops, ignored negative results, and incorrect tool-result interpretation
Task fulfillment Missing required actions, skipped verification, constraint failures, and premature completion
Answer grounding Unsupported final answers and unsupported external claims
Conversation health Runaway context growth and trajectories that fail to converge

Each Rule exposes these controls:

  • Keywords: only evaluate conversations containing at least one configured keyword.
  • Minimum signal score: discard weak signals before adjudication and causal grouping.
  • Minimum diagnostic confidence: publish only candidates meeting the confidence threshold.
  • Adjudication mode: use deterministic signals plus model adjudication, or deterministic signals only.
  • Adjudication model: override the deployment default for signals that require judgment.
  • Signal overrides: enable or disable individual signal kinds and override their severity.

deterministic-only performs no adjudication model calls. It is useful for predictable, cost-sensitive screening, but intentionally has lower recall for signals that require semantic judgment.

Automatic And Manual Runs#

Automatic scheduling is disabled when a project is first created. Enable Auto-diagnose on the Findings page to schedule Runs for newly ingested conversations.

Automatic behavior:

  1. New trace activity is mapped to a stable conversation identifier.
  2. Enabled Rules are queued for that conversation activity snapshot.
  3. Each Run becomes available after 10 minutes of inactivity.
  4. Newer activity supersedes an older automatic snapshot.
  5. Duplicate Runs for the same Rule, conversation, and activity snapshot are not created.

For existing data, open Rules and run an enabled Rule against the last 24 hours. The API also supports explicit time-window backfills and on-demand conversation Runs.

Reading A Finding#

The inbox sorts Findings by repair handoff state and diagnostic score. Expand a row to inspect:

  • Severity: critical, high, medium, low, or info.
  • Diagnostic score: the strength and impact used to rank the diagnosed problem.
  • Diagnostic confidence: how strongly the available evidence supports the diagnosis.
  • Cause assessment: the most likely supported cause.
  • Remediation hint: a focused repair direction, not an automatic code change.
  • Evidence: trace, span, or aggregate references with locations and quotes when available.
  • Evidence window: the time and conversations used for the diagnosis.
  • Detection count: repeated observations collapsed under the same Finding instance key.

Score and confidence are separate. A high-impact problem can still have low confidence, and a well-supported issue can be low severity.

Review And Repair Lifecycle#

Review state and repair state are deliberately independent:

text
unreviewed -> confirmed
           -> dismissed

confirmed + repairHandoffRef -> handed off
handed off + resolvedAt      -> externally resolved
  • Confirm accepts the Finding as actionable.
  • Dismiss removes it from the default inbox without deleting its evidence.
  • Create repair issue opens a GitHub Issue and confirms the Finding in one action.
  • Resolved means the linked GitHub Issue was closed or an associated pull request was merged.

Provon never checks out the repository or applies a patch. A user, Claude Code, Codex, or another controlled coding-agent workflow consumes the GitHub Issue and produces the pull request.

GitHub Repair Handoff#

The current repair gateway is GitHub. Configure an enabled project connector with a default owner and repository before creating a repair issue.

Each issue contains:

text
Goal
What is wrong
Cause assessment
Evidence
Delivery requirement
Provon correlation metadata

All Provon-created repair issues receive the provon-repair label. The stable reference stored on the Finding has the form github:owner/repo#number.

See Connectors for OAuth, token, and repository configuration.

Diagnostics API#

Project-scoped diagnostics routes use:

text
GET    /v1/projects/:projectId/diagnostics/findings
GET    /v1/projects/:projectId/diagnostics/findings/:findingId
PATCH  /v1/projects/:projectId/diagnostics/findings/:findingId
GET    /v1/projects/:projectId/diagnostics/settings
PATCH  /v1/projects/:projectId/diagnostics/settings
GET    /v1/projects/:projectId/diagnostics/rules
PATCH  /v1/projects/:projectId/diagnostics/rules/:ruleId
POST   /v1/projects/:projectId/diagnostics/rule-runs
POST   /v1/projects/:projectId/diagnostics/rules/:ruleId/backfills
POST   /v1/projects/:projectId/diagnostics/findings/:findingId/repair-handoff

API-key callers need diagnostics:read or diagnostics:write as appropriate. Signed-in Workbench users are authorized through project permissions.

Data Requirements#

Diagnosis works best when telemetry preserves:

  • a stable gen_ai.conversation.id;
  • the user's goal and constraints;
  • ordered assistant, model, and tool events;
  • tool inputs, results, failures, and retries;
  • a terminal answer or explicit non-completion;
  • trace and span identifiers for evidence links.

If conversation identity or raw span evidence is missing, Provon may still show telemetry while skipping diagnosis because no usable trajectory can be reconstructed.

CLI Diagnosis#

provon diagnose <source> runs the same built-in Rule definitions over a trace source. Results are returned to the caller only: they are not persisted, do not enter the Findings inbox, and cannot create a repair issue. Persisted Findings are produced separately by server-side Rule Runs over ingested project evidence.

See Trace diagnosis with CLI for deterministic screening and hash-bound Agent adjudication.