Tracing

agent activity -> traces -> conversations -> diagnostic Runs -> Findings

A trace records what happened. A Finding is a supported conclusion produced from one or more traces. Healthy traces and weak evidence should not produce Findings. Trace capture, investigation, diagnosis, and Finding persistence are separate product stages.

Start Here#

Choose A Capture Path#

Use one primary capture path for each model call. Sending an application span and a separate Gateway record for the same operation can produce duplicate evidence unless the application deliberately models them as different operations.

The paths can coexist in one project. For example, production agents can use OTLP while persisted coding-agent sessions arrive through transcript sync.

Provon does not require a proprietary tracing SDK. Direct instrumentation uses OpenTelemetry SDKs, framework instrumentors, or a Collector that can export OTLP/HTTP. Compatibility means the emitted telemetry follows the OTLP transport contract; diagnosis quality still depends on the trace structure and GenAI evidence.

What Each Path Can See#

Gateway capture is the shortest integration path, but it observes the model boundary rather than the complete application. It cannot infer a tool result, retrieval decision, or application-side verification step that never crosses the Gateway.

OTLP instrumentation gives the application control over the complete trace tree. Use it when diagnosis depends on:

• agent orchestration and handoffs;
• retrieval queries and returned evidence;
• tool arguments, results, failures, and retries;
• application validation or verification steps;
• work performed before or after a model request.

Transcript sync is designed for persisted coding-agent history. It is a source adapter, not a live SDK, and uploads only files whose projected content has changed.

The Evidence Model#

Provon organizes tracing data into these levels:

project
  conversation
    trace
      span
        event

• A trace is one bounded unit of work, such as one agent run or one conversation turn.
• A span is one ordered operation inside that work, such as model inference or tool execution.
• A conversation groups related traces across turns or requests.
• A user is an optional analysis dimension, not part of the parent-child trace tree.
• An event records a point-in-time fact inside a span.

Use gen_ai.conversation.id to keep all turns of one conversation together. Preserve OTel trace and parent span IDs to keep causal order inside each trace.

See Trace model and instrumentation for boundaries and a diagnosis-ready trace structure. Use the Tracing attribute reference for the exact field contract and Diagnosis-ready tracing for the practical quality checklist.

From Trace To Finding#

Trace ingestion and diagnosis are separate stages:

1. Provon accepts Gateway evidence, OTLP telemetry, or projected transcripts.
2. The trace read model makes spans searchable and groups them into conversations.
3. Enabled diagnostic Rules reconstruct an ordered conversation trajectory.
4. Deterministic signals and optional model adjudication assess supported problems.
5. Candidates above the Rule thresholds become Findings.

This separation is intentional. A successful OTLP response proves only that ingest accepted the payload. A visible trace proves that the read path is ready. A Finding additionally requires usable conversation evidence and a supported diagnosis.

Verify Your First Trace#

Open Traces, select the new trace, and verify:

• the root span represents the complete unit of work;
• span names are stable operation names rather than request-specific values;
• model and tool spans have the expected parent-child order;
• gen_ai.conversation.id is present when work spans multiple traces;
• input, output, tool result, status, token, cost, and error fields appear when emitted;
• sensitive values have been removed before export.

Then investigate the evidence in the Workbench, with the CLI, or through the Trace read API.

Production Next Steps#

Before increasing traffic:

• define the trace and conversation boundary;
• set a stable service.name;
• decide which content must be retained for diagnosis;
• choose a conversation-consistent sampling strategy;
• batch exports and flush short-lived processes;
• test one known failure from ingest through Finding publication.

See Production tracing for sampling, privacy, batching, and retention.

Documentation Path#

Use the tracing documentation in this order:

1. Start: complete the Tracing quickstart.
2. Capture: choose Gateway, OTLP, or transcript sync.
3. Instrument: define boundaries with the trace model, select fields from the attribute reference, and apply the diagnosis-ready checklist.
4. Scale the topology: preserve participant ownership with multi-agent and distributed tracing.
5. Investigate: use the Workbench, CLI, or Trace read API.
6. Diagnose: understand Findings or run trace diagnosis with CLI.
7. Operate: set sampling, privacy, and retention in Production tracing, using the OTLP API as the transport reference.