# Diagnosis-Ready Tracing

A trace can be valid OpenTelemetry and still be too weak for diagnosis. Provon needs enough
structured evidence to reconstruct the goal, execution path, failure, recovery, and terminal
outcome without guessing from raw logs.

Use this guide after [sending your first trace](./tracing-quickstart.md). For boundaries, see
[Trace model and instrumentation](./tracing-model.md). For the field-level contract, use the
[Tracing attribute reference](./tracing-attributes.md).

## Start With One Representative Run

Instrument one real workflow before broadening coverage:

1. Choose a run with a clear user goal, at least one model call, and one tool or retrieval step.
2. Capture the complete run, including a failure or negative result when possible.
3. Open the trace in the Workbench and compare it with the application's execution.
4. Fix missing boundaries, ownership, inputs, results, and errors.
5. Confirm that the conversation view tells the same story across turns.
6. Run the relevant diagnostic Rule and inspect the evidence it can support.

Volume does not compensate for weak structure. One complete trace is a better integration test than
thousands of model-call records without tool results or a terminal answer.

## Define Stable Boundaries

Use one trace for one bounded unit of work:

- one agent run;
- one conversation turn;
- one request from user input to terminal answer;
- one workflow execution with a defined end.

Use a new trace for each later turn or independent run. Keep `gen_ai.conversation.id` stable when
those traces belong to the same user goal or interaction.

Do not keep an OTel trace open for an unbounded conversation. Long-lived traces are difficult to
flush, sample, inspect, and retry, and they obscure which turn introduced a failure.

## Make The Root Span Reviewable

The root span should answer these questions at a glance:

1. What goal or constraint entered the system?
2. What terminal answer or action left the system?
3. Did the bounded run complete successfully?

Use `gen_ai.input.messages` and `gen_ai.output.messages` for readable user and assistant content.
Use child spans for model, retrieval, tool, delegation, and verification work.

Do not use an HTTP server span as the only root-level explanation when the important unit of work is
an agent run. Infrastructure spans can remain in the trace, but the agent operation must be
identifiable.

## Treat Names As A Query Contract

Use stable, verb-first names:

```text
invoke-support-agent
classify-intent
retrieve-policy
execute-refund-check
generate-answer
verify-answer
```

Keep request IDs, users, models, retry numbers, and timestamps out of span names. Put dynamic values
in attributes instead.

| Avoid                     | Prefer                                         |
| ------------------------- | ---------------------------------------------- |
| `request-01J7...`         | `invoke-support-agent`                         |
| `gpt-5-mini-call`         | `generate-answer` plus `gen_ai.request.model`  |
| `search-retry-2`          | `search-knowledge-base` plus `retry.number=2`  |
| `tool-user-123-order-456` | `execute-order-lookup` plus bounded attributes |

Stable names keep searches, dashboards, and diagnostic targeting meaningful across releases.

## Capture Execution Facts, Not Conclusions

Record objective evidence on the span where it happened:

- the model and provider used;
- the messages or input that drove the operation;
- tool arguments and the result actually returned;
- retrieved documents or bounded evidence references;
- token, cost, and latency data when available;
- span status and `error.type`;
- retry and recovery attempts as separate spans;
- the final answer or explicit non-completion.

Do not write derived labels such as `hallucinated=true` or `root_cause=bad_prompt` into execution
spans as if they were observed facts. Provon Rules derive diagnostic conclusions from the evidence
and publish them as Findings.

## Preserve Causality

Parent-child structure should match execution ownership:

```text
invoke-agent
|-- retrieve-context
|-- generate-plan
|-- execute-tool attempt=1 status=ERROR
|-- execute-tool attempt=2 status=OK
`-- verify-answer
```

- Propagate W3C trace context across service and queue boundaries.
- Keep each failed attempt; do not replace it with the successful retry.
- Use the same `gen_ai.tool.call.id` to correlate a requested tool call with its result.
- Set the failed span status to `ERROR` and include a stable `error.type`.
- Keep verification after the action it verifies.

Timestamps help order events, but they do not replace parent span IDs, tool call IDs, and explicit
participant identity.

## Attribute Every Participant

For multi-agent systems, set `gen_ai.agent.id` on every agent-owned span. Add
`gen_ai.agent.name` and `gen_ai.agent.version` when available.

Do not flatten several agents into one anonymous orchestration span. A diagnosis must be able to
distinguish the participant that delegated work, the participant that acted, and the participant
that produced the terminal answer.

See [Multi-agent and distributed tracing](./tracing-multi-agent.md) for topology and handoff
patterns.

## Keep Useful Evidence, Remove Noise

Keep a span when it changes how a reviewer understands the run:

- agent and workflow boundaries;
- model inference;
- retrieval;
- tool execution;
- external side effects;
- validation and verification;
- retry, fallback, and recovery;
- errors and policy decisions.

Consider filtering repetitive framework, database, and HTTP internals when they do not explain
agent behavior. Preserve them when they establish latency, ownership, or a failure boundary.

Do not duplicate the same prompt or response across every child span. Put evidence on the operation
that owns it and use the root span for the bounded run's user-facing input and output.

## Bound Structured Content

Prompts, tool results, retrieved documents, and raw payloads can be both large and sensitive:

- redact credentials and personal data before export;
- truncate or summarize unbounded tool output;
- retain stable references when full documents live elsewhere;
- avoid inline binary content;
- choose payload retention before production traffic;
- test whether the retained projection still supports the enabled Rules.

Redaction after ingestion cannot prevent sensitive data from entering the telemetry pipeline.

## Quality Levels

| Level       | Evidence                                                                    | Result                                     |
| ----------- | --------------------------------------------------------------------------- | ------------------------------------------ |
| Visible     | IDs, timestamps, names, parent relationships                                | Trace tree and timing                      |
| Searchable  | Service, operation, conversation, user, agent, model, status                | Filters, summaries, and conversation views |
| Diagnosable | Goal, ordered messages, tool inputs/results, failures, retries, final state | Supported diagnostic Rules and Findings    |

Aim for diagnosable traces on workflows that feed Findings. Infrastructure-only telemetry can
remain visible or searchable without carrying private model content.

## Common Failure Patterns

| Pattern                                      | Why it fails                                                |
| -------------------------------------------- | ----------------------------------------------------------- |
| One span per model request only              | Tool, retrieval, orchestration, and verification are absent |
| One trace for an entire long-lived chat      | Boundaries, flushing, and per-turn causality become unclear |
| New conversation ID on every turn            | Cross-turn evidence cannot be reconstructed                 |
| One participant identity for all agents      | Ownership and handoff failures become ambiguous             |
| Successful retry overwrites the failed span  | Recovery evidence disappears                                |
| Error text without error span status         | Error filters and deterministic signals can miss it         |
| Request-specific span names                  | Searches and aggregates fragment                            |
| Full raw payload copied to every span        | Cost, noise, and privacy exposure increase                  |
| Gateway and application both record one call | Duplicate evidence can distort counts and diagnosis         |

## Review Checklist

- The trace covers one bounded run or turn.
- The root input states the real user goal.
- The root output records the terminal answer or explicit failure.
- Span names are stable and low-cardinality.
- Model, retrieval, tool, and verification work appear in causal order.
- Every agent-owned span has an explicit participant ID.
- Tool calls, results, failures, and retries remain distinct.
- Errors use span status `ERROR` and `error.type`.
- Related traces share one `gen_ai.conversation.id`.
- Sensitive and unbounded content is removed before export.
- The Workbench trace and conversation views match the application's execution.
- A known failure can support a Finding without relying on unstated assumptions.

## Related Docs

- [Tracing overview](./tracing.md)
- [Tracing quickstart](./tracing-quickstart.md)
- [Trace model and instrumentation](./tracing-model.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Multi-agent and distributed tracing](./tracing-multi-agent.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [Production tracing](./tracing-production.md)
- [Findings](./findings.md)