# Explore Traces

Use the Provon Workbench to move from a symptom to the first unexpected operation, then expand to
the complete conversation when the cause crosses trace boundaries.

For automation, use the [Tracing CLI](./tracing-cli.md) or [Trace read API](./trace-api.md).

## Traces

The **Traces** view is the execution index. Trace summaries can include:

- root operation, start time, and duration;
- span and error counts;
- conversation and observed-user identity;
- provider, model, agent, workflow, type, and environment dimensions;
- input, output, reasoning, and cache token totals;
- cost totals;
- input and output previews.

Open a trace to inspect its span tree. The tree should preserve the causal structure of the bounded
run:

```text
invoke-agent
|-- retrieve-context
|-- generate-plan
|-- execute-tool
`-- verify-answer
```

Select a span to review status, timing, attributes, resource attributes, events, retained content,
and attachments.

## Conversations

The **Conversations** view groups traces by `gen_ai.conversation.id` or another recognized
conversation attribute. Use it when the current run depends on:

- an earlier user constraint;
- a previous tool result;
- a separate participant's handoff;
- a later recovery or review run;
- several turns that contribute to one outcome.

One conversation can contain several independently bounded traces. If related turns appear as
separate conversations, fix their emitted identity rather than merging them in the UI.

## Users

The **Users** view aggregates activity by observed `user.id`. It is intended for users of the
instrumented application, not Provon workspace members.

Use it to compare trace, conversation, error, model, token, cost, and environment dimensions for one
observed user while respecting the deployment's privacy policy.

## Search

Trace search supports:

| Search type       | Matches                     |
| ----------------- | --------------------------- |
| `trace-id`        | Exact trace identity        |
| `span-id`         | Exact span identity         |
| `user-id`         | Observed user               |
| `conversation-id` | Conversation identity       |
| `span-name`       | Span operation name         |
| `operation-name`  | `gen_ai.operation.name`     |
| `content`         | Retained searchable content |
| `input`           | Input content               |
| `output`          | Output content              |

Combine identity, content, level, structured filters, and a bounded time range to reduce the result
set before opening a trace.

Level filters are `ok`, `error`, and `unknown`. A span with error text but no `ERROR` status may not
appear under the error level.

## Investigation Workflow

1. Start with a trace ID, conversation ID, error, latency, cost, user, operation, or content search.
2. Open the trace and identify the first unexpected span, not only the final error.
3. Inspect its parent, inputs, result, status, and any later retry or recovery.
4. Move to the conversation when the required evidence crosses trace boundaries.
5. Attribute each action and terminal answer to an explicit participant.
6. Compare the objective evidence with any published Finding.
7. Export the trace only when repository context or Agent adjudication is needed.

Do not infer causality only from timestamps. Parent span IDs, tool call IDs, participant identity,
status, and ordered conversation evidence provide the stronger chain.

## Reading A Trace For Diagnosis

Check these questions in order:

| Question                                     | Evidence                                                   |
| -------------------------------------------- | ---------------------------------------------------------- |
| What was the bounded goal?                   | Root input and conversation context                        |
| Which participant owned each action?         | `gen_ai.agent.*` and parent structure                      |
| What did the model or tool actually receive? | Input messages, tool arguments, retrieval query            |
| What did it return?                          | Output messages, tool result, retrieved documents          |
| Where did behavior first diverge?            | Earliest unexpected span or event                          |
| Was the fault corrected later?               | Retry, fallback, verification, or later participant result |
| What terminal claim was made?                | Root output and final participant output                   |

A final error is not always the cause. A failed tool can be corrected, while an earlier ignored
negative result can remain the decisive fault.

## Trace Quality Problems

| Symptom                            | Likely instrumentation problem                            |
| ---------------------------------- | --------------------------------------------------------- |
| Flat list of model calls           | Missing orchestration and parent context                  |
| Empty trace preview                | Root input or output is absent                            |
| Related turns appear separately    | Missing or unstable conversation ID                       |
| Several agents look like one actor | Missing `gen_ai.agent.id`                                 |
| Retry appears only as success      | Failed attempt was overwritten                            |
| Error search misses a failed span  | Status was not set to `ERROR`                             |
| Tool result cannot be reviewed     | Result was dropped, over-redacted, or payload-only        |
| Finding has insufficient evidence  | Goal, result, terminal answer, or conversation is missing |

Use [Diagnosis-ready tracing](./tracing-best-practices.md) to correct the source instrumentation.

## Continue With Automation

- Query and export evidence with the [Tracing CLI](./tracing-cli.md).
- Build product integrations with the [Trace read API](./trace-api.md).
- Run built-in Rules with [Trace diagnosis with CLI](./trace-diagnosis.md).
- Review exact fields in the [Tracing attribute reference](./tracing-attributes.md).

## Related Docs

- [Tracing overview](./tracing.md)
- [Trace model and instrumentation](./tracing-model.md)
- [Multi-agent and distributed tracing](./tracing-multi-agent.md)
- [Findings](./findings.md)
- [Tracing troubleshooting](./troubleshooting.md)