# Trace Model And Instrumentation

Trace structure determines whether Provon can only display telemetry or can reconstruct evidence
for diagnosis. This page defines execution, conversation, and span boundaries for agent traces.

Use the [Tracing attribute reference](./tracing-attributes.md) for exact fields and
[Diagnosis-ready tracing](./tracing-best-practices.md) for the practical review workflow and common
anti-patterns.

## Boundaries

### One Trace

Use one trace for one bounded unit of work:

- one agent run against a task;
- one request from user input to terminal answer;
- one conversation turn;
- one workflow execution with a defined start and end.

Do not keep one trace open for an unbounded multi-turn conversation. Use a new trace for each turn
or run and keep `gen_ai.conversation.id` stable across them.

### One Conversation

A conversation groups related traces in time order. Provon prefers:

```text
gen_ai.conversation.id
```

The normalizer also recognizes `conversation.id`, `conversation_id`,
`provon.conversation_id`, `thread.id`, and `thread_id`.

A trace without a conversation ID remains visible. For diagnosis, Provon falls back to the trace ID
as the trajectory boundary, so cross-trace conversation evidence is unavailable.

### One Span

A span represents one operation with one owner and one outcome. Common GenAI operation names are:

| `gen_ai.operation.name`     | Provon span type    |
| --------------------------- | ------------------- |
| `invoke_agent`              | Agent execution     |
| `invoke_workflow`           | Workflow execution  |
| `create_agent`              | Agent creation      |
| `execute_tool`              | Tool execution      |
| `retrieval`                 | Retrieval           |
| `embeddings`                | Embedding operation |
| Other values such as `chat` | Inference           |

Use child spans for work that is causally inside the parent operation. Use separate traces with the
same conversation ID for independent agent runs or later conversation turns.

## Recommended Trace Tree

```text
invoke-agent
|-- classify-intent       gen_ai.operation.name=chat
|-- retrieve-context      gen_ai.operation.name=retrieval
|-- search-knowledge-base gen_ai.operation.name=execute_tool
|-- generate-answer       gen_ai.operation.name=chat
`-- verify-answer         application validation span
```

The root span should cover the complete unit of work. Model, retrieval, tool, and verification spans
should appear in execution order beneath the orchestrating span.

For multi-agent systems, set `gen_ai.agent.id` on every agent-owned span. Do not rely on the span
name to identify the participant.

Keep independently scheduled, resumed, or retried agent runs in separate traces and group them with
`gen_ai.conversation.id`. Use one distributed trace only when all participants are part of the same
bounded execution and trace context is propagated across the boundary.

See [Multi-agent and distributed tracing](./tracing-multi-agent.md) for participant, handoff, tool
ownership, and queue-boundary patterns.

## Stable Names

Treat span names as a query contract:

- use action names such as `retrieve-context` and `verify-answer`;
- keep request IDs, user IDs, model names, and retry numbers out of names;
- place dynamic values in attributes;
- keep names stable across releases so searches and dashboards continue to match.

Use attributes such as `retry.number=2`, not a span name such as `generate-answer-retry-2`.

## Identity Attributes

| Purpose           | Preferred field          | Notes                                    |
| ----------------- | ------------------------ | ---------------------------------------- |
| Workload          | Resource `service.name`  | Stable application or agent service name |
| Conversation      | `gen_ai.conversation.id` | Same value on every related trace        |
| Observed user     | `user.id`                | Use an internal opaque ID when possible  |
| Agent participant | `gen_ai.agent.id`        | Stable participant identity              |
| Agent display     | `gen_ai.agent.name`      | Human-readable name                      |
| Agent release     | `gen_ai.agent.version`   | Version of agent behavior                |
| Workflow          | `gen_ai.workflow.name`   | Stable workflow name                     |

Provon also normalizes `user.email`, `user.full_name`, `user.hash`, `user.name`, and `user.roles`.
Only send personal data that the deployment is allowed to retain.

## GenAI Evidence Attributes

Provon preserves unknown OTel attributes and projects recognized fields into GenAI read models:

| Evidence  | Attributes                                                                                    |
| --------- | --------------------------------------------------------------------------------------------- |
| Operation | `gen_ai.operation.name`, `gen_ai.provider.name`                                               |
| Model     | `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`                         |
| Messages  | `gen_ai.input.messages`, `gen_ai.output.messages`, `gen_ai.system_instructions`               |
| Request   | stream, choice count, seed, token limit, temperature, top-k, top-p, penalties, stop sequences |
| Usage     | input, output, reasoning, cache creation, and cache read token fields                         |
| Tool      | `gen_ai.tool.name`, call ID, type, description, arguments, and result                         |
| Retrieval | data source ID, query text, and retrieved documents                                           |
| Agent     | ID, name, version, and description                                                            |
| Result    | finish reasons, output type, span status, and `error.type`                                    |

Follow the current OpenTelemetry GenAI semantic conventions where possible. If a language SDK
cannot set structured attribute values directly, JSON-encode them. Provon parses valid JSON-looking
strings for messages, tool data, definitions, and retrieved documents.

See the [Tracing attribute reference](./tracing-attributes.md) for exact field names, accepted
conversation aliases, operation mapping, structured-value handling, and cost compatibility fields.

```ts
span.setAttribute(
  'gen_ai.input.messages',
  JSON.stringify([{ role: 'user', content: 'Find the failed deployment.' }]),
);
span.setAttribute(
  'gen_ai.output.messages',
  JSON.stringify([{ role: 'assistant', content: 'The canary health check failed.' }]),
);
```

## Tool Evidence

Use one `execute_tool` span per attempt. Record:

- a stable `gen_ai.tool.call.id`;
- `gen_ai.tool.name`;
- the arguments actually used;
- the returned result or structured error;
- span status `ERROR` and `error.type` for failures.

Keep retries as separate spans. Do not overwrite a failed result with the later successful result;
diagnostics need both attempts and their order.

## Root Input And Output

The evidence should let a reviewer answer two questions without reconstructing raw provider
payloads:

1. What goal or constraint did the user provide?
2. What terminal answer or action did the agent produce?

Put user and assistant content in `gen_ai.input.messages` and `gen_ai.output.messages`. Keep raw
transport envelopes in separate attributes only when they are needed and allowed by the retention
policy.

## Errors And Completion

For a failed operation:

- set the OTel span status to `ERROR`;
- set `error.type` to a stable category;
- record the exception event when the SDK supports it;
- keep the failed tool or provider result;
- represent recovery as later spans rather than changing the original fact.

For an incomplete run, end the root span with an error or explicit non-completion result. A final
assistant message that claims success after a failed operation is valuable diagnostic evidence and
must not erase the underlying error.

## Diagnosis Readiness

| Level       | Minimum evidence                                                              | Provon result                                |
| ----------- | ----------------------------------------------------------------------------- | -------------------------------------------- |
| Visible     | Trace ID, span ID, timestamps, names                                          | Trace tree and timing                        |
| Searchable  | Service, operation, conversation, user, model, status                         | Lists, filters, conversations, and summaries |
| Diagnosable | Goal, ordered messages, tool inputs/results, errors, retries, terminal answer | Conversation trajectory and diagnostic Rules |

Start by making the trace diagnosable. Sampling, redaction, and projected-only retention can
intentionally reduce this level; document that trade-off before production rollout.

## Related Docs

- [Tracing overview](./tracing.md)
- [Tracing quickstart](./tracing-quickstart.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Multi-agent and distributed tracing](./tracing-multi-agent.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [OTLP/HTTP API](./otlp-api.md)
- [Production tracing](./tracing-production.md)
- [Findings](./findings.md)