# Multi-Agent And Distributed Tracing

Multi-agent traces must preserve two different structures:

- the **physical execution structure**: which process or agent ran which operation;
- the **logical conversation structure**: which runs contributed to the same user goal.

Provon uses OTel trace and parent span IDs for execution causality,
`gen_ai.conversation.id` for cross-trace grouping, and `gen_ai.agent.*` for participant attribution.
Do not collapse those dimensions into one identifier.

## Choose The Topology

### Independent Agent Runs

Use separate traces when agents are independently scheduled, retried, resumed, or executed at
different times:

```text
conversation: repair-42

trace A: planner run
trace B: implementation agent run
trace C: review agent run
```

Every trace uses the same `gen_ai.conversation.id=repair-42`, but keeps its own trace ID and bounded
root span. This is the default for durable workflows, queued handoffs, later turns, and coding-agent
sessions.

Physical separation prevents one participant's retries, status, and terminal answer from being
mistaken for another participant's execution.

### One Distributed Run

Use one trace only when all participating services and agents are part of the same bounded causal
execution and W3C trace context is propagated across every boundary:

```text
invoke-workflow
|-- invoke-agent planner
|   `-- generate-plan
|-- invoke-agent researcher
|   `-- execute-tool search
`-- invoke-agent writer
    `-- generate-answer
```

Each agent-owned span still needs its own `gen_ai.agent.id`. Sharing a trace ID does not provide
participant attribution.

### Decision Rule

| Boundary                                                   | Model                                   |
| ---------------------------------------------------------- | --------------------------------------- |
| Synchronous child work with propagated context             | Child span in the same trace            |
| Queued, resumed, independently retried, or later agent run | New trace with the same conversation ID |
| New user turn in the same interaction                      | New trace with the same conversation ID |
| Unrelated goal or user interaction                         | New trace and new conversation ID       |

Do not create parent-child relationships across different trace IDs. An OTel parent span ID is
meaningful only inside its trace.

## Participant Identity

Set participant attributes on every agent-owned span:

| Purpose  | Attribute              | Guidance                            |
| -------- | ---------------------- | ----------------------------------- |
| Agent    | `gen_ai.agent.id`      | Stable agent identity               |
| Display  | `gen_ai.agent.name`    | Human-readable role or product name |
| Release  | `gen_ai.agent.version` | Behavior or deployment version      |
| Workflow | `gen_ai.workflow.name` | Stable orchestration workflow name  |

Example:

```ts
span.setAttribute('gen_ai.operation.name', 'invoke_agent');
span.setAttribute('gen_ai.agent.id', 'security-reviewer');
span.setAttribute('gen_ai.agent.name', 'Security Reviewer');
span.setAttribute('gen_ai.agent.version', '2026-08-01');
span.setAttribute('gen_ai.conversation.id', 'repair-42');
```

Use stable IDs such as `planner`, `security-reviewer`, or a configured agent definition ID. Do not
use a process ID, request ID, or generated span ID as the agent ID.

If several instances run the same agent behavior, keep one agent ID and put instance identity in a
separate operational attribute.

## Model Handoffs As Evidence

Represent a handoff with an `invoke_agent` span owned by the invoked participant:

```text
invoke-workflow
|-- invoke-agent planner
|   `-- generate-plan
`-- invoke-agent implementer
    |-- execute-tool edit-file
    `-- execute-tool run-tests
```

Record the delegated goal in `gen_ai.input.messages` and the participant's returned result in
`gen_ai.output.messages`. Preserve the parent-child relationship when the handoff is synchronous.

For an asynchronous handoff:

1. end the delegating trace with the handoff request as objective evidence;
2. start a new trace when the target participant begins;
3. reuse the conversation ID;
4. put the delegated goal on the new root input;
5. retain an application correlation ID as a normal attribute when needed.

Do not encode an entire handoff protocol into span names. Names should remain stable operations such
as `delegate-research` or `invoke-reviewer`.

## Tool Ownership

Use one `execute_tool` span per tool attempt and nest it beneath the agent operation that requested
it:

```text
invoke-agent researcher
|-- generate-query
|-- execute-tool search attempt=1 status=ERROR
|-- execute-tool search attempt=2 status=OK
`-- generate-summary
```

Set:

- `gen_ai.tool.name`;
- `gen_ai.tool.call.id`;
- `gen_ai.tool.call.arguments`;
- `gen_ai.tool.call.result`;
- span status `ERROR` and `error.type` on failures.

Provon attributes the tool event to the tool and uses the parent chain to preserve which agent
invoked it. A shared tool call ID does not replace the parent-child relationship.

## Conversation Identity

Use one conversation ID for all traces needed to understand the same user goal. Good boundaries
include:

- one chatbot thread;
- one support case;
- one repair task from Finding to review;
- one multi-agent report or workflow instance.

Do not use:

- a global constant shared by all users;
- the trace ID copied into every later trace;
- the agent ID;
- an email address or other unnecessary personal identifier.

Provon prefers `gen_ai.conversation.id`. It also recognizes `conversation.id`, `conversation_id`,
`provon.conversation_id`, `thread.id`, and `thread_id`.

## Distributed Context

For synchronous calls, propagate the standard W3C `traceparent` header so downstream services join
the active trace. Propagate only bounded, reviewed values through W3C baggage.

For queues and durable workflows:

- preserve the originating trace and span IDs in job metadata for external correlation;
- start a new trace when the queued run is an independent execution;
- carry the stable conversation ID and participant ID explicitly;
- keep enqueue, dequeue, retry, and dead-letter facts visible when they explain behavior.

Current Provon span projection preserves trace IDs, parent span IDs, events, and attributes, but not
OTel span links. Do not rely on span links alone for participant attribution or diagnostic
causality.

## Failure And Recovery

Keep failures local to the participant that experienced them:

```text
trace B: implementation agent
|-- execute-tool apply-patch status=OK
|-- execute-tool test status=ERROR
`-- output: implementation incomplete

trace C: review agent
`-- output: rejected because tests failed
```

Do not mark the whole conversation successful only because a later agent produced an answer.
Preserve:

- the failed participant span;
- the error type and result;
- later recovery as a separate span or trace;
- each participant's terminal output;
- the final workflow outcome.

This lets diagnostic Rules distinguish an execution failure, an ignored negative result, a failed
handoff, and a false claim of success.

## Gateway Context

For Gateway-captured model calls, pass participant and conversation context as headers:

```bash
-H "x-otel-gen-ai-conversation-id: repair-42" \
-H "x-otel-gen-ai-agent-id: security-reviewer" \
-H "x-otel-gen-ai-agent-name: Security Reviewer" \
-H "x-otel-gen-ai-agent-version: 2026-08-01" \
-H "x-otel-gen-ai-workflow-name: repair-workflow"
```

Gateway capture observes the model boundary. It cannot create application-side tool, handoff, queue,
or verification spans that it never sees. Use direct OTLP instrumentation for the orchestration
tree when those operations matter to diagnosis.

## Review Checklist

- Every independently scheduled agent run has its own bounded trace.
- Related traces share one stable `gen_ai.conversation.id`.
- Every agent-owned span has `gen_ai.agent.id`.
- Agent IDs describe behavior, not process instances.
- Synchronous child work has valid parent-child relationships.
- No parent span ID points into another trace.
- Handoff inputs and returned results are retained.
- Tool spans are nested under the invoking participant.
- Failed and recovered attempts remain separate.
- The final answer can be attributed to one participant.
- Conversation evidence does not merge unrelated users or goals.
- Diagnosis does not depend only on OTel span links.

## Related Docs

- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Trace model and instrumentation](./tracing-model.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Gateway evidence](./gateway-observability.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [Production tracing](./tracing-production.md)