# Gateway Evidence And Trace Capture

Gateway Trace Capture turns a model call into the evidence used by Provon traces, conversations,
diagnostic Rules, and Findings. It is separate from request success: inference can succeed while
trace persistence is filtered or disabled.

```text
Gateway request -> root operation span -> provider attempt spans
                                +-> usage, cost, errors, guardrail events
                                +-> conversation and agent context
```

## What A Gateway Trace Contains

The root span can include:

- Gateway route, status, duration, and time to first chunk;
- selected provider, Provider Key, credential fingerprint, and model mapping;
- requested and response models;
- input, output, cache, and reasoning tokens;
- normalized input, output, and total cost;
- OpenTelemetry GenAI request and response fields;
- conversation, user, session, agent, and workflow context;
- normalized provider error classification;
- guardrail mode, decision, and evidence;
- captured textual request and response bodies.

Each upstream attempt is a child span with:

- provider and upstream URL;
- Gateway and upstream model IDs;
- target and credential identity;
- fallback index and retry count;
- status, duration, and normalized failure classification.

Provider secrets and raw credential values are never stored as trace attributes.

## Add Diagnostic Context

Send context as Gateway request headers:

```bash
-H "x-otel-gen-ai-conversation-id: conversation-123" \
-H "x-otel-user-id: user-456" \
-H "x-otel-session-id: session-789" \
-H "x-otel-gen-ai-agent-id: support-agent" \
-H "x-otel-gen-ai-agent-name: Support Agent" \
-H "x-otel-gen-ai-agent-version: 2026-08-01"
```

The mapping is defined in [Gateway API](./gateway-api.md#request-context).

Use one stable conversation ID across all turns that belong to the same user goal. Provon uses that
identity to reconstruct ordered conversation trajectories. Without it, each Gateway request may be
diagnosed as an isolated trace.

Use pseudonymous user IDs unless the Workbench genuinely needs direct identity. Header values are
telemetry and follow the project's retention boundary.

## Capture Modes

Configure **Providers** -> **Request traces**:

| Mode         | Behavior                                                 |
| ------------ | -------------------------------------------------------- |
| **All**      | Persist every Gateway request                            |
| **Filtered** | Persist requests matching at least one configured filter |
| **Off**      | Do not persist Gateway traces                            |

Default mode is **All**.

Filtered mode supports:

| Filter                  | Match                                                        |
| ----------------------- | ------------------------------------------------------------ |
| Failed responses        | Non-successful final response                                |
| Blocked requests        | Guardrail or Gateway-policy block recorded during execution  |
| Recovered requests      | A later attempt recovered the request                        |
| Slow requests           | Duration at or above a threshold                             |
| High cost               | Calculated request cost at or above a USD threshold          |
| High tokens             | Provider-reported token count at or above a threshold        |
| Specific status codes   | Exact status codes or ranges                                 |
| Provider error category | Normalized categories such as rate limit or upstream timeout |

When Filtered mode has no explicit filters, Provon defaults to failed, blocked, and recovered
requests.

## Choose A Capture Strategy

Use **All** when:

- validating a new integration;
- reconstructing multi-turn conversations;
- running diagnostic Rules over representative production behavior;
- comparing healthy and unhealthy trajectories;
- investigating intermittent routing or model changes.

Use **Filtered** when:

- telemetry volume is more important than complete conversation reconstruction;
- only failures, recoveries, slow calls, or high-cost calls need evidence;
- the selected filters still preserve the context required by downstream Rules.

Use **Off** only when Gateway routing is needed without persisted request evidence.

Filtering individual calls can fragment a conversation. A diagnostic Rule cannot reason about turns
that were never captured. For projects that rely on conversation Findings, prefer **All** and use
project retention to control storage duration.

## Request And Response Bodies

For a trace selected by capture policy:

- textual request bodies are attached to the root span;
- textual response bodies are attached when capture completes within the runtime limit;
- oversized or truncated response bodies are not attached as full payloads;
- normalized GenAI fields may still be available independently of the attachment.

The default response capture limit is 2 MiB and can be changed by the deployment runtime.

Treat prompts, responses, tool arguments, and context headers as potentially sensitive. Use Gateway
guardrails to monitor, redact, or block supported PII and secret patterns before they become
provider traffic and trace evidence. Apply project telemetry retention to the resulting data.

## Trace Semantics

Gateway traces use OpenTelemetry-aligned GenAI attributes where a semantic convention exists:

| Evidence     | Attributes                                                       |
| ------------ | ---------------------------------------------------------------- |
| Operation    | `gen_ai.operation.name`                                          |
| Provider     | `gen_ai.provider.name`                                           |
| Models       | `gen_ai.request.model`, `gen_ai.response.model`                  |
| Usage        | `gen_ai.usage.*`                                                 |
| Cost         | `gen_ai.cost.*.usd`                                              |
| Conversation | `gen_ai.conversation.id`                                         |
| Agent        | `gen_ai.agent.id`, `gen_ai.agent.name`, `gen_ai.agent.version`   |
| HTTP         | `http.request.method`, `http.response.status_code`, `http.route` |

Gateway-specific routing, target, credential, guardrail, and error facts use `gateway.*` and
`provon.guard.*` attributes. They remain objective execution evidence; diagnostic labels are
derived later by Rules and Findings.

## Verify The Evidence Path

After a request:

1. Record the `x-provon-request-id` response header.
2. Open **Traces** and locate the root span with the same trace ID.
3. Confirm the conversation, user, and agent context.
4. Expand provider attempt spans.
5. Compare requested, Gateway, and upstream model IDs.
6. Inspect usage, cost, error, retry, fallback, and guardrail attributes.
7. Confirm the trace appears in the expected conversation.

If the request succeeded but no trace appears:

- confirm Trace Capture is not **Off**;
- in **Filtered** mode, confirm at least one filter matched;
- confirm the request used the same project API key as the Workbench project;
- confirm the deployment has a Gateway telemetry sink or trace store;
- check the jobs or telemetry runtime if capture is asynchronous.

## From Trace To Finding

Gateway capture does not label every unsuccessful request as a Finding. Provon first:

1. reconstructs the conversation trajectory;
2. extracts objective facts from model and tool spans;
3. runs deterministic diagnostic signals;
4. applies model adjudication only where configured and needed;
5. publishes a Finding only when Rule score and confidence thresholds pass.

This separation keeps Gateway execution facts distinct from diagnostic conclusions.

See [Findings](./findings.md) for the Rule, Run, Finding lifecycle.

## Related Docs

- [Tracing overview](./tracing.md)
- [Trace model and instrumentation](./tracing-model.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Multi-agent and distributed tracing](./tracing-multi-agent.md)
- [Production tracing](./tracing-production.md)
- [Gateway quickstart](./gateway-quickstart.md)
- [Migration guide](./gateway-migration.md)
- [Integrate SDKs](./gateway-integrations.md)
- [Gateway API](./gateway-api.md)
- [Gateway governance](./gateway-governance.md)
- [Production guide](./gateway-production.md)
- [Gateway troubleshooting](./gateway-troubleshooting.md)
- [OpenTelemetry setup](./opentelemetry.md)
- [Telemetry architecture](./telemetry.md)