Production Tracing

Sampling Strategy#

Provon accepts the sampling decision made by the Gateway, SDK, or Collector. It does not reconstruct dropped spans.

Preserve Complete Conversations#

Provon diagnoses ordered conversation trajectories. Independently sampling each trace can retain one turn while dropping the earlier goal, failed tool result, or final answer.

For projects that use diagnostic Rules:

• prefer full capture while validating instrumentation;
• make one deterministic keep or drop decision per conversation;
• apply that decision to every trace in the conversation;
• preserve all spans in each sampled trace;
• keep known failures, recoveries, expensive runs, and slow runs.

A standard trace-ID ratio sampler is useful for traffic control, but separate turns usually have different trace IDs. Use an application-level conversation decision when conversation completeness matters.

Head Sampling#

Head sampling decides before the operation runs. It is cheap and predictable but cannot know whether the run will fail.

For distributed traces, use a parent-based sampler so child services honor the root decision:

import { ParentBasedSampler, TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-base';
import { NodeSDK } from '@opentelemetry/sdk-node';

const sdk = new NodeSDK({
  sampler: new ParentBasedSampler({
    root: new TraceIdRatioBasedSampler(0.2),
  }),
  // Configure the Provon OTLP/HTTP exporter and batch processor.
});

This retains approximately 20 percent of root traces. It does not by itself keep all traces in a multi-trace conversation.

Collector Tail Sampling#

Tail sampling can inspect status, latency, and attributes before deciding. It is useful for keeping errors and high-value traces while reducing healthy traffic.

Tail sampling still operates on individual OTel traces. If one conversation spans many traces, combine it with an application-level conversation decision or accept that conversation diagnosis will be incomplete.

Gateway Capture Filters#

Gateway trace capture supports:

• all;
• off;
• filtered by failed, blocked, or recovered requests, duration, cost, tokens, status code, or provider error category.

filtered is appropriate for request-level Gateway investigation. It is not equivalent to conversation-consistent sampling and can remove the healthy or earlier turns needed by diagnostic Rules.

Context Propagation#

Propagate W3C traceparent across synchronous HTTP and RPC boundaries so downstream work joins the active trace. Use W3C baggage only for bounded, non-sensitive correlation values.

At queue and durable workflow boundaries, decide whether the consumer is a child operation in the same bounded execution or a new independently scheduled run:

• keep the same trace only when trace context remains valid and the producer waits on that causal work;
• start a new trace for resumed, delayed, independently retried, or separately terminated work;
• preserve gen_ai.conversation.id and gen_ai.agent.id across the new run;
• do not set a parent span ID that belongs to another trace.

See Multi-agent and distributed tracing for participant and handoff topologies.

Batching And Delivery#

Use a batch span processor in long-running services. For short-lived commands, jobs, and serverless invocations, flush or shut down the provider before the process exits.

An OTLP 200 response from Provon means the decoded payload was staged and the ingest job was queued. Trace summaries become visible asynchronously.

Handle ingest responses as follows:

The default decoded OTLP request limit is 10 MiB. Runtime configuration can lower or raise it. Prefer smaller bounded batches over requests close to the limit.

An ingest 200 is asynchronous acceptance, not record-level validation or query readiness. See the OTLP/HTTP API for the exact response, background validation boundary, and retry contract.

Content Capture#

Prompts, responses, system instructions, tool arguments, tool results, retrieved documents, and attachments can contain:

• credentials and access tokens;
• personal or regulated data;
• source code and proprietary documents;
• untrusted model output;
• large binary or multimodal payloads.

Redact before export whenever possible. Collector-side processors are useful for centralized policy, but source-side redaction prevents sensitive data from entering any telemetry queue.

Do not put project API keys, provider credentials, connector tokens, or authentication headers in span attributes, events, log bodies, or OTel baggage.

Payload Retention#

Worker deployments expose:

PROVON_TELEMETRY_PAYLOAD_RETENTION=all
PROVON_TELEMETRY_PAYLOAD_RETENTION=projected-only

all retains projected query fields and raw payload evidence. projected-only retains projected fields used for lists, search, summaries, dashboards, and usage queries without full raw payload bodies.

Projected-only retention reduces exposure and storage, but it can remove the exact message, tool result, or provider body needed for span inspection and diagnosis. Test the enabled Rules against the chosen mode before production rollout.

Telemetry data retention and payload retention answer different questions:

• Data retention controls how long telemetry rows remain available.
• Payload retention controls whether complete raw evidence is stored.

Configure both deliberately. Deleting old projected rows does not replace source-side minimization, and dropping raw payloads does not remove all identity or usage fields.

Cardinality And Payload Bounds#

• Keep span names low-cardinality and stable.
• Put request-specific IDs in attributes.
• Bound tool outputs and retrieved documents before export.
• Store large supported media as attachments rather than repeated inline values.
• Avoid copying the same prompt or response onto every child span.
• Use opaque user and conversation IDs.

High-cardinality values are valid evidence attributes, but poor operation names and repeated large payloads make search, storage, and review less useful.

Environment Isolation#

Use separate Provon projects and API keys for production, staging, and development when access, retention, or diagnostics policy differs. Also emit a stable environment attribute for filtering.

Never reuse a production project API key in local examples or client-side browser code. Rotate project keys independently from model-provider credentials.

Release Checklist#

• One agent run or turn maps to one bounded trace.
• Conversation sampling keeps all required turns.
• Parent-child context survives service and queue boundaries.
• Every agent-owned span has an explicit participant ID.
• Exporters batch, retry, and flush correctly.
• Errors set span status and error.type.
• Private content is redacted before export.
• Tool and message payload sizes are bounded.
• Payload and data retention are configured.
• One known failure remains diagnosable end to end.
• One healthy run produces no false Finding.

Related Docs#

• Trace model and instrumentation
• Tracing attribute reference
• Diagnosis-ready tracing
• Multi-agent and distributed tracing
• OpenTelemetry setup
• OTLP/HTTP API
• Trace read API
• Gateway
• Telemetry architecture
• Troubleshooting