OpenTelemetry Setup

Use the Gateway instead when Provon should route the model call and enforce request-time policy. Read Tracing first when choosing between Gateway, OTLP, and agent transcript capture.

Use the Tracing quickstart to send one transparent OTLP JSON example before adding an SDK.

Before instrumenting production workloads, define the trace boundary and evidence fields in Trace model and instrumentation and the Tracing attribute reference.

Choose An Instrumentation Path#

Provon compatibility is protocol-based. Any instrumentor that emits valid OTLP/HTTP can send telemetry, but not every framework emits the messages, tool results, participant identity, and terminal outcome needed by diagnostic Rules. Verify the result against Diagnosis-ready tracing before treating an integration as complete.

Quick Start#

Create a project and project API key in the Workbench, then configure the exporter:

export PROVON_API_KEY="your_project_api_key"
export PROVON_OTEL_URL="https://otel.provon.dev/v1"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="$PROVON_OTEL_URL/traces"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $PROVON_API_KEY"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_RESOURCE_ATTRIBUTES="service.name=support-agent,deployment.environment.name=development"

The hosted OTEL URL already includes /v1. Use signal-specific exporter endpoints so an SDK does not append a second /v1. Provon accepts:

POST https://otel.provon.dev/v1/traces

The ingest contract is:

OTLP/gRPC is not exposed.

See the OTLP/HTTP API for response semantics, limits, status codes, signal availability, and retry behavior.

For local development, set PROVON_OTEL_URL to http://127.0.0.1:3000/v1.

If the SDK requires a complete trace URL:

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otel.provon.dev/v1/traces"

Run the instrumented agent, flush the exporter, then verify the new record under Traces. A successful response means the payload was staged and queued; trace summaries can appear asynchronously.

Python#

Install the OpenTelemetry SDK and OTLP/HTTP exporter:

pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http

import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider(
    resource=Resource.create({"service.name": "support-agent"}),
)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=f"{os.environ['PROVON_OTEL_URL']}/traces",
            headers={"Authorization": f"Bearer {os.environ['PROVON_API_KEY']}"},
        )
    )
)
trace.set_tracer_provider(provider)

Call provider.shutdown() before a short-lived process exits so the final batch is exported.

Node.js#

Install the Node SDK and protobuf OTLP exporter:

pnpm add @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-proto \
  @opentelemetry/resources @opentelemetry/semantic-conventions

import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { NodeSDK } from '@opentelemetry/sdk-node';
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [ATTR_SERVICE_NAME]: 'support-agent',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${process.env.PROVON_OTEL_URL}/traces`,
    headers: {
      Authorization: `Bearer ${process.env.PROVON_API_KEY}`,
    },
  }),
});

sdk.start();

Initialize the SDK before importing modules that register framework or model-client instrumentation. Otherwise early calls can run before an active tracer provider exists.

Call await sdk.shutdown() during graceful process termination.

OpenTelemetry Collector#

Route an existing Collector to Provon when you want centralized batching, retries, sampling, and redaction:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:

exporters:
  otlphttp/provon:
    traces_endpoint: https://otel.provon.dev/v1/traces
    headers:
      Authorization: Bearer ${env:PROVON_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/provon]

Keep the project API key in the Collector's secret environment. Do not place it in source control.

Framework And Agent SDK Integrations#

Provon does not require a framework-specific tracing package. An integration is compatible when it can export valid OTLP/HTTP to Provon.

Use this order:

1. Enable the framework's native OpenTelemetry instrumentation when available.
2. Point its OTLP/HTTP exporter or Collector at Provon.
3. Run one representative agent workflow.
4. Inspect the emitted operation, participant, message, tool, retrieval, usage, and error fields.
5. Add manual spans or attributes only for evidence the instrumentor does not emit.

Protocol compatibility does not imply diagnosis readiness. Some instrumentors capture only model requests; others use legacy or framework-specific attributes. Provon preserves unknown attributes, but built-in projections and diagnostic Rules depend on the normalized fields in the Tracing attribute reference.

Avoid enabling two instrumentors for the same model call. In particular, decide whether the Gateway record and an application span represent one duplicated operation or two intentional layers of the execution.

Add Diagnosis Evidence#

Valid OTel spans are enough for trace visibility. Diagnostic Rules additionally need the user goal, ordered model and tool activity, failures, recovery, and terminal answer.

At minimum:

• set a stable resource service.name;
• put one bounded agent run or conversation turn in each trace;
• use gen_ai.conversation.id across related traces;
• preserve parent span IDs and tool call IDs;
• emit GenAI message, model, usage, tool, retrieval, and error attributes.

See Trace model and instrumentation for boundaries and naming rules. Use the Tracing attribute reference for exact normalized fields and structured-value handling.

Complete Examples#

The repository includes minimal OTLP/HTTP agent services:

• TypeScript
• Python
• Go

These examples demonstrate exporter setup, W3C context extraction, a root agent span, a child tool span, and graceful shutdown. Extend their attributes using the trace model before treating them as diagnosis-ready production instrumentation.

Production#

Use Production tracing to choose:

• conversation-consistent sampling;
• SDK or Collector batching and retry;
• payload bounds and source-side redaction;
• raw payload and data retention;
• environment and API-key isolation.

Troubleshooting#

• 401 or 403: verify the project API key, Bearer header, and telemetry:ingest.
• No traces: confirm OTLP/HTTP rather than OTLP/gRPC and flush the exporter.
• Only traces: configure log and metric pipelines separately.
• No conversation grouping: add a stable recognized conversation attribute.
• No Findings: verify trajectory evidence, Rule scheduling, thresholds, and adjudication.
• Rejected request: inspect content type, content encoding, body size, and protobuf availability.
• Accepted but missing data: inspect background ingest logs for envelope or record validation errors.

See the full troubleshooting guide.

Related Docs#

• Tracing overview
• Tracing quickstart
• OTLP/HTTP API
• Trace model and instrumentation
• Tracing attribute reference
• Diagnosis-ready tracing
• Multi-agent and distributed tracing
• Production tracing