# OTLP/HTTP API

Provon accepts OpenTelemetry traces, logs, and metrics over OTLP/HTTP. This page defines the
transport and delivery contract. For SDK and Collector setup, start with
[OpenTelemetry setup](./opentelemetry.md).

## Endpoints

| Signal  | Method | Path          |
| ------- | ------ | ------------- |
| Traces  | `POST` | `/v1/traces`  |
| Logs    | `POST` | `/v1/logs`    |
| Metrics | `POST` | `/v1/metrics` |

Use the Provon deployment origin as the OTLP base endpoint:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://provon.example.com"
```

Most OTel SDKs append the signal path. If an exporter requires a complete URL:

```bash
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://provon.example.com/v1/traces"
```

OTLP/gRPC is not exposed.

## Authentication

Send a project API key as a Bearer token:

```http
Authorization: Bearer <project API key>
```

The key needs the `telemetry:ingest` capability. Provider credentials, connector tokens, and
signed-in browser sessions are not OTLP credentials.

For standard exporter environment variables:

```bash
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $PROVON_API_KEY"
```

The API key selects the destination organization and project. Clients must not send or override an
`organization_id` or `project_id` in telemetry attributes to choose a tenant.

## Encodings

| Property           | Supported values                               |
| ------------------ | ---------------------------------------------- |
| Protocol           | OTLP over HTTP                                 |
| Body encoding      | OTLP protobuf or OTLP JSON                     |
| `Content-Type`     | `application/x-protobuf` or `application/json` |
| `Content-Encoding` | `identity`, `gzip`, or `deflate`               |
| Response body      | JSON                                           |

The JSON body must use the standard OTLP JSON envelope, such as `resourceSpans` for traces. It is
not Provon's normalized internal span JSON.

Protobuf decoding and individual signals must be enabled by the deployed runtime. The standard Node
and Cloudflare runtime assemblies provide the decoders and enable configured signals.

## Request Size

The default maximum decoded request body is 10 MiB:

```text
PROVON_OTLP_MAX_BYTES=10485760
```

The limit applies after `gzip` or `deflate` decompression. A small compressed request can therefore
still receive `413` if its decoded payload is too large.

Keep exporter batches well below the deployment limit. Smaller bounded batches reduce retry cost,
queue pressure, and the impact of one malformed request.

## Success Response

Provon returns `200` after both of these operations succeed:

1. the decoded request body is durably written to OTLP staging storage;
2. a small ingest job pointing to that body is accepted by the ingest queue.

The response has this shape:

```json
{
  "partialSuccess": {},
  "queued": true,
  "jobId": "ing_..."
}
```

`200` is an **asynchronous acceptance result**. It does not mean that every record has been parsed,
normalized, written to telemetry storage, or materialized into a trace summary.

The request path validates authentication, signal availability, content type, content encoding, and
decoded size before returning. The background worker performs envelope decoding and record-level
validation later. A syntactically malformed OTLP body can therefore be accepted into the queue and
fail during background ingest.

Verify delivery by finding the trace, log, or metric in the same project. Do not use the presence of
`jobId` as a query-readiness signal.

## Error Responses

Errors use:

```json
{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable message",
    "details": {}
  }
}
```

| Status         | Example code                   | Meaning                                                    | Client action                              |
| -------------- | ------------------------------ | ---------------------------------------------------------- | ------------------------------------------ |
| `401` or `403` | Authentication or capability   | Invalid key or missing `telemetry:ingest`                  | Fix credentials; do not retry unchanged    |
| `402`          | `BILLING_USAGE_LIMIT_EXCEEDED` | Organization ingestion is suspended                        | Resolve the usage limit                    |
| `413`          | `PAYLOAD_TOO_LARGE`            | Encoded declaration or decoded stream exceeds the limit    | Reduce exporter batch size                 |
| `422`          | `VALIDATION_FAILED`            | Unsupported content type, encoding, or protobuf capability | Fix exporter or runtime configuration      |
| `501`          | `OTLP_SIGNAL_NOT_ENABLED`      | This runtime does not accept the requested signal          | Enable the signal or stop sending it       |
| `503`          | `OTLP_STAGING_UNAVAILABLE`     | Durable staging is temporarily unavailable                 | Retry with exponential backoff             |
| `503`          | `INGEST_QUEUE_BACKPRESSURE`    | Queue admission is temporarily saturated                   | Honor `Retry-After` and retry with backoff |

Authentication errors can use the shared API error shape rather than the OTLP-specific error body.
Clients should use the HTTP status as the primary retry decision.

## Retry And Idempotency

Use the exporter's bounded exponential backoff:

- retry network failures and `503`;
- honor `Retry-After` when present;
- do not retry `200`;
- do not retry `401`, `402`, `403`, `413`, `422`, or `501` without changing configuration or the
  request.

Provon hashes each staged payload and derives an idempotency key from the project, signal, and
payload hash. Preserve stable trace and span IDs when an exporter retries an uncertain delivery.
This replay handling does not make OTLP delivery globally exactly-once; query and summary paths use
stable telemetry identities to tolerate append-oriented ingestion.

## Partial And Invalid Records

Background ingest can accept valid records from an envelope while rejecting invalid records. The
synchronous response cannot report those record-level failures because parsing happens after queue
acceptance.

During integration:

- send a small known-good trace first;
- verify the exact trace ID in the Workbench or read API;
- inspect background ingest logs when `200` requests never become visible;
- test invalid credentials, oversized payloads, and queue backpressure separately.

For production, monitor staging, queue, dead-letter, and summary-materialization failures. An HTTP
success metric alone is not an end-to-end ingest health metric.

## Example Collector Exporter

```yaml
exporters:
  otlphttp/provon:
    endpoint: https://provon.example.com
    headers:
      Authorization: Bearer ${env:PROVON_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp/provon]
```

Add separate logs and metrics pipelines only when the deployment has those signals enabled.

## Security

- Keep project API keys in server-side secret storage.
- Do not send an OTLP key from browser code.
- Redact credentials and private content before export.
- Use separate projects and keys where environment, access, or retention policy differs.
- Rotate project keys independently from model-provider credentials.
- Treat OTel baggage as exported telemetry and keep it bounded and non-sensitive.

## Related Docs

- [OpenTelemetry setup](./opentelemetry.md)
- [Tracing overview](./tracing.md)
- [Tracing quickstart](./tracing-quickstart.md)
- [Tracing attribute reference](./tracing-attributes.md)
- [Diagnosis-ready tracing](./tracing-best-practices.md)
- [Production tracing](./tracing-production.md)
- [Explore traces](./trace-explorer.md)
- [Trace read API](./trace-api.md)
- [Troubleshooting](./troubleshooting.md)