Telemetry Architecture

This is an implementation and operations reference. For application instrumentation, start with Tracing and OpenTelemetry setup. For the public ingest contract, use the OTLP/HTTP API. For public trace queries, use the Trace read API. For normalized log and metric queries, use the Telemetry Read API.

The current production Worker path is Cloudflare-native:

• Cloudflare Workers expose API, OTLP, Gateway, jobs, and warehouse routing surfaces.
• Cloudflare Queues decouple OTLP request acceptance from decoding and warehouse writes.
• R2 stores staged OTLP payloads and trace attachments.
• R2 Data Catalog and Iceberg store telemetry tables.
• Cloudflare R2 SQL serves read queries.
• A Rust Cloudflare Container owns Iceberg table creation, append writes, summary row replacement, and retention deletes.

Node deployments still use the same application interfaces, but their runtime storage adapter is DuckDB-backed. The Worker path is the canonical production path described below.

Core Rules#

Telemetry follows a small set of rules that keep the storage model understandable and scalable:

• Telemetry is physically isolated by organization id. The organization id is used as the Iceberg namespace and must be identifier-safe; built-in records use the same opaque, immutable ID format as user-created workspaces. SQL emitters quote namespace and table identifiers before execution.
• Table names are canonical. Production should use the built-in table names and should not invent per-deployment telemetry table names.
• Project isolation is a row-level field inside the organization namespace. Every projected row carries project_id.
• Every warehouse row carries event_date as a YYYY-MM-DD UTC string. Time-bounded reads and retention deletes must push this column so R2 SQL and Iceberg can prune partitions.
• Writes are append-oriented. Raw signal tables are append-only; read code deduplicates by stable keys. Summary tables are rebuildable derived read models and are replaced by key.
• Worker reads default to the summaries read model. Production should not silently fall back to direct append-only span scans when summary tables are missing.
• The API and OTLP surfaces resolve auth, project context, and queueing. The warehouse executor owns table creation, append idempotency, Iceberg mutations, and catalog/storage execution.

Runtime Pieces#

Physical Tables#

Each organization namespace contains the same canonical tables:

The canonical fully qualified table name is:

<orgId>.<tableName>

For example:

default.spans
default.trace_summaries

Payload retention is controlled by PROVON_TELEMETRY_PAYLOAD_RETENTION:

• all writes projected fields and retained raw span payload data.
• projected-only writes queryable projected fields without retained raw span payload data.

The span projection and its optional raw JSON now share one spans row; there is no separate span payload table. Logs and metrics intentionally retain only projected hot columns. Projected fields are enough for list, search, summaries, dashboard, and usage queries. Raw span payload data is needed for complete span detail.

Write Path#

Telemetry reaches the warehouse through two main write paths: external OTLP ingest and internal Gateway trace capture.

flowchart TD
  subgraph otlpPath["External OTLP ingest"]
    otlpRequest["OTLP HTTP request"] --> otlpRuntime["OTLP runtime validates auth, size, type, and encoding"]
    otlpRuntime --> stagedPayload["R2 staged payload"]
    otlpRuntime --> ingestJob["Ingest queue job"]
    stagedPayload --> ingestWorker["Jobs runtime / IngestWorker"]
    ingestJob --> ingestWorker
    ingestWorker --> decodedSignals["Decoded telemetry records"]
  end

  subgraph gatewayPath["Gateway trace capture"]
    gatewayRequest["Gateway request"] --> capture["Capture request, response, attempts, usage, cost, and guardrails"]
    capture --> gatewayDelivery["RPC or Gateway telemetry queue"]
    gatewayDelivery --> gatewayProjection["Normalize to trace spans"]
  end

  decodedSignals --> signalStores["Trace, log, and metric stores"]
  gatewayProjection --> signalStores
  signalStores --> dirtyRefs["Dirty trace refs"]
  dirtyRefs --> summaryQueue["Telemetry summary queue"]

OTLP Ingest#

OTLP routes are mounted only when both a blob store and an ingest queue are wired.

Accepted endpoints:

• POST /v1/traces
• POST /v1/logs
• POST /v1/metrics

The request must resolve to a project context and carry telemetry:ingest. Public API-key requests are checked for ingestion suspension before the payload is accepted.

The request path is:

1. The OTLP route validates the signal, auth context, body size, content type, and content encoding.
2. JSON and protobuf OTLP payloads are accepted. gzip, deflate, and identity transport encodings are supported.
3. The decoded payload is streamed into the blob store under an otlp-staging key.
4. The route computes a payload hash and enqueues an IngestJob with project context, signal, format, blob key, hash, idempotency key, byte size, and creation time.
5. The producer receives 200 only after the blob write and queue send have both succeeded.
6. If producer-side queue admission is saturated, the route returns 503 with INGEST_QUEUE_BACKPRESSURE and an optional Retry-After.

The jobs runtime consumes ingest messages through the Cloudflare Queue handler:

1. The queue handler partitions mixed queue batches and passes ingest messages to IngestWorker.
2. IngestWorker claims a bounded batch, drops oversized jobs as invalid, and defers jobs outside the per-tick byte budget.
3. It loads staged blobs, decodes OTLP JSON or protobuf, validates records, and groups accepted records by (orgId, projectId, signal).
4. Trace records are normalized, trace attachments are externalized into the blob store, and dirty trace refs are collected.
5. Groups are flushed to the trace, log, or metric store with source provenance that includes job id, idempotency key, and record range.
6. Successful jobs are acked and their staged OTLP blobs are deleted.
7. Failed jobs are nacked with retry backoff. Exhausted jobs are dead-lettered and their staged blobs are deleted.
8. After successful trace writes, dirty traces are enqueued onto the summary maintenance queue.

Gateway Trace Capture#

Gateway calls can write telemetry without going through external OTLP staging.

The Gateway path:

1. Gateway evaluates trace-capture settings and captures request, response, attempt, usage, cost, error, guardrail, and attachment data.
2. In the Cloudflare split runtime, captured Gateway telemetry is sent to the OTel runtime through GATEWAY_TELEMETRY_QUEUE. Node and custom compositions can instead write through their configured Gateway telemetry sink or trace store.
3. Gateway telemetry is converted to normalized trace spans.
4. Request and response body attachments are externalized into blob storage when present.
5. The spans are appended to the same trace store as OTLP spans.
6. The project is marked activated and the affected trace is marked dirty for summary maintenance.

Gateway telemetry and external OTLP telemetry therefore converge in the same organization namespace and project-scoped tables.

Warehouse Append#

Worker stores do not write Iceberg tables directly. They send row batches to the internal warehouse executor through the PROVON_TELEMETRY_WAREHOUSE_EXECUTOR Service Binding.

The internal append call is:

POST /append

The request carries the organization id. The warehouse Worker hashes the organization namespace onto one of 64 stable warehouse-v1-* Container shards. Append and retention mutations share this routing function, so one organization cannot have concurrent writers in different shards. The Container then:

flowchart LR
  stores["Worker stores"] --> binding["Warehouse executor Service Binding"]
  binding --> warehouse["Warehouse Worker"]
  warehouse --> container["Bounded Container shard pool"]
  container --> namespace["Organization Iceberg namespace"]
  namespace --> tables["Canonical telemetry tables"]
  namespace --> markers["Source-range markers"]

1. Normalizes the organization namespace.
2. Splits the request into table-specific row groups.
3. Ensures the target namespace and table exist.
4. Coalesces table writes by window, max rows, and max bytes to reduce small commits.
5. Appends projected rows and optional raw span payload data into the canonical Iceberg tables.
6. Uses source provenance as an in-memory append ledger so replayed job segments are not appended twice while durable snapshot markers remain the source of truth across process restarts.
7. Coalesces summary rows by natural key and explicit (updated_at, last_ingested_at) version.
8. Appends the surviving summary rows and all accepted source-range markers in one snapshot.

Summary tables remain append-only because Cloudflare R2 SQL does not currently read equality-delete files safely. Query-time readers therefore select the latest version per natural key. Data Catalog compaction performs copy-on-write file replacement to control physical duplication and small-file growth without introducing equality-delete manifests.

Append coalescing defaults to a short window and bounded batch size. It can be tuned with:

• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MS
• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MAX_ROWS
• PROVON_TELEMETRY_WAREHOUSE_APPEND_COALESCE_MAX_BYTES
• PROVON_TELEMETRY_WAREHOUSE_CACHE_MAX_TABLES

Summary Maintenance#

trace_summaries is the default derived read model. It exists so common reads do not need to scan raw append-only spans.

The default Worker dependency chain is:

flowchart LR
  spans["spans"] --> traceSummaries["trace_summaries"]
  traceSummaries --> conversations["query-time conversation rollups"]
  traceSummaries --> users["query-time user rollups"]
  traceSummaries --> dashboards["query-time dashboard rollups"]

After trace writes, the ingest and Gateway paths call markTracesDirty() with trace ids and, when available, event_date. Dirty trace refs are normalized and placed on TELEMETRY_SUMMARY_QUEUE.

The jobs runtime drains summary jobs by project:

flowchart TD
  traceWrites["Trace writes"] --> dirty["markTracesDirty()"]
  dirty --> queue["TELEMETRY_SUMMARY_QUEUE"]
  queue --> jobs["Jobs runtime"]
  jobs --> project["Group by organization and project"]
  project --> r2sql["Project trace_summaries through R2 SQL"]
  r2sql --> append["Warehouse append path"]
  append --> summaries["Updated trace_summaries"]

1. It groups dirty trace jobs by (orgId, projectId).
2. It projects trace_summaries from append-only spans through R2 SQL.
3. It writes trace summary rows back through the warehouse append path.
4. The warehouse executor coalesces concurrent versions of the same natural key before appending.
5. Jobs are acked only after the warehouse append succeeds. Failures retry with backoff and are eventually acknowledged with an error log after max attempts.

The Worker default is:

PROVON_TELEMETRY_QUERY_SOURCE=summaries

When this mode is active in writer Workers, TELEMETRY_SUMMARY_QUEUE is required. If a summary table has not been materialized for an organization that has no telemetry, summary queries return an empty read model. They do not fall back to append-only span aggregation. Other R2 SQL failures remain errors.

Higher-level conversation, user, and dashboard views are aggregated from trace_summaries at query time. This keeps the Worker/Iceberg write path to one derived index instead of expanding writes into dashboard-specific gold tables.

The Node/DuckDB runtime follows the same default: dirty trace maintenance rebuilds only trace_summaries; conversation, user, and dashboard views are derived when queried.

Read Path#

The public read surface is API-key and session aware. Public /v1/* routes require API-key access; project-scoped Workbench routes resolve the requested project and still require telemetry:read where API-key capabilities apply.

Important read endpoints:

Reads use the R2 SQL executor for Worker deployments. This executor:

• Inlines bound parameters into SQL literals.
• Rejects mutating SQL such as INSERT, UPDATE, DELETE, CREATE, and DROP.
• Sends read SQL to Cloudflare R2 SQL.
• Enforces a maximum response size and asks callers to narrow projections or add limits.
• Logs query fingerprints, row counts, response sizes, durations, and failures.

Trace detail reads intentionally use summary rows to locate the trace's event_date. In append-only Worker mode, a missing summary row causes trace reads to return empty detail rather than scanning all span partitions. This preserves the rule that summary materialization must exist before production trace detail reads rely on append-only spans.

Logs and metrics require an explicit time range:

• range=<named range>
• or startMs=<epoch ms>&endMs=<epoch ms>

The API rejects log and metric list calls without a range. Trace list and search calls default to the last 24 hours when range is omitted.

Search And Listing#

Trace listing is keyset-paginated. Cursors encode the last row position instead of using offsets:

<started_at>\t<trace_id>

Supported trace search types:

• trace-id
• span-id
• user-id
• conversation-id
• span-name
• operation-name
• content
• input
• output

Trace list and search ranges are converted to both millisecond predicates and event_date predicates so the query can prune Iceberg partitions.

Trace list filters can also include:

• level filters: ok, error, unknown
• structured filter state compiled into SQL conditions
• exact trace ids for workflow/event consumers

Conversation and observed-user lists are also keyset-paginated and can use range, text search, and structured filters. They read summary tables by default.

Log and metric lists are keyset-paginated independently:

• Log cursor: <time_ms>\t<id>
• Metric cursor: <time_ms>\t<metric_name>\t<id>

Retention And Reclaim#

Retention has two different reclaim concerns:

• Queue and blob cleanup for transient ingest payloads.
• Warehouse retention for persisted telemetry rows.

Staged OTLP Payload Reclaim#

Staged OTLP blobs are temporary. They are deleted after:

• successful ingest job ack,
• invalid oversized job ack,
• or final failed/dead-letter handling.

If queue enqueue fails after a blob was staged, the route attempts to delete the staged blob before returning the failure.

Warehouse Data Retention#

The Cloudflare jobs Worker runs scheduled maintenance and calls runDataRetentionOnce().

The Worker retention path:

1. Reads organization retention targets from the meta-store.
2. Resolves retention days from billing plan and cloud config.
3. Computes a UTC cutoff date from now - retentionDays.
4. Processes organizations in bounded batches.
5. Sends table-specific data-file delete requests through the warehouse executor.
6. Retries each organization with a per-organization timeout.
7. Accumulates rows-deleted and Iceberg maintenance stats.

Cloudflare warehouse retention prunes expired data files from:

• spans
• logs
• metrics
• trace_summaries

The delete predicate is:

event_date < <cutoff-date>

The Worker production path keeps append and retention operations for an organization on the same executor shard because telemetry tables are physically isolated by organization namespace. The Rust executor serializes mutations per table, plans matching Iceberg data files with the cutoff predicate, writes a ManifestStatus::Deleted data manifest, and commits an Operation::Delete snapshot. It does not create row-level equality delete files for warehouse retention.

Retention knobs:

• PROVON_DATA_RETENTION_INTERVAL_MS
• PROVON_DATA_RETENTION_BATCH_SIZE
• PROVON_DATA_RETENTION_DELETE_TIMEOUT_MS
• PROVON_DATA_RETENTION_MAX_RETRIES
• PROVON_DATA_RETENTION_DELETE_MAX_FILES

The executor reports maintenance stats:

• Iceberg data files deleted
• Iceberg rows deleted
• Iceberg manifest files written
• Iceberg snapshots committed
• whether the data-file delete limit was reached

Cloudflare warehouse retention currently removes expired telemetry data files from the active Iceberg snapshot. OTLP staging blobs are reclaimed by the ingest worker. Trace attachment blob retention is implemented in the DuckDB project-retention path; the Cloudflare organization-retention path should be extended separately if attachment blobs need lifecycle deletion independent of warehouse retention.

Operational Configuration#

Worker telemetry depends on these binding groups:

Important ingest knobs:

• PROVON_OTLP_MAX_BYTES
• PROVON_OTLP_MAX_BATCH_BYTES
• PROVON_OTLP_MAX_JOB_BYTES
• PROVON_OTLP_WORKER_BATCH_SIZE
• PROVON_OTLP_WORKER_LOCK_MS
• PROVON_OTLP_FLUSH_MAX_ROWS
• PROVON_OTLP_FLUSH_MAX_BYTES
• PROVON_OTLP_MAX_PENDING_JOBS

Important read-model knobs:

• PROVON_TELEMETRY_QUERY_SOURCE=summaries
• PROVON_TELEMETRY_PAYLOAD_RETENTION=all

R2 Data Catalog maintenance is configured with:

pnpm configure:cf:telemetry-maintenance
pnpm configure:cf:telemetry-maintenance:apply

The production defaults reduce metadata and small-file overhead:

• compaction target: 64 MB; supported overrides are 64, 128, 256, or 512 MB
• snapshot expiration: snapshots older than 15 days
• minimum retained snapshots: 10

The 15-day floor preserves source-range idempotency markers across the Cloudflare Queue replay window. After applying maintenance, verify the catalog settings with Wrangler and compare repeated fixed trace-list queries before and after compaction. Also verify that logical summary reads still return one latest row per natural key and that replaying the same source range does not append another physical row.

R2 SQL remains an analytical read path: even a compacted, direct summary query can have multi-second tail latency. User-facing sub-500 ms reads require a rebuildable telemetry serving projection rather than additional Iceberg query-time machinery.

Production should keep the canonical table names. Environment hooks for table-name overrides exist in some runtime types for compatibility and tests, but the current architecture relies on organization namespaces plus canonical table names as the single source of truth.

Failure Semantics#

Telemetry writes are at-least-once at the queue layer and idempotent by stable row/source identity at the append layer.

Important failure behavior:

• OTLP request acceptance is durable only after both blob staging and queue enqueue succeed.
• Enqueue failure attempts to delete the staged blob.
• Queue retries use exponential backoff.
• Exhausted ingest jobs are dead-lettered.
• Ingest job failures do not block other jobs in the same batch.
• Summary compaction jobs retry independently from raw telemetry ingest.
• Summary enqueue failure is treated as part of ingest success handling so the raw job can be retried.
• R2 SQL reads are read-only and size-limited.
• Warehouse retention failures are retried per organization and then surfaced to scheduled maintenance as failures.