Self-hosting Operations

Define Service Objectives#

Set explicit targets for:

The targets determine alert thresholds and capacity. Do not infer them from default polling intervals.

Health Checks#

GET /healthz returns runtime status, version, and deployment target:

curl --fail --silent https://provon.example.com/healthz

Use it for process or Worker liveness. It does not prove:

• metadata queries work;
• telemetry can be written or read;
• queues are draining;
• a model provider is reachable;
• diagnostics are completing;
• a Connector can perform its external action.

Add synthetic checks with a dedicated project:

1. Query an authenticated API endpoint.
2. Send a small OTLP trace with a unique trace ID.
3. Poll until the trace is queryable within the evidence-visibility objective.
4. Send a low-cost Gateway canary and correlate x-provon-request-id to its trace.
5. Periodically run a deterministic diagnostic Rule over known evidence.

Keep synthetic data clearly labeled and outside production diagnostic inputs when possible.

Monitor The Pipeline#

Node#

Capture stdout and stderr from the Node process. Alert on repeated labels for:

• ingest-worker;
• gateway-telemetry-worker;
• telemetry-summary-materialization;
• diagnostic-rules;
• repair-status-sync;
• data-retention;
• Gateway usage or credit reconciliation.

Also monitor process restarts, filesystem free space, file-descriptor usage, memory, CPU, DuckDB latency, and MotherDuck connectivity when enabled.

Cloudflare#

Monitor each surface independently:

• API, Gateway, OTel, Jobs, and warehouse Worker errors;
• Queue backlog, oldest message age, retry exhaustion, and the ingest DLQ;
• scheduled trigger success;
• Container starts, failures, and write latency;
• R2 SQL query errors;
• Data Catalog append, compaction, snapshot-expiration, and retention failures;
• D1, R2, KV, Durable Object, Worker, Queue, and Container usage.

A healthy API Worker can coexist with a stalled Jobs Worker. Alert on the oldest unfinished work, not only on request error rates.

Back Up The Recovery Set#

Define one recovery point that includes:

1. metadata;
2. telemetry;
3. blobs and staged payloads;
4. persistent queue or runtime state;
5. AUTH_SECRET and required storage credentials;
6. deployment configuration and application version.

Store the secret recovery copy separately from encrypted metadata. Test restoration in an isolated environment on a schedule.

Node Backup#

For local SQLite and DuckDB, use a quiesced backup:

1. Stop new Gateway and OTLP traffic at the ingress.
2. Allow in-flight requests and background work to finish.
3. Send SIGTERM and wait for graceful shutdown.
4. Snapshot the complete persistent data directory.
5. Record the Provon version and configuration revision.
6. Restart and run the normal verification checks.

Include:

• the file behind PROVON_META_DB_URL;
• the DuckDB file when local;
• PROVON_LOCAL_BLOB_DIR and its parent runtime-state files;
• PROVON_SELF_HOSTED_DIR when local models must be recoverable.

Use the libSQL or MotherDuck provider's consistent export or snapshot mechanism for remote stores. Do not copy a live DuckDB file from storage with uncertain snapshot semantics.

Cloudflare Backup#

Treat each managed service according to its recovery model:

• export D1 metadata before upgrades and test D1 point-in-time recovery procedures;
• retain the Wrangler configs, Durable Object migration history, and resource IDs;
• protect R2 blobs and the Data Catalog bucket with account-level access and lifecycle policy;
• preserve Data Catalog maintenance settings;
• record Queue and DLQ state expectations in the recovery plan;
• keep deployment and secret inventory outside the Cloudflare account for account-recovery events.

Create a pre-upgrade D1 export:

pnpm exec wrangler d1 export provon-meta \
  --remote \
  --output provon-meta-pre-upgrade.sql \
  --config services/cloudflare-worker/wrangler.api.toml

R2 Data Catalog tables are transactionally managed Iceberg data. Do not manually delete or rewrite catalog objects as a backup or cleanup method; doing so can leave table metadata inconsistent.

Restore#

Restore into an isolated origin first:

1. Restore metadata with the original AUTH_SECRET.
2. Restore telemetry and blobs to the matching recovery point.
3. Restore Node runtime state or recreate Cloudflare bindings and queues.
4. Start the same application version that created the backup.
5. Verify sign-in, project membership, API keys, stored provider credentials, traces, Findings, and Connector decryption.
6. Upgrade only after the recovered version is healthy.
7. Switch traffic after the end-to-end synthetic checks pass.

If only metadata is restored, trace and attachment references may be missing. If only telemetry is restored, tenant ownership and access records may be missing. Document which degraded restore modes are acceptable.

Upgrade Node#

Use an immutable release or container tag.

1. Read release and migration notes.
2. Build the new Node image or source artifact.
3. Run the Node server test and relevant workspace checks.
4. Quiesce traffic and create a recovery set.
5. Stop the old process gracefully.
6. Start one new process against the persistent data.
7. Let startup migrations complete.
8. Verify health, sign-in, API, OTLP, Gateway, traces, diagnostics, and Connectors.
9. Retain the prior artifact until the release is accepted.

Do not run old and new Node processes concurrently against one local data directory during a rolling upgrade.

Upgrade Cloudflare#

1. Read release and migration notes.
2. Run pnpm test:cf.
3. Export D1 and record current Worker versions and config.
4. Apply any required Data Catalog maintenance change.
5. Run pnpm deploy:cf.
6. Verify each surface and the complete asynchronous evidence path.
7. Watch queue lag, D1 errors, warehouse writes, and R2 SQL reads through the rollout window.

pnpm deploy:cf applies D1 migrations before application surfaces. Database migrations are forward changes. Rolling application code back does not reverse the database schema; confirm schema compatibility before using a Worker version rollback.

Never rename, delete, or regenerate a migration that reached a persistent environment. Create a new forward migration.

Capacity And Scaling#

Node#

Scale vertically and control evidence volume:

• CPU and memory for request handling and DuckDB queries;
• disk capacity and IOPS for metadata, telemetry, blobs, and queues;
• body and batch limits for ingest;
• trace capture and payload retention;
• background loop batch sizes.

When one process no longer meets availability or throughput objectives, move to shared adapter infrastructure or the Cloudflare runtime. Starting more default Node replicas creates independent local work queues and schedulers.

Cloudflare#

Scale each constrained service:

• Gateway Worker concurrency and provider deadlines;
• OTel admission and Queue backlog;
• Jobs consumer concurrency and batch size;
• warehouse Container instances and append coalescing;
• R2 SQL read latency and summary freshness;
• D1 and Durable Object hot keys;
• Data Catalog small-file count and compaction.

Tune from platform telemetry. Raising producer admission without increasing consumer and warehouse capacity only moves the bottleneck into queue age.

Retention Operations#

Verify three different mechanisms:

Retention is not backup expiration. Keep recovery copies under a separate policy and document when they are destroyed.

For Cloudflare, run the maintenance configuration in dry-run mode after configuration changes:

pnpm configure:cf:telemetry-maintenance

The checked-in script enforces a minimum Data Catalog snapshot window needed by ingest idempotency. Do not shorten it independently.

Incident Procedures#

Gateway Incident#

1. Preserve x-provon-request-id and the failing request shape.
2. Pause affected project traffic or route the application back to its prior provider endpoint.
3. Check provider attempts, guardrail events, and usage reservations.
4. Keep captured evidence for diagnosis.

Ingest Backlog#

1. Confirm acceptance errors versus post-acceptance visibility lag.
2. Measure oldest pending work and dependency errors.
3. Stop increasing admission limits.
4. Restore the writer, telemetry store, or queue consumer.
5. Verify replay and deduplication before clearing dead letters.

Credential Exposure#

1. Revoke the exposed workload, provider, Connector, storage, or deployment credential at its authority.
2. Disable the affected integration or project path.
3. Search logs and trace evidence for accidental secret capture.
4. Replace the credential and verify the new path.
5. Treat AUTH_SECRET exposure as deployment-wide encrypted-credential exposure and plan a full credential reauthorization or re-encryption event.

Storage Recovery#

1. Stop writes.
2. Preserve current failed state for investigation.
3. Restore into an isolated environment.
4. Validate tenant ownership and cross-store references.
5. Resume traffic only after end-to-end checks pass.

Release Gate#

• Recovery set and restore procedure are current.
• Health, queue, background job, storage, and provider alerts are active.
• Synthetic API, OTLP, and Gateway checks pass.
• The new version and config revision are recorded.
• Forward migrations completed once.
• Evidence visibility and diagnostic latency remain within objectives.
• Retention and Data Catalog maintenance completed successfully.
• The previous artifact and application rollback procedure remain available.

Related Docs#

• Self-hosting security
• Self-hosting configuration
• Production tracing
• Production Gateway
• Troubleshooting