# Self-hosting Architecture

Provon separates product responsibilities into logical planes even when one runtime co-locates them.
This keeps storage, scaling, and security decisions explicit without requiring a distributed
deployment for every installation.

## Logical Planes

```mermaid
flowchart LR
  users["Users"] --> workbench["Workbench"]
  apps["Agent applications"] --> gateway["Gateway"]
  apps --> otlp["OTLP ingest"]

  workbench --> api["Control API"]
  gateway --> providers["Model providers"]
  gateway --> evidence["Evidence pipeline"]
  otlp --> evidence

  api --> meta["Metadata store"]
  api --> metering["Metering store"]
  evidence --> blobs["Blob / staging store"]
  evidence --> telemetry["Telemetry store"]
  evidence --> jobs["Background execution"]
  jobs --> telemetry
  jobs --> meta
  jobs --> metering
  jobs --> connectors["External connectors"]
```

| Plane                | Responsibilities                                                                      |
| -------------------- | ------------------------------------------------------------------------------------- |
| Workbench            | User-facing configuration, trace investigation, Findings, and operations              |
| Control API          | Identity, organizations, projects, API keys, provider settings, Rules, and connectors |
| Request data plane   | Gateway request policy, provider routing, guardrails, OTLP acceptance, and staging    |
| Background execution | Ingest decoding, trace summaries, diagnostics, retention, and status sync             |
| Storage              | Metadata, telemetry, blobs, queue state, and runtime state                            |

The planes are logical ownership boundaries. Node runs them in one process. Cloudflare maps them to
separate deployable surfaces and managed services.

## Node Mapping

```mermaid
flowchart TD
  proxy["Reverse proxy"] --> node["node-server"]
  node --> app["Workbench + API + Gateway + OTLP"]
  node --> loops["In-process background loops"]
  node --> meta["libSQL / SQLite metadata"]
  node --> telemetry["DuckDB or MotherDuck telemetry"]
  node --> blobs["Local blobs and persistent runtime state"]
```

The Node entrypoint is `services/node-server`. It composes the shared API and Node runtime, applies
metadata migrations at startup, serves the built Workbench, and starts local background loops.

Node is intentionally a single-instance architecture with the default adapters:

- metadata can be local SQLite or remote libSQL;
- telemetry can be local DuckDB or MotherDuck;
- blobs, ingest queues, Gateway telemetry queues, summary queues, and other runtime state remain in
  the configured local data directory;
- model process lifecycle is owned by the Node runtime, which controls optional standalone Python
  services for inference (`services/python-inference`) and fine-tuning
  (`services/python-fine-tuning`) over HTTP.

Remote metadata or telemetry alone does not make the runtime horizontally scalable. Multiple active
processes would have independent local queues and duplicate schedulers.

## Cloudflare Mapping

```mermaid
flowchart TD
  pages["Cloudflare Pages<br/>Workbench"] --> api["API Worker<br/>identity, control API, reads"]
  gateway["Gateway Worker<br/>model traffic and policy"] --> kv["KV<br/>Gateway runtime cache"]
  gateway --> queues["Queues<br/>ingest, summaries"]
  otlp["OTLP Worker<br/>acceptance and staging"] --> r2["R2<br/>blobs and staged payloads"]
  otlp --> queues
  queues --> jobs["Jobs Worker<br/>queue consumers, diagnostics, summaries, cron"]
  jobs --> d1["D1<br/>metadata"]
  api --> d1
  jobs --> durable["Durable Objects<br/>Gateway usage and target state"]
  jobs --> warehouse["Warehouse Worker<br/>deterministic telemetry shard routing"]
  warehouse --> container["Bounded Container pool<br/>Iceberg writes and retention deletes"]
  container --> catalog["R2 Data Catalog / Iceberg"]
  container --> r2
  api --> r2sql["R2 SQL<br/>telemetry reads"]
```

The split runtime keeps hot request paths independent:

- Workbench availability does not sit on the Gateway request path.
- OTLP acceptance can queue work before telemetry projection completes.
- the Jobs Worker owns queue consumption, diagnostics, summaries, retention, and scheduled
  maintenance;
- telemetry mutation runs behind an internal Service Binding and deterministic warehouse shards
  while preserving organization-scoped data isolation.

All resources live in the operator's Cloudflare account. There is no Provon-hosted administration
plane that must be reachable for the deployment to function.

## Request And Data Flow

### Gateway Request

```mermaid
flowchart LR
  app["Agent application"] --> auth["Authenticate project API key"]
  auth --> policy["Load routing, provider, usage, and guardrail policy"]
  policy --> credential["Decrypt selected provider credential"]
  credential --> provider["Model provider"]
  provider --> response["Provider-compatible response"]
  response --> evidence["Queue trace evidence"]
  evidence --> telemetry["Project telemetry and summaries"]
```

Prompts and responses leave the deployment when the selected provider is external. A private Provon
deployment does not make an external model provider private.

### OTLP Ingest

```mermaid
flowchart LR
  exporter["OTel exporter"] --> validate["Validate auth, body size, and signal support"]
  validate --> stage["Stage accepted payload"]
  stage --> queue["Queue ingest job"]
  queue --> worker["Decode and normalize"]
  worker --> store["Write spans, logs, or metrics"]
  store --> summaries["Trace summaries and diagnostic Rules"]
```

Acceptance and visibility are asynchronous. A successful ingest response proves durable acceptance
for that runtime; trace visibility is a separate verification gate.

### Control And Connector Flow

Workbench sessions call the Control API. Provider keys, connector credentials, and OAuth tokens are
encrypted before they are stored in metadata. Connector actions and OAuth handshakes create outbound
requests to the configured external systems.

## Storage Boundaries

| Data class      | Contents                                                            | Node                     | Cloudflare                        |
| --------------- | ------------------------------------------------------------------- | ------------------------ | --------------------------------- |
| Metadata        | Users, projects, keys, Rules, Findings, settings, encrypted secrets | SQLite/libSQL            | D1                                |
| Metering        | Idempotent billable usage events                                    | Separate SQLite/libSQL   | Separate D1                       |
| Telemetry       | Spans, logs, metrics, summaries, and diagnostic evidence            | DuckDB or MotherDuck     | R2 Data Catalog / Iceberg, R2 SQL |
| Blobs           | OTLP staging, attachments, exports, and large payloads              | Local filesystem         | R2                                |
| Runtime state   | Ingest jobs, summary work, Gateway telemetry, and local leases      | Persistent files         | Queues, D1, KV, Durable Objects   |
| Model artifacts | Models, engines, process state, and health                          | Filesystem and processes | Not provided                      |

Back up metadata, metering, and telemetry independently at a coordinated recovery point. Metadata
contains the tenant model and encrypted credentials; metering contains billing facts; telemetry
contains the evidence those records reference.

## Network Boundary

Required inbound paths depend on enabled features:

| Caller            | Surface                                       |
| ----------------- | --------------------------------------------- |
| Browser           | Workbench and authentication callbacks        |
| Agent application | `/gateway/v1/*`                               |
| OTel exporter     | `/v1/traces`, `/v1/logs`, `/v1/metrics`       |
| Service client    | `/v1/*` API routes allowed by its project key |

Potential outbound paths include:

- configured model providers;
- sign-in identity providers;
- Connector OAuth and API endpoints;
- MotherDuck when selected by the Node runtime;
- package registries and Cloudflare APIs during build or deployment.

The core runtime can remain on a private network when all callers are private, model execution is
local, and external identity and connectors are disabled. Document each enabled egress dependency
instead of assuming an air-gapped deployment.

## Failure Boundaries

| Failure                      | Expected effect                                                           |
| ---------------------------- | ------------------------------------------------------------------------- |
| Workbench unavailable        | API, Gateway, and OTLP may continue if their runtime surfaces are healthy |
| Model provider unavailable   | Gateway request fails or follows configured retry/fallback policy         |
| Telemetry writer delayed     | Accepted evidence remains queued; trace visibility lags                   |
| Metadata unavailable         | Sign-in, project lookup, policy, and most control operations fail         |
| Background execution stopped | Ingest projection, summaries, diagnostics, retention, and status sync lag |
| Blob storage unavailable     | New staged ingest and attachment operations fail                          |

Design health checks and alerts around these boundaries. `/healthz` reports runtime liveness and
version; it is not a complete dependency or end-to-end evidence check.

## Next Steps

- [Deploy Node](./node.md)
- [Deploy on Cloudflare](./cloudflare.md)
- [Configure self-hosting](./configuration.md)
- [Secure the deployment](./security.md)
- [Operate the deployment](./operations.md)