Self-hosted Model Runtime

This is different from self-hosting the Provon platform. A model-runtime target is one upstream choice inside the Gateway, while self-hosting determines where the Workbench, API, evidence, and jobs run.

When To Use It#

Use the self-hosted model runtime when:

• prompts or models must stay inside your network;
• you want to run open-weights models on your own GPU/CPU hardware;
• you are fine-tuning models and want to deploy the resulting checkpoints behind the Gateway;
• a cloud provider does not expose the model you need.

Use a cloud Provider Key when operational simplicity, elastic scale, or managed model access matter more.

Architecture#

flowchart LR
  app["Agent application"] --> gateway["AI Gateway"]
  gateway --> self["self/<model-id>"]
  self --> node["Node runtime"]
  node --> catalog["Model catalog"]
  node --> engines["Engine installs"]
  node --> downloads["Artifact downloads"]
  node --> python["services/python-inference"]
  python --> gpu["GPU / CPU inference"]

The Node runtime:

• discovers models through the canonical @provon/model-catalog package;
• installs and updates inference engines such as llama.cpp, vLLM, and SGLang;
• downloads model artifacts and tracks progress;
• starts, stops, and health-checks the Python inference service per profile;
• exposes each logical model as a stable self/<gateway-model-id> Gateway route.

Node does not spawn Python child processes for inference. It calls services/python-inference over HTTP, which keeps engine lifecycle and Python dependencies out of the Node process.

Supported Engines#

Engine support depends on the host OS, GPU drivers, and whether the engine is installed. The runtime reports supportedOnThisMachine and installMode for each engine so the Workbench can show the available path.

Import A Model#

Models are imported from a catalog source. The primary source is Hugging Face Hub:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/imports/huggingface" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"repoId": "organization/model-name"}'

Importing creates a runtime record and selects downloadable profiles. The model is not downloaded automatically; start a download for the profile you want to serve.

Start And Serve A Model#

After import, start the download:

curl -X POST \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/downloads" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Then start the inference service:

curl -X PUT \
  "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/service" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Model identity is split by responsibility:

Once a compatible profile reports running, route Gateway requests to the logical model ID:

curl "$PROVON_API_URL/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "self/qwen3.5-2b",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

The Gateway resolves catalog.gatewayModelId to a healthy running profile that supports the requested endpoint, then rewrites the upstream request to that profile's routeModel. Applications therefore do not need to include engine, format, quantization, or profile details in the model ID.

Runtime Status And Activity#

Get the runtime overview:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \
  -H "Authorization: Bearer $PROVON_API_KEY"

List active downloads and engine installs:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \
  -H "Authorization: Bearer $PROVON_API_KEY"

The response includes engine capabilities, installed versions, download progress, and service health.

Catalog Queries#

List models that the runtime can import:

curl "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog?query=llama&limit=20" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Query parameters follow the canonical catalog format: query, pipelineTag, format, engine, limit, and cursor fields. Refresh the catalog from upstream when needed:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Engine Installation#

Install an engine through the runtime:

curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/engines/llama.cpp/installations" \
  -H "Authorization: Bearer $PROVON_API_KEY"

Installation is asynchronous for most engines. Poll activity and runtime endpoints for progress. Cancel or remove an engine with the matching generation path.

Configuration#

Node runtime storage variables:

The model runtime state file is fixed at <data-dir>/data/model-runtime.json and is not configurable.

The Python inference service is configured separately:

Point Node at the Python inference service through the runtime configuration. The exact variable name depends on the Node runtime adapter; see Self-hosting configuration.

Capability Requirements#

The project API key needs:

• workspace:read to inspect runtime status, catalog, and activity;
• models:manage to install engines, import models, start downloads, and control services;
• gateway:invoke to send inference requests through the Gateway.

Limitations#

• Self-hosted model runtime is only available in the Node runtime. Cloudflare Workers do not provide local model execution.
• One Node process should own the configured data directory. Multiple active Node processes sharing the directory can conflict on downloads and service ports.
• Managed Provon Cloud models and self-hosted models are separate provider paths; adding a pricing definition does not make a local model routable.

Next Steps#

• AI Models for the complete model lifecycle.
• Model providers for cloud and managed targets.
• Model fine-tuning for training adapters and deploying checkpoints.
• Self-hosting Node for runtime deployment.
• Models API for runtime lifecycle endpoints.
• Gateway API for the inference request contract.