# Self-hosted Model Runtime Provon can serve local and private models behind the same OpenAI-compatible Gateway used for cloud providers. The Node runtime owns model catalog, engine installation, artifact download, and service lifecycle; the actual inference runs in `services/python-inference`, a standalone Python service that Node controls remotely over HTTP. This is different from [self-hosting the Provon platform](../self-hosting/index.md). A model-runtime target is one upstream choice inside the Gateway, while self-hosting determines where the Workbench, API, evidence, and jobs run. ## When To Use It Use the self-hosted model runtime when: - prompts or models must stay inside your network; - you want to run open-weights models on your own GPU/CPU hardware; - you are fine-tuning models and want to deploy the resulting checkpoints behind the Gateway; - a cloud provider does not expose the model you need. Use a cloud Provider Key when operational simplicity, elastic scale, or managed model access matter more. ## Architecture ```mermaid flowchart LR app["Agent application"] --> gateway["AI Gateway"] gateway --> self["self/<model-id>"] self --> node["Node runtime"] node --> catalog["Model catalog"] node --> engines["Engine installs"] node --> downloads["Artifact downloads"] node --> python["services/python-inference"] python --> gpu["GPU / CPU inference"] ``` The Node runtime: - discovers models through the canonical `@provon/model-catalog` package; - installs and updates inference engines such as llama.cpp, vLLM, and SGLang; - downloads model artifacts and tracks progress; - starts, stops, and health-checks the Python inference service per profile; - exposes each logical model as a stable `self/` Gateway route. Node does **not** spawn Python child processes for inference. It calls `services/python-inference` over HTTP, which keeps engine lifecycle and Python dependencies out of the Node process. ## Supported Engines | Engine | Formats typically served | Installation mode | | --------- | ------------------------ | --------------------------- | | llama.cpp | GGUF | bundled / guided / external | | vLLM | Safetensors | container / external | | SGLang | Safetensors | container / external | Engine support depends on the host OS, GPU drivers, and whether the engine is installed. The runtime reports `supportedOnThisMachine` and `installMode` for each engine so the Workbench can show the available path. ## Import A Model Models are imported from a catalog source. The primary source is Hugging Face Hub: ```bash curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/imports/huggingface" \ -H "Authorization: Bearer $PROVON_API_KEY" \ -H "Content-Type: application/json" \ -d '{"repoId": "organization/model-name"}' ``` Importing creates a runtime record and selects downloadable profiles. The model is not downloaded automatically; start a download for the profile you want to serve. ## Start And Serve A Model After import, start the download: ```bash curl -X POST \ "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/downloads" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` Then start the inference service: ```bash curl -X PUT \ "$PROVON_API_URL/projects/$PROJECT_ID/models/$MODEL_ID/profiles/$PROFILE_ID/service" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` Model identity is split by responsibility: | ID | Purpose | | --------------------------- | ----------------------------------------------- | | `catalog.id` | Internal catalog record | | `runtimeProfile.id` | Exact install and lifecycle target | | `catalog.gatewayModelId` | Stable model ID used by applications | | `runtimeProfile.routeModel` | Exact model ID exposed by the inference service | Once a compatible profile reports `running`, route Gateway requests to the logical model ID: ```bash curl "$PROVON_API_URL/gateway/v1/chat/completions" \ -H "Authorization: Bearer $PROVON_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "self/qwen3.5-2b", "messages": [{"role": "user", "content": "Hello"}] }' ``` The Gateway resolves `catalog.gatewayModelId` to a healthy running profile that supports the requested endpoint, then rewrites the upstream request to that profile's `routeModel`. Applications therefore do not need to include engine, format, quantization, or profile details in the model ID. ## Runtime Status And Activity Get the runtime overview: ```bash curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` List active downloads and engine installs: ```bash curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` The response includes engine capabilities, installed versions, download progress, and service health. ## Catalog Queries List models that the runtime can import: ```bash curl "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog?query=llama&limit=20" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` Query parameters follow the canonical catalog format: `query`, `pipelineTag`, `format`, `engine`, `limit`, and cursor fields. Refresh the catalog from upstream when needed: ```bash curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` ## Engine Installation Install an engine through the runtime: ```bash curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/engines/llama.cpp/installations" \ -H "Authorization: Bearer $PROVON_API_KEY" ``` Installation is asynchronous for most engines. Poll `activity` and `runtime` endpoints for progress. Cancel or remove an engine with the matching `generation` path. ## Configuration Node runtime storage variables: | Variable | Default | Purpose | | ------------------- | -------------------------- | ----------------------------------------------------- | | `PROVON_BLOBS_DIR` | `.provon/blobs` | Staging data; resolves `data/` and `models/` defaults | | `PROVON_MODELS_DIR` | Sibling `models` directory | Model weights and artifacts | The model runtime state file is fixed at `/data/model-runtime.json` and is not configurable. The Python inference service is configured separately: | Variable | Default | Purpose | | ------------------- | ---------- | ----------------------------------------- | | `PROVON_HOST` | `0.0.0.0` | Bind host | | `PROVON_PORT` | `8000` | Bind port | | `PROVON_API_KEY` | none | Bearer token shared with Node | | `PROVON_MODELS_DIR` | `./models` | Root directory for cached model artifacts | Point Node at the Python inference service through the runtime configuration. The exact variable name depends on the Node runtime adapter; see [Self-hosting configuration](../self-hosting/configuration.md). ## Capability Requirements The project API key needs: - `workspace:read` to inspect runtime status, catalog, and activity; - `models:manage` to install engines, import models, start downloads, and control services; - `gateway:invoke` to send inference requests through the Gateway. ## Limitations - Self-hosted model runtime is only available in the Node runtime. Cloudflare Workers do not provide local model execution. - One Node process should own the configured data directory. Multiple active Node processes sharing the directory can conflict on downloads and service ports. - Managed Provon Cloud models and self-hosted models are separate provider paths; adding a pricing definition does not make a local model routable. ## Next Steps - [AI Models](./index.md) for the complete model lifecycle. - [Model providers](../ai-gateway/providers.md) for cloud and managed targets. - [Model fine-tuning](./fine-tuning.md) for training adapters and deploying checkpoints. - [Self-hosting Node](../self-hosting/node.md) for runtime deployment. - [Models API](../api/models.md) for runtime lifecycle endpoints. - [Gateway API](../api/gateway.md) for the inference request contract.