Skip to content

AI Models Quickstart

Run a local open-weight model behind the Provon Gateway in a few minutes. This guide uses a small GGUF model so you can follow it on a laptop with CPU inference.

View as Markdown Open the plain-text version of this page.

Prerequisites#

  • A running Provon Node server or the provon CLI.
  • A running services/python-inference process with the llama.cpp engine installed.
  • A project API key with models:manage and gateway:invoke capabilities.

Start The Python Inference Service#

In one terminal, start the inference service:

bash
cd services/python-inference
pip install -e ".[llama.cpp]"
PROVON_PORT=8000 PROVON_MODELS_DIR=./models python -m provon_inference

Verify it is healthy:

bash
curl http://127.0.0.1:8000/health

Configure The CLI#

Point the CLI at the Python inference service and at your Provon workspace:

bash
export PROVON_PYTHON_INFERENCE_URL=http://127.0.0.1:8000
export PROVON_API_URL=https://api.provon.dev/v1
export PROVON_API_KEY=your-project-api-key

If you are running the Node server locally, use http://127.0.0.1:3000 for PROVON_API_URL.

Import A Model#

Search for a small GGUF model:

bash
provon model search --query "qwen2.5 0.5b gguf" --format gguf --limit 5

Import the exact repository:

bash
provon model import Qwen/Qwen2.5-0.5B-Instruct-GGUF

The command prints the selected profile id. Save it for the next step.

Download And Serve#

Install (download) the model artifacts:

bash
provon model install Qwen/Qwen2.5-0.5B-Instruct-GGUF --profile <profile-id>

Then serve it in the foreground:

bash
provon model serve Qwen/Qwen2.5-0.5B-Instruct-GGUF --profile <profile-id> --foreground

The command prints the Gateway model id, typically self/qwen2.5-0.5b-instruct-gguf.

Send A Request Through The Gateway#

In another terminal, send a chat completion request:

bash
curl "$PROVON_API_URL/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $PROVON_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "self/qwen2.5-0.5b-instruct-gguf",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

If the Gateway resolves the model to your running profile, the response comes from the local Python inference service.

Verify The Trace#

Open the Provon Workbench or query the Traces API for the request you just sent. The trace shows:

  • the Gateway model id (self/...);
  • the upstream routeModel the runtime selected;
  • the response latency and token usage.

This confirms the local model is integrated into the evidence loop.

Send A Request Directly (Optional)#

For debugging, you can also call services/python-inference directly using the profile's routeModel:

bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<route-model>",
    "messages": [{"role": "user", "content": "Say hello"}]
  }'

Use provon model get <model-id> --profile <profile-id> to read the routeModel.