# AI Models Troubleshooting

This guide covers common failures when importing, serving, fine-tuning, or exporting local models.

## Gateway returns 503 for a `self/<model-id>` request

**What it looks like:** The Gateway responds with a model-unavailable error even though the model
shows as installed.

**Common causes:**

- `services/python-inference` is not running or `PROVON_PYTHON_INFERENCE_URL` points to the wrong host/port.
- The model service was started but has not reached the `running` state yet.
- The requested endpoint (for example, `chat/completions`) is not listed in the profile's `endpoints`.

**What to do:**

1. Check the Python inference service health directly:
   ```bash
   curl "$PROVON_PYTHON_INFERENCE_URL/health"
   ```
2. Check the runtime status to see the service state and the selected profile:
   ```bash
   curl "$PROVON_API_URL/projects/$PROJECT_ID/models/runtime" \
     -H "Authorization: Bearer $PROVON_API_KEY"
   ```
3. Call the Python service directly with the profile's `routeModel` to rule out Gateway routing:
   ```bash
   curl "$PROVON_PYTHON_INFERENCE_URL/v1/chat/completions" \
     -H "Authorization: Bearer $PROVON_PYTHON_INFERENCE_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model": "<route-model>", "messages": [{"role":"user","content":"hi"}]}'
   ```

## Runtime reports an engine as unsupported

**What it looks like:** `provon model install` fails with "Engine X is not supported on this machine",
or the API status shows `supportedOnThisMachine: false`.

**Common causes:**

- `PROVON_PYTHON_INFERENCE_URL` is not configured.
- The Python inference service is reachable but the engine extra is not installed (for example, you are
  trying to serve a Safetensors model but only installed the `llama.cpp` extra).

**What to do:**

1. Confirm the URL:
   ```bash
   echo "$PROVON_PYTHON_INFERENCE_URL"
   curl "$PROVON_PYTHON_INFERENCE_URL/health"
   ```
2. Reinstall the Python inference service with the required engine extra:
   ```bash
   pip install -e ".[llama.cpp]"      # for GGUF
   pip install -e ".[vllm]"           # for Safetensors via vLLM
   pip install -e ".[sglang]"         # for Safetensors via SGLang
   pip install -e ".[all]"            # install every engine
   ```

Remember that Node and the CLI never install engine binaries themselves; they only report whether
the remote Python inference service can serve a given engine family.

## Model download is interrupted or stuck

**What it looks like:** `provon model install` or the download API returns a job that stays in
`running` forever, or the CLI exits before completion.

**What to do:**

1. Check `activity` to see the download job state:
   ```bash
   curl "$PROVON_API_URL/projects/$PROJECT_ID/models/activity" \
     -H "Authorization: Bearer $PROVON_API_KEY"
   ```
2. Partial downloads are stored as `.part` files in `PROVON_MODELS_DIR`. If a job is stuck, cancel it
   and start a new download; the runtime resumes from the existing `.part` file when possible.
3. If you are using the CLI, the process must stay alive while the download runs. For long downloads,
   prefer using the Node server API instead, where downloads continue as long as the server process runs.

## Catalog is empty after restarting the Node server or CLI

**What it looks like:** `provon model list --all` returns no models, or the catalog query returns an
empty page after a restart.

**Why:** The catalog cache and runtime state are kept in memory only. When the process restarts, the
cache is gone. Imported models that have been downloaded are rediscovered from local `manifest.json`
files, but the upstream catalog browseable list must be refreshed.

**What to do:**

```bash
curl -X POST "$PROVON_API_URL/projects/$PROJECT_ID/models/catalog-refreshes" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

Or with the CLI:

```bash
provon model search --query llama --limit 20
```

## A checkpoint cannot be deployed

**What it looks like:** `POST /fine-tuning/checkpoints/:checkpointId/deploy` returns
`checkpoint "..." is not registered for inference` or a deployment timeout.

**Common causes:**

- The checkpoint does not have a `profileId`, which means the training job did not produce a runtime
  profile. Check the job output format and whether the Python fine-tuning service reported success.
- `services/python-inference` is not reachable.
- The checkpoint path is not readable from the Python inference service host.

**What to do:**

1. Read the checkpoint to confirm it has a `profileId`:
   ```bash
   curl "$PROVON_API_URL/projects/$PROJECT_ID/fine-tuning/checkpoints/$CHECKPOINT_ID" \
     -H "Authorization: Bearer $PROVON_API_KEY"
   ```
2. Confirm the Python inference service is healthy and can access `PROVON_MODELS_DIR`.
3. Try starting the model manually with the profile id to get a clearer error:
   ```bash
   curl -X PUT "$PROVON_API_URL/projects/$PROJECT_ID/models/$PROFILE_ID/profiles/$PROFILE_ID/service" \
     -H "Authorization: Bearer $PROVON_API_KEY"
   ```

## Fine-tuning job fails immediately

**What it looks like:** The job status becomes `failed` seconds after creation.

**Common causes:**

- The Dataset does not use the `chat/v1` schema or has zero examples.
- The base model is not known to the runtime.
- Node cannot reach `services/python-fine-tuning`.
- `PROVON_MODEL_FINE_TUNING_WORK_DIR` is set differently for Node and the Python service, so the
  manifest path is invalid from one side.

**What to do:**

1. Verify the Dataset schema and example count.
2. Confirm the base model is imported and, if needed, downloaded.
3. Check the job error message and the Python fine-tuning service logs.
4. Ensure Node and the Python service share the same absolute work directory.

## GGUF export fails

**What it looks like:** An export job with `format: "gguf"` fails during conversion or quantization.

**Common causes:**

- `PROVON_LLAMA_CPP_PATH` points to a directory that is missing `convert_hf_to_gguf.py` or
  `llama-quantize`.
- The requested `quantMethod` is not supported for the source model architecture.

**What to do:**

1. Confirm the llama.cpp checkout has the required scripts:
   ```bash
   ls "$PROVON_LLAMA_CPP_PATH"/convert_hf_to_gguf.py
   ls "$PROVON_LLAMA_CPP_PATH"/build/bin/llama-quantize
   ```
2. Use a common quantization such as `q4_k_m` for your first export.
3. Read the export job record for the exact error message.

## Multiple Node processes share the same data directory

**What it looks like:** Downloads restart unexpectedly, service ports conflict, or models flip between
running and stopped states.

**Why:** Only one Node process should own a given `PROVON_MODELS_DIR` at a time. Runtime state
(downloads, service bookkeeping, catalog cache) is in memory, so two processes cannot coordinate.

**What to do:** Run a single Node server for each `PROVON_MODELS_DIR`. If you need isolation, use
separate `PROVON_MODELS_DIR` values for different environments.

## Getting more help

If these steps do not resolve the issue:

1. Collect the runtime status, activity, and the relevant job or service records.
2. Collect logs from `services/python-inference` or `services/python-fine-tuning`.
3. Open an issue with the request id, model id, profile id, and the exact error response.

For deployment topology questions, see [Self-hosting Node](../self-hosting/node.md).