# Gateway Troubleshooting

Troubleshoot the Gateway in request order. First establish whether Provon accepted the client,
then whether it resolved a target, then whether the provider attempt succeeded, and finally whether
trace evidence was persisted.

## Fast Triage

For one failing request, record:

- HTTP status and complete response error body;
- `x-provon-request-id`, when present;
- request endpoint and model value;
- whether the failure happens before or after streaming starts;
- the project API key ID, not its secret;
- the selected trace and provider-attempt spans, when present.

Then follow this sequence:

1. **No request ID and `401` or `403`**: verify client authentication and key capability.
2. **Gateway error before an upstream attempt**: inspect model resolution, limits, guardrails, or
   project state.
3. **Provider attempt exists**: inspect its status, error category, selected Provider Key, and
   upstream model.
4. **Model response succeeds but no trace appears**: inspect Trace Capture and telemetry delivery.

Do not enable retries or fallback until the primary failure is understood.

## Common Statuses

Use `error.code` as the primary signal. The same status can represent different layers.

| Status | Typical Gateway cause                                        | First check                                       |
| ------ | ------------------------------------------------------------ | ------------------------------------------------- |
| `400`  | Invalid JSON, body shape, endpoint, feature, or model field  | Request body and endpoint contract                |
| `401`  | Missing or invalid Provon project API key                    | Bearer token                                      |
| `403`  | Project key lacks the required capability                    | `gateway:invoke` or control API capability        |
| `402`  | Insufficient Provon Cloud AI Gateway credits                 | Organization credit balance                       |
| `404`  | No eligible upstream target or unsupported legacy path       | Model ID, Provider Key, mapping, and capability   |
| `409`  | Enforcing guardrail blocked the exchange                     | Guardrail plugin, direction, category, and action |
| `413`  | Request body exceeds the runtime limit                       | Payload size and endpoint                         |
| `415`  | Unsupported media type or request encoding                   | `Content-Type` and uncompressed request body      |
| `429`  | Request, token, cost, or ingestion policy rejected admission | API-key limit and reset headers                   |
| `503`  | Gateway paused or no usable runtime path                     | Project Gateway state and deployment health       |

An upstream provider can return the same statuses. When a trace exists, the root span shows the
final Gateway result and child attempt spans show upstream responses.

## Authentication Fails With `401` Or `403`

Verify all of these:

```text
Base URL:      https://<origin>/gateway/v1
Authorization: Bearer <Provon project API key>
Capability:    gateway:invoke
```

Common mistakes:

- sending the upstream provider key to Provon;
- pointing the client at the provider's original base URL;
- using a session cookie or connector credential in an application;
- configuring a client that sends only `x-api-key`;
- appending a second `/v1`;
- using a key from a different Provon project.

Discovery calls under `/v1/gateway/*` require `workspace:read`. Provider Key write calls require
`gateway:manage`. These are separate from inference.

## Gateway Returns `UPSTREAM_NOT_CONFIGURED`

`UPSTREAM_NOT_CONFIGURED` means model resolution produced no eligible attempts. Check:

1. The request uses the intended form: `<provider>/<model>`, a configured plain model, or
   `provon/auto`.
2. The Provider Key is enabled and has a usable credential.
3. **Specific** mapping mode contains an enabled mapping for the request model.
4. The mapping allows the requested endpoint.
5. Required features such as streaming, tools, structured output, vision, or audio are declared.
6. A Model Policy or Auto Router allowlist is not excluding the target.
7. Target health or circuit state has not removed every candidate.

Query the current project truth:

```bash
curl "$PROVON_BASE_URL/v1/gateway/capability-matrix?includeDisabled=false" \
  -H "Authorization: Bearer $PROVON_API_KEY"

curl \
  "$PROVON_BASE_URL/v1/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

These calls require `workspace:read`.

## A Managed Model Returns `402`

`INSUFFICIENT_GATEWAY_CREDITS` applies to Provon Cloud paid-through-balance model requests. Open
organization **Billing**, inspect **Cloud AI Gateway credits**, and add credit before retrying.

BYOK Provider Keys do not consume this balance. They use the upstream provider account and its
billing boundary.

## Gateway Returns `429`

Inspect:

- `error.code`: `rate_limit_exceeded`, `gateway_budget_exceeded`, or another project-state error;
- `error.reason` and `error.policyId`;
- `Retry-After`;
- `RateLimit-Limit`, `RateLimit-Remaining`, and `RateLimit-Reset`.

Do not automatically retry a cost or lifetime budget rejection. Retry a resettable rate limit only
when the application can wait until the indicated window.

If `error.code` is `INGESTION_SUSPENDED`, resolve the project ingestion state instead of changing an
API-key limit.

## A Guardrail Returns `409`

An enforced guardrail block uses:

```text
error.code: PROVON_GATEWAY_GUARD_TRIGGERED
x-provon-guard-triggered: true
x-provon-guard-plugin-id: <plugin>
x-provon-guard-action: <action>
```

Do not retry the same payload. Inspect the rule direction, category, match evidence, and action.
Move a new rule back to **Monitor** while evaluating false positives. Keep application
authorization separate from Gateway tool-shape policy.

## The Provider Attempt Fails

Open the request trace and inspect the child attempt:

- Provider Key and credential fingerprint;
- Gateway and upstream model IDs;
- upstream status and normalized error category;
- attempt timeout;
- retry count and fallback index;
- circuit-breaker or target-health events.

Typical actions:

| Error class            | Action                                                      |
| ---------------------- | ----------------------------------------------------------- |
| Authentication         | Rotate or replace the upstream Provider Key credential      |
| Model permission       | Confirm the upstream account and model mapping              |
| Invalid request        | Compare translated request shape with provider requirements |
| Rate limit             | Reduce load or configure bounded recovery to another target |
| Timeout or network     | Check upstream health and the per-attempt/overall deadline  |
| Unsupported capability | Select a target that declares the endpoint and feature      |

Do not configure fallback for upstream authentication, permission, or malformed-request errors.

## Streaming Stops Or Cannot Recover

Test the same request with streaming disabled first. Then verify:

- the target declares `streaming`;
- the client accepts the provider-compatible event stream;
- reverse proxies do not buffer or terminate the stream;
- client, Provider Key, and overall Gateway deadlines are compatible;
- response guardrails have the intended streaming action.

Retry or fallback is only safe before response bytes have been committed. Once a stream has started,
the client may receive a partial response instead of a transparent recovery.

## The Model Call Succeeds But No Trace Appears

Check the evidence path independently from inference:

1. Open **Providers** and set **Request traces** to **All**.
2. Confirm the request used the same project API key as the Workbench project.
3. Search **Traces** by timestamp and `x-provon-request-id`.
4. Allow for asynchronous telemetry delivery in split-runtime deployments.
5. Verify the Gateway telemetry queue or sink and the telemetry/jobs runtime.
6. Inspect dead-letter or runtime logs when delivery continues to fail.

In **Filtered** mode, a successful request is persisted only when at least one filter matches.
**Off** intentionally persists no Gateway trace.

## The Trace Exists But Conversation Diagnosis Is Incomplete

Verify:

- every turn uses the same `x-otel-gen-ai-conversation-id`;
- unrelated user goals do not share one conversation ID;
- agent IDs identify the actual participant;
- application tool, handoff, retrieval, and queue spans are emitted separately;
- the same model operation is not recorded by both Gateway and application instrumentation;
- sampling or filtered capture did not remove required turns.

Gateway capture sees model requests. It cannot reconstruct application operations that were never
instrumented.

## Escalation Record

Preserve this non-secret bundle when escalating:

```text
timestamp:
project ID:
request ID:
endpoint:
model:
HTTP status:
error code:
streaming:
Provider Key ID:
attempt error category:
trace URL or trace ID:
```

Remove prompts, responses, credentials, raw authorization headers, and personal data unless they
are explicitly required and approved for the investigation.

## Related Docs

- [Gateway API](./gateway-api.md)
- [Model providers](./model-providers.md)
- [Routing and reliability](./gateway-routing.md)
- [Gateway governance](./gateway-governance.md)
- [Gateway evidence](./gateway-observability.md)
- [Production guide](./gateway-production.md)