# Run The Gateway In Production

A production rollout is complete only when model execution, policy enforcement, and trace evidence
all work under the same project boundary. Treat the provider response and the Provon trace as
separate verification gates.

Complete the [Gateway quickstart](./gateway-quickstart.md) and
[migration guide](./gateway-migration.md) before moving application traffic.

## 1. Define The Boundary

Use a separate Provon project API key for each application and environment boundary. The workload
receives the project key; Provon stores upstream credentials in project-scoped Provider Keys.

```text
application
  -> Provon project API key
  -> Gateway policy and routing
  -> Provider Key
  -> upstream provider
```

For production:

- expose the Gateway over HTTPS;
- keep Workbench sessions and application API keys separate;
- grant the application key `gateway:invoke`, plus only the other capabilities it needs;
- rotate project and provider keys independently;
- use **Specific** model mappings when the target requires a strict allowlist.

## 2. Validate Request Compatibility

Inventory each request shape before rollout:

- endpoint family;
- provider-qualified or logical model ID;
- streaming;
- tool calling;
- structured output or JSON mode;
- vision, audio, or other multimodal input;
- maximum request and response size;
- client timeout and cancellation behavior.

Query the exact endpoint and features:

```bash
curl \
  "$PROVON_BASE_URL/v1/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"
```

Do not infer compatibility from a provider name. Provon filters candidates by declared endpoint and
feature support, and upstream providers may impose stricter limits.

## 3. Make Routing Explicit

Start a rollout with a provider-qualified model such as `openai/gpt-5-mini`. This isolates Gateway
transport from routing-policy changes.

Add dynamic routing only when its behavior is intentional:

| Requirement                       | Mechanism                               |
| --------------------------------- | --------------------------------------- |
| Pin one provider                  | `<provider>/<model>`                    |
| Expose a stable application model | Plain model plus a project Model Policy |
| Select across mapped targets      | `provon/auto` plus Auto Router          |
| Recover on transient failure      | Bounded retry and ordered fallback      |
| Avoid repeatedly failing targets  | Circuit breaker and target health       |

Retries and fallback are disabled unless configured. Keep recovery inside the overall request
deadline, and do not retry authentication, invalid-request, guardrail, or hard-budget failures.

## 4. Set Hard Resource Limits

Attach request, token, and cost limits to the project API key before increasing traffic.

Use:

- request rate limits for throughput boundaries;
- token rate limits for burst control;
- token or cost budgets for cumulative spend;
- model-scoped limits when expensive routes need a separate boundary.

Provon reserves estimated usage before routing and settles against provider-reported usage when
available. Test both the allowed path and the expected `429` rejection contract.

For Provon Cloud managed models, fund and monitor the organization **Cloud AI Gateway credits**
balance separately. A depleted balance returns `402`; BYOK requests remain on the upstream
provider's billing boundary.

See [Gateway governance](./gateway-governance.md#usage-limits).

## 5. Introduce Guardrails In Monitor Mode

Run new guardrails in **Monitor** against representative traffic before enforcing them. Review:

- match categories and false positives;
- input versus output direction;
- redaction or adjustment behavior;
- blocked-request evidence;
- client handling of the `409` block contract.

Application authorization remains the source of user permissions. Gateway tool policies are an
additional request boundary, not a replacement for application checks.

## 6. Choose Evidence And Privacy Policy

Keep **Request traces** set to **All** during rollout. Verify that every test request produces:

- a root Gateway span;
- the expected provider-attempt spans;
- requested, Gateway, and upstream model IDs;
- status, latency, usage, and cost when available;
- conversation and agent context;
- guardrail and recovery events.

Before steady state, decide:

- whether full request and response bodies may be retained;
- which pseudonymous user and conversation identifiers are required;
- whether **All** or **Filtered** capture preserves enough context for diagnostic Rules;
- the project telemetry retention period.

Filtered capture reduces volume but can fragment multi-turn conversations. If Provon should diagnose
conversation-level failures, prefer complete capture and control storage with retention.

See [Gateway evidence](./gateway-observability.md).

## 7. Roll Out In Stages

A practical sequence is:

1. Send synthetic requests through one Provider Key.
2. Move a small production cohort with a provider-qualified model.
3. Compare response correctness, error rate, latency, tokens, and cost.
4. Verify trace completeness and request-ID correlation.
5. Enable bounded same-model recovery.
6. Introduce Model Policies or Auto Router only if required.
7. Increase traffic after each stage passes.

Log `x-provon-request-id` with the application request. It connects a client-visible failure to the
Gateway root span and provider attempts.

## 8. Prepare Rollback

Keep rollback independent from diagnosis:

- preserve the previous provider endpoint configuration during the rollout window;
- make the Gateway base URL an application configuration value;
- keep provider-qualified models available while testing policy-based routing;
- know how to pause the project Gateway from **Providers**;
- do not delete Provider Keys or mappings until the new path is stable.

Restoring the prior base URL rolls model traffic back. It does not remove Gateway traces already
captured, so the failed rollout remains available as diagnostic evidence.

## Release Gate

Do not complete the rollout until all items pass:

- Gateway endpoint uses HTTPS.
- Project and provider credentials are separated.
- Required endpoint and features appear in discovery.
- Provider-qualified canary requests succeed.
- Streaming, tools, and structured outputs are verified where used.
- Request, token, and cost limits reject as designed.
- Guardrails have been reviewed with representative traffic.
- Traces contain conversation context and provider attempts.
- Retention and payload policy match the data classification.
- The application logs `x-provon-request-id`.
- Rollback has been exercised.

## Related Docs

- [Integrate SDKs](./gateway-integrations.md)
- [Migration guide](./gateway-migration.md)
- [Model providers](./model-providers.md)
- [Routing and reliability](./gateway-routing.md)
- [Gateway governance](./gateway-governance.md)
- [Gateway evidence](./gateway-observability.md)
- [Gateway troubleshooting](./gateway-troubleshooting.md)
- [Self-hosting](./self-hosting.md)
- [Self-hosting security](./self-hosting-security.md)