Skip to content

Migrate To The Provon Gateway

This guide moves an existing model call behind Provon without changing model behavior and routing policy at the same time. The migration is complete only when the application respo

View as Markdown Open the plain-text version of this page.
On this page Browse sections

Complete the Gateway quickstart against the target project first.

Migration Strategy#

Change one responsibility at a time:

  1. Keep the current provider and model.
  2. Move the provider credential into Provon.
  3. Change the application base URL and API key.
  4. Verify response compatibility and Gateway trace evidence.
  5. Add stable conversation and agent context.
  6. Add logical model names, recovery, Auto Router, limits, or guardrails only when needed.

Use a provider-qualified model during steps 1 through 5. For example, migrate gpt-5-mini to openai/gpt-5-mini, not to provon/auto.

1. Inventory The Current Calls#

Record every request shape the application uses:

Field Examples
Endpoint Chat Completions, Responses, Messages, embeddings, rerank
Provider and model OpenAI and gpt-5-mini
Features Streaming, tools, structured output, vision, audio
Authentication Bearer, x-api-key, cloud-specific signing
Client behavior Timeout, cancellation, retries, error parsing
Payload limits Largest request, response, file, image, or audio body
Required provider fields Provider-specific body fields or response metadata

Do not assume that an OpenAI-compatible chat request implies compatibility for files, batches, realtime, fine-tuning, or provider-native extensions.

2. Verify Target Capability#

Connect the upstream credential under Providers, then query the exact endpoint and feature set:

bash
curl \
  "$PROVON_BASE_URL/v1/gateway/model-selection?endpoint=chat-completions&features=streaming,tool-calling" \
  -H "Authorization: Bearer $PROVON_API_KEY"

This discovery call requires workspace:read. Inference requires gateway:invoke.

Check every request shape independently. A provider may be native for one endpoint, translated for another, bridged through a different endpoint, or unsupported. See the Gateway API discovery contract.

3. Separate The Credentials#

Before migration:

text
application -> upstream provider key -> provider

After migration:

text
application -> Provon project API key -> Gateway -> project Provider Key -> provider

Create a project API key for the application and environment boundary. Grant gateway:invoke and only the other capabilities the workload needs. Store the upstream provider key in a project-scoped Provider Key.

Rotate the two credentials independently. Never send the upstream key to the Gateway inference endpoint.

4. Change The Client#

For an OpenAI-compatible client, replace:

Setting Before After
Base URL Provider API origin <PROVON_ORIGIN>/gateway/v1
API key Upstream provider key Provon project API key
Model Provider model ID Provider-qualified Gateway model ID
Auth Provider-specific mechanism Authorization: Bearer <PROVON_API_KEY>

Environment-only migrations commonly become:

bash
export OPENAI_BASE_URL="$PROVON_BASE_URL/gateway/v1"
export OPENAI_API_KEY="$PROVON_API_KEY"

Do not append another /v1 when the configured base URL already ends in /gateway/v1.

Clients that hard-code provider authentication, sign requests, or reject custom base URLs need an adapter or direct HTTP integration. See SDK integrations.

5. Preserve Model Intent#

Start with a provider-qualified model:

json
{
  "model": "openai/gpt-5-mini"
}

This keeps provider intent explicit while Provon selects among eligible Provider Keys for that provider.

After the migration is stable, choose deliberately:

Desired behavior Model form and configuration
Keep provider selection in application code <provider>/<model>
Give the application a stable logical name Plain model plus a project Model Policy
Let Provon select across mapped targets provon/auto plus Auto Router

Changing to a logical model or Auto Router is a routing change, not part of the transport migration. Test it separately.

6. Add Diagnostic Identity#

Gateway capture can diagnose a single call without extra headers, but conversation Rules need stable identity across turns.

Send request-scoped values:

text
x-otel-gen-ai-conversation-id: one stable ID for the user goal
x-otel-user-id: a pseudonymous user ID
x-otel-session-id: the application session
x-otel-gen-ai-agent-id: the logical agent participant
x-otel-gen-ai-agent-version: the deployed agent version
x-otel-gen-ai-workflow-name: the workflow or graph name

Use the same conversation ID for every turn in one goal. Do not reuse one conversation ID across unrelated users or tasks. Keep user, session, and conversation values out of process-wide default headers when a client instance serves multiple requests.

See Gateway evidence and multi-agent tracing.

7. Avoid Duplicate Model Evidence#

If the application already emits an OpenTelemetry model-call span, decide which source owns that operation:

  • use the Gateway span as the model-boundary record; or
  • keep the application model span and disable equivalent Gateway persistence.

Keep application-side spans for tools, handoffs, queues, retrieval, and business operations. The Gateway cannot reconstruct work that never crosses the model boundary.

Recording the same model call twice can distort token, cost, latency, and diagnostic aggregates. See diagnosis-ready tracing.

8. Canary And Verify#

Make the Gateway base URL a runtime configuration value and move a small cohort first.

For each canary request:

  1. Compare status, response shape, output, and provider model with the direct path.
  2. Test streaming, tools, structured output, and multimodal requests separately.
  3. Log x-provon-request-id with the application request.
  4. Open the trace and verify the root span and every provider attempt.
  5. Check latency, time to first chunk, tokens, cost, and normalized errors.
  6. Confirm conversation and agent identity are correct.

Do not enable fallback to hide a compatibility failure. First make the intended primary path pass.

9. Add Controls In Order#

After compatibility is established:

  1. Attach request, token, and cost limits to the application key.
  2. Run new guardrails in Monitor mode.
  3. Add bounded retries for transient failures.
  4. Add same-model recovery before cross-model substitution.
  5. Introduce Model Policies or Auto Router only for an explicit routing requirement.
  6. Choose the steady-state Trace Capture and retention policy.

Each control changes behavior. Verify its client contract and trace evidence before adding the next.

Rollback#

Keep these independent switches during the rollout:

  • the prior provider base URL;
  • the prior application secret reference;
  • the provider-qualified model;
  • a Gateway pause control;
  • unchanged Provider Keys and mappings until the rollout is complete.

Rollback restores the previous base URL and provider credential. It does not delete Gateway traces, so failed canaries remain available for diagnosis.

Migration Gate#

  • Every endpoint and feature appears in discovery.
  • Project and provider credentials are separate.
  • A provider-qualified request passes before policy-based routing is enabled.
  • Client timeouts, cancellations, streaming, and error parsing behave as expected.
  • x-provon-request-id is logged.
  • Gateway traces contain provider attempts and required diagnostic identity.
  • Duplicate model-call telemetry has a defined owner.
  • Limits and guardrails have been tested with expected rejections.
  • The previous direct-provider path has been exercised as rollback.

Continue with the production guide. Use Gateway troubleshooting when a verification gate fails.