Improve A Model From Trace Evidence
The model improvement loop turns reviewed production behavior into Dataset examples, a fine-tuned checkpoint, and a new Gateway target whose behavior can be observed again.
Before You Start#
You need:
- a Node deployment with model inference and fine-tuning services;
- a base model supported by the local runtime;
- diagnosis-ready conversations;
- a Dataset with reviewed chat examples.
Self-hosted model execution and training are not available in the Cloudflare runtime.
1. Choose What To Learn#
Create a Dataset with one collection objective:
- preserve successful behavior;
- correct failed conversations;
- learn from user corrections;
- teach project knowledge.
Use filters to constrain the evidence when the objective supports them. Collection is based on trace-backed producers; Findings and Knowledge remain independent domain artifacts rather than Dataset source types.
See the Dataset quickstart.
2. Review The Examples#
Inspect the input, expected output, optional rejected output, tags, and evidence provenance. Remove examples that are unsupported, duplicated, private beyond the intended training boundary, or inconsistent with the desired behavior.
Do not use example count alone as a readiness signal. A smaller coherent Dataset is preferable to a larger contradictory one.
3. Train A Checkpoint#
Open model fine-tuning, select a base model and Dataset, then create an SFT job. Provon freezes the selected examples into an immutable run manifest so later Dataset edits do not change the meaning of the job.
Track the job until it completes and inspect its checkpoint lineage. See Model fine-tuning.
4. Deploy Through The Gateway#
Deploy the checkpoint to the local model runtime. Once its inference service is healthy, route a
provider-qualified or project-owned logical model name to the new self/ target.
Keep the previous target available until the new checkpoint passes representative requests. See AI Models and Gateway routing.
5. Close The Loop#
Send new traffic through the deployed target and compare:
- task completion and Finding recurrence;
- latency, token usage, and errors;
- behavior on held-out evaluation examples;
- regressions outside the behavior represented in the training Dataset.
The new traces become evidence for the next review, repair, or training cycle.