Skip to content

Dataset Best Practices

A well-curated Dataset improves model behavior more reliably than a large but noisy one. These practices help you collect, review, and maintain Examples that stay useful across tra

View as Markdown Open the plain-text version of this page.

Prefer Quality Over Quantity#

A smaller coherent Dataset almost always beats a larger contradictory one. Before keeping an Example, ask:

  • Is the input complete enough that the expected output makes sense without extra context?
  • Is the expected output actually correct, or is it just the last message the assistant produced?
  • Would I be comfortable if this Example were used verbatim in training?

If the answer to any question is no, edit or remove the Example.

Review Checklist#

For each Example before it is used for training or evaluation:

  • Input preserves the task. The user message, system prompt, and tool context needed for the task are present and not truncated.
  • Expected output is correct. It solves the task, follows project conventions, and is free of hallucinations.
  • Rejected output is genuinely worse. For preference data, the rejected response is a plausible but inferior alternative, not an arbitrary wrong answer.
  • Provenance is intact. A conversation source has trace IDs and a conversation ID; a manual or imported source has an external reference.
  • Privacy boundary is respected. PII, credentials, tokens, and internal identifiers are not present unless they are the explicit subject of the task.
  • No duplication. The same evidence and policy revision produces an idempotent Example, but manual or imported entries can still duplicate semantically identical cases.
  • No contradictions. Similar inputs do not lead to opposite expected outputs.

Choose The Right Objective#

Match the collection objective to the behavior you want to teach:

If you want to... Use this objective Prerequisite
Keep behavior that already works preserve_successful_behavior Completed, high-quality conversations
Fix a diagnosed failure pattern correct_failed_conversations Findings with corrected expected responses
Learn from explicit user feedback learn_from_user_corrections Rejected answers and verified corrections
Ground responses in project facts teach_project_knowledge Reviewed Knowledge Goals

Avoid mixing objectives inside one Dataset. Each Dataset has a single collection policy, and mixed signals make review and export harder.

Tag Conventions#

Tags are optional but make large Datasets manageable. Consider a small, consistent vocabulary:

Prefix Example Meaning
objective: objective:preserve_successful_behavior Why the Example was collected
source: source:manual, source:conversation How the Example entered the Dataset
review: reviewed, review:pending Review state
domain: domain:support, domain:api Functional area
split: split:train, split:eval Intended use for an evaluation run

Keep tags short, lowercase, and hyphen-separated. Tags must be unique within an Example and cannot be empty.

Manual Examples vs. Automatic Collection#

Automatic collection scales, but it can only capture what already happened in traces. Add manual Examples when:

  • The desired behavior has not yet appeared in production.
  • You want to teach a narrow edge case without waiting for it to occur naturally.
  • You need a canonical reference that downstream reviewers can compare against.

When adding a manual Example, always set source.externalRef or source.traceIds so reviewers can understand where it came from.

Balance Coverage#

A Dataset that only contains the most recent or most common conversations will overfit to recent behavior. Aim for:

  • Temporal spread. Include Examples from different time windows.
  • Difficulty spread. Include both routine and edge cases.
  • Outcome spread. For preserve_successful_behavior, include different successful paths, not just the shortest one.

Use the Workbench filters or API list endpoints to inspect the distribution of source kinds, tags, and creation dates.

Training And Evaluation Splits#

Provon does not split a Dataset into train/eval automatically. When you prepare a fine-tuning or evaluation run:

  • Reserve a held-out evaluation set before sampling for training.
  • Use tags such as split:eval to mark Examples that should not be consumed by training runs.
  • Keep the evaluation set representative of the production distribution you care about.

Fine-tuning materializes a run manifest from the selected Examples, so the split is frozen at run creation time.

Privacy And Compliance#

Treat a Dataset as a training artifact. Any data inside it can end up in a model checkpoint or exported JSONL file.

  • Remove or mask personal identifiers, API keys, and internal hostnames.
  • Review conversation sources against your data-retention policy.
  • Disable collection when you no longer want new production data entering the Dataset.

Keep Datasets Focused#

One Dataset should teach one thing. Creating separate Datasets for separate behaviors makes it easier to:

  • review Examples with consistent criteria;
  • mix and match Datasets in fine-tuning jobs;
  • retire outdated behavior without affecting unrelated examples.