Synthetic data is excellent for expanding a known category. It is weak at telling you which categories matter.
Use it to generate paraphrases, languages, formatting variations, adversarial inputs, and combinations of conditions around a human-defined behavior. Keep the source case and generation method so duplicates and bias remain visible.
Do not let the same model create the cases, expected answers, and judge without human calibration. The suite may reward the generator’s assumptions and miss the messy behavior of real users.
Mix synthetic cases with designed product-contract examples and production failures. Compare category distributions with live traffic. Sample generated cases for human review and remove unrealistic or ambiguous expectations.
Synthetic data creates breadth cheaply. Real traces create relevance. The strongest evaluation set uses both: humans define what good means, production reveals what actually happens, and generation fills deliberate coverage gaps.