The first hundred eval cases should describe your product, not the model benchmark leaderboard.
I would allocate roughly half to common user workflows. These establish whether the feature completes its main job. Use another twenty for known failures from production, support, and manual testing. Add fifteen edge cases involving missing context, unusual formats, multilingual input, and conflicting evidence. Use the final fifteen for policy boundaries, adversarial requests, tool failures, and safe abstention.
Tag every case by task and risk. Overall averages hide a collapse in one important category. Include deterministic checks for structure and permissions, rubric-based checks for semantic quality, and human labels for genuinely ambiguous cases.
Do not freeze the set at one hundred. Retire duplicates, strengthen weak expectations, and add real failures. A useful suite becomes harder over time because users keep discovering cases the team did not imagine.
The number is only a forcing function. Coverage, labels, and connection to release decisions are what make the dataset valuable.