Building fair evaluation sets is a combinatorial problem. Here’s how to solve it exactly.

Overall accuracy is a weighted average — and the weights are whatever your evaluation set happens to contain. If your eval set is 90% group…