Our benchmarks evaluate generative model performance purely based on human feedback. Below we elaborate and describe the framework in detail.
1. Scoring
It has been shown in the literature, that humans perform poorly and inconsistently on absolute pointwise judgements, but are much better at comparative judgements [1]. To rank and rate models we therefore employ the well-established approach of computing a final rating based on the judgements of a series of pair-wise matches [2]. We calculate the final scores based on the Bradley-Terry model which models the probability of model m beating model m′:
The BT scores are scaled to match the familiar Elo range:
However, note that an important property of the BT model over the typical Elo rating scheme, is that it jointly optimizes over all matches. Therefore it is not sensitive to the match order, and in turn more stable.
Confidence intervals are calculated based on bootstrapping, i.e., resampling from the votes with replacement and reporting the 95% CI.
2. Matchmaking
A match is a pairing of generated outputs from two models based on the same input (e.g., prompt). Evaluating all permutations of matches is infeasible, so instead matchups are sampled from the total pool of permutations without replacement. For major benchmarks we ensure at least 10,000 matches per model participant.
3. Prompts
We use the term prompts generally referring to the inputs supplied for generation. In the text-to-image case, this would be simply text, but generally it could be text, images, videos, etc., or a combination of these.
3.1. Prompt selection
The prompts used for the benchmarks are manually selected and curated to be relevant to the specific evaluation criteria, and to provide good coverage of actual meaningful use cases. This means that we also favor commercial applications over ‘viral prompts’ (like Ghiblification or MS Paint trend). To strike a balance between coverage and tractability we curate ~1500 prompts for the major benchmarks.
For the initial general benchmarks we strictly select prompts written in English. We consider this valid without loss of generality, as translating a prompt into English can be solved through an additional LLM step. We leave evaluating specific language capabilities as specific benchmarks. As annotators may not be proficient in English, by default they are shown the prompt translated based on their device language, however they can toggle between the translation and the original.
4. Annotator reliability
Avoiding noisy or low-effort responses entirely is not possible on any crowdsourcing platform, so we employ several measures. Annotators qualify on validation tasks with known answers before their judgements count, and each match collects multiple judgements (by default five) with the winner decided by majority. A reliability estimator combines validation performance and continuous signals into a per-annotator user score used to weight judgements.
5. Capability dimensions
Most arena-style frameworks measure only end-to-end preference. In a research setting one usually wants more granular insight into where a model is strong or weak. A dimension is the combination of a prompt set and a single criterion on which annotators choose. We hypothesize that carefully constructed dimensions represent atomic capabilities that can be studied individually or recombined with weights.