This benchmark ranks image-to-3D models on how faithfully they rebuild an object as a textured 3D asset from a single image. The rankings come from 52,667 pairwise matches, decided by 263,258 votes. Quality is evaluated in two dimensions: Geometry and Faithfulness. All rankings are fitted with a Bradley–Terry model and reported on the Elo scale. All data and judgments are public; see the notes for details.
We evaluate image-to-3D models purely on human feedback. Humans perform inconsistently on absolute judgements, but much better in a comparative setting. Our rating is therefore based on pair-wise matches, the results of which are jointly fit using the Bradley–Terry model where the probability of model m beating model m′ is given by:
A match pairs the 3D assets two models generated from the same input image. As evaluating all permutations is infeasible, matches are sampled without replacement, ensuring broad coverage per model. The resulting scores are scaled to the familiar Elo range via Rm = 400 βm + 1000 (2), which stretches standard Elo by ln 10 ≈ 2.3 to better highlight differences in model capability.
We curated 357 input images, each showing an object to reconstruct, held fixed to ensure reproducibility of the results. Every asset is shown to annotators as a turntable video rendered identically for every model, next to the input image.
To ensure trustworthy ratings we employ several measures, notably annotator pre-filtering via gold-standard 3D comparisons, and redundancy, collecting a minimum of m responses per match (m = 5) and deciding the winner by majority. Read more
Table 1. Overall standings, Image-to-3D benchmark. Scores from a Bradley–Terry fit rescaled to Elo; intervals are 95% bootstrap percentiles. Figure 1 shows how a match is judged.
“Which video reflects the shape of the object in the photo?”
▸
Annotators: 5–0 of 5
“Which video looks more like the object in the photo?”
▸
Annotators: 3–2 of 5
Figure 1. How a match is judged. One pair of models on a different input image for each dimension: (a) clay render, (b) textured render, each with the question annotators answered. Annotators saw the two sides unlabelled and in random order. The highlighted asset is the one annotators chose.
We use crowd intelligence to deliver real, diverse human feedback at scale for model evaluation and post-training.
