CountBench & PixMo-Count

Evaluates dense object counting and point-grounding in natural scenes.27 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
Molmo-72B →AI2	91.2% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	Leads this displayed comparison. Source notes Source cell: 91.2
Claude-3.5 Sonnet →Anthropic	89.7% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#2 of 27 in this displayed comparison; 1.5 pp below the leader. Source notes Source cell: 89.7
Molmo-7B-O →AI2	89% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#3 of 27 in this displayed comparison; 2.2 pp below the leader. Source notes Source cell: 89.0
Molmo-7B-D →AI2	88.5% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#4 of 27 in this displayed comparison; 2.7 pp below the leader. Source notes Source cell: 88.5
GPT-4o-0513 →OpenAI	87.9% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#5 of 27 in this displayed comparison; 3.3 pp below the leader. Source notes Source cell: 87.9
MolmoE-1B →	87.2% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#6 of 27 in this displayed comparison; 4 pp below the leader. Source notes Source cell: 87.2
Gemini 1.5 Pro →Google	85.8% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#7 of 27 in this displayed comparison; 5.4 pp below the leader. Source notes Source cell: 85.8
LLaVA OneVision-72B† →	84.3% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#8 of 27 in this displayed comparison; 6.9 pp below the leader. Source notes Source cell: 84.3
Claude-3 Opus →Anthropic	83.6% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#9 of 27 in this displayed comparison; 7.6 pp below the leader. Source notes Source cell: 83.6
Claude-3 Haiku →Anthropic	83% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#10 of 27 in this displayed comparison; 8.2 pp below the leader. Source notes Source cell: 83.0
xGen-MM-interleave-4B† →	81.9% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#11 of 27 in this displayed comparison; 9.3 pp below the leader. Source notes Source cell: 81.9
Gemini 1.5 Flash →Google	81.6% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#12 of 27 in this displayed comparison; 9.6 pp below the leader. Source notes Source cell: 81.6
PaliGemma-mix-3B →Google	80.6% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#13 of 27 in this displayed comparison; 10.6 pp below the leader. Source notes Source cell: 80.6
Qwen2-VL-72B →Qwen	80.4% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#14 of 27 in this displayed comparison; 10.8 pp below the leader. Source notes Source cell: 80.4
LLaVA OneVision-7B† →	78.8% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#15 of 27 in this displayed comparison; 12.4 pp below the leader. Source notes Source cell: 78.8
Pixtral-12B →Mistral	78.8% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#15 of 27 in this displayed comparison; 12.4 pp below the leader. Source notes Source cell: 78.8
Llama-3.2V-90B-Instruct →Meta	78.5% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#17 of 27 in this displayed comparison; 12.7 pp below the leader. Source notes Source cell: 78.5
Qwen2-VL-7B →Qwen	76.5% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#18 of 27 in this displayed comparison; 14.7 pp below the leader. Source notes Source cell: 76.5
Cambrian-1-8B† →	76.4% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#19 of 27 in this displayed comparison; 14.8 pp below the leader. Source notes Source cell: 76.4
Cambrian-1-34B† →	75.6% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#20 of 27 in this displayed comparison; 15.6 pp below the leader. Source notes Source cell: 75.6
InternVL2-Llama-3-76B →Meta	74.7% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#21 of 27 in this displayed comparison; 16.5 pp below the leader. Source notes Source cell: 74.7
Llama-3.2V-11B-Instruct →Meta	73.1% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#22 of 27 in this displayed comparison; 18.1 pp below the leader. Source notes Source cell: 73.1
GPT-4V →OpenAI	69.9% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#23 of 27 in this displayed comparison; 21.3 pp below the leader. Source notes Source cell: 69.9
Phi3.5-Vision-4B →Microsoft	64.6% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#24 of 27 in this displayed comparison; 26.6 pp below the leader. Source notes Source cell: 64.6
InternVL2-8B →	57.8% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#25 of 27 in this displayed comparison; 33.4 pp below the leader. Source notes Source cell: 57.8
LLaVA-1.5-13B →	47.1% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#26 of 27 in this displayed comparison; 44.1 pp below the leader. Source notes Source cell: 47.1
LLaVA-1.5-7B →	40.1% 1 report	CountBench & PixMo-Count · CountBenchQA · test accuracy[1]Molmo and PixMo · Table 1 comparison	#27 of 27 in this displayed comparison; 51.1 pp below the leader. Source notes Source cell: 40.1
Molmo-72B →AI2	85.2% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	Leads this displayed comparison. Source notes Source cell: 85.2
Molmo-7B-D →AI2	84.8% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#2 of 27 in this displayed comparison; 0.4 pp below the leader. Source notes Source cell: 84.8
Molmo-7B-O →AI2	83.3% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#3 of 27 in this displayed comparison; 1.9 pp below the leader. Source notes Source cell: 83.3
MolmoE-1B →	79.6% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#4 of 27 in this displayed comparison; 5.6 pp below the leader. Source notes Source cell: 79.6
Gemini 1.5 Pro →Google	64.3% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#5 of 27 in this displayed comparison; 20.9 pp below the leader. Source notes Source cell: 64.3
Gemini 1.5 Flash →Google	61.1% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#6 of 27 in this displayed comparison; 24.1 pp below the leader. Source notes Source cell: 61.1
LLaVA OneVision-72B† →	60.7% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#7 of 27 in this displayed comparison; 24.5 pp below the leader. Source notes Source cell: 60.7
PaliGemma-mix-3B →Google	60% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#8 of 27 in this displayed comparison; 25.2 pp below the leader. Source notes Source cell: 60.0
GPT-4o-0513 →OpenAI	59.6% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#9 of 27 in this displayed comparison; 25.6 pp below the leader. Source notes Source cell: 59.6
Llama-3.2V-90B-Instruct →Meta	58.5% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#10 of 27 in this displayed comparison; 26.7 pp below the leader. Source notes Source cell: 58.5
Claude-3.5 Sonnet →Anthropic	58.3% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#11 of 27 in this displayed comparison; 26.9 pp below the leader. Source notes Source cell: 58.3
Qwen2-VL-72B →Qwen	55.7% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#12 of 27 in this displayed comparison; 29.5 pp below the leader. Source notes Source cell: 55.7
InternVL2-Llama-3-76B →Meta	54.6% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#13 of 27 in this displayed comparison; 30.6 pp below the leader. Source notes Source cell: 54.6
LLaVA OneVision-7B† →	54.4% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#14 of 27 in this displayed comparison; 30.8 pp below the leader. Source notes Source cell: 54.4
Pixtral-12B →Mistral	51.7% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#15 of 27 in this displayed comparison; 33.5 pp below the leader. Source notes Source cell: 51.7
Cambrian-1-34B† →	50.7% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#16 of 27 in this displayed comparison; 34.5 pp below the leader. Source notes Source cell: 50.7
xGen-MM-interleave-4B† →	50.2% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#17 of 27 in this displayed comparison; 35 pp below the leader. Source notes Source cell: 50.2
Qwen2-VL-7B →Qwen	48% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#18 of 27 in this displayed comparison; 37.2 pp below the leader. Source notes Source cell: 48.0
Llama-3.2V-11B-Instruct →Meta	47.4% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#19 of 27 in this displayed comparison; 37.8 pp below the leader. Source notes Source cell: 47.4
Cambrian-1-8B† →	46.6% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#20 of 27 in this displayed comparison; 38.6 pp below the leader. Source notes Source cell: 46.6
GPT-4V →OpenAI	45% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#21 of 27 in this displayed comparison; 40.2 pp below the leader. Source notes Source cell: 45.0
Claude-3 Haiku →Anthropic	43.9% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#22 of 27 in this displayed comparison; 41.3 pp below the leader. Source notes Source cell: 43.9
InternVL2-8B →	43.9% 1 report	CountBench & PixMo-Count · PixMo-Count · test accuracy[1]Molmo and PixMo · Table 1 comparison	#22 of 27 in this displayed comparison; 41.3 pp below the leader. Source notes Source cell: 43.9

Sources

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung et al. · 2024
All 50 authors
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha
Dates & evaluation details
CountBench & PixMo-Count · CountBenchQA · test accuracy · Molmo and PixMo · Table 1 comparison · 2026-10-27
Published model comparison, 2024. Molmo and PixMo · Table 1 comparison. Reported academic benchmark accuracy. API-model versions and open-model sizes are retained. CountBenchQA and PixMo-Count are different image sets and remain separate measures; the paper notes that prompting and preprocessing can materially change results. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
CountBench & PixMo-Count · test accuracy · Molmo and PixMo · Table 1 comparison · 2026-10-27
Published model comparison, 2024. Molmo and PixMo · Table 1 comparison. Reported academic benchmark accuracy. API-model versions and open-model sizes are retained. CountBenchQA and PixMo-Count are different image sets and remain separate measures; the paper notes that prompting and preprocessing can materially change results. Table 1; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

Evaluates dense object counting and point-grounding in natural scenes.

Reading the measurement

Study what information the model receives and which inference the answer requires. A question may mix recall, visual perception, deduction and search. Use task breakdowns to find the mechanism behind a headline gain.

Published evaluation

Reported academic benchmark accuracy. API-model versions and open-model sizes are retained. CountBenchQA and PixMo-Count are different image sets and remain separate measures; the paper notes that prompting and preprocessing can materially change results.

Where this fits in an evaluation suite

Ask what the task actually requires: recall, deduction, search, or a mixture. Separate closed-book reasoning from retrieval and tool-assisted problem solving.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models — published result table ↗
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung et al. · 2024
All 50 authors
Deitke, Matt, Clark, Christopher, Lee, Sangho, Tripathi, Rohun, Yang, Yue, Park, Jae Sung, Salehi, Mohammadreza, Muennighoff, Niklas, Lo, Kyle, Soldaini, Luca, Lu, Jiasen, Anderson, Taira, Bransom, Erin, Ehsani, Kiana, Ngo, Huong, Chen, YenSung, Patel, Ajay, Yatskar, Mark, Callison-Burch, Chris, Head, Andrew, Hendrix, Rose, Bastani, Favyen, VanderBilt, Eli, Lambert, Nathan, Chou, Yvonne, Chheda, Arnavi, Sparks, Jenna, Skjonsberg, Sam, Schmitz, Michael, Sarnat, Aaron, Bischoff, Byron, Walsh, Pete, Newell, Chris, Wolters, Piper, Gupta, Tanmay, Zeng, Kuo-Hao, Borchardt, Jon, Groeneveld, Dirk, Nam, Crystal, Lebrecht, Sophie, Wittlif, Caitlin, Schoenick, Carissa, Michel, Oscar, Krishna, Ranjay, Weihs, Luca, Smith, Noah A., Hajishirzi, Hannaneh, Girshick, Ross, Farhadi, Ali, Kembhavi, Aniruddha

Reviewed 2026-10-27. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All reasoning benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

Artificial Analysis Intelligence Index v5.0

Humanity’s Last Exam

MMMU-Pro

ARC-AGI-2