MVBench

20 dynamic video tasks specifically constructed so static single-frame shortcuts cannot solve them (action order, moving direction, counterfactuals).26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
VideoChat2 →MVBench · original evaluation · GPT-4V take 16 frames as input, and the resolution is 512×512, while others use small resolution of 224×224. · LLM: Mistral-7B	60.4% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · GPT-4V take 16 frames as input, and the resolution is 512×512, while others use small resolution of 224×224. · LLM: Mistral-7B	Leads this displayed comparison. Source notes Source cell: 60.4
VideoChat2 →MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	51.1% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	#2 of 16 in this displayed comparison; 9.3 pp below the leader. Source notes Source cell: 51.1
GPT-4V [59] →OpenAI	43.5% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · GPT-4V take 16 frames as input, and the resolution is 512×512, while others use small resolution of 224×224. · LLM: GPT-4	#3 of 16 in this displayed comparison; 16.9 pp below the leader. Source notes Source cell: 43.5
LLaVA [47] →	36% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: Vicuna-7B	#4 of 16 in this displayed comparison; 24.4 pp below the leader. Source notes Source cell: 36.0
VideoChat [42] →	35.5% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	#5 of 16 in this displayed comparison; 24.9 pp below the leader. Source notes Source cell: 35.5
VideoChat2text →	34.7% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	#6 of 16 in this displayed comparison; 25.7 pp below the leader. Source notes Source cell: 34.7
VideoLLaMA [101] →	34.1% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	#7 of 16 in this displayed comparison; 26.3 pp below the leader. Source notes Source cell: 34.1
Otter-I [38] →	33.5% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: MPT-7B	#8 of 16 in this displayed comparison; 26.9 pp below the leader. Source notes Source cell: 33.5
VideoChatGPT [51] →	32.7% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: Vicuna-7B	#9 of 16 in this displayed comparison; 27.7 pp below the leader. Source notes Source cell: 32.7
InstructBLIP [11] →	32.5% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: Vicuna-7B	#10 of 16 in this displayed comparison; 27.9 pp below the leader. Source notes Source cell: 32.5
LLaMA-Adapter [103] →Meta	31.7% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: LLaMA-7B	#11 of 16 in this displayed comparison; 28.7 pp below the leader. Source notes Source cell: 31.7
BLIP2 [39] →	31.4% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: FlanT5-XL	#12 of 16 in this displayed comparison; 29 pp below the leader. Source notes Source cell: 31.4
mPLUG-Owl-V [94] →	29.7% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: LLaMA-7B	#13 of 16 in this displayed comparison; 30.7 pp below the leader. Source notes Source cell: 29.7
mPLUG-Owl-I [94] →	29.4% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: LLaMA-7B	#14 of 16 in this displayed comparison; 31 pp below the leader. Source notes Source cell: 29.4
Otter-V [38] →	26.8% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Video MLLMs: All models take 16 frames as input, with the exception of VideoChatGPT, which uses 100 frames. · LLM: LLaMA-7B	#15 of 16 in this displayed comparison; 33.6 pp below the leader. Source notes Source cell: 26.8
MiniGPT-4 [104] →	18.8% 1 report	MVBench · Average task accuracy[1]MVBench · original evaluation · Image MLLMs: Following [11], all models take 4 frames as input, with the output embeddings concatenated before feeding into the LLM. · LLM: Vicuna-7B	#16 of 16 in this displayed comparison; 41.6 pp below the leader. Source notes Source cell: 18.8

Sources

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — published result table ↗
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu et al. · 2023
All 12 authors
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao
Dates & evaluation details
MVBench · Average task accuracy · original evaluation · 2026-10-26
Published model comparison, 2023. MVBench · original evaluation. Reported average across twenty short-video understanding tasks. LLM backbone and the source input condition are preserved. Table 2; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

20 dynamic video tasks specifically constructed so static single-frame shortcuts cannot solve them (action order, moving direction, counterfactuals).

Reading the measurement

Study what information the model receives and which inference the answer requires. A question may mix recall, visual perception, deduction and search. Use task breakdowns to find the mechanism behind a headline gain.

Published evaluation

Reported average across twenty short-video understanding tasks. LLM backbone and the source input condition are preserved.

Where this fits in an evaluation suite

Ask what the task actually requires: recall, deduction, search, or a mixture. Separate closed-book reasoning from retrieval and tool-assisted problem solving.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Reported average across twenty short-video understanding tasks. LLM backbone and the source input condition are preserved.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — published result table ↗
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu et al. · 2023
All 12 authors
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, Yu Qiao

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All reasoning benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

Artificial Analysis Intelligence Index v5.0

Humanity’s Last Exam

MMMU-Pro

ARC-AGI-2