AppWorld

Interactive world of 9 day-to-day apps (Gmail, Spotify, Venmo, Amazon, Splitwise, Todoist) and 457 APIs governed by 100 simulated users.26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
GPT-4O →OpenAIAppWorld · Test-C · ReAct	13% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · ReAct	Leads this displayed comparison. Source notes Source cell: 30.2 \| 13.0
GPT-4O →OpenAIAppWorld · Test-C · FullCodeRefl	12.2% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · FullCodeRefl	#2 of 16 in this displayed comparison; 0.8 pp below the leader. Source notes Source cell: 19.2 \| 12.2
GPT-4O →OpenAIAppWorld · Test-C · IPFunCall	10.1% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · IPFunCall	#3 of 16 in this displayed comparison; 2.9 pp below the leader. Source notes Source cell: 18.0 \| 10.1
GPT-4Trb →OpenAIAppWorld · Test-C · IPFunCall	9.3% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · IPFunCall	#4 of 16 in this displayed comparison; 3.7 pp below the leader. Source notes Source cell: 14.6 \| 9.30
GPT-4O →OpenAIAppWorld · Test-C · PlanExec	7.9% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · PlanExec	#5 of 16 in this displayed comparison; 5.1 pp below the leader. Source notes Source cell: 19.7 \| 7.9
GPT-4Trb →OpenAIAppWorld · Test-C · FullCodeRefl	7.2% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · FullCodeRefl	#6 of 16 in this displayed comparison; 5.8 pp below the leader. Source notes Source cell: 12.5 \| 7.2
GPT-4Trb →OpenAIAppWorld · Test-C · ReAct	5.8% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · ReAct	#7 of 16 in this displayed comparison; 7.2 pp below the leader. Source notes Source cell: 17.5 \| 5.8
LLaMA3 →AppWorld · Test-C · FullCodeRefl	4.3% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · FullCodeRefl	#8 of 16 in this displayed comparison; 8.7 pp below the leader. Source notes Source cell: 7.0 \| 4.3
GPT-4Trb →OpenAIAppWorld · Test-C · PlanExec	3.6% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · PlanExec	#9 of 16 in this displayed comparison; 9.4 pp below the leader. Source notes Source cell: 11.0 \| 3.6
DeepSeek →DeepSeekAppWorld · Test-C · FullCodeRefl	2.9% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · FullCodeRefl	#10 of 16 in this displayed comparison; 10.1 pp below the leader. Source notes Source cell: 5.8 \| 2.9
DeepSeek →DeepSeekAppWorld · Test-C · ReAct	0.7% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · ReAct	#11 of 16 in this displayed comparison; 12.3 pp below the leader. Source notes Source cell: 2.9 \| 0.7
LLaMA3 →AppWorld · Test-C · PlanExec	0.7% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · PlanExec	#11 of 16 in this displayed comparison; 12.3 pp below the leader. Source notes Source cell: 2.4 \| 0.7
DeepSeek →DeepSeekAppWorld · Test-C · PlanExec	0% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · PlanExec	#13 of 16 in this displayed comparison; 13 pp below the leader. Source notes Source cell: 0.7 \| 0.0
LLaMA →Meta	0% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · ToolLLaMA	#13 of 16 in this displayed comparison; 13 pp below the leader. Source notes Source cell: 0.0 \| 0.0
LLaMA3 →AppWorld · Test-C · ReAct	0% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · ReAct	#13 of 16 in this displayed comparison; 13 pp below the leader. Source notes Source cell: 3.4 \| 0.0
Mistral-7B →Mistral	0% 1 report	AppWorld · Test-C · Scenario goal completion[1]AppWorld · Test-C · CodeAct	#13 of 16 in this displayed comparison; 13 pp below the leader. Source notes Source cell: 0.0 \| 0.0
GPT-4O →OpenAIAppWorld · Test-C · ReAct	30.2% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · ReAct	Leads this displayed comparison. Source notes Source cell: 30.2 \| 13.0
GPT-4O →OpenAIAppWorld · Test-C · PlanExec	19.7% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · PlanExec	#2 of 16 in this displayed comparison; 10.5 pp below the leader. Source notes Source cell: 19.7 \| 7.9
GPT-4O →OpenAIAppWorld · Test-C · FullCodeRefl	19.2% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · FullCodeRefl	#3 of 16 in this displayed comparison; 11 pp below the leader. Source notes Source cell: 19.2 \| 12.2
GPT-4O →OpenAIAppWorld · Test-C · IPFunCall	18% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · IPFunCall	#4 of 16 in this displayed comparison; 12.2 pp below the leader. Source notes Source cell: 18.0 \| 10.1
GPT-4Trb →OpenAIAppWorld · Test-C · ReAct	17.5% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · ReAct	#5 of 16 in this displayed comparison; 12.7 pp below the leader. Source notes Source cell: 17.5 \| 5.8
GPT-4Trb →OpenAIAppWorld · Test-C · IPFunCall	14.6% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · IPFunCall	#6 of 16 in this displayed comparison; 15.6 pp below the leader. Source notes Source cell: 14.6 \| 9.30
GPT-4Trb →OpenAIAppWorld · Test-C · FullCodeRefl	12.5% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · FullCodeRefl	#7 of 16 in this displayed comparison; 17.7 pp below the leader. Source notes Source cell: 12.5 \| 7.2
GPT-4Trb →OpenAIAppWorld · Test-C · PlanExec	11% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · PlanExec	#8 of 16 in this displayed comparison; 19.2 pp below the leader. Source notes Source cell: 11.0 \| 3.6
LLaMA3 →AppWorld · Test-C · FullCodeRefl	7% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · FullCodeRefl	#9 of 16 in this displayed comparison; 23.2 pp below the leader. Source notes Source cell: 7.0 \| 4.3
DeepSeek →DeepSeekAppWorld · Test-C · FullCodeRefl	5.8% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · FullCodeRefl	#10 of 16 in this displayed comparison; 24.4 pp below the leader. Source notes Source cell: 5.8 \| 2.9
LLaMA3 →AppWorld · Test-C · ReAct	3.4% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · ReAct	#11 of 16 in this displayed comparison; 26.8 pp below the leader. Source notes Source cell: 3.4 \| 0.0
DeepSeek →DeepSeekAppWorld · Test-C · ReAct	2.9% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · ReAct	#12 of 16 in this displayed comparison; 27.3 pp below the leader. Source notes Source cell: 2.9 \| 0.7
LLaMA3 →AppWorld · Test-C · PlanExec	2.4% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · PlanExec	#13 of 16 in this displayed comparison; 27.8 pp below the leader. Source notes Source cell: 2.4 \| 0.7
DeepSeek →DeepSeekAppWorld · Test-C · PlanExec	0.7% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · PlanExec	#14 of 16 in this displayed comparison; 29.5 pp below the leader. Source notes Source cell: 0.7 \| 0.0
LLaMA →Meta	0% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · ToolLLaMA	#15 of 16 in this displayed comparison; 30.2 pp below the leader. Source notes Source cell: 0.0 \| 0.0
Mistral-7B →Mistral	0% 1 report	AppWorld · Test-C · Task goal completion[1]AppWorld · Test-C · CodeAct	#15 of 16 in this displayed comparison; 30.2 pp below the leader. Source notes Source cell: 0.0 \| 0.0
GPT-4O →OpenAIAppWorld · Test-N · ReAct	32.1% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · ReAct	Leads this displayed comparison. Source notes Source cell: 48.8 \| 32.1
GPT-4O →OpenAIAppWorld · Test-N · FullCodeRefl	26.8% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · FullCodeRefl	#2 of 16 in this displayed comparison; 5.3 pp below the leader. Source notes Source cell: 33.9 \| 26.8
GPT-4O →OpenAIAppWorld · Test-N · PlanExec	23.2% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · PlanExec	#3 of 16 in this displayed comparison; 8.9 pp below the leader. Source notes Source cell: 44.6 \| 23.2
GPT-4Trb →OpenAIAppWorld · Test-N · IPFunCall	21.4% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · IPFunCall	#4 of 16 in this displayed comparison; 10.7 pp below the leader. Source notes Source cell: 30.4 \| 21.4
GPT-4Trb →OpenAIAppWorld · Test-N · FullCodeRefl	19.6% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · FullCodeRefl	#5 of 16 in this displayed comparison; 12.5 pp below the leader. Source notes Source cell: 25.6 \| 19.6
LLaMA3 →AppWorld · Test-N · FullCodeRefl	17.9% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · FullCodeRefl	#6 of 16 in this displayed comparison; 14.2 pp below the leader. Source notes Source cell: 24.4 \| 17.9
GPT-4O →OpenAIAppWorld · Test-N · IPFunCall	16.1% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · IPFunCall	#7 of 16 in this displayed comparison; 16 pp below the leader. Source notes Source cell: 32.1 \| 16.1
GPT-4Trb →OpenAIAppWorld · Test-N · PlanExec	16.1% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · PlanExec	#7 of 16 in this displayed comparison; 16 pp below the leader. Source notes Source cell: 32.7 \| 16.1
GPT-4Trb →OpenAIAppWorld · Test-N · ReAct	12.5% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · ReAct	#9 of 16 in this displayed comparison; 19.6 pp below the leader. Source notes Source cell: 26.8 \| 12.5
DeepSeek →DeepSeekAppWorld · Test-N · FullCodeRefl	8.9% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · FullCodeRefl	#10 of 16 in this displayed comparison; 23.2 pp below the leader. Source notes Source cell: 13.1 \| 8.90
LLaMA3 →AppWorld · Test-N · ReAct	8.9% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · ReAct	#10 of 16 in this displayed comparison; 23.2 pp below the leader. Source notes Source cell: 20.8 \| 8.9
DeepSeek →DeepSeekAppWorld · Test-N · ReAct	1.8% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · ReAct	#12 of 16 in this displayed comparison; 30.3 pp below the leader. Source notes Source cell: 07.1 \| 1.8
LLaMA3 →AppWorld · Test-N · PlanExec	1.8% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · PlanExec	#12 of 16 in this displayed comparison; 30.3 pp below the leader. Source notes Source cell: 8.9 \| 1.8
DeepSeek →DeepSeekAppWorld · Test-N · PlanExec	0% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · PlanExec	#14 of 16 in this displayed comparison; 32.1 pp below the leader. Source notes Source cell: 1.8 \| 0.0
LLaMA →Meta	0% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · ToolLLaMA	#14 of 16 in this displayed comparison; 32.1 pp below the leader. Source notes Source cell: 0.0 \| 0.0
Mistral-7B →Mistral	0% 1 report	AppWorld · Test-N · Scenario goal completion[1]AppWorld · Test-N · CodeAct	#14 of 16 in this displayed comparison; 32.1 pp below the leader. Source notes Source cell: 0.0 \| 0.0
GPT-4O →OpenAIAppWorld · Test-N · ReAct	48.8% 1 report	AppWorld · Test-N · Task goal completion[1]AppWorld · Test-N · ReAct	Leads this displayed comparison. Source notes Source cell: 48.8 \| 32.1
GPT-4O →OpenAIAppWorld · Test-N · PlanExec	44.6% 1 report	AppWorld · Test-N · Task goal completion[1]AppWorld · Test-N · PlanExec	#2 of 16 in this displayed comparison; 4.2 pp below the leader. Source notes Source cell: 44.6 \| 23.2

Sources

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents — published result table ↗
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li et al. · 2024
All 9 authors
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian
Dates & evaluation details
AppWorld · Test-N · Task goal completion · 2026-10-26
Published model comparison, 2024. AppWorld · Test-N. The source cell lists TGC | SGC. This observation uses only Task goal completion; the two measures and test splits are never averaged together. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
AppWorld · Test-N · Scenario goal completion · 2026-10-26
Published model comparison, 2024. AppWorld · Test-N. The source cell lists TGC | SGC. This observation uses only Scenario goal completion; the two measures and test splits are never averaged together. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
AppWorld · Test-C · Task goal completion · 2026-10-26
Published model comparison, 2024. AppWorld · Test-C. The source cell lists TGC | SGC. This observation uses only Task goal completion; the two measures and test splits are never averaged together. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
AppWorld · Test-C · Scenario goal completion · 2026-10-26
Published model comparison, 2024. AppWorld · Test-C. The source cell lists TGC | SGC. This observation uses only Scenario goal completion; the two measures and test splits are never averaged together. Table 3; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

What it measures

Interactive world of 9 day-to-day apps (Gmail, Spotify, Venmo, Amazon, Splitwise, Todoist) and 457 APIs governed by 100 simulated users.

Reading the measurement

The useful outcome is a verified change or deliverable. Inspect the tool interface, initial state, user simulator and policy constraints. An agent can claim completion without reaching the required state, or reach it through an action the protocol forbids.

Published evaluation

The source cell lists TGC | SGC. This observation uses only Task goal completion; the two measures and test splits are never averaged together.

Where this fits in an evaluation suite

A transcript that says “done” is weaker evidence than the verified final state. Track policy violations, recoveries, and invalid attempts alongside success.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

The source cell lists TGC | SGC. This observation uses only Task goal completion; the two measures and test splits are never averaged together.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents — published result table ↗
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li et al. · 2024
All 9 authors
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, Niranjan Balasubramanian

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All agents & tool use benchmarks →

Sources

What it measures

Reading the measurement

Published evaluation

Where this fits in an evaluation suite

A reproducible comparison

Continue exploring

AA-Briefcase v1.1

GDPval-AA v2.1

APEX-Agents-AA

AA-AnalystAgent