Agents orchestrate workflows in simulated SaaS systems while respecting operational constraints.
34 model results
One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
91.8% of objectives completed overall, but 46.0% of workflows fully passed. 0.35 guardrail violations per task. #2 of 28 in this displayed comparison; 4.22 pp below the leader.
88.8% of objectives completed overall, but 38.1% of workflows fully passed. 0.40 guardrail violations per task. #3 of 28 in this displayed comparison; 4.99 pp below the leader.
91.0% of objectives completed overall, but 45.5% of workflows fully passed. 0.34 guardrail violations per task. #4 of 28 in this displayed comparison; 5.74 pp below the leader.
89.4% of objectives completed overall, but 35.2% of workflows fully passed. 0.47 guardrail violations per task. #5 of 28 in this displayed comparison; 9.97 pp below the leader.
81.3% of objectives completed overall, but 36.7% of workflows fully passed. 0.36 guardrail violations per task. #6 of 28 in this displayed comparison; 10.87 pp below the leader.
87.7% of objectives completed overall, but 37.7% of workflows fully passed. 0.64 guardrail violations per task. #7 of 28 in this displayed comparison; 11.96 pp below the leader.
88.4% of objectives completed overall, but 37.0% of workflows fully passed. 0.57 guardrail violations per task. #8 of 28 in this displayed comparison; 12.65 pp below the leader.
87.2% of objectives completed overall, but 29.4% of workflows fully passed. 0.55 guardrail violations per task. #9 of 28 in this displayed comparison; 15.31 pp below the leader.
85.9% of objectives completed overall, but 33.0% of workflows fully passed. 0.60 guardrail violations per task. #10 of 28 in this displayed comparison; 17.35 pp below the leader.
75.7% of objectives completed overall, but 32.1% of workflows fully passed. 0.39 guardrail violations per task. #11 of 28 in this displayed comparison; 17.58 pp below the leader.
84.8% of objectives completed overall, but 23.6% of workflows fully passed. 0.63 guardrail violations per task. #12 of 28 in this displayed comparison; 17.61 pp below the leader.
87.6% of objectives completed overall, but 25.4% of workflows fully passed. 0.74 guardrail violations per task. #13 of 28 in this displayed comparison; 18.89 pp below the leader.
Published observations span 25.93 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#14 of 28 in this displayed comparison; 19.15 pp below the leader.
Published breakdowns
89.3% of objectives completed overall, but 45.4% of workflows fully passed. 0.49 guardrail violations per task.
87.8% of objectives completed overall, but 26.6% of workflows fully passed. 0.81 guardrail violations per task. #15 of 28 in this displayed comparison; 21.32 pp below the leader.
Published observations span 28.34 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#16 of 28 in this displayed comparison; 22.14 pp below the leader.
Published breakdowns
90.4% of objectives completed overall, but 41.2% of workflows fully passed. 0.42 guardrail violations per task.
Published observations span 26.89 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#17 of 28 in this displayed comparison; 22.47 pp below the leader.
Published breakdowns
88.8% of objectives completed overall, but 41.6% of workflows fully passed. 0.44 guardrail violations per task.
Published observations span 28.99 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#18 of 28 in this displayed comparison; 23.92 pp below the leader.
Published breakdowns
87.6% of objectives completed overall, but 39.1% of workflows fully passed. 0.46 guardrail violations per task.
82.0% of objectives completed overall, but 20.2% of workflows fully passed. 0.82 guardrail violations per task. #19 of 28 in this displayed comparison; 24.35 pp below the leader.
78.5% of objectives completed overall, but 17.7% of workflows fully passed. 0.77 guardrail violations per task. #20 of 28 in this displayed comparison; 26.56 pp below the leader.
76.3% of objectives completed overall, but 20.9% of workflows fully passed. 0.91 guardrail violations per task. #21 of 28 in this displayed comparison; 27.81 pp below the leader.
77.9% of objectives completed overall, but 17.0% of workflows fully passed. 1.12 guardrail violations per task. #22 of 28 in this displayed comparison; 29.06 pp below the leader.
79.9% of objectives completed overall, but 15.7% of workflows fully passed. 1.03 guardrail violations per task. #23 of 28 in this displayed comparison; 29.27 pp below the leader.
Published observations span 27.28 percentage points. Configuration and source differences contribute to this spread; it is not measured run-to-run uncertainty.
#24 of 28 in this displayed comparison; 31.78 pp below the leader.
Published breakdowns
88.1% of objectives completed overall, but 32.1% of workflows fully passed. 0.71 guardrail violations per task.
66.0% of objectives completed overall, but 9.7% of workflows fully passed. 1.41 guardrail violations per task. #25 of 28 in this displayed comparison; 40.31 pp below the leader.
75.6% of objectives completed overall, but 13.9% of workflows fully passed. 0.84 guardrail violations per task. #26 of 28 in this displayed comparison; 42.1 pp below the leader.
40.6% of objectives completed overall, but 2.0% of workflows fully passed. 2.93 guardrail violations per task. #27 of 28 in this displayed comparison; 62.7 pp below the leader.
5.8% of objectives completed overall, but 0.3% of workflows fully passed. 0.22 guardrail violations per task. #28 of 28 in this displayed comparison; 72.53 pp below the leader.
Publisher-selected recent-model cohort from the public evaluation page. Model labels preserve effort and fallback configurations. Values are a dated snapshot, not a live feed. Consult the publisher’s methodology for the harness, task split and judging; do not merge this cohort with vendor-run results.
Source-reported model settings are max unless a cell says otherwise. The original page reports numbers without a uniform percent symbol; original formatting is retained. OSWorld includes two scoring rules in each cell: leaderboard ordering uses partial credit only. The internal composite and AA index use separate scales; their scores cannot be averaged.
AutomationBench-AA · Automation Bench-AA · Kimi K3.1 Pro · complete release comparison · 2026-10-27
Published model comparison, 2026. Kimi K3.1 Pro · complete release comparison. Exact values from the publisher’s embedded full comparison table. Maximum effort, fallback settings and dated model versions remain in model labels. Blank cells stay missing. Source footnotes specify mixed harnesses and externally quoted comparisons; this is a launch-reported cohort rather than a new independent evaluation. Benchmarks · full embedded comparison table · row 22; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
TRACE THE SCORE · Per-model notes use published breakdowns where available.
Reported in recent model launches
These model reports include this benchmark. Their harnesses, task subsets, scoring policies, and resource budgets can differ; follow each report’s evaluation notes before joining the numbers.
Agents orchestrate workflows in simulated SaaS systems while respecting operational constraints.
How to read the result
Measure both the desired final state and policy adherence. Finishing an objective through a forbidden action should not earn success.
Limits and comparison traps
API discovery, tool schemas, and guardrail checks are part of the evaluation contract.
Published evaluation
Exact values from the publisher’s embedded full comparison table. Maximum effort, fallback settings and dated model versions remain in model labels. Blank cells stay missing. Source footnotes specify mixed harnesses and externally quoted comparisons; this is a launch-reported cohort rather than a new independent evaluation.
Where this fits in an evaluation suite
A transcript that says “done” is weaker evidence than the verified final state. Track policy violations, recoveries, and invalid attempts alongside success.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Exact values from the publisher’s embedded full comparison table. Maximum effort, fallback settings and dated model versions remain in model labels. Blank cells stay missing. Source footnotes specify mixed harnesses and externally quoted comparisons; this is a launch-reported cohort rather than a new independent evaluation.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.