Aider coding benchmarks

Programming and refactoring tasks test edits to real code. Polyglot and refactoring results retain their own task sets and attempt budgets.26 Oct 2026

One row per model and compatible score measure. Mean = equal-weight arithmetic average of imported reports. Distinct configurations reported for the same model in a source table remain separate. Reports can overlap; the range is not a confidence interval. Unbundle to inspect each observation.
		Evidence	What the result shows
GPT-5 →OpenAIAider · two-attempt programming tasks	88% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks	Leads this displayed comparison (tied). Source notes Source cell: 88.0%
GPT-5 →OpenAIgpt-5-2025-08-07_high	88% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gpt-5-2025-08-07_highReleased 2025-08-07	Leads this displayed comparison (tied).
GPT-5 →OpenAIAider · two-attempt programming tasks · Dirname : 2025-08-23-15-47-21--gpt-5-high Test cases : 225 Model : gpt-5 (high) Edit format : diff Commit hash : 32faf82 Reasoning effort : high Pass rate 1 : 52.0 Pass rate 2 : 88.0 Pass num 1 : 117 Pass num 2 : 198 Percent cases well formed : 91.6 Error outputs : 23 Num malformed responses : 22 Num with malformed responses : 19 User asks : 96 Lazy comments : 3 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2675561 Completion tokens : 2623429 Test timeouts : 3 Total tests : 225 Command : aider --model openai/gpt-5 Date : 2025-08-23 Versions : 0.86.2.dev Seconds per case : 194.0 Total cost : 29.0829	86.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-08-23-15-47-21--gpt-5-high Test cases : 225 Model : gpt-5 (high) Edit format : diff Commit hash : 32faf82 Reasoning effort : high Pass rate 1 : 52.0 Pass rate 2 : 88.0 Pass num 1 : 117 Pass num 2 : 198 Percent cases well formed : 91.6 Error outputs : 23 Num malformed responses : 22 Num with malformed responses : 19 User asks : 96 Lazy comments : 3 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2675561 Completion tokens : 2623429 Test timeouts : 3 Total tests : 225 Command : aider --model openai/gpt-5 Date : 2025-08-23 Versions : 0.86.2.dev Seconds per case : 194.0 Total cost : 29.0829	#3 of 130 in this displayed comparison; 1.3 pp below the leader.
GPT-5 →OpenAIgpt-5-2025-08-07_medium	86.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gpt-5-2025-08-07_mediumReleased 2025-08-07	#3 of 130 in this displayed comparison; 1.3 pp below the leader.
o3-pro →OpenAI	84.9% 2 reports	Aider coding benchmarks · Polyglot · percent correct[3][2] View 2 reports 84.9% · Aider coding benchmarks · Polyglot · percent correct · o3-pro-2025-06-10_high [3] 84.9% · Aider coding benchmarks · Polyglot · percent correct · Aider · two-attempt programming tasks · Dirname : 2025-08-25-13-23-27--gpt-5-medium Test cases : 225 Model : gpt-5 (medium) Edit format : diff Commit hash : 32faf82 Reasoning effort : medium Pass rate 1 : 49.8 Pass rate 2 : 86.7 Pass num 1 : 112 Pass num 2 : 195 Percent cases well formed : 88.4 Error outputs : 40 Num malformed responses : 40 Num with malformed responses : 26 User asks : 102 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2827261 Completion tokens : 1468799 Test timeouts : 0 Total tests : 225 Command : aider --model openai/gpt-5 Date : 2025-08-25 Versions : 0.86.2.dev Seconds per case : 118.7 Total cost : 17.693 [2]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #5 of 130 in this displayed comparison; 3.1 pp below the leader.
Gemini 2.5 Pro Preview (Jun 2025) →Googlegemini-2.5-pro-preview-06-05_32K	83.1% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gemini-2.5-pro-preview-06-05_32KReleased 2025-06-05	#6 of 130 in this displayed comparison; 4.9 pp below the leader.
gemini-2.5-pro-preview-06-05 (32k think) →Google	83.1% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-28-00-38-18--o3-pro-high Test cases : 225 Model : o3-pro (high) Edit format : diff Commit hash : 5318380 Reasoning effort : high Pass rate 1 : 43.6 Pass rate 2 : 84.9 Pass num 1 : 98 Pass num 2 : 191 Percent cases well formed : 97.8 Error outputs : 20 Num malformed responses : 8 Num with malformed responses : 5 User asks : 100 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2372636 Completion tokens : 1235902 Test timeouts : 1 Total tests : 225 Command : aider --model o3-pro Date : 2025-06-28 Versions : 0.85.1.dev Seconds per case : 449.0 Total cost : 146.3249	#6 of 130 in this displayed comparison; 4.9 pp below the leader.
GPT-5 →OpenAIAider · two-attempt programming tasks · Dirname : 2025-06-06-16-36-21--gemini0605-32k-think-diff-fenced Test cases : 225 Model : gemini-2.5-pro-preview-06-05 (32k think) Edit format : diff-fenced Commit hash : f827f22 Thinking tokens : 32768 Pass rate 1 : 46.2 Pass rate 2 : 83.1 Pass num 1 : 104 Pass num 2 : 187 Percent cases well formed : 99.6 Error outputs : 1 Num malformed responses : 1 Num with malformed responses : 1 User asks : 112 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2719961 Completion tokens : 4648227 Test timeouts : 0 Total tests : 225 Command : aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k Date : 2025-06-06 Versions : 0.84.1.dev Seconds per case : 200.3 Total cost : 49.8822	81.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-06-16-36-21--gemini0605-32k-think-diff-fenced Test cases : 225 Model : gemini-2.5-pro-preview-06-05 (32k think) Edit format : diff-fenced Commit hash : f827f22 Thinking tokens : 32768 Pass rate 1 : 46.2 Pass rate 2 : 83.1 Pass num 1 : 104 Pass num 2 : 187 Percent cases well formed : 99.6 Error outputs : 1 Num malformed responses : 1 Num with malformed responses : 1 User asks : 112 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2719961 Completion tokens : 4648227 Test timeouts : 0 Total tests : 225 Command : aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k Date : 2025-06-06 Versions : 0.84.1.dev Seconds per case : 200.3 Total cost : 49.8822	#8 of 130 in this displayed comparison; 6.7 pp below the leader.
GPT-5 →OpenAIgpt-5-2025-08-07_low	81.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gpt-5-2025-08-07_lowReleased 2025-08-07	#8 of 130 in this displayed comparison; 6.7 pp below the leader.
o3 →OpenAIo3-2025-04-16_high	81.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]o3-2025-04-16_highReleased 2025-04-16	#8 of 130 in this displayed comparison; 6.7 pp below the leader.
o3 →OpenAIAider · two-attempt programming tasks · Dirname : 2025-08-25-14-16-37--gpt-5-low Test cases : 225 Model : gpt-5 (low) Edit format : diff Commit hash : 32faf82 Reasoning effort : low Pass rate 1 : 43.1 Pass rate 2 : 81.3 Pass num 1 : 97 Pass num 2 : 183 Percent cases well formed : 86.7 Error outputs : 46 Num malformed responses : 46 Num with malformed responses : 30 User asks : 113 Lazy comments : 1 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2534059 Completion tokens : 779568 Test timeouts : 1 Total tests : 225 Command : aider --model openai/gpt-5 Date : 2025-08-25 Versions : 0.86.2.dev Seconds per case : 62.4 Total cost : 10.3713	81.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-08-25-14-16-37--gpt-5-low Test cases : 225 Model : gpt-5 (low) Edit format : diff Commit hash : 32faf82 Reasoning effort : low Pass rate 1 : 43.1 Pass rate 2 : 81.3 Pass num 1 : 97 Pass num 2 : 183 Percent cases well formed : 86.7 Error outputs : 46 Num malformed responses : 46 Num with malformed responses : 30 User asks : 113 Lazy comments : 1 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2534059 Completion tokens : 779568 Test timeouts : 1 Total tests : 225 Command : aider --model openai/gpt-5 Date : 2025-08-25 Versions : 0.86.2.dev Seconds per case : 62.4 Total cost : 10.3713	#8 of 130 in this displayed comparison; 6.7 pp below the leader.
Grok 4 →xAI	79.6% 2 reports	Aider coding benchmarks · Polyglot · percent correct[2][3] View 2 reports 79.6% · Aider coding benchmarks · Polyglot · percent correct · grok-4-0709_high [2] 79.6% · Aider coding benchmarks · Polyglot · percent correct · grok-4-0709 [3]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #12 of 130 in this displayed comparison; 8.4 pp below the leader.
grok-4 →xAI	79.6% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-25-21-04-24--o3-price-reduction-high Test cases : 225 Model : o3 (high) Edit format : diff Commit hash : c48fea6 Reasoning effort : high Pass rate 1 : 40.0 Pass rate 2 : 81.3 Pass num 1 : 90 Pass num 2 : 183 Percent cases well formed : 94.7 Error outputs : 25 Num malformed responses : 23 Num with malformed responses : 12 User asks : 116 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 3148932 Completion tokens : 2047615 Test timeouts : 2 Total tests : 225 Command : aider --model o3 --reasoning-effort high Date : 2025-06-25 Versions : 0.84.1.dev Seconds per case : 197.3 Total cost : 21.2259	#12 of 130 in this displayed comparison; 8.4 pp below the leader.
Gemini 2.5 Pro Preview (Jun 2025) →Googlegemini-2.5-pro-preview-06-05	79.1% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gemini-2.5-pro-preview-06-05Released 2025-06-05	#14 of 130 in this displayed comparison; 8.9 pp below the leader. Source notes No thinking parameter set; default thinking token length
gemini-2.5-pro-preview-06-05 (default think) →Google	79.1% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-07-11-19-37-40--xai-or-grok4-high Test cases : 225 Model : grok-4 (high) Edit format : diff Commit hash : f7870b6-dirty Reasoning effort : high Pass rate 1 : 40.9 Pass rate 2 : 79.6 Pass num 1 : 92 Pass num 2 : 179 Percent cases well formed : 97.3 Error outputs : 11 Num malformed responses : 8 Num with malformed responses : 6 User asks : 133 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2815347 Completion tokens : 3411480 Test timeouts : 0 Total tests : 225 Command : aider --model openrouter/x-ai/grok-4 Date : 2025-07-11 Versions : 0.85.2.dev Seconds per case : 403.2 Total cost : 59.6182	#14 of 130 in this displayed comparison; 8.9 pp below the leader.
o3 (high) + gpt-4.1 →OpenAI	78.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-06-18-38-56--gemini0605-diff-fenced Test cases : 225 Model : gemini-2.5-pro-preview-06-05 (default think) Edit format : diff-fenced Commit hash : 4c161f9-dirty Pass rate 1 : 44.9 Pass rate 2 : 79.1 Pass num 1 : 101 Pass num 2 : 178 Percent cases well formed : 100.0 Error outputs : 4 Num malformed responses : 0 Num with malformed responses : 0 User asks : 105 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 4 Prompt tokens : 2751296 Completion tokens : 4142197 Test timeouts : 1 Total tests : 225 Command : aider --model gemini/gemini-2.5-pro-preview-06-05 Date : 2025-06-06 Versions : 0.84.1.dev Seconds per case : 175.2 Total cost : 45.5961	#16 of 130 in this displayed comparison; 9.8 pp below the leader.
Gemini 2.5 Pro Preview (Jun 2025) →Googlegemini-2.5-pro-preview-05-06	76.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gemini-2.5-pro-preview-05-06Released 2025-05-06	#17 of 130 in this displayed comparison; 11.1 pp below the leader.
Gemini 2.5 Pro Preview 05-06 →Google	76.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-25-20-30-16--o3-price-reduction Test cases : 225 Model : o3 Edit format : diff Commit hash : c48fea6 Pass rate 1 : 40.9 Pass rate 2 : 76.9 Pass num 1 : 92 Pass num 2 : 173 Percent cases well formed : 93.8 Error outputs : 22 Num malformed responses : 22 Num with malformed responses : 14 User asks : 108 Lazy comments : 2 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2893189 Completion tokens : 1154767 Test timeouts : 1 Total tests : 225 Command : aider --model o3 Date : 2025-06-25 Versions : 0.84.1.dev Seconds per case : 101.7 Total cost : 13.7517	#17 of 130 in this displayed comparison; 11.1 pp below the leader.
o3 →OpenAIAider · two-attempt programming tasks · Dirname : 2025-06-27-23-53-57--o3-mini-high-diff-arch Test cases : 224 Model : o3 (high) + gpt-4.1 Edit format : architect Commit hash : 4f4f00f-dirty Editor model : gpt-4.1 Editor edit format : editor-diff Reasoning effort : high Pass rate 1 : 34.8 Pass rate 2 : 78.2 Pass num 1 : 78 Pass num 2 : 176 Percent cases well formed : 100.0 Error outputs : 18 Num malformed responses : 0 Num with malformed responses : 0 User asks : 172 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 1306877 Completion tokens : 1327154 Test timeouts : 1 Total tests : 225 Command : aider --model o3 Date : 2025-06-27 Versions : 0.85.1.dev Seconds per case : 121.8 Total cost : 17.5518	76.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-27-23-53-57--o3-mini-high-diff-arch Test cases : 224 Model : o3 (high) + gpt-4.1 Edit format : architect Commit hash : 4f4f00f-dirty Editor model : gpt-4.1 Editor edit format : editor-diff Reasoning effort : high Pass rate 1 : 34.8 Pass rate 2 : 78.2 Pass num 1 : 78 Pass num 2 : 176 Percent cases well formed : 100.0 Error outputs : 18 Num malformed responses : 0 Num with malformed responses : 0 User asks : 172 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 1306877 Completion tokens : 1327154 Test timeouts : 1 Total tests : 225 Command : aider --model o3 Date : 2025-06-27 Versions : 0.85.1.dev Seconds per case : 121.8 Total cost : 17.5518	#17 of 130 in this displayed comparison; 11.1 pp below the leader.
o3 →OpenAIo3-2025-04-16_medium	76.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]o3-2025-04-16_mediumReleased 2025-04-16	#17 of 130 in this displayed comparison; 11.1 pp below the leader.
o3 →OpenAI	76.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]o3-2025-04-16_unknownReleased 2025-04-16	#17 of 130 in this displayed comparison; 11.1 pp below the leader.
DeepSeek V3.2 →DeepSeekdeepseek-reasoner	74.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]deepseek-reasonerReleased 2025-12-01	#22 of 130 in this displayed comparison; 13.8 pp below the leader.
DeepSeek V3.2 Exp →DeepSeekDeepSeek-V3.2-Exp_thinking	74.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]DeepSeek-V3.2-Exp_thinkingReleased 2025-09-29	#22 of 130 in this displayed comparison; 13.8 pp below the leader.
DeepSeek V3.2 Exp (Reasoner) →DeepSeek	74.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-05-07-19-32-40--gemini0506-diff-fenced-completion_cost Test cases : 225 Model : Gemini 2.5 Pro Preview 05-06 Edit format : diff-fenced Commit hash : 3b08327-dirty Pass rate 1 : 36.4 Pass rate 2 : 76.9 Pass num 1 : 82 Pass num 2 : 173 Percent cases well formed : 97.3 Error outputs : 15 Num malformed responses : 7 Num with malformed responses : 6 User asks : 105 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 2 Total tests : 225 Command : aider --model gemini/gemini-2.5-pro-preview-05-06 Date : 2025-05-07 Versions : 0.82.4.dev Seconds per case : 165.3 Total cost : 37.4104	#22 of 130 in this displayed comparison; 13.8 pp below the leader.
Gemini 2.5 Pro Exp (Mar 2025) →Google	72.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]gemini-2.5-pro-exp-03-25Released 2025-03-25	#25 of 130 in this displayed comparison; 15.1 pp below the leader.
Gemini 2.5 Pro Preview (Mar 2025) →Google	72.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]gemini-2.5-pro-preview-03-25Released 2025-03-31	#25 of 130 in this displayed comparison; 15.1 pp below the leader.
Gemini 2.5 Pro Preview 03-25 →Google	72.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-10-03-09-45-34--deepseek-v3.2-reasoner Test cases : 225 Model : DeepSeek-V3.2-Exp (Reasoner) Edit format : diff Commit hash : cbb5376 Pass rate 1 : 39.6 Pass rate 2 : 74.2 Pass num 1 : 89 Pass num 2 : 167 Percent cases well formed : 97.3 Error outputs : 8 Num malformed responses : 6 Num with malformed responses : 6 User asks : 67 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 2191446 Completion tokens : 1645129 Test timeouts : 1 Total tests : 225 Command : aider --model deepseek/deepseek-reasoner Date : 2025-10-03 Versions : 0.86.2.dev Seconds per case : 291.2 Total cost : 1.3045	#25 of 130 in this displayed comparison; 15.1 pp below the leader.
Claude Opus 4 →Anthropicclaude-opus-4-20250514_32K	72% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]claude-opus-4-20250514_32KReleased 2025-05-22	#28 of 130 in this displayed comparison; 16 pp below the leader.
Claude Opus 4-20250514 →Anthropic	72% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-04-12-04-55-50--gemini-25-pro-diff-fenced Test cases : 225 Model : Gemini 2.5 Pro Preview 03-25 Edit format : diff-fenced Commit hash : 0282574 Pass rate 1 : 40.9 Pass rate 2 : 72.9 Pass num 1 : 92 Pass num 2 : 164 Percent cases well formed : 92.4 Error outputs : 21 Num malformed responses : 21 Num with malformed responses : 17 User asks : 69 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 2 Total tests : 225 Command : aider --model gemini/gemini-2.5-pro-preview-03-25 Date : 2025-04-12 Versions : 0.81.3.dev Seconds per case : 45.3 Total cost : 0	#28 of 130 in this displayed comparison; 16 pp below the leader. Source notes Source cell: 72.0%
o4-mini →OpenAI	72% 2 reports	Aider coding benchmarks · Polyglot · percent correct[3][2] View 2 reports 72% · Aider coding benchmarks · Polyglot · percent correct · o4-mini-2025-04-16_high [3] 72% · Aider coding benchmarks · Polyglot · percent correct · Aider · two-attempt programming tasks · Dirname : 2025-05-25-20-40-51--opus4-diff-exuser Test cases : 225 Model : claude-opus-4-20250514 (32k thinking) Edit format : diff Commit hash : 9ef3211 Thinking tokens : 32000 Pass rate 1 : 37.3 Pass rate 2 : 72.0 Pass num 1 : 84 Pass num 2 : 162 Percent cases well formed : 97.3 Error outputs : 10 Num malformed responses : 6 Num with malformed responses : 6 User asks : 97 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2567514 Completion tokens : 363142 Test timeouts : 4 Total tests : 225 Command : aider --model claude-opus-4-20250514 Date : 2025-05-25 Versions : 0.83.3.dev Seconds per case : 44.1 Total cost : 65.7484 [2]	The imported reports agree at the published precision. Repeated citations may describe the same underlying evaluation. #28 of 130 in this displayed comparison; 16 pp below the leader.
DeepSeek R1 (0528) →DeepSeek	71.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-04-16-22-01-58--o4-mini-high-diff-exsys Test cases : 225 Model : o4-mini (high) Edit format : diff Commit hash : b66901f-dirty Pass rate 1 : 19.6 Pass rate 2 : 72.0 Pass num 1 : 44 Pass num 2 : 162 Percent cases well formed : 90.7 Error outputs : 26 Num malformed responses : 24 Num with malformed responses : 21 User asks : 66 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Test timeouts : 2 Total tests : 225 Command : aider --model o4-mini Date : 2025-04-16 Versions : 0.82.1.dev Seconds per case : 176.5 Total cost : 19.6399	#31 of 130 in this displayed comparison; 16.6 pp below the leader.
DeepSeek-R1 (May 2025) →DeepSeek	71.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]DeepSeek-R1-0528Released 2025-05-28	#31 of 130 in this displayed comparison; 16.6 pp below the leader.
Claude Opus 4 →Anthropicclaude-opus-4-20250514	70.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]claude-opus-4-20250514Released 2025-05-22	#33 of 130 in this displayed comparison; 17.3 pp below the leader.
Claude Opus 4-20250514 (no think) →Anthropic	70.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-06-06-16-47-07--r1-diff Test cases : 224 Model : DeepSeek R1 (0528) Edit format : diff Commit hash : 4c161f9-dirty Pass rate 1 : 34.4 Pass rate 2 : 71.4 Pass num 1 : 77 Pass num 2 : 160 Percent cases well formed : 94.6 Error outputs : 28 Num malformed responses : 15 Num with malformed responses : 12 User asks : 105 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2644169 Completion tokens : 1842168 Test timeouts : 2 Total tests : 225 Command : aider --model deepseek/deepseek-reasoner Date : 2025-06-06 Versions : 0.84.1.dev Seconds per case : 716.6 Total cost : 4.8016	#33 of 130 in this displayed comparison; 17.3 pp below the leader.
DeepSeek V3.2 →DeepSeekdeepseek-chat	70.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]deepseek-chatReleased 2025-12-01	#35 of 130 in this displayed comparison; 17.8 pp below the leader.
DeepSeek V3.2 Exp →DeepSeekDeepSeek-V3.2-Exp	70.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]DeepSeek-V3.2-ExpReleased 2025-09-29	#35 of 130 in this displayed comparison; 17.8 pp below the leader.
DeepSeek V3.2 Exp (Chat) →DeepSeek	70.2% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-05-25-19-57-20--opus4-diff-exuser Test cases : 225 Model : claude-opus-4-20250514 (no think) Edit format : diff Commit hash : 9ef3211 Pass rate 1 : 32.9 Pass rate 2 : 70.7 Pass num 1 : 74 Pass num 2 : 159 Percent cases well formed : 98.7 Error outputs : 3 Num malformed responses : 3 Num with malformed responses : 3 User asks : 105 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2671437 Completion tokens : 380717 Test timeouts : 3 Total tests : 225 Command : aider --model claude-opus-4-20250514 Date : 2025-05-25 Versions : 0.83.3.dev Seconds per case : 42.5 Total cost : 68.6253	#35 of 130 in this displayed comparison; 17.8 pp below the leader.
Claude 3.7 Sonnet →Anthropicclaude-3-7-sonnet-20250219_32K	64.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]claude-3-7-sonnet-20250219_32KReleased 2025-02-24	#38 of 130 in this displayed comparison; 23.1 pp below the leader.
claude-3-7-sonnet-20250219 →AnthropicAider · two-attempt programming tasks · Dirname : 2025-10-03-09-21-36--deepseek-v3.2-chat Test cases : 225 Model : DeepSeek-V3.2-Exp (Chat) Edit format : diff Commit hash : cbb5376 Pass rate 1 : 38.7 Pass rate 2 : 70.2 Pass num 1 : 87 Pass num 2 : 158 Percent cases well formed : 98.2 Error outputs : 6 Num malformed responses : 4 Num with malformed responses : 4 User asks : 60 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 2266868 Completion tokens : 573477 Test timeouts : 4 Total tests : 225 Command : aider --model deepseek/deepseek-chat Date : 2025-10-03 Versions : 0.86.2.dev Seconds per case : 104.0 Total cost : 0.8756	64.9% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-10-03-09-21-36--deepseek-v3.2-chat Test cases : 225 Model : DeepSeek-V3.2-Exp (Chat) Edit format : diff Commit hash : cbb5376 Pass rate 1 : 38.7 Pass rate 2 : 70.2 Pass num 1 : 87 Pass num 2 : 158 Percent cases well formed : 98.2 Error outputs : 6 Num malformed responses : 4 Num with malformed responses : 4 User asks : 60 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Prompt tokens : 2266868 Completion tokens : 573477 Test timeouts : 4 Total tests : 225 Command : aider --model deepseek/deepseek-chat Date : 2025-10-03 Versions : 0.86.2.dev Seconds per case : 104.0 Total cost : 0.8756	#38 of 130 in this displayed comparison; 23.1 pp below the leader.
DeepSeek R1 + claude-3-5-sonnet-20241022 →Anthropic	64% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-02-24-21-47-23--sonnet37-diff-think-32k-64k Test cases : 225 Model : claude-3-7-sonnet-20250219 (32k thinking tokens) Edit format : diff Commit hash : 60d11a6, 93edbda Pass rate 1 : 29.3 Pass rate 2 : 64.9 Pass num 1 : 66 Pass num 2 : 146 Percent cases well formed : 97.8 Error outputs : 66 Num malformed responses : 5 Num with malformed responses : 5 User asks : 5 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 1 Total tests : 225 Command : aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k Date : 2025-02-24 Versions : 0.75.1.dev Seconds per case : 105.2 Total cost : 36.8343	#40 of 130 in this displayed comparison; 24 pp below the leader. Source notes Source cell: 64.0%
o1 →OpenAI	61.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]o1-2024-12-17_highReleased 2024-12-17	#41 of 130 in this displayed comparison; 26.3 pp below the leader.
o1-2024-12-17 →OpenAI	61.7% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-01-23-19-14-48--r1-architect-sonnet Test cases : 225 Model : DeepSeek R1 + claude-3-5-sonnet-20241022 Edit format : architect Commit hash : 05a77c7 Editor model : claude-3-5-sonnet-20241022 Editor edit format : editor-diff Pass rate 1 : 27.1 Pass rate 2 : 64.0 Pass num 1 : 61 Pass num 2 : 144 Percent cases well formed : 100.0 Error outputs : 2 Num malformed responses : 0 Num with malformed responses : 0 User asks : 392 Lazy comments : 6 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 5 Total tests : 225 Command : aider --architect --model r1 --editor-model sonnet Date : 2025-01-23 Versions : 0.72.3.dev Seconds per case : 251.6 Total cost : 13.2933	#41 of 130 in this displayed comparison; 26.3 pp below the leader.
Claude Sonnet 4 →Anthropicclaude-sonnet-4-20250514_32K	61.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]claude-sonnet-4-20250514_32KReleased 2025-05-22	#43 of 130 in this displayed comparison; 26.7 pp below the leader.
Claude Sonnet 4-20250514 →AnthropicAider · two-attempt programming tasks · Dirname : 2024-12-21-19-23-03--polyglot-o1-hard-diff Test cases : 224 Model : o1-2024-12-17 (high) Edit format : diff Commit hash : a755079-dirty Pass rate 1 : 23.7 Pass rate 2 : 61.7 Pass num 1 : 53 Pass num 2 : 139 Percent cases well formed : 91.5 Error outputs : 25 Num malformed responses : 24 Num with malformed responses : 19 User asks : 16 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 2 Total tests : 225 Command : aider --model openrouter/openai/o1 Date : 2024-12-21 Versions : 0.69.2.dev Seconds per case : 133.2 Total cost : 186.4958	61.3% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2024-12-21-19-23-03--polyglot-o1-hard-diff Test cases : 224 Model : o1-2024-12-17 (high) Edit format : diff Commit hash : a755079-dirty Pass rate 1 : 23.7 Pass rate 2 : 61.7 Pass num 1 : 53 Pass num 2 : 139 Percent cases well formed : 91.5 Error outputs : 25 Num malformed responses : 24 Num with malformed responses : 19 User asks : 16 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 2 Total tests : 225 Command : aider --model openrouter/openai/o1 Date : 2024-12-21 Versions : 0.69.2.dev Seconds per case : 133.2 Total cost : 186.4958	#43 of 130 in this displayed comparison; 26.7 pp below the leader.
Claude 3.7 Sonnet →Anthropicclaude-3-7-sonnet-20250219	60.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]claude-3-7-sonnet-20250219Released 2025-02-24	#45 of 130 in this displayed comparison; 27.6 pp below the leader.
claude-3-7-sonnet-20250219 →AnthropicAider · two-attempt programming tasks · Dirname : 2025-05-24-22-10-36--sonnet4-diff-exuser-think32k Test cases : 225 Model : claude-sonnet-4-20250514 (32k thinking) Edit format : diff Commit hash : e3cb907 Thinking tokens : 32000 Pass rate 1 : 25.8 Pass rate 2 : 61.3 Pass num 1 : 58 Pass num 2 : 138 Percent cases well formed : 97.3 Error outputs : 10 Num malformed responses : 10 Num with malformed responses : 6 User asks : 111 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2863068 Completion tokens : 1271074 Test timeouts : 6 Total tests : 225 Command : aider --model claude-sonnet-4-20250514 Date : 2025-05-24 Versions : 0.83.3.dev Seconds per case : 79.9 Total cost : 26.5755	60.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-05-24-22-10-36--sonnet4-diff-exuser-think32k Test cases : 225 Model : claude-sonnet-4-20250514 (32k thinking) Edit format : diff Commit hash : e3cb907 Thinking tokens : 32000 Pass rate 1 : 25.8 Pass rate 2 : 61.3 Pass num 1 : 58 Pass num 2 : 138 Percent cases well formed : 97.3 Error outputs : 10 Num malformed responses : 10 Num with malformed responses : 6 User asks : 111 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Prompt tokens : 2863068 Completion tokens : 1271074 Test timeouts : 6 Total tests : 225 Command : aider --model claude-sonnet-4-20250514 Date : 2025-05-24 Versions : 0.83.3.dev Seconds per case : 79.9 Total cost : 26.5755	#45 of 130 in this displayed comparison; 27.6 pp below the leader.
o3-mini →OpenAIo3-mini-2025-01-31_high	60.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[3]o3-mini-2025-01-31_highReleased 2025-01-31	#45 of 130 in this displayed comparison; 27.6 pp below the leader.
o3-mini →OpenAIAider · two-attempt programming tasks · Dirname : 2025-02-24-19-54-07--sonnet37-diff Test cases : 225 Model : claude-3-7-sonnet-20250219 (no thinking) Edit format : diff Commit hash : 75e9ee6 Pass rate 1 : 24.4 Pass rate 2 : 60.4 Pass num 1 : 55 Pass num 2 : 136 Percent cases well formed : 93.3 Error outputs : 16 Num malformed responses : 16 Num with malformed responses : 15 User asks : 12 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 0 Total tests : 225 Command : aider --model sonnet Date : 2025-02-24 Versions : 0.74.4.dev Seconds per case : 28.3 Total cost : 17.7191	60.4% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-02-24-19-54-07--sonnet37-diff Test cases : 225 Model : claude-3-7-sonnet-20250219 (no thinking) Edit format : diff Commit hash : 75e9ee6 Pass rate 1 : 24.4 Pass rate 2 : 60.4 Pass num 1 : 55 Pass num 2 : 136 Percent cases well formed : 93.3 Error outputs : 16 Num malformed responses : 16 Num with malformed responses : 15 User asks : 12 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 0 Test timeouts : 0 Total tests : 225 Command : aider --model sonnet Date : 2025-02-24 Versions : 0.74.4.dev Seconds per case : 28.3 Total cost : 17.7191	#45 of 130 in this displayed comparison; 27.6 pp below the leader.
Qwen3 235B A22B diff, no think, Alibaba API →Qwen	59.6% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Aider · two-attempt programming tasks · Dirname : 2025-01-31-20-42-47--o3-mini-diff-high Test cases : 224 Model : o3-mini (high) Edit format : diff Commit hash : b0d58d1-dirty Pass rate 1 : 21.0 Pass rate 2 : 60.4 Pass num 1 : 47 Pass num 2 : 136 Percent cases well formed : 93.3 Error outputs : 26 Num malformed responses : 24 Num with malformed responses : 15 User asks : 19 Lazy comments : 0 Syntax errors : 0 Indentation errors : 0 Exhausted context windows : 1 Test timeouts : 7 Total tests : 225 Command : aider --model o3-mini --reasoning-effort high Date : 2025-01-31 Versions : 0.72.4.dev Seconds per case : 124.6 Total cost : 18.1584	#49 of 130 in this displayed comparison; 28.4 pp below the leader.
Qwen3-235B-A22B →Qwen	59.6% 1 report	Aider coding benchmarks · Polyglot · percent correct[2]Qwen3-235B-A22BReleased 2025-04-28	#49 of 130 in this displayed comparison; 28.4 pp below the leader. Source notes no cost data

Sources

Aider Polyglot ↗
Epoch AI
Dates & evaluation details
Aider coding benchmarks · Polyglot · percent correct · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: aider_polyglot_external.csv; score column: Percent correct; fraction scale: 0.01. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Aider code editing benchmark ↗
Paul Gauthier, Aider contributors
Dates & evaluation details
Aider coding benchmarks · Polyglot · percent correct · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: aider_polyglot_external.csv; score column: Percent correct; fraction scale: 0.01. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Aider coding benchmarks · Polyglot · percent correct · 2026-10-26
Published model comparison. Aider · two-attempt programming tasks. Polyglot · percent correct; Aider · two-attempt programming tasks. Results retain their published native scale and task setting. Published result table; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.
Aider LLM Leaderboards — Polyglot coding benchmark ↗
Aider
Dates & evaluation details
Aider coding benchmarks · Polyglot · percent correct · 2026-10-26
Epoch AI public data snapshot, CC BY 4.0. Externally collected result; original evaluation provider is linked. CSV: aider_polyglot_external.csv; score column: Percent correct; fraction scale: 0.01. Model version is retained in configuration. Original source authors retain credit and any additional licensing terms.
Aider refactoring benchmark ↗
Paul Gauthier, Aider contributors
Dates & evaluation details
Aider coding benchmarks · Refactoring · percent correct · 2026-10-26
Published model comparison. Aider · refactoring leaderboard. Refactoring · percent correct; Aider · refactoring leaderboard. Results retain their published native scale and task setting. Published result table; source precision and any reported uncertainty retained. These are source-reported observations, not evaluations run by RLTasks.

TRACE THE SCORE · Per-model notes use published breakdowns where available.

Score vs. release date

Epoch AI · 72 observations

Aider coding benchmarks · Polyglot · percent correct · higher is better

Dates are model release dates recorded by Epoch AI, not evaluation dates. Each point retains its published score and configuration. Separate releases are not a matched longitudinal experiment. Missing dates are omitted. Explore all reported results ↓

The dashed line traces the best observed score through the model release dates. It describes this selection of reports, with their original configurations. It does not estimate unmeasured models or costs.

View plotted values and sources (72)

Aider coding benchmarks · Aider coding benchmarks · Polyglot · percent correct. Published measurements in their original score units.
Model	Score	Model release date	Configuration & source
gpt-oss-120b →	41.8%	2025-08-05	gpt-oss-120b_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Grok 4 →	79.6%	2025-07-09	grok-4-0709_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3 (unknown thinking) →	76.9%	2025-04-16	o3-2025-04-16_unknown Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3.2 (Thinking) →	74.2%	2025-12-01	deepseek-reasoner Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Pro Preview (Mar 2025) →	72.9%	2025-03-31	gemini-2.5-pro-preview-03-25 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3.2 →	70.2%	2025-12-01	deepseek-chat Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen3-235B-A22B →	59.6%	2025-04-28	Qwen3-235B-A22B Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen3-235B-A22B-Instruct (Jul 2025) →	59.6%	2025-07-25	Qwen3-235B-A22B-Instruct-2507 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Kimi K2 0905 (Novita) →	59.1%	2025-09-05	moonshotai/kimi-k2-0905 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
gpt-oss-120b →	41.8%	2025-08-05	openai/gpt-oss-120b_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o1-mini (unknown thinking) →	32.9%	2024-09-12	o1-mini-2024-09-12_unknown Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-5 (high) →	88%	2025-08-07	gpt-5-2025-08-07_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-5 (medium) →	86.7%	2025-08-07	gpt-5-2025-08-07_medium Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3-pro →	84.9%	2025-06-10	o3-pro-2025-06-10_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Pro Preview (Jun 2025) →	83.1%	2025-06-05	gemini-2.5-pro-preview-06-05_32K Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-5 (low) →	81.3%	2025-08-07	gpt-5-2025-08-07_low Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3 (high) →	81.3%	2025-04-16	o3-2025-04-16_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Grok 4 →	79.6%	2025-07-09	grok-4-0709 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Pro Preview (Jun 2025) →	79.1%	2025-06-05	gemini-2.5-pro-preview-06-05 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3 (medium) →	76.9%	2025-04-16	o3-2025-04-16_medium Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Pro Preview (Jun 2025) →	76.9%	2025-05-06	gemini-2.5-pro-preview-05-06 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3.2-Exp →	74.2%	2025-09-29	DeepSeek-V3.2-Exp_thinking Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Pro Exp (Mar 2025) →	72.9%	2025-03-25	gemini-2.5-pro-exp-03-25 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude Opus 4 →	72%	2025-05-22	claude-opus-4-20250514_32K Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o4-mini (high) →	72%	2025-04-16	o4-mini-2025-04-16_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-R1 (May 2025) →	71.4%	2025-05-28	DeepSeek-R1-0528 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude Opus 4 →	70.7%	2025-05-22	claude-opus-4-20250514 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3.2-Exp →	70.2%	2025-09-29	DeepSeek-V3.2-Exp Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude 3.7 Sonnet (32k thinking) →	64.9%	2025-02-24	claude-3-7-sonnet-20250219_32K Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o1 (high) →	61.7%	2024-12-17	o1-2024-12-17_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude Sonnet 4 →	61.3%	2025-05-22	claude-sonnet-4-20250514_32K Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude 3.7 Sonnet →	60.4%	2025-02-24	claude-3-7-sonnet-20250219 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3-mini (high) →	60.4%	2025-01-31	o3-mini-2025-01-31_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Kimi K2 Instruct →	59.1%	2025-07-12	Kimi-K2-Instruct Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-R1 →	56.9%	2025-01-20	DeepSeek-R1 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude Sonnet 4 →	56.4%	2025-05-22	claude-sonnet-4-20250514 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Flash (May 2025) →	55.1%	2025-05-20	gemini-2.5-flash-preview-05-20_23K Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3 (Mar 2025) →	55.1%	2025-03-24	DeepSeek-V3-0324 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
o3-mini (medium) →	53.8%	2025-01-31	o3-mini-2025-01-31_medium Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Grok 3 →	53.3%	2025-04-09	grok-3-beta Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4.1 →	52.4%	2025-04-14	gpt-4.1-2025-04-14 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude 3.5 Sonnet (Oct 2024) →	51.6%	2024-10-22	claude-3-5-sonnet-20241022 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Grok-3 mini →	49.3%	2025-04-09	grok-3-mini-beta_high Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V3 →	48.4%	2024-12-26	DeepSeek-V3 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Flash Preview (Apr 2025) →	47.1%	2025-04-17	gemini-2.5-flash-preview-04-17 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4o (Mar 2025) →	45.3%	2025-03-27	chatgpt-4o-03-27-2025 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4.5 Preview (Feb 2025) →	44.9%	2025-02-27	gpt-4.5-preview-2025-02-27 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.5 Flash (May 2025) →	44%	2025-05-20	gemini-2.5-flash-preview-05-20 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen3-32B →	40%	2025-04-28	Qwen3-32B Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.0 Flash (Dec 2024) →	38.2%	2024-12-06	gemini-exp-1206 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.0 Pro Exp (Feb 2025) →	35.6%	2025-02-05	gemini-2.0-pro-exp-02-05 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Grok-3 mini →	34.7%	2025-04-09	grok-3-mini-beta_low Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4.1 mini →	32.4%	2025-04-14	gpt-4.1-mini-2025-04-14 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Claude 3.5 Haiku (Oct 2024) →	28%	2024-10-22	claude-3-5-haiku-20241022 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4o (Jan 2025) →	27.1%	2025-01-29	chatgpt-4o-01-29-2025 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4o (Aug 2024) →	23.1%	2024-08-06	gpt-4o-2024-08-06 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.0 Flash (Dec 2024) →	22.2%	2024-12-11	gemini-2.0-flash-exp Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen2.5-Max →	21.8%	2025-01-25	qwen-max-2025-01-25 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
QwQ-32B →	20.9%	2025-03-05	QwQ-32B Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemini 2.0 Flash Thinking Exp →	18.2%	2025-01-21	gemini-2.0-flash-thinking-exp-01-21 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4o (Nov 2024) →	18.2%	2024-11-20	gpt-4o-2024-11-20 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
DeepSeek-V2.5 (Sep 2024) →	17.8%	2024-09-06	DeepSeek-V2.5 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen2.5-Coder-32B-Instruct →	16.4%	2024-11-21	Qwen2.5-Coder-32B-Instruct Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Llama 4 Maverick →	15.6%	2025-04-05	Llama-4-Maverick-17B-128E-Instruct Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Yi-Lightning →	12.9%	2024-12-02	yi-lightning Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Cohere Command A →	12%	2025-03-13	c4ai-command-a-03-2025 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Codestral →	11.1%	2025-01-13	codestral-2501 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
openhands-lm-32b-v0.1 →	10.2%	2024-03-26	openhands-lm-32b-v0.1 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4.1 nano →	8.9%	2025-04-14	gpt-4.1-nano-2025-04-14 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Qwen2.5-Coder-32B-Instruct →	8%	2024-11-21	Qwen2.5-Coder-32B-Instruct Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
Gemma 3 27B →	4.9%	2025-03-12	gemma-3-27b-it Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗
GPT-4o mini →	3.6%	2024-07-18	gpt-4o-mini-2024-07-18 Epoch AI · Aider coding benchmarks · Polyglot · percent correct ↗

Epoch data and evaluation setting

Epoch AI publishes this benchmark in its capability collection. Imported rows retain the published score column and model configuration. Original authors and evaluation providers remain credited in the result references.

How to read the result

Compare the named task version, tools and scoring rule. Epoch-run evaluations and externally collected results have different provenance. ECI inclusion does not make a benchmark score an ECI value.

Epoch attribution and score coverage

Data is attributed to Epoch AI under CC BY 4.0, with original evaluation providers credited in the linked source snapshots. Scores are imported only where a published mapping identifies the field and its scale; missing or incompatible records are omitted rather than assigned zero.

Versions and evaluation settings

Unite the Aider evaluation family. Polyglot and refactoring are different task sets and remain separately keyed measures.

Where this fits in an evaluation suite

The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.

Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.

A reproducible comparison

Retain Epoch’s task version, source score column, model version, tool setting and sampling budget. External observations retain their original publication URL.

Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.

PRIMARY EVIDENCE & FURTHER READING

Aider Polyglot ↗
Epoch AI
Capabilities & benchmarking — retained external evaluation records ↗
Epoch AI · 2026
Aider code editing benchmark ↗
Paul Gauthier, Aider contributors
Aider refactoring benchmark ↗
Paul Gauthier, Aider contributors

Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.

Continue exploring

All code & engineering benchmarks →