What it measures
GPU kernel-generation tasks evaluate whether a model can replace reference operations with correct, efficient implementations.
How to read the result
Correctness is the first gate. Then compare speedup on the same hardware, inputs, compiler, and timing protocol.
Limits and comparison traps
A faster but numerically invalid kernel is not an optimization success. Different accelerators and performance thresholds produce different score meanings.
Published evaluation
Fraction of generated kernels that are correct and faster than the default torch.compile baseline at the named task level. This is hardware- and baseline-specific; other backend settings are distinct columns.
Where this fits in an evaluation suite
The agent wrapper is part of the system. Pin tools, dependencies, timeouts, tests, and the repository revision. Compare contest coding separately from repository maintenance.
Start with a capability hypothesis, then choose a second test with a different failure mode. Keep the checkpoint, reasoning effort, agent tools, sampling policy, and scoring procedure fixed when measuring a training intervention. Record failures and missing outputs in the denominator.
A reproducible comparison
Fraction of generated kernels that are correct and faster than the default torch.compile baseline at the named task level. This is hardware- and baseline-specific; other backend settings are distinct columns.
Publish the exact run configuration, item coverage, score units, and available uncertainty. If a source supplies only an aggregate or a single run, preserve that limit. A new model name or a rounded score alone does not establish a capability improvement.
PRIMARY EVIDENCE & FURTHER READING
- KernelBench: Can LLMs Write Efficient GPU Kernels? — published result table ↗
All 7 authors
Ouyang, Anne, Guo, Simon, Arora, Simran, Zhang, Alex L., Hu, William, Ré, Christopher, Mirhoseini, Azalia
- Primary benchmark documentation or model evaluation report ↗
Reviewed 2026-10-26. Explanation and interpretation by RLTasks. Benchmark ownership remains with its original authors.