Overview
Most code models pass tests but write slow code. This project measures how efficient open source LLM-generated code really is — not just whether it works.
Based on ENAMEL (ICLR 2025), we evaluate 13 open source models on 161 programming problems. The result: no open source model reaches expert efficiency. The best one (Code Llama 7B) only matches expert speed on 33% of problems.
Latest Results
Experiment History
Methodology
Efficiency Score
For each problem, code is run against 4 levels of test cases (8 + 4 + 4 + 4 cases). The per-level score is:
f = (T_i − max_m t[i,j,l,m]) / (T_i − max_m t*[i,l,m])
where T_i = α · maxl,m t*i,l,m with α = 2. Levels are combined by hardness-weighted mean with h = (3, 3, 4).
eff@k Estimator
The Rao–Blackwellized estimator over n ≥ k samples, using binomial weights over order statistics (Algorithm 1 from the paper). Theorem 1 proves unbiasedness and a variance bound of (k/n)·Var[max].
Robust Timing
Each test case runs R = 6 repeats, aggregated by the Hodges–Lehmann estimator for robustness to outliers. Candidates are executed in a sandboxed subprocess with resource limits.
Statistical Rigor
Every reported score carries a bootstrap confidence interval (10,000 resamples). No comparison is claimed without a pairwise test. The α-sweep and h-sweep are reported alongside the headline.
Project Status — All Milestones Complete
Metric Core
Algorithm 1 recurrence, Eq. (1)–(6), verified exactly against brute-force enumeration.
DoneParity Gate
17 models compared. Ranking preserved (τ = 0.783). All models score higher than published.
DoneHonest Statistics
Bootstrap CIs, censored scoring, full α and h sweeps in every report.
DoneContainerization
Dockerfile + docker-compose for reproducible measurement environments.
DoneReference Audit
30 references audited, 600/600 test cases pass. All reference solutions correct.
DoneAdversarial Generation
Per-candidate worst-case search. Average 1.60× slowdown found via input mutation.
DoneTwo-Axis Reporting
Memory tracking via ru_maxrss in every test case. Peak memory recorded alongside time.
DoneContamination Study
Paraphrased problem generator. Tests memorization vs algorithm understanding.
Doneq-Distribution (§2.2)
Measured: level 3 carries 71% sensitivity, levels 1–2 are near-flat. Confirmed.
DoneNewer Models — pass@1 from EvalPlus Leaderboard
The models below are from the EvalPlus Leaderboard (125 models total). Their code efficiency has not been measured with ENAMEL yet — only correctness (pass@1) is shown. The gap between pass@1 and eff@1 is what ENAMEL measures. Green = open source, Blue = commercial.