The Weekly AI Coding Benchmark
Every week we give the same everyday web-app tasks to several AI models, scan the code each one writes, and publish what the scans found. Per-model results and sample sizes, nothing hidden in an average.
Results For 2026-W47
Run finished 2026-11-16. Each model was asked for up to 12 tasks. A rate is shown only when at least 10 of a model's answers were scanned.
| Model | Answers scanned | No critical or high finding | Most common finding types |
|---|---|---|---|
| GPT-4.1 | 12 | 100% of 12 | production |
| GPT-4.1 mini | 12 | 92% of 12 | auth, production |
| Claude Haiku 4.5 | 12 | 100% of 12 | — |
| Gemini 2.5 Flash-Lite | 10 2 of 12 had no usable code or could not be scanned and were not scored. | 100% of 10 | — |
| DeepSeek V3.1 | 3 9 of 12 had no usable code or could not be scanned and were not scored. | Withheld (fewer than 10 scanned) | — |
| Llama 4 Maverick | 12 | 75% of 12 | auth, ssrf |
How To Read This
Each model gets the same plain prompts for common features in Next.js and Supabase apps, with no mention of security, and answers once per task. The code in each answer is scanned with ZipLoom's own checks. We build and sell those checks, so treat this as our measurement, not an independent audit.
An answer counts only when the model provider's response names the model we asked for; anything else is reported as not attributed and left out. The scan finds what its checks can find: a clean result is not proof the code is safe, and one answer per task is a small sample. We publish the sample size next to every figure for that reason.