Macroscope’s 2025 study
- Publisher
- Macroscope
- Evaluation
- September 2025
- Tested
- August 25–September 14, 2025
- Focus
- Known runtime bugs across eight languages
- Bugs evaluated
- 72 to 118, depending on the tool
ASTER / THE BENCHMARK LAB
Aster hasn’t published a benchmark yet. When it does, you’ll see the dataset, the bug counts and the test date beside it. Until then, here’s how to read any score, including ours.
HOW TO READ THIS
Every rate you see is bugs found divided by bugs evaluated. Look at the second number: tools in this study weren’t all tested on the same count.
57 found ÷ 118 evaluated = 48.3%
THE NUMBERS
| Tool | Detection rate | Bugs found / evaluated | Source |
|---|---|---|---|
| Macroscope | 48.3% | 57 / 118 | Macroscope · 2025 study |
| CodeRabbit | 45.8% | 54 / 118 | Macroscope · 2025 study |
| Cursor Bugbot | 42.4% | 50 / 118 | Macroscope · 2025 study |
| Greptile | 23.6% | 17 / 72 | Macroscope · 2025 study |
| Graphite Diamond | 18.3% | 21 / 115 | Macroscope · 2025 study |
STUDY DETAILS
No Aster number appears on this site until all of these exist.
WHAT MAKES A SCORE USEFUL
Before you trust any score, including ours, check these four things.
Ask which repos, commits and languages were tested. Then check the tool versions, settings and test date.
Each finding should match a known bug. A detection rate alone won’t tell you how many suggestions were wrong or noisy.
Failed and skipped cases matter. Look for how many bugs each tool was actually tested on, next to the percentage.
Can you read the study yourself? Macroscope’s is linked here. Aster’s will be published the same way, with the evidence, or not at all.
The cited study tests known runtime bugs across eight languages. Aster checks something else: leaked keys, Supabase and Firebase access rules, package versions with known holes, JavaScript/TypeScript syntax and some risky code patterns. It doesn’t yet check webhooks, admin access or rate limits. A high bug-detection score, from any tool, is not a security guarantee. See what Aster checks.
START WITH YOUR OWN CODE
Judge Aster on your own code. You get each finding critical first, with the file, the line and a plain-English why. 1 free audit every month.