A Code-Risk Score That Failed Calibration — and the Narrower Claim That Held Up 6-7x
Key Points
What is Hotspots?
Hotspots is a free, open-source CLI that scores every function and file in a codebase by structural complexity (cyclomatic complexity, nesting depth, fan-out, non-structured exits), then layers in git history (churn, touch frequency, recency) to separate active risk from dormant debt. It runs locally in under a minute, with no account and no code leaving your machine — hotspots.dev (source at github.com/Stephen-Collins-tech/hotspots).
What is an odds ratio, in plain terms?
It's how many times more likely one group is to have an outcome compared to another group. An odds ratio of 6 means a flagged file was 6 times more likely to need a later fix than an unflagged one — not a 6% difference, a 6x difference.
Why does "6-7x more likely" matter more than a correlation number?
A correlation score (like Spearman's rho, written ρ) tells you how well two rankings move together across an entire dataset — it's abstract and easy to misread. An odds ratio answers the question an engineer actually has: "if this file gets flagged, how much more should I worry about it?"
Does this mean file_risk_score is a reliable probability of a bug?
No, and that's the point of this post. The raw score failed a direct calibration test (a far more rigorous bar than "has some correlation"). What survived is a narrower, relative claim — whether a file lands in the riskiest 10% of the files you're already touching in one diff — which is a different and more defensible statement than "this file has an X% chance of a bug."
Where can I see the full methodology and numbers?
The complete writeup, every cutoff tested, and a script to reproduce the result against any repo you choose is public at github.com/Stephen-Collins-tech/hotspots-public-research.
Most tools that claim to score code risk never publish the check for whether the score means anything. I built one of those tools, so I ran the check on my own formula — and the honest answer was “not the way I was using it.” What’s interesting is what survived once I stopped asking the wrong question.
The claim I’d been making without testing it
hotspots has had a file_risk_score for a while — a composite of a file’s worst function complexity, its average complexity, and how many functions it has, rolled into one number per file. I’d always treated it the way these scores usually get treated: higher number, riskier file, done. I’d never directly tested whether the number itself behaved like a probability.
So I tested it. I pulled real fix-commit history from a pool of public repos, matched it against file_risk_score values at the time those fixes landed, and checked whether the score calibrated — meaning files with similar scores should have similar, predictable rates of later needing a fix, smoothly scaling from low scores (rarely fixed) to high scores (often fixed).
The correlation (ρ, a 0-to-1 measure of how tightly two rankings track each other) came out at 0.515, against a bar of 0.8 I’d set before running the test specifically so I couldn’t talk myself into a lower one afterward. That’s a clear miss. As an absolute, standalone probability of “this file will need a fix,” file_risk_score doesn’t hold up.
The part of the failure that wasn’t flat
Here’s where it got interesting. The calibration curve wasn’t uniformly bad — it had a sharp, specific shape. Files in roughly the riskiest 10% of the score distribution (the “top decile” — a decile is just a tenth of the data) had a fix rate wildly out of line with everything below them. That’s not what a broken signal usually looks like; a broken signal is flat noise everywhere. A signal with one real, concentrated effect and noise everywhere else looks exactly like this.
So the real question changed from “does this score calibrate” (no) to “is this top-decile effect real, or is it an artifact of exactly where I happened to draw the 90th-percentile line.”
Making the result hard to fake
I didn’t trust a single clean number, so I tried to break it four separate ways:
1. Does it survive moving the cutoff? I swept the threshold — top 20%, 15%, 10%, 5% — instead of trusting one arbitrary cut:
| Cutoff | Fix rate if flagged | Base rate (if you ignored the score) | Odds ratio | How unlikely by chance |
|---|---|---|---|---|
| top 20% | 22.1% | 10.9% | 3.56x | 1-in-roughly-3-trillion |
| top 15% | 25.5% | 10.9% | 4.37x | 1-in-roughly-800-trillion |
| top 10% | 30.3% | 10.9% | 5.57x | astronomically unlikely by chance |
| top 5% | 36.3% | 10.9% | 7.08x | astronomically unlikely by chance |
The effect got stronger as I tightened the cutoff, not weaker. An artifact of where a bin boundary happens to fall usually fades as you move away from that boundary. This sharpened monotonically — consistently in one direction — across a 4x range of cutoff widths. That’s the signature of a real effect, not a coincidence of binning.
2. Is one term secretly doing all the work? I pulled each ingredient of the formula out individually — max complexity, average complexity, function count — and re-ran the test each time. Every single one independently weakened the signal when removed. No term was dead weight riding along for free.
3. Does it survive repos the method has never seen? This was the test that mattered most. Every check up to this point still drew from the same 25-repo pool I’d originally used to find the pattern — and a held-out split of one pool is not the same thing as a truly new repo. So I picked five repos — one each in Rust, Go, TypeScript, Java, and Python — that had never been part of any prior check in this line of work, mined fresh fix-commit data from scratch, and ran the identical top-decile test with the identical pre-registered bar.
It replicated. Pooled odds ratio 6.68, with odds against chance in the billions. Every one of the five repos pointed the same direction individually, even though (expectedly, with small individual samples) not every one cleared statistical significance alone.
4. Does it beat the obvious cheap alternative? On that same never-before-seen data, I ran the same test against just counting lines changed — the simplest possible proxy for “this is a big, risky change.” file_risk_score won head-to-head at every cutoff tested, from 3.9x vs. 1.2x at the loosest cutoff up to 10.0x vs. 4.2x at the tightest.
What actually shipped
The raw score still doesn’t calibrate as a probability, and I’m not shipping it as one. What I shipped is the narrower, better-earned claim: hotspots diff now ranks each changed file’s file_risk_score against only the other files touched in that same diff — not the whole repo — and flags the ones at or above the 90th percentile of that diff-local ranking.
{
"file": "src/parser/tokenizer.rs",
"file_risk_score": 7.8,
"file_risk_diff_percentile": 0.94,
"file_risk_top_decile": true
}A file with no score at all (deleted, or an unscoreable type) is excluded from the ranking rather than defaulting to a misleadingly low value. And a diff that only touches one scoreable file gets no flag at all — there’s no “relative to the rest of this diff” in a population of one.
What I’m still not claiming
This is a statement about which of the files you’re already touching deserves the closest look — not an absolute verdict on any file in isolation. The same file could land in the top decile of a small, otherwise-clean diff and nowhere near it in a diff that also touches three genuinely gnarlier files. And five out-of-sample repos is real replication, not an exhaustive one; I’d treat a sixth or seventh repo landing outside this range as informative, not as a contradiction of the result.
The point of writing this up in this much detail — including the failed calibration test I could have quietly skipped — is that a risk score is only worth trusting if the people building it are willing to publish the check that could have killed it. The full methodology, every cutoff table, and a script that runs the live flag against any repo you choose are public at hotspots-public-research, alongside the rest of the findings — including ones that didn’t survive, and features that got removed after shipping once the evidence stopped supporting them.