I labeled 800 smart contracts as risky if they were exploited and safe if they weren't. Trained a model to predict risk based on code patterns, audit status, and deployment characteristics. It achieved 82% accuracy on the test set.
Then it flagged a thoroughly audited, well-designed protocol as low risk. That protocol was exploited two weeks later for $4.1 million.
The labeling mistake
Not being hacked doesn't mean a contract is safe. It might just mean attackers haven't found the vulnerability yet or the protocol isn't valuable enough to target. My safe labels included contracts that were actually vulnerable but hadn't been exploited yet.
The model learned to recognize contracts that had been attacked, not contracts that were attackable. Those are completely different things.
Why the data poisoned everything
Machine learning can only be as good as the labels you provide. I had given it outcome data and asked it to predict risk. But outcomes depend on attacker behavior, not just contract quality.
Some risky contracts never get exploited because the TVL is too low. Some safe contracts get attacked anyway through novel methods. The model couldn't learn actual risk because the training signal was contaminated with randomness.
What makes labels useful
Manual expert review of contract architecture would have provided better labels than exploit history. A security researcher evaluating code quality gives you signal about actual risk, not just whether someone happened to attack it.
That approach doesn't scale easily, which is why I avoided it. But unscalable accurate labels beat scalable garbage labels every time. I learned that the expensive way.