I trained a classifier on 200 documented flash loan attacks. The model could identify known attack patterns with 91% precision. Investors started using it to screen protocols before depositing funds.
Then a new attack happened using a price oracle manipulation method we'd never seen. The detector flagged it as safe. Users lost $2.3 million.
The fundamental problem
Machine learning finds patterns in existing data. Attackers specifically design new exploits to avoid those patterns. You're training on yesterday's attacks to predict tomorrow's creativity.
The model was excellent at recognizing reentrancy attacks and oracle manipulations it had seen before. But flash loan attack design evolves adversarially. Each new exploit deliberately looks different from previous ones.
Why this keeps happening
High accuracy on historical data creates false confidence. My 91% number was meaningless because it measured performance on attack types that were already public knowledge. The important question wasn't whether I could detect known attacks, but whether I could detect unknown ones.
I couldn't. No supervised learning model can.
What provides actual value
Anomaly detection based on normal protocol behavior works better than pattern matching on attacks. Instead of learning what attacks look like, the system learns what healthy operation looks like and flags deviations.
It generates more false positives, but it catches novel exploits. That tradeoff is worth making when the cost of missing an attack is catastrophic.