Measuring success of a model - practical case of anomaly detection on financial data

In a field that rewards better models, the thing that fixed my anomaly detector was a threshold rule.

The numbers that started it: on one run, ROC AUC read 0.9998 while precision sat at 0.12. Both are correct. Same model, same data, same run. One of them says ship it and the other says nobody will use this.

Everyone already knows accuracy is useless here — a model that flags nothing scores over 99.9% on this data, and that failure is obvious enough that people stopped reporting it. ROC AUC fails the same way and still gets treated as the serious metric.

The denominator#

ROC plots true positive rate against false positive rate.

TPR = TP / (TP + FN)
FPR = FP / (FP + TN)

True negatives sit in the FPR denominator. In rare-event detection they are effectively the whole dataset, so FPR stays small no matter how many false alarms you generate. The curve cannot express the thing you care about.

Precision has no TN term. That is the entire difference.

The ceiling is arithmetic, not modelling#

The detector warns about this before you waste a day on it:

WARNING  p99.9 flags ~27 of 26,988 normal items but only 12 anomalies exist,
         so precision cannot exceed 0.308 however well the model ranks.
         Raise the percentile (~p99.956 balances them) or set an alert budget.

Nothing about the model is in that calculation. A percentile flags a fixed fraction of everything, so on a large dataset it flags hundreds of normal items regardless of how good the ranking is. With 12 real anomalies in the set, precision is capped at 0.308 before the model contributes anything.

The fix is to flag a fixed count instead of a fixed fraction. Set the budget to the number of cases you can actually investigate, and the alert list stays that size. On the demo data, switching from a percentile to a budget of 20 moved precision from 0.12 to 0.60 — same model, five times fewer false alarms.

That is the part worth sitting with. The model didn’t change. The single largest improvement in the system came from deciding how many alerts a person could work through in a day.

The model wasn’t the differentiator either#

I ran the LSTM against ARIMA and a matrix profile on the same windows, the same entity-wise split, the same labels, averaged over three seeds.

On a single-variable anomaly — a turnover figure that spikes or collapses:

Method	PR-AUC	Precision@50	Recall@50
LSTM	0.97	0.71	1.00
Matrix profile	0.89	0.62	0.88
ARIMA	0.60	0.45	0.64

The LSTM leads, but a training-free matrix profile is close behind. With fewer epochs and a noisier feature set I have seen the two draw level. For this kind of anomaly the deep model is not buying much.

On a relationship anomaly — assets that normally track last month’s turnover moving the wrong way, while every individual value stays inside its own normal range:

Method	PR-AUC	Precision@50	Recall@50
LSTM	0.69	0.71	0.66
Matrix profile	0.03	0.03	0.03
ARIMA	0.02	0.03	0.03

The single-series methods collapse to near-random. There is nothing to see in the one series each of them watches — the anomalous values sit within 1.2 to 1.5 standard deviations of their own history. The break is a large conditional surprise and an unremarkable marginal one.

So the LSTM is not better in general. It is better at one specific thing, and on that thing the alternatives are blind rather than merely worse. That is a narrower claim than “we used deep learning” and a more useful one.

What the evaluation had to get right#

Four properties, each fixing a mistake that produces good-looking numbers that mean nothing.

Splits partition companies, not windows. Sliding windows overlap by five of their six months, so splitting windows at random puts near-duplicates on both sides of the boundary and the test set grades the model on data it has already seen. Splits are stratified on whether a company contains any anomaly, so the rare positives reach all three sets.

The threshold comes from held-out data. Error measured on training data is in-sample, understates normal error, drags the cutoff down, and floods the results with false positives.

A percentile, not a normalised midpoint. Min-max normalising test error against validation error and cutting at the midpoint lets one extreme validation sample define the decision boundary. A percentile is stable and has a direct reading: p99 means roughly 1% of normal windows are expected to be flagged.

PR-AUC is the headline metric, reported with the number of positives in the test set. The run also refuses to pretend when an evaluation is degenerate — no anomalies in the test split, or too few for metrics to be stable across seeds.

Caveats#

The dataset behind these numbers is simulated, with a known generative process. The headline figures describe a toy problem, not yours. The multivariate result is robust across seeds and configurations; the size of the LSTM’s univariate edge is not. Real data is the proper next test.

The splits are also by entity, which answers “does this generalise to new companies?” If your question is “will this catch next month’s problem?”, split by time instead — train on the past, score the future.

The takeaway#

ROC AUC answers a question about ranking quality. Nobody’s operational decision depends on it. The metric that matters is the one that describes what the person reading the alert list will experience, and the threshold that matters is the one set by how many alerts that person can get through.

Code, the full methodology, and the baseline write-up: github.com/MAY2704/anomaly_detection