A fine-tuned small model against a large one on the same governance task. The headline gap was inside the error bars. The useful signal was which way each model got things wrong.
The task was narrow and real: given a database column’s name and description, decide whether it holds sensitive data. It is a data-governance problem, the kind where you point a classifier at a data dictionary instead of at the customer data itself, so the model never sees a single row it shouldn’t.
Two ways to solve it. Fine-tune a small model, RoBERTa, on historical classifications. Or prompt a large one, Gemini 2.5 Flash, with the policy documents in context. The small model scored an F1 of about 0.73 on held-out data. The large one scored 0.90.
That looks like a clear win for the large model. It mostly isn’t, and the reason is worth more than the result.
The held-out test set had 27 rows. Nine of them were sensitive.
On a split that size, a single flipped prediction moves F1 by roughly 0.05. So the gap between two models, or between two runs of the same model, is comfortably inside the noise before you have said anything about either model. A 0.73 and a 0.90 on 27 rows are not two measurements you can rank. They are two draws from a distribution wide enough to swap their order.
The pipeline made this hard to ignore. On the same fine-tuned model, the test split scored 0.727 and a separate unseen set scored 0.821. The unseen set is supposed to be the harder one, names the model never trained on, paraphrased rather than copied. It scored higher. That should not happen if either number were precise. It happens because both are computed on a couple of dozen rows, and at that size the ordering is random.
The honest reading of the whole exercise is the one that resists the headline: on this data, at this size, the F1 comparison cannot rank anything. Not RoBERTa against Gemini, not test against unseen, not one run against the next. This is the same failure as post one’s 0.9998 ROC AUC and post two’s perfect parity match. A real number, computed correctly, that does not support the decision people want to hang on it.
There is a more useful thing in the results than the aggregate score, and it survives the small sample better: which way each model is wrong.
On the test split the fine-tuned model over-flags. Recall 0.889, precision 0.615. It catches 8 of 9 sensitive columns and raises 5 false alarms. On the unseen split it flips and under-flags, recall 0.800 against precision 0.842.
For a governance control those two errors are not priced the same. A false alarm costs a reviewer a minute. A miss ships unprotected personal data into production. A model that over-flags is annoying and safe. A model that under-flags is quiet and dangerous. The fine-tuned model does not yet hold a steady bias toward the cheap error, and that is a real defect, more real than a 0.1 gap in F1.
It is also a fixable one, and not by training longer. It is a threshold on the model’s confidence, tuned to buy recall at the cost of precision, because on this task recall is the number you protect. More epochs would not have found that. Reading the errors did.
The seven unseen mistakes are more diagnostic than any score, because you can see what the model actually learned.
missed (predicted safe, actually sensitive):
birth_date_of_client, kyc_risk_classification,
otp_seed_value, geo_location_latitude
over-flagged (predicted sensitive, actually safe):
consent_marketing_flag, account_nickname, office_location_codeotp_seed_value and birth_date_of_client are plainly sensitive, and a real deployment that missed them would be a finding, not a rounding error. The pattern across the misses is that the model has learned surface lexical cues, the presence of words like name, email, card, more than the policy categories underneath them. Which is exactly what a small pile of synthetic training rows can be expected to teach it, and exactly the kind of thing an aggregate F1 hides and a list of misses reveals.
This is the argument for the small model that the score does not make. The fine-tuned model is cheap to run, sees only metadata, and every one of its errors is a column name you can read and reason about. When it is wrong, you can see why. That auditability is worth as much as a tenth of an F1 point in a setting where the cost of a miss is regulatory.
The real answer is the boring one, and the trade is not the F1 gap. The small model is cheaper, runs on a laptop, and fails legibly, but it needs a decent pile of historical labels and it learns lexical shortcuts when the data is thin. The large model needs no training data and brings general knowledge to a cold problem, at higher cost per call and with errors that are harder to trace to a cause. Pick the small model when you have the labels and want cost and auditability. Pick the large one when you do not have the labels and want breadth.
But that choice was never going to be settled by 0.727 versus 0.90. Those numbers came from 27 rows, and 27 rows cannot rank two models. What the results actually told me was narrower and more useful: neither model yet errs in the safe direction on purpose, and that, not the leaderboard, is the thing to fix before either one guards real data.
The fine-tuned RoBERTa pipeline is open source at github.com/MAY2704/sensitivity_scan. The committed dataset is synthetic banking metadata, so the numbers demonstrate that the pipeline learns the task rather than benchmarking a production control. Bring your own column inventory before trusting a score.