Writing
On evaluation design, data quality, and the architecture around AI systems that have to actually work.
The decision gate for agent-proposed code
AI reviewers leave comments and policy engines return deny, but neither reads a diff and decides on it. A gate that routes sixty agent-proposed changes into applied, refused, or one reviewer item — with the model kept strictly advisory.
- The tests never reached the bug: evaluating a legacy migration done by AI A pipeline scored a perfect parity match and failed the artifact anyway. Here is why, and why a confidence score should be arithmetic a person can re-run.
- A small model against a large one on a narrow task: what the mistakes showed A fine-tuned small model against a large one on the same governance task. The headline gap was inside the error bars. The useful signal was which way each model got things wrong.
- Measuring an anomaly detector: ROC AUC 0.9998, precision 0.12 Why ROC AUC flatters rare-event detectors, and why the alert budget ended up mattering more than the model.