Case studies in measuring whether AI systems work.
Evaluations of anomaly detection, model comparisons and agent-written code, with the numbers that mattered and the ones that didn’t.
Precision of one anomaly detector under two alert rules
Flag a fixed percentile 0.12
Flag the top 20 (alert budget) 0.60
Topics
- Connecting the dots Reasoning & decisions, experiences worth sharing.
- Evaluation of AI Assessing the performance and reliability of AI systems.
- Context engineering Turning scattered enterprise data into a queryable source of truth.
- Integration Wiring AI into the systems around it while scaling the demo
- Open-source & open-weight AI Building on open models and open-source tooling
- Looking ahead My take on evolution of technology
Recent writing
All posts- The decision gate for agent-proposed code AI reviewers leave comments and policy engines return deny, but neither reads a diff and decides on it. A gate that routes sixty agent-proposed changes into applied, refused, or one reviewer item — with the model kept strictly advisory.
- The tests never reached the bug: evaluating a legacy migration done by AI A pipeline scored a perfect parity match and failed the artifact anyway. Here is why, and why a confidence score should be arithmetic a person can re-run.
- A small model against a large one on a narrow task: what the mistakes showed A fine-tuned small model against a large one on the same governance task. The headline gap was inside the error bars. The useful signal was which way each model got things wrong.