Field notes · Topic
Reliability.
Launch is where the real work starts: sources change, models are updated and habits drift. These notes cover the routines that keep quality, cost and trust where they were on the day the system went live.
7 notes
Notes on this topic
-
What to monitor after an AI system goes live, and why quality slips.
Quality slips without anyone touching the system: sources change, the case mix shifts, providers update models. Watch quality, cost, speed and use together, and treat a rising override rate as the earliest warning you will get.
-
When AI gets it wrong: an incident plan for the first hours.
AI incidents are rarely outages. They are wrong answers that look right. Set severity by impact, make the stop switch and fallback routine, tell affected people quickly, and turn every incident into new evaluation cases.
-
How to switch language models without breaking what works.
Models get retired, prices change and better options appear, so plan to switch from day one. With an evaluation set, a model-agnostic layer and side-by-side runs, a migration becomes a test result, not a leap of faith.
-
Every override is a lesson: building a feedback loop that improves AI.
Thumbs-up buttons collect noise. Structured feedback from the review step (what was changed and why), triaged weekly by the owner, turns daily corrections into better sources, prompts and evaluation cases.
-
Why reviewers stop checking AI output, and how to keep review real.
When a system is right most of the time, people stop looking. Evidence-first screens, known-answer cases, rotation and metrics that notice when approval becomes a reflex keep human oversight meaningful.
-
LLMOps and MLOps: what changes when the model is a language model?
Much of MLOps carries over, but with language models you rarely train: you manage prompts, retrieval sources, external model versions and human review. The center of gravity moves from training pipelines to evaluation and change control.
-
What a support agreement for an AI system should promise.
You can’t promise accuracy the way you promise uptime. A useful agreement commits to quality thresholds on the evaluation set, incident response times, rules for model and prompt changes, cost reporting and a regular improvement review.