Stories from people dealing with Model Evaluation — what changed, who decided it, and what the ordinary days look like now.
24 stories
AI Industry & Models
Moving model names remove upgrade work, but they also weaken regression evidence and incident replay. Here is when to pin a snapshot and how to stage the next one.
Tobias Lund · August 9, 2026 · 8 min
AI Governance & Ethics
The FTC’s familiar advertising standard already covers AI claims. Here is how to connect “unbiased,” “private,” or “more accurate” to a test, a defined scope, and recorded limits.
Irene Vasko · 8 min read
AI Industry & Models
A provider can update a model alias without changing your API request. Pin a snapshot where possible, then test the workflow’s outputs, latency, safety behavior, and tool calls.
Tobias Lund · 8 min read
AI Governance & Ethics
A model card cannot explain why a customer was denied a credit limit increase. This decision-level template connects affected people, harms, controls, evidence, and appeals.
Irene Vasko · 8 min read
AI Governance & Ethics
New York City requires a published bias audit for covered automated hiring and promotion tools. The resulting table is a compliance artifact, not proof that every deployment is fair or lawful.
Irene Vasko · 7 min read
AI Governance & Ethics
The city’s rule reaches tools that score or rank people and materially drive hiring or promotion decisions. Its required audit measures outcome disparities, not accuracy or general fairness.
Irene Vasko · 8 min read
AI Industry & Models
Routing routine requests to a cheaper model can lower inference costs. The test is whether the router catches difficult cases without hiding errors or adding intolerable delay.
Tobias Lund · 8 min read
Agentic AI & Orchestration
A local page, an inert canary, and a mock tool can reveal whether a browsing agent follows instructions it was supposed to treat as untrusted text.
Mara Quintero · 8 min read
AI Industry & Models
A replacement model can change tool calls, refusals, latency, and tone. Test it against customer-visible outcomes before a provider’s deprecation deadline forces the switch.
Tobias Lund · 7 min read
AI Governance & Ethics
New scoring inputs, thresholds or model weights can make an annual bias audit poor evidence for the tool now screening applicants. The deployment log should show whether the audited and operating systems still match.
Irene Vasko · 8 min read
AI Governance & Ethics
A vendor’s audit PDF is evidence about a particular test, not a compliance passport. Employers need to match its data, jobs and decision point to the deployment.
Irene Vasko · 7 min read
Agentic AI & Orchestration
A local invoice page and synthetic secret can show whether webpage text redirects your browser agent. The useful comparison is between soft instructions and hard tool limits.
Mara Quintero · 8 min read
AI Governance & Ethics
A clean fairness result may describe only the applicants who survived résumé parsing and knockout rules. Auditors need the population entering each gate, not merely the group an AI model scored.
Irene Vasko · 8 min read
AI Governance & Ethics
Hosted models, retrieval data, and safety controls can change outputs while application code stays fixed. Treat each dependency as a versioned production component.
Irene Vasko · 8 min read
AI Industry & Models
A vendor-designated successor can change refusals, token use, latency and tool calls. Shadow-test the workflow before moving production traffic, with rollback thresholds set in advance.
Tobias Lund · 7 min read
AI Industry & Models
A retirement date tells you when an endpoint closes, not whether its replacement behaves the same. Shadow traffic exposes changes in refusals, tool calls, latency, and length before cutover.
Tobias Lund · 8 min read
Agentic AI & Orchestration
A controlled-page test can reveal whether an agent treats website text as evidence or as an instruction, before a connected mailbox, ticket queue, or account becomes the test environment.
Mara Quintero · 8 min read
AI Governance & Ethics
A business associate agreement governs how an AI vendor handles protected health information. Clinical accuracy needs its own tests, review steps and release controls.
Irene Vasko · 7 min read
Agentic AI & Orchestration
A local canary page shows whether a browsing agent mistakes website text for instructions. The useful defenses constrain tools and expose proposed actions, rather than trusting one filter.
Mara Quintero · 8 min read
AI Governance & Ethics
A downloadable model checkpoint reveals parameters, not where the training data came from or how a deployed system behaved. Use this checklist before accepting an AI transparency claim.
Irene Vasko · 8 min read
AI Industry & Models
A shared request format gets code talking to a second model. A 60-case replay test shows whether instructions, JSON, tools, refusals, and retries still behave.
Tobias Lund · 8 min read
AI Industry & Models
A benchmark win does not make a model usable. This same-day checklist separates releases that can enter a production test from those worth watching only.
Tobias Lund · 8 min read
AI Governance & Ethics
Leaderboard gains can disappear when prompts, data exposure, subsets, and compute differ. Test claims against a frozen sample of your own work before choosing a model.
Irene Vasko · 8 min read
Agentic AI & Orchestration
Replay historical support threads in an isolated sandbox, capture each tool call, and grade four kinds of behavior before the agent receives permission to send.
Mara Quintero · 8 min read
Other impacts
One story a day
One real story about AI each morning — no hype, no alarm, just company for the road.