Is âhuman in the loopâ a safety feature, or just a place to put the blame?
Dr. Sam Ashoo â an EM physician and clinical informaticist â draws a line this week between two ways to protect a clinicianâs judgment: a nicely-worded principle, and a clause in a contract.
The National Nurses United âNurses and Patientsâ Bill of Rightsâ asserts a right to override AI decisions âwithout the threat of discipline or discharge.â Every professional society has a version of that sentence.
The problem: a mission statement has never once stopped a manager. A contract can.
Hereâs the part builders skip. Operational AI â acuity scoring, predictive staffing, automated adjudication â is often sold to a CFO because it canât be easily overridden. Non-negotiability isnât a bug in that sale. Itâs the pitch. One health system cut contract-labor spend 15% within six months of turning on predictive staffing; the savings depend on the algorithm winning the argument.
So when we draw âhuman in the loopâ on the architecture slide, we should be honest about which human, with what authority, absorbing what risk when they hit override. If the answer is âa nurse who gets written up for deviating,â the loop is decorative.
đ¤ âUnions are self-interested â this is a labor play dressed up as patient safety.â Sure, itâs a labor play. Itâs also the only enforceable override protection currently at the bedside, which tells you something uncomfortable about how weak the voluntary ones are. Two things can be true.
đ¤ âOur tool always lets the clinician override.â Letting them click âdismissâ isnât the same as protecting them when they do it 40 times a shift and someone runs the report. Design the override; then go find out what happens to the person who uses it.
Interactives
https://www.clinicians.dev/interactives/
CMS just reprimanded its own AI â in Medicare.
Six months into the WISeR prior-authorization pilot, CMS ordered its contractor Virtix Health to file a Corrective Action Plan for missing turnaround-time standards. Reporting indicates all six pilot states and all six contractors fell short, and 60+ members of Congress are moving to wind the program down.
This is the government running the exact automated-denial playbook it lets Medicare Advantage plans run â and stumbling on the boring part, turnaround time, not the AI part.
The failure mode wasnât a hallucination. It was an SLA.
đ¤ âSo AI prior auth doesnât work?â It means this one missed its clock. Read that as a spec, not an obituary.
The effect vanished before it reached the patient.
The most rigorous test yet of embedded LLM decision support â a cluster-randomized trial of GPT-4o âAI Consultâ across 16 Penda Health clinics, published in Nature Medicine â found no significant difference in 14-day treatment failure (2.2% vs 2.0%). The trial defined treatment failure as any of the following events within 14 days of an encounter: persistent symptoms requiring a return visit, an unplanned escalation to emergency care, or a clinically significant safety event (e.g., a missed diagnosis, inappropriate prescription, or severe adverse outcome).
Clinicians fully followed only 19.5% of the appropriate red alerts. (Red alerts were prompted when the LLMâs analysis indicated that the documentation, diagnosis, or treatment plan was incorrect or potentially harmful)
This trial ran in Kenya, not the US, so hold the generalization loosely â but the shape of the result is the lesson: an alert nobody acts on is an override by omission.
đĄ 80/20: Stop reporting âaccuracy.â Report action rate â what fraction of the modelâs correct, high-priority flags actually changed a decision. Build that number into your eval from day one; itâs the only one that separates a tool that helps from a tool that documents.
This weekâs cost lesson: route, donât just pick.
Two signals landed together. Meta imposed internal caps on token spend after 2026 inference costs approached the billions â and if a hyperscaler is rationing tokens, your side project has no excuse. In the same window, Nate Jones published a âmodel-pickerâ routing prompt that sends each task to the cheapest model that can still do it well.
đĄ 80/20: For any clinical workflow, most steps donât need your best model â triage-routing a summarization step to a small model and reserving the frontier model for the reasoning step can cut cost by an order of magnitude with no quality loss youâd notice. Meter it per-task, not per-app.
The security agents are here â and so are the models that make them necessary.
Cognition launched Devin Security Swarm, an agent that scans a codebase, tests whether a vulnerability is actually exploitable, and opens the fix PR itself. Meanwhile a researcher argues the GLM-5.2 open-weight model is a genuine security emergency precisely because its safeguards strip off so easily.
đ¤ âAutonomous agents fixing my security? Hard pass.â Fair â an agent that can write the fix can write the hole. But if youâre a clinician shipping healthcare code, âI ran an exploitability scan on every PRâ is a sentence your security review wants to hear. Supervised, on non-PHI repos, this is a gate, not a co-pilot.
Ultra-shorts:
Together AI raised an $800M Series C at an $8.3B valuation, adding ~500 MW of compute â the open-model inference layer keeps consolidating capital.
xAI shipped a no-code Grok Voice Agent Builder at roughly $0.05/min. Patient-reminder and intake lines just got a weekend-project price tag (synthetic data only â do not point this at a real panel without a BAA).
đď¸ From the Pods
đď¸ The 229 Podcast â âWhy the Browser Is Your Health Systemâs Biggest Security Gapâ (This Week Health, w/ Andrew Rollo)
The browser â not the endpoint â is now where the clinical data actually lives, which means your browser update cadence is your real vulnerability-patch cycle. Schedule a 10,000-endpoint browser refresh around network load, not against it, or youâll starve the EHR at the wrong hour.
đ Speaker Blindspot: Appeal to authority plus a quiet false dichotomy â a Chrome Enterprise engineer frames the managed enterprise browser as the inevitable answer to a threat heâs also defining, skipping past thin-client, VDI, and app-isolation alternatives entirely.
đď¸ Tradeoffs â âMaking Medical Decisions with Uncertain Scienceâ (Aspen Ideas Health panel)
As AI gets embedded everywhere, informed consent is quietly collapsing into mere disclosure, and the field hasnât decided whether thatâs acceptable. The panelâs sharpest line for builders: âthe algorithm is trained to sound certain,â which hands clinicians an uphill battle against a confident machine.
đ Speaker Blindspot: False equivalence â ârejecting AI is like refusing electricity in the ERâ smuggles inevitability past the very consent debate the panel is trying to have. Electricity has no bias, no accountability gap, and no de-skilling risk; AI has all three.
What are you building this week? Email and tell me (kevin@clinicians.build) â I read every one.
â Kevin & AI





