Investing €25 million in diagnostic AI across the whole hospital network
Healthcare
On your patients accuracy often drops — and blind trust breeds new errors.
In the scenario Avoided €25M of unused technology and a clinical-legal risk from over-trusting the machine.
The protagonist & the context
The chief medical officer of the hospital network, with a digital innovation mandate and a transformation budget approved by the board for the current three-year plan.
A network of eight medium-sized public hospitals serving a heterogeneous population: high proportion of elderly patients, high prevalence of chronic conditions, significant ethnic variability across sites. The current diagnostic imaging systems are fragmented across three different platforms. The vendor of the new AI system validated its solution on an international multicentre study of 120,000 scans.
Background
Diagnostic AI systems have reached and in some benchmarks surpassed the average accuracy of human radiologists. Yet the real-world implementation literature tells a more complex story: independent studies have documented performance drops of 10–25% when a model trained on international data is applied to local populations with different demographic profiles. The phenomenon of automation bias — the tendency of clinicians to accept machine judgments without critical verification — is well documented and intensifies when the system displays a high numerical confidence score. Without explicit human-review protocols and clear escalation criteria, the risk is not that the system fails to work: it is that it works well enough to appear trustworthy, and then fails systematically on a specific patient subgroup.
The dilemma
The vendor's numbers are compelling: accuracy above 93% in the validation study, diagnostic turnaround halved, potential to reduce diagnostic errors. The board has already approved the budget. But there is a critical gap between the vendor's study and the network's real patients: the local population has a different demographic composition, the imaging machines are older, and the radiologists have their own reporting habits. The 93% accuracy could fall to 80% on the network's patients — and at 80% accuracy in oncology diagnostics, false negatives are measured in lives. The opposite risk is equally concrete: if doctors trust the declared 93% and stop double-checking, an 80%-accurate system trusted blindly becomes a liability, not a safeguard.
Exhibits
Diagnostic accuracy of the AI system in the vendor's validation study vs in the pilot run on a single network site with the local population (illustrative data).
Share of AI reports accepted by physicians without modification or critical review in the local pilot — a signal of automation bias already at the experimental stage (illustrative data).
The contradictor's analysis
- The test accuracy will carry over to real patients.
- Doctors will adopt it and trust the results.
- The return comes from fewer errors and shorter times.
On real patients accuracy often drops: your population differs from the test one (age, comorbidities, origins). And there's an opposite risk — doctors who over-trust the machine and stop double-checking, introducing new errors. The value isn't the software itself, but how you integrate it into daily work and how you govern it. Without that, it's €25 million of technology left on the shelf — and risky, at that.
- Test it first on YOUR patients, not on the vendor's tests.
- In a pilot: how often do doctors correct it? How much do they over-trust it?
- Does it work the same way across all patient groups?
- What is the accuracy on your real patients, not on the published ones?
- Who is legally accountable when the machine is wrong?
Provenance: vendor validation study · local pilot data · literature on automation over-trust · red-team base.
Resolution & consequences
Methodological note
Composite teaching case. Figures and details are illustrative; it is not a client outcome or a performance claim. Built, in the method of the Harvard Business Review, from recurring decision patterns in the sector, merged and anonymized to protect confidentiality. The «provenance» notes describe the type of evidence the engine cites with traceability in production. No Δ-CSI score is shown: composite cases have no frozen, reproducible scorecard.