CASE-09 · Healthcare & new technologies
Investing €25 million in diagnostic AI across the whole hospital network
66
Healthcare
Healthcare
On your patients accuracy often drops — and blind trust breeds new errors.
In one line Avoided €25M of unused technology and a clinical-legal risk from over-trusting the machine.
The chief medical officer of the hospital network, with a digital innovation mandate and a transformation budget approved by the board for the current three-year plan.
A network of eight medium-sized public hospitals serving a heterogeneous population: high proportion of elderly patients, high prevalence of chronic conditions, significant ethnic variability across sites. The current diagnostic imaging systems are fragmented across three different platforms. The vendor of the new AI system validated its solution on an international multicentre study of 120,000 scans.
Diagnostic AI systems have reached and in some benchmarks surpassed the average accuracy of human radiologists. Yet the real-world implementation literature tells a more complex story: independent studies have documented performance drops of 10–25% when a model trained on international data is applied to local populations with different demographic profiles. The phenomenon of automation bias — the tendency of clinicians to accept machine judgments without critical verification — is well documented and intensifies when the system displays a high numerical confidence score. Without explicit human-review protocols and clear escalation criteria, the risk is not that the system fails to work: it is that it works well enough to appear trustworthy, and then fails systematically on a specific patient subgroup.
The vendor's numbers are compelling: accuracy above 93% in the validation study, diagnostic turnaround halved, potential to reduce diagnostic errors. The board has already approved the budget. But there is a critical gap between the vendor's study and the network's real patients: the local population has a different demographic composition, the imaging machines are older, and the radiologists have their own reporting habits. The 93% accuracy could fall to 80% on the network's patients — and at 80% accuracy in oncology diagnostics, false negatives are measured in lives. The opposite risk is equally concrete: if doctors trust the declared 93% and stop double-checking, an 80%-accurate system trusted blindly becomes a liability, not a safeguard.
Diagnostic accuracy of the AI system in the vendor's validation study vs in the pilot run on a single network site with the local population (illustrative data).
Share of AI reports accepted by physicians without modification or critical review in the local pilot — a signal of automation bias already at the experimental stage (illustrative data).
On real patients accuracy often drops: your population differs from the test one (age, comorbidities, origins). And there's an opposite risk — doctors who over-trust the machine and stop double-checking, introducing new errors. The value isn't the software itself, but how you integrate it into daily work and how you govern it. Without that, it's €25 million of technology left on the shelf — and risky, at that.
Provenance: vendor validation study · local pilot data · literature on automation over-trust · red-team base.
Composite cases, in the method of the Harvard Business Review: reconstructions based on real, recurring situations in each sector, merged and anonymized to protect confidentiality. The decision dynamics are authentic; names, figures and details are altered and not traceable to any single client or case. The «provenance» notes describe the type of evidence the engine cites with traceability in production. The Δ-CSI values illustrate the intensity of the pressure the contradiction put on the assumptions.