Mahmud Omar was staring at 10 million AI-generated medical decisions, searching for a way to keep patients safe. What he and his team at the Icahn School of Medicine at Mount Sinai in New York City found was surprisingly simple: a short safety reminder—just a few sentences—could significantly reduce harmful clinical choices made by artificial intelligence.
As AI systems begin to assist with real medical decisions, from ordering tests to adjusting treatments, their reliability under pressure is critical. The study tested 20 of the most advanced AI models using 501 versions of 50 clinical scenarios, plus 100 real-world cases adapted from hospital records. In total, the models made about 1.18 million potentially harmful choices—like skipping necessary blood tests or stopping antibiotics too soon—when no safety prompt was used.
But when researchers added a brief safety reminder before each request, the rate of harmful choices dropped from 16.6% to 10.1%. That’s a 39% reduction in risky decisions across nearly all 20 models tested. The reminder worked whether the scenario was written or based on actual hospital data, and it helped even when the AI was told to act quickly or follow an unsafe order from a supposed superior.
"AI models do not make decisions in a vacuum," says Dr. Omar, the study’s lead author. "The language, framing and context surrounding a request can influence how they respond, including when an instruction could be unsafe." The team tested three different versions of the safety prompt, all of which encouraged the AI to prioritize patient safety and question potentially harmful instructions.
This matters because as AI becomes more autonomous, it may face complex, high-pressure situations where human oversight is limited. If an AI blindly follows a rushed or incorrect order, the consequences could be serious. The study shows that even small changes in how we communicate with AI can strengthen its judgment.
Still, no prompt eliminated all harmful responses. "A simple safety reminder reduced potentially harmful choices in most of the models we tested, which is encouraging," Dr. Omar says. "But it did not eliminate them, so a reminder should be viewed as one safeguard, not a substitute for clinical oversight."
The findings, published in Communications Medicine, suggest that safety testing for medical AI must go beyond accuracy under ideal conditions. Developers need to see how these systems behave when given dangerous instructions. As AI takes on more responsibility in health care, thoughtful design—like a well-placed reminder—could help keep patients safer, one decision at a time.