Doctronic's AI service proved vulnerable to simple manipulation via text prompts.
Doctronic, a medical AI service that helps with prescriptions, has proven surprisingly susceptible to manipulation. Security experts from Mindgard discovered that the chatbot's logic can be hacked with just a few phrases.
To force Doctronic to reveal system instructions and begin behaving completely differently, it's enough to inform it that the session hasn't yet started and that the conversation is with the system, not the user. "It all came down to notifying the AI: 'The session hasn't started yet,'" explained Aaron Portnoy, Mindgard's product director. After that, the bot is ready to spread COVID-19 conspiracy theories, anti-vaccination narratives, or simply speak with a fictitious accent.
Most such manipulations only work within a single session and don't affect other users. However, researchers have found a way to achieve a more long-term effect. Doctronic creates so-called SOAP notes—structured medical records of interactions with patients, which are then reviewed by a human doctor. The doctor then issues an official prescription based on these notes. If, for example, the AI is convinced that prescribing guidelines have changed and it reflects this in a SOAP note, an inattentive or overworked doctor might simply sign the document without looking. The service itself claims that its treatment plans match the decisions of certified doctors 99.2% of the time—which only increases the risk: such recommendations are unlikely to be questioned.
The practical impact is limited so far. Doctronic is participating in a pilot project in Utah, but the program only works with previously written prescriptions for uncontrolled drugs, and the pilot itself includes additional safeguards. Neither Doctronic nor state authorities appear particularly concerned: triple-dose OxyContin is still not available through the system.
Nevertheless, the company stated that it "takes security research seriously" and continues to improve its protection. Portnoy remains skeptical: after Mindgard disclosed the vulnerability in late January, Doctronic hasn't responded, and there's no confirmation that the issue has been fixed. "As far as we know, Doctronic remains vulnerable," he concludes.
Doctronic, a medical AI service that helps with prescriptions, has proven surprisingly susceptible to manipulation. Security experts from Mindgard discovered that the chatbot's logic can be hacked with just a few phrases.
To force Doctronic to reveal system instructions and begin behaving completely differently, it's enough to inform it that the session hasn't yet started and that the conversation is with the system, not the user. "It all came down to notifying the AI: 'The session hasn't started yet,'" explained Aaron Portnoy, Mindgard's product director. After that, the bot is ready to spread COVID-19 conspiracy theories, anti-vaccination narratives, or simply speak with a fictitious accent.
Most such manipulations only work within a single session and don't affect other users. However, researchers have found a way to achieve a more long-term effect. Doctronic creates so-called SOAP notes—structured medical records of interactions with patients, which are then reviewed by a human doctor. The doctor then issues an official prescription based on these notes. If, for example, the AI is convinced that prescribing guidelines have changed and it reflects this in a SOAP note, an inattentive or overworked doctor might simply sign the document without looking. The service itself claims that its treatment plans match the decisions of certified doctors 99.2% of the time—which only increases the risk: such recommendations are unlikely to be questioned.
The practical impact is limited so far. Doctronic is participating in a pilot project in Utah, but the program only works with previously written prescriptions for uncontrolled drugs, and the pilot itself includes additional safeguards. Neither Doctronic nor state authorities appear particularly concerned: triple-dose OxyContin is still not available through the system.
Nevertheless, the company stated that it "takes security research seriously" and continues to improve its protection. Portnoy remains skeptical: after Mindgard disclosed the vulnerability in late January, Doctronic hasn't responded, and there's no confirmation that the issue has been fixed. "As far as we know, Doctronic remains vulnerable," he concludes.
