For better or worse, chatbots are changing the way we think, learn and perceive the world around us. This kind of disruption is manifest in many areas of life, but perhaps one of the most sensitive and often concerning is the growing use of generative AI (GenAI) tools for healthcare. Alongside a number of freely available AI chatbots, major technology companies have moved into consumer-facing health AI with the launches of services such as Copilot Health, ChatGPT Health, and Amazon’s HealthAI that models help users interpret their medical records and ask questions about their symptoms, lab results and treatment options.

But there are risks to expecting an AI tool to take on the role of your physician. Also, the risk is not only that users receive the wrong advice, but that they may share deeply sensitive personal information with systems whose privacy protections, data-sharing practices and legal obligations may differ from those of a doctor or hospital, as well as that their data may be exposed to unexpected entities. Misuse of AI chatbots in general is now the number one health technology hazard out there, according to one US patient safety organization.

From theory to practice

The reason why the model-builders are launching in this space is obvious: chatbots have become a hugely popular way to search for medical advice. According to Microsoft, people talk about their health and the health of their loved ones more than any other topic on their mobile devices. Chatbots are there 24/7 with an answer for everything, dispensed in a confident tone that helps to put nervous patients at their ease.

At a time when national healthcare systems are under growing strain, many individuals would probably self-diagnose with the help of AI before deciding whether to seek medical attention. The time, effort and potential cost of entering the labyrinthine health system rather than triaging at home makes this a popular way of doing things.

Yet concerns are already emerging. The first is of hallucinations or incorrect advice. An Oxford University study from February published in Nature Medicine found:

  • Users often didn’t know what information they should share with the LLM
  • LLMs provided very different answers, even if the questions posed to them varied only slightly
  • Models often provided both good and bad advice, but users struggled to distinguish between the two

“Despite all the hype, AI just isn't ready to take on the role of the physician,” warned the study’s lead medical practitioner, Dr Rebecca Payne. “Patients need to be aware that asking a large language model about their symptoms can be dangerous, giving wrong diagnoses and failing to recognize when urgent help is needed.”

Uncovering the privacy risks

There are also non-health related risks which should encourage individuals to pause for thought. The most obvious is that sharing sensitive medical information with a publicly available chatbot may mean that data is used to train the model and therefore gets regurgitated out to others. Even unintentionally, models have been known to accidentally expose data typed in by their users.

Some providers may use data to improve their models unless users opt out, while others make stronger promises not to use health-related information for training. Either way, everybody should know what kind of service they’re dealing with before uploading anything sensitive. Your health data is not like a stolen credit card that can be frozen while the details are replaced and reissued. It’s yours for life, and once shared with an AI tool, it may become a permanent digital record.

On the other hand, most of the main health-focused chatbots promise not to use this data for training purposes. Still, training is only one part of the privacy picture, and the services may not make the same promises about third-party data sharing. Your personal medical information may up in the hands of a data aggregator, a third party that sits between the model provider and your healthcare provider. It might also be shared with advertisers, either directly or via one of these aggregators, although it will usually be anonymized prior to use. Even so, people should be cautious: health data is unusually sensitive, and anonymization doesn’t always remove every risk.

When breach risk multiplies

The problem with sensitive data traversing so many organizations is that there’s a greater chance it could be exposed to digital thieves and fraudsters. US lawmakers claim to have identified $21 billion in losses tied to a handful of breaches at data broker firms. Health data is highly monetizable by fraudsters for several reasons:

  • It retains its value for long periods of time, as it can’t usually be replaced or reissued
  • It could include insurance information with which to submit fake claims or receive medical services in your name
  • It could be used to blackmail you

The more companies that hold this data, the more opportunities there are for hackers to compromise them and steal it. The challenge is that most healthcare AI tools are not regulated by HIPAA as they are classed as consumer rather than enterprise-grade services. That means the providers may not be subject to the kind of strict data protection rules you would normally expect.

Advice for patients

So how can you minimize your exposure to the risks of healthcare GenAI? If you are concerned about a medical condition, avoid general-purpose bots and look instead for ones specially designed for answering health-related questions. Review whether the service explains how it handles your data, whether it uses your prompts for training, whether it shares information with third parties, and whether it is covered by HIPAA or an equivalent privacy regime in your country.

Don’t blindly trust the output unless there are citation links to verify it. And even then, don’t take its answers as gospel: always check with a medical professional, and/or an official website (e.g., NHS, MedlinePlus).

To protect your privacy, consider the following:

  • Never share/upload medical documents, lab results or other sensitive documents with an AI tool unless you understand how the tool handles that data.
  • Avoid entering names, addresses, insurance details, patient numbers or other identifiers.
  • Ensure training and chat-history features are switched off.
  • Share only the minimum information needed for the task.
  • Assume everything you type in could be retained or exposed, and adjust your prompts accordingly.

Ultimately, AI chatbots may be useful for brainstorming questions about a specific condition to ask your doctor, or for explaining a medical term you’re not familiar with. But there’s a big difference between using AI to prepare for care and using it as a substitute for care. Don’t treat a confident answer as a diagnosis, and don’t ignore urgent symptoms because a machine sounded reassuring.