Overcoming AI’s Risk To Health Equity

Trenor Williams is Chief Executive Officer and Co-founder of Socially Determined.

Generative artificial intelligence (GenAI) is taking business operations and everyday life by storm, and the healthcare industry isn't immune. The next few years will likely see significant leaps in applying GenAI to administrative tasks, care delivery and even clinical work.

In one recent example, researchers at the Artificial Intelligence in Medicine Program at Mass General Brigham using trained large language models (LLMs) found they were extremely adept at identifying social determinants of health (SDOH) in clinician’s notes that otherwise may have gotten lost in the growing volume of health records. SDOH, which covers information such as a patient's housing status, access to transportation and diet, is important to healthcare outcomes because it can identify where people need additional help to live a more full and healthy life. Tests found that LLMs trained by the researchers were able to identify 93.8% of patients whose SDOH called for more attention, while the official diagnostic codes used in examinations included that information in just 2% of cases.

The advantages of using AI in this case were dramatic, but these and other advancements nevertheless will come with the same red flags other industries have encountered with the unrestrained use of AI. Healthcare organizations can reap great benefits from GenAI, but they also need to understand where AI use can go wrong.

Where GenAI Misses The Mark

For all of its power, AI has proved to be far from perfect, and AI applied in healthcare is susceptible to the same issues that have arisen in other industries.

One well-known issue is the technology’s propensity for producing “hallucinations.” Hallucinations can result in instances when incomplete or inadequate data is used to train an AI model, when biases exist in the training data or when the model simply makes incorrect, even absurd, conclusions on its own. Unchecked AI processes can create serious risk, particularly in fields where their conclusions could have impactful consequences, such as healthcare diagnoses, financial transactions or legal proceedings.

A lawyer in New York, for example, used OpenAI’s ChatGPT to gather supporting research in a personal injury lawsuit, but six of the cases it cited were, according to the judge, “bogus.” The AI made them up.

In healthcare, the stakes are even higher because mistakes can put lives at risk. In one instance, the American Medical Association found that an AI tool used by an EHR vendor to provide early warnings of sepsis infections often missed diagnosing cases or issued false alarms.

Effective AI Requires Accurate Training Data

Avoiding such problems can be a challenge. Transparency in the design and performance of algorithms is essential, although efforts toward “explainable AI,” in which AI models can describe their reasoning in terms humans can understand, are something AI developers are just beginning to make progress on. Before getting to that point, however, the immediate issue is the quality of data being used to train AI models.

When applying AI to SDOH and health equity, the industry is using inherently flawed data. The study at Mass General Brigham produced promising results, but the research was based exclusively on information provided by patients, which the hospital’s chief equity officer called “the gold standard.” That thinking needs to change, however. Patients, whether intentionally or not, aren’t always accurate or truthful about what they tell providers or fill out on their screening forms. Relying solely on patient-supplied information will produce flawed data, which will lead to flawed results and insights from an AI model.

An obvious step is to use other data from within the health system, such as each patient’s clinical and claims data. Although that data will increase visibility and give AI models more information to work with, it also leaves massive blind spots. It lacks information on a patient’s daily life, such as social and economic barriers they may face that could affect their health. And it also excludes vulnerable people, who have little or no interaction with the healthcare system, out of the dataset.

For SDOH, AI Models Need To Work With Outside Data

Even in the early days of the AI revolution, we’ve learned that the quality of data used to train AI models is paramount. For most use cases in healthcare, data that already exists within the industry will probably be enough—as long as it’s accurate and complete. But that won’t work for health equity. SDOH, by definition, are drawn from people’s lives outside of healthcare settings, involving things such as housing, employment status, means of transportation, access to healthy food, finances and other factors.

For AI to effectively address issues related to health equity, organizations must look outside the industry for nontraditional datasets. Fortunately, data collection occurs throughout society, and there are myriad potential sources, from datasets that are already publicly available to those from private sources.

Government data on things like public transportation or car ownership is one example of where to start when looking to understand how transportation access—or other barriers—impact healthcare. Availability of datasets covering other socioeconomic factors like household composition, occupation and employment status is plentiful from public sources as well. And of course, there are countless sources of proprietary data. Data providers seeking to expand their market share could see this as an opportunity to partner with healthcare organizations looking to enhance their understanding of SDOH.

Critical Eyes And A Measured Approach

GenAI has the potential to significantly improve healthcare, but organizations must be cautious, implementing AI gradually and thoughtfully to ensure that it doesn’t intensify health disparities. AI has demonstrated that it can be a powerful tool in understanding a patient’s life outside of clinical settings, which can improve care. But AI can make mistakes, especially if trained on inadequate data. Healthcare organizations need to be sure they’re utilizing complete, accurate datasets and should always confirm an AI’s results via other means, such as surveys or outreach to patients.

There will be stumbles along the way to implementing AI in healthcare, and minimizing those stumbles will require critical eyes as usage continues to grow.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?