The Creative Nexus

AI and Rare Disorders: Tackling Data Scarcity

How do you train a diagnostic model when only a few people have the condition? Rare disorders expose the limits of medical AI built on large datasets.

Read in Norwegian

For optimal functioning, AI models necessitate access to substantial volumes of data. This leads us to a central question: How is the development of AI tools affected in fields where available data is limited? In this edition of the newsletter, The Creative Nexus, on AI communication and creativity, I investigate this question through the lens of my rare disorders field.

Two researchers examine images at a laboratory table.

Rare disorders encompass a multitude of congenital and hereditary conditions. Worldwide, between 6,000 and 8,000 rare disorders are spread across various medical disciplines. While each diagnosis is rare, the collective prevalence of rare disorders is substantial.

Do rare diagnoses risk being overlooked in the development of medical AI?

AI's Entrance into Healthcare

We currently stand at a crossroads where more and more people see that AI can change how healthcare approaches diagnosis and treatment. AI is well on its way to becoming a powerful tool in modern medicine, with significant potential to enhance diagnostics and therapy through data analysis and pattern identification.

AI-based tools are now changing medical practices in various domains. In radiology, AI has contributed to improved image analysis, enabling doctors to identify and diagnose conditions with greater precision and speed. In pathology, the technology has streamlined the process of identifying cancer cells, potentially leading to faster and more accurate diagnoses and, consequently, more precise treatment.

AI is also making its way into mental health. For example, AI tools analyze patients' verbal and non-verbal communication to detect early signs of mental disorders such as depression or anxiety. Chatbots provide valuable information to patients and healthcare professionals, while automation on the administrative side has streamlined routine tasks, including those related to patient records.

Similar Challenges in Other Fields

Amidst the optimism surrounding using artificial intelligence to improve medical treatment, questions arise about whether the technology can also be applied to rare disorders. Doubt is present among both professionals and patients. Effective AI use typically requires access to extensive data, which is naturally scarce in the case of many rare disorders.

However, such challenges are not unique to the rare disorders field. In many areas, from astronomy to archaeology, researchers face similar problems. Studies of rare cosmic phenomena or archaeological analyses based on small findings also involve minimal data. This applies to art restoration, research on endangered species, and linguistic studies of rare or endangered languages, where researchers often have to deal with small, specialized datasets.

Thus, perspectives and solutions from various fields, when viewed together, can also be relevant and informative when considering the way forward for AI technology.

Lack of Knowledge and Limited Resources

A diagnosis is rare if fewer than 1 in 2000 individuals have it. Today, more than 300 million people worldwide live with rare disorders.

Read more about the prevalence of rare disorders.

Rare disorders pose numerous challenges related to knowledge and expertise. Often, more information is needed for patients and healthcare professionals. Limited knowledge leads to diagnostic delays.

Stories of lengthy and frustrating procedures to obtain a correct diagnosis are common. Even with a verifiable diagnosis, many patients lack access to specialized knowledge, hindering access to necessary treatment and adequate follow-up.

Research Already Well Underway

There are several approaches to addressing the issue of little access to data. One possible solution is to use "few-shot learning" or "one-shot learning," where AI can learn from a small amount of data. Datasets can also be expanded through data generation and augmentation. Additionally, existing AI technology can help organize the scattered information available.

The research article The Impact of Artificial Intelligence in the Odyssey of Rare Disorders from 2023 provides an overview of current research on how AI is used in rare diseases. It discusses the challenges of small and fragmented datasets and possible solutions involving machine learning (ML), deep learning (DL), and related techniques.

According to the article, AI technology is under development for several rare diagnoses, including Duchenne muscular dystrophy, a genetic disorder leading to progressive muscle wasting, Fanconi anaemia, a rare genetic aberration resulting in bone marrow failure, and Huntington's disease (HD). This hereditary disorder gradually breaks down the brain's nerve cells, affecting cognition, behaviour, and physical function.

You can find more information about these and other rare disorders on Orphanet, a portal for rare disorders and orphan drugs.

Researchers in Mahmood Lab at the Brigham and Women’s Hospital have developed a tool that can assist doctors in diagnosing rare disorders faster and more accurately. This tool, called SISH (Self-guided Image Search for Histology), is a search engine that can scan through extensive medical images from previous patient cases and learn from the process. In this way, the intelligent search engine can assist doctors in finding answers to new topics.

Such tools will not only make it easier to identify disorders. Still, they will also help to merge information from different medical fields to provide an even more complete picture of the patient's health.

Researchers at the Stanford Institute for Human-Centered AI have demonstrated that training machines to identify rare disorders with limited data is possible by developing a POPDX tool.

The core of the progress is POPDx's ability to use previous knowledge, text, and the Human Disease Ontology disease catalogue to make informed assumptions based on available information.

Furthermore, the tool can draw from a wide range of information, including medical history, demographic information such as age and gender, results of physical examinations, laboratory tests, and questionnaire responses. 

This accumulation of fragmented knowledge enables POPDx patients to have a comprehensive picture of a patient's health, thereby increasing the chances of correctly identifying rare disorders.

Researchers envision that POPDx will monitor patients' health over time, allowing it to learn when specific disorders are most likely to develop. This could pave the way for more targeted health interventions, where treatment can commence earlier and be tailored to the individual.

There is currently notable optimism surrounding the idea of a technology-driven healthcare system. How this optimism will materialize for rare disorders remains to be seen. The technological possibilities exist, and a significant amount of research and development activity is underway. However, even if sufficient data were available to develop practical AI tools, a substantial task remains in making them accessible to those who need them. This responsibility extends far beyond the scope of individual healthcare institutions, research groups, or technology companies and will require effort and commitment at a broader societal level.

Use of AI

This newsletter is updated with the assistance of OpenAI's text generator, ChatGPT, and has been carefully edited and quality-checked by me, the author, to ensure accuracy and relevance. The visualizations were created using the AI tool Midjourney based on my requests and choices. Although AI tools have been a great help, my human judgment and professional insight guide the final content.

Mulige interessekonflikter: Artikkelforfatteren er ansatt som spesialrådgiver ved Nasjonalt senter for sjeldne diagnoser.

Read the original on Substack ↗