Machine Learning in Emergency Medicine - Should it Copy Doctors or Forge it's Own Way?

Paging Doctor Smith

The decision of whether to send someone home from an emergency room is one that can keep emergency room doctors awake at night wondering if they made the right decision.

This perspective comes from Dr. Ziad Obermeyer who served as the inspiration for this article due to his appearance on the Freakonomics MD podcast episode “The E.R. Doctor’s Dilemma”. Dr. Obermeyer spoke about his research into measuring how many patients died after they were sent home from an emergency department in the US and whether these could be improved through applying machine learning techniques [1].

Dr. Obermeyers’ study looked at over 10 million emergency room visits between 2007-2012, an estimated 20% sample of all emergency room visits. From these they identified ~12,000 deaths which occurred in the 7 days following the emergency room visit. From these visits they reviewed risk factors and applied basic machine learning to try to better understand what could be done to reduce the number of deaths.

One key insight that caught my attention was the question:

“Should we be training machine learning (ML) models to make the same decisions as doctors or directly diagnose patients?”

These may sound the same but there is an important distinction I am going to explore in this article.

Now imagine this…

You’re standing in front of a microwave, you’ve put your food in and set it for 2 minutes. There’s a line up of people behind you. The buzzer rings. You now have to decide whether you think you got your time right OR to test the temperature, put it back in, and make the people behind you wait longer.

What do you do?

You’re likely wondering what this has to do with machine learning in emergency medicine, but I promise it’s relevant.

Now imagine you’re an emergency room doctor. You’re with a patient who’s sitting in front of you (your metaphorical food in the microwave). You’ve done your initial assessment and diagnosis of their presenting complaint (set the timer and let the microwave run and the buzzer rings). Now you need to decide, do I send this person home? Or do I admit them to the hospital to run more tests which delays the queue of other sick people (the line up of people behind you at the microwave).

How Can Machine Learning Help Emergency Doctors?

First we must understand how machine learning (ML) is able to learn and become useful for doctors. In the example above it would be useful to have a “second opinion” on whether or not a patient should be sent home. This “second opinion” is typically given by another doctor but is where an ML system can step in to provide that second opinion.

A literature review of the “Influence of artificial intelligence on the work design of emergency department clinicians” [2] found that applications focused on one of 3 primary areas:

1. Predict or Identify
2. Enhance Emergency Department (ED) Triage
3. Improve Decision Making

The following chart shows these common approaches and some of their possible consequences:

This illustrates how setting the goal of machine learning systems can have dramatically different consequences, putting doctors out of work or just improving their current performance. But how do you make that decision?

Predicting Outcomes vs Improving Decision Making

The difference between predicting outcomes vs improving decision making each require their own flavor of machine learning (ML) to solve the problem.

Predicting outcomes is likely the one you’re most familiar with. For example, a ML model can be shown a large number of X-rays that are labelled with what the patient had a broken bone or not. Then when the model is shown a brand new X-ray and it will attempt to predict whether that X-ray contains a broken bone or not.

Improving decision making is more nuanced in that it provides additional information for a doctor to consider. What this can look like is given a patients information and presenting concerns what are they at highest risk of? This kind of tool can be useful as it may surface conditions a doctor did not originally consider and prevent missing indicators.

For example, in the ED the research done by Dr. Ziad Obermeyer found that typically, patients that died with in 7 days of leaving the ED were correctly discharged by common best practice standards [1]. BUT they were also able to see in the analysis that a unique combination of uncommon features indicated that person was at higher risk. For example, patients presenting with non-pain syndromes such as shortness of breath (dyspnea) were at highest risk, especially compared to pain syndromes such as chest pain or pneumonia [1].

The large number of features to consider can be seen from the risk ratios for early death for the 20 most common symptoms/diagnoses in emergency departments. The risk ratio per diagnosis or symptom means if you were diagnosed or presented with X and discharged, how frequently do people with X die vs. those that were not discharged.

So a risk ratio in the table below of ~4 for “Altered Mental Status” means if you enter the ED and with altered mental status as your primary symptom then discharged, you are 4 times more likely to die in the following 7 days than an individual who was admitted to the hospital.

What this shows is that there are ~15 different potential symptoms or diagnoses that increase patient risk when discharged.

A doctor is unlikely to be able to account for all of the minor risk categories but this is where machine learning shines by being able to reduce the cognitive load of the doctors and thus has the opportunity to assist in their decision making.

Predict or Advise, that is the Question

Setting a goal is one of, if not the most, critical part of any project. So the question becomes for ML in emergency medicine should the goal be to diagnose patients and replace doctors (predict) or assist doctors by providing more information than they would normally be able to consider (advise)?

Advising doctors is likely the most achievable option and can be seen from the number of publications on the topic from the literature review [2] mentioned above that it is where academia is focussing their efforts:

Why is 90%+ of emergency medicine and machine learning research focussed on advising instead of replacement?

For prediction of patient diagnoses there are two primary drawbacks:

1. Giving doctors a prediction of diagnoses can induce diagnostic momentum [3]
   A common form of diagnostic momentum is when a doctor receives a patient with an existing diagnosis and misses other issues because they only focused on the diagnosis provided. This can be an issue with learners or less experienced doctors who fall prey to this when diagnoses are handed off by senior doctors or supervisors. An entire area of research is ongoing in how to prevent this with more info of the different types here: First10 EM Cognitive Errors

2. It is much more difficult for ML models to predict on unseen data as they don’t know the underlying mechanisms
   If a patient presents with a new combination of symptoms the model hasn’t encountered before it’s difficult for it to come up with a meaningful prediction. A doctor is able to do this as they have the foundational knowledge of physiology and body systems, whereas the models just have the examples of inputs and outcomes from existing patients.

One of the biggest upsides to patient diagnoses prediction reducing the barriers to accessing healthcare.

• With doctor burnout on the rise there is a shortage of doctors in both urban and rural communities as well as developing countries. By being able to offload some of the work to machine learning systems an emergency doctors time can be distributed and focussed on patients with the most challenging diagnoses.

For advising doctors it is comparatively much simpler to imagine how an interpretable machine learning model could provide a doctor with the most likely outcomes for a patient along with the contributing factors. Those contributing factors can be the additional depth of information where generally inconsequential features could be missed by a doctor not using such a system.

This comparison of the two approaches is summarized in the following table:

Conclusion: We Must Walk Before We Can Run

The goal of this post was to provide a high level overview of the current state of machine learning in emergency medicine. The current state of the art is still in the early stages and there is a lot of work to be done before we can see the full potential of machine learning in emergency medicine.

Based upon the literature review, it is my opinion that advising doctors is the correct goal for machine learning in emergency medicine at this point in time. This is because it is much easier to explain the outputs of a model to a doctor than it is to explain how a model has learned medicine well enough to replace a doctor. Additionally, regulation is lagging significant advances in the technology sector and the potential for unexpected impacts on patients must be better understood before delivering a doctor-as-a-service solution.

I believe the stretch goal needs to be to replace some doctors with machine learning systems and free them up to have a larger impact on communities as a whole. The short term goal should be to reduce the cognitive load of doctors and assist them in their decision making to prove the efficacy of machine learning in emergency medicine. Building trust and incrementally building the evidence base is the best way to ensure that machine learning systems are used in a way that benefits patients and doctors.

I see the development of machine learning in emergency medicine as following somewhat of a spiral development process where you start with initial goals, iterate quickly and reassess the goals. This is a process that I have seen in my own work and I believe it is the best way to ensure that the work is done in a way that is beneficial to patients and doctors.

References

1. Obermeyer Z, Cohn B, Wilson M, Jena A B, Cutler D M. Early death after discharge from emergency departments: analysis of national US insurance claims data BMJ 2017; 356 :j239 https://doi.org/10.1136/bmj.j239
2. Boonstra, A., Laven, M. Influence of artificial intelligence on the work design of emergency department clinicians a systematic literature review. BMC Health Serv Res 22, 669 (2022). https://doi.org/10.1186/s12913-022-08070-7
3. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78:(8)775-80. PMID: 12915363