patient journey machine learning

Accessibility Links

• Skip to content
• Skip to search IOPscience
• Skip to Journals list
• Accessibility help
• Accessibility Help

Click here to close this panel.

Purpose-led Publishing is a coalition of three not-for-profit publishers in the field of physical sciences: AIP Publishing, the American Physical Society and IOP Publishing.

Together, as publishers that will always put purpose above profit, we have defined a set of industry standards that underpin high-quality, ethical scholarly communications.

We are proudly declaring that science is our only shareholder.

Machine learning in patient flow: a review

Rasheed El-Bouri 1 , Thomas Taylor 1 , Alexey Youssef 1 , Tingting Zhu 1 and David A Clifton 1

Published 22 February 2021 • © 2021 The Author(s). Published by IOP Publishing Ltd Progress in Biomedical Engineering , Volume 3 , Number 2 Citation Rasheed El-Bouri et al 2021 Prog. Biomed. Eng. 3 022002 DOI 10.1088/2516-1091/abddc5

Article metrics

6799 Total downloads

Share this article

Author e-mails.

[email protected]

Author affiliations

1 Institute of Biomedical Engineering, University of Oxford, Oxford, United Kingdom

Rasheed El-Bouri https://orcid.org/0000-0001-6494-9504

Thomas Taylor https://orcid.org/0000-0002-4017-7113

• Received 30 October 2020
• Accepted 20 January 2021
• Published 22 February 2021

Peer review information

Method : Single-anonymous Revisions: 1 Screened for originality? Yes

Buy this article in print

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

Export citation and abstract BibTeX RIS

Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 license . Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

1. Introduction

When a country's population and average age increase every year, it is inevitable that a strain is placed upon its healthcare system. This is due to the clinical attention that is generally required by older people and the increasing size of the ageing population. This is the situation faced by many countries in the world today (Andrews 2001 , Tinker 2002 , Oliver et al 2014 ). National media outlets can be particularly vocal about the performance of healthcare systems which makes the desire for a solution to poor efficiency in healthcare systems not only technically and economically desirable, but also politically important. The ability to cope with the demand for efficient healthcare has recently further been compromised due to the coronavirus pandemic that has swept the world which has shut down the normal operation of many healthcare institutions and reduced their capacity to treat patients significantly in many cases (Chen et al 2020 , Hick et al 2020 , Janbabai et al 2020 ). This has consequently increased the pressure placed on healthcare institutions as well as extending the waiting times faced by patients (Propper et al 2020 ). Despite numerous attempts by governments and hospitals to apply traditional management techniques and lean practices to improve the throughput of patients through hospitals, very little has proven effective in the long-term running of the hospital (Hall 2013 , Rutman et al 2015 ). Even fewer techniques developed have proved easily extendable to multiple hospitals as a simple solution to maximising flow throughput.

It is common today for hospitals today to have digital systems in which all patient data is recorded. These are called the electronic health records and store information on the patients passing through the hospital as well as the state of the hospital at a given time. With the abundance of this data, it has become increasingly feasible to adopt algorithmic approaches to the running of hospitals. As a result, many researchers have turned to utilising machine learning amongst other algorithmic approaches in order to tackle the issue of maximising patient flow through hospitals. In using this algorithmic approach, researchers hope to create solutions which can extend to any hospital which has an electronic health record system, thereby making their solutions 'generalisable' to the rest of the industry. In this review we aim to provide an understanding of the landscape of research that has been developed in the field of machine learning applied to the patient flow problem.

1.1. What is patient flow?

Patient Flow is a term used within healthcare services to refer to the way in which patients are moved through a healthcare facility. It involves the medical care, resources, and internal systems needed to get patients from admission to discharge while maintaining a standard of quality of care and satisfaction for the patient (Hall 2013 ). Many works have shown that patient flow can be predictable using machine learning techniques. These works aim to use these predictions to improve the flow of patients and resources in order to provide a faster and better service to patients.

2. Motivation

Patient flow is a topic that has been studied extensively by various researchers of differing backgrounds. As a result the literature associated with the improvement of patient flow is vast and a diverse range of techniques from different disciplines are employed in an attempt to tackle the problem. In this review, we will primarily focus on the history of how patient flow has been handled, as well as techniques that involve the use of machine learning methods. This is, however, by no means an exhaustive review of all methods used for the improvement of patient flow. It should also be noted that this review is not intended to summarise the machine learning methods that have been applied to patient flow or the best performing models for each task (as seen in Chen et al 2020 ) and so the performances of the models will not be included. Rather, it is to provide some structure to the field of machine learning applied to patient flow, to allow researchers to see how machine learning has already been applied to the patient flow problem and where there are (to the best of our knowledge) gaps in the literature.

While some authors have attempted to tackle patient flow as a single system through a hospital, most researchers break the problem into smaller constituent problems to tackle. These constituent parts are usually associated with the key flow bottlenecks in hospitals and these are: (a) prediction of patient admissions and demand on emergency departments (EDs), (b) prediction of flow through the emergency-to-inpatient interface (i.e. handover from ED to the hospital), (c) prediction of movement of patients (and associated resource) within the hospital and (d) prediction of length-of-stay. In this review we will discuss the work published in all of these topics and how they have been used to improve patient flow through hospitals.

Figure 1 shows the process of hospitalisation for many hospitals with an ED (although many hospitals may also receive patients from different EDs). Hospital visits can be decomposed into two overarching types of admission: elective (planned) and emergency (unplanned). It is generally the unplanned emergency admissions which cause the greatest disruptions to patient flow through hospitals (Tancrez et al 2009 ).

Figure 1.  A visualisation of the process of hospitalisation and the main considerations at each stage from a patient flow perspective.

Download figure:

Elective admissions are planned prior to their admission. As a result, the resource for these patients has been planned and there is bed space should it be needed. Elective patients have also been shown to have consistent lengths of stay in hospitals meaning they cause minimal disruption to the flow of the hospital (Kelly et al 2012 ).

Due to each emergency case being different there can be no estimate of the resource required or how long each patient will stay in hospital prior to their arrival. These therefore have become popular topics for the use of machine learning for prediction. Should these patients need hospitalisation, there is again little warning and so adapting the planning of the hospital becomes difficult.

In the following sections we will look at the work that has been carried out in applying machine learning to all of these sections of the hospitalisation process, the techniques that have been used and where we believe researchers should focus their attention on in the future to further improve patient flow.

4. How patient flow is currently managed

The effect of poor resource management on patient flow within the hospital is well known. Conceptually, high patient flow can be achieved by the effective balance of supply and demand within the system. If the supply of beds, staff and equipment is readily available to meet the needs of patients arriving at the door, then few perceivable barriers exist to prevent their immediate usage. However, studies of waiting lists have long shown that increasing supply in fact leads to a proportional stimulation of demand, highlighting the inadequacies of using relative need for services solely for the basis of resource provision (Feldstein and Severson 1964 ). If increasing supply cannot satiate demand, the optimisation of existing resources is an obvious and necessary strategy. Oredsson et al ( 2011 ) reviewed modern triage-based interventions designed to improve patient flow in emergency departments, demonstrating that the most significant improvements are observed through the use of fast-track and team triage approaches, indicating the importance of casemix as a fundamental consideration.

Current approaches to the management of patient flow in hospitals are typically driven by the need to report and improve upon key performance objectives. Within the United Kingdom National Health Service (NHS), the introduction of the Patient's Charter allowed providers greater flexibility to curate local operational policies, whilst imposing stricter performance and reporting structures across the system (NHS England 2015 ). By specifying the metrics required to deliver an adequate level of care, the identification and treatment of bottlenecks in the system naturally become a focus of attention. Such metrics are often objective and time-based, such as the time taken for acute arrivals to be admitted or discharged. Perhaps the most significant of these targets introduced within the NHS was that of the 4 h waiting limit for ED arrivals, stipulating the need to admit, transfer or discharge a patient within this timeframe (Stevens 2004 ). The most widely used approach to fulfil this target in the UK is the use of the 'See and Treat' framework, which encourages rapid on-arrival assessment of the patients needs by an individual clinician, and allows full autonomy to that clinician to decide the treatments, referrals and investigations necessary to facilitate their care, or be discharged as appropriate. Saint Lamont ( 2005 ) discussed the benefits and limitations of this approach, including the barriers to adoption observed when additional resources or suitably trained staff are unavailable.

Anecdotally, a lack of efficiency and poor patient flow is typically perceived to correlate with a reduction in staff availability. This observation is particularly valid where patient satisfaction is concerned. A study by Thompson et al ( 1996 ) showed positive overall satisfaction was associated with the perception of short waiting times and accurate information delivery, rather than actual waiting times. Whilst increasing staff within the emergency department may improve turnaround times for rapid triage and discharge of non-urgent cases, it is less likely to result in an improvement for patients requiring admission, as shown by Bucheli and Martina ( 2004 ), indicating that the true bottlenecks exist further in the pathway beyond the emergency department. This fact has been clearly recognised in recent guidance, where the focus on enabling patient flow has shifted away from the performance of the ED and towards acute networks and support services (Ham 2017 ). At the one end, Clinical Streaming has been introduced as the process by which patients are assigned to one of several parallel pathways, according to their care requirements, allowing for more structured and reliable coordination of support services within the hospital. At the other, Discharge to Assess (D2A) models emphasise the need to address unnecessary delays in discharging clinically optimised patients from hospital, due to a lack of funding or support within the community (Hyslop 2020 ).

5. Machine learning for patient admissions

5.1. prediction of emergency admissions.

The number of patient admissions to the hospital is arguably one of the most important aspects of patient flow. This determines the demand that is placed upon the hospital and therefore affects how patients can be treated. The importance of predicting patient admissions is reflected by the number of publications in this area. However, with little information on patients prior to their arrival it is also one of the most difficult areas of patient flow to create accurate predictions.

Boyle et al ( 2008 , 2012 ) and Batal et al ( 2001 ) predict the number of emergency admissions using multiple regression. They frame the problem such that they forecast for daily admissions as well as weekly and monthly admissions. The use of regression is for interpretability of the predictions as well as the development of a simple model to improve the chance of being able to generalise to other hospitals. As mentioned previously, due to limited information on these patients prior to arrival, the authors use the days of the week and national holidays as features.

Whereas the aforementioned studies approach the problem as a static prediction (i.e. using information from a snapshot in time to make predictions), Tandberg and Qualls ( 1994 ), Au-Yeung et al ( 2009 ), and Schweigler et al ( 2009 ) treat the problem as a time-series. They use autoregressive models to account for the trajectory of the numbers of patients. This approach is more likely to be successful than a static approach due to the incorporation of data close to the event of interest. However, the benefit of a static approach (if the model is accurate) is that a prediction can be made at an early stage and action can be taken based on that prediction without needing to wait for the time-series to unfold. These time-series approaches also perform regressions to predict patient volumes in the coming days, weeks and months.

While the seasonal features such as weather and time of the year have been shown to be helpful with predicting patient numbers, they are not patient-specific and therefore are limited in their use for predicting when a patient will be admitted to hospital. As a result, LaMantia et al ( 2010 ) and Artetxe et al ( 2020 ) consider predicting patient readmissions to the ED instead of predicting any given admission. In doing so they are able to utilise the wealth of data already recorded by the hospitals on individual patients and identify markers that indicate high risk of readmission in an emergency. Hosseinzadeh et al ( 2013 ) use Naive Bayes and a decision tree in order to classify patients who are going to be readmitted to hospital using their health records as features. These works generally pose the readmission problem as predicting readmission within the next 30 d as this has the most impact on the health and welfare of the patient, as well as the scheduling of the hospital (Leppin et al 2014 ).

A problem that can arise due to these readmission predictions is that patients can be readmitted for various issues (for example a patient who was hospitalised for cardiac issues might need rehospitalisation for breaking their leg). Considering this type of readmission is not very useful for the hospital or the patient, as it is not indicative of an underlying condition and so the health records of the patient will not be useful for this prediction. To get around this issue, many authors have conditioned their prediction of admission on subsets of patients with certain underlying conditions. Shameer et al ( 2017 ) use a naive Bayes classifier to predict readmission and only considers a subset of patients with heart failure. They only consider a readmission to be valid if the patients are readmitted with heart failure within 30 d. Kalagara et al ( 2018 ) also condition their problem on a subset of patients who have had a neurosurgical procedure carried out and compare the performance of their model (trained used gradient boosted trees) using features available during the patients stay versus features that were obtained after the patients discharge. Naturally the model with access to features after the patient discharge performed better, however it is very difficult in most situations to obtain features post-discharge. Min et al ( 2019 ) carry out a similar study but consider patients suffering from COPD. They investigate various machine learning methods and find that gradient boosted trees offer the best prediction of readmission accuracy for their dataset. They also utilise recurrent neural networks in order to treat the problem as a time-series problem but the performance is significantly worse. In fact there are very few works that treat the prediction of readmission as a time-series due to the difficulty of obtaining data on patients post-discharge (Arora et al 2010 ).

5.2. Scheduling instead of admissions

The ultimate aim of all of the works mentioned in section 5.1 is to provide the hospital with an understanding of the volumes of patients that may be attending the ED. By forecasting this (and if the model is accurate) the hospitals may then plan the appropriate resource (including staff, tests and making equipment available) in order to be able to cope with the demand placed on them. For low numbers forecast, hospitals may also then reduce the required resource that is on standby which can lead to cost savings (Thungjaroenkul et al 2007 ).

Some authors however approach the problem from the scheduling perspective. This is different in that whereas predicting admissions makes the assumption that resource can be altered to meet demand, the scheduling approach does not. With this approach authors assume that there is fixed resource and how it is used can be optimised with varying patient numbers.

Rosemarin et al ( 2019 ) define the ED scheduling problem as needing to satisfy the following constraints: the schedule must minimise the risk of adverse consequences, minimise patient waiting time, minimise patient length-of-stay, minimise ED crowding and minimise interruption to caregivers. They use a mixture of health record data of the patients and data on the status the ED to reconstruct the state of the ED when the patients were there. They then use a mixed integer linear program to optimise these scenarios, maximising throughput while being constrained by the aforementioned constraints. They then train a deep learning architecture on this optimised data and use it as a ranking system to predict the optimal patient-caregiver pair in the ED.

Some authors prefer to allow the machine learning algorithms to discover the optimal policies instead of optimising the problem themselves to learn from. This is seen in Lee and Lee ( 2020a ) where a deep Q network (a reinforcement learning algorithm) is used to learn the optimal policy of treating patients in the ED. In order to do this a simulation is made of the ED which will allow the agent to take exploratory moves essential for reinforcement learning. The state of the model is defined as the distribution of acuity (sum of patients at each acuity level) within the ED as well as the distribution of needed treatment type. The action of the agent is to rank the next patient that needs to be seen meaning it is also a patient priority-ranking system. Krämer et al ( 2019 ) also present a priority ranking system based on severity prediction, but go as far as predicting whether patient presentations to the ED should be treated as elective visits given their low severity. They do this using the primary diagnosis code of the patient, however there may be difficulties in expanding this tool to other hospitals given that many hospitals assign diagnosis codes after the patient is discharged from hospital and not at admission.

Yeh and Lin ( 2007 ) and Arisha and Abo-Hamad ( 2013 ) approach the scheduling problem slightly differently in that instead of ranking the priority patients in the ED, they instead aim to design the staffing schedules. They do this using genetic algorithms and allowing the staffing schedules to be updated and 'evolve' to a point where they are suitable for the demand placed on the ED. This approach makes the assumption that should the staffing level be predicted accurately, then there will be no need to prioritise patients in the ED as there will be enough staff (and resource) to process them.

Figure 2 shows the works that have been conducted so far on the prediction of admissions and scheduling in the ED. This is by no means an exhaustive summary but we aim to provide some structure to help other researchers understand what work has been conducted in the field of machine learning for patient flow through the ED. Table 1 further outlines the problems that have readily available datasets for prediction, and what models are popularly used to tackle the prediction problem in the literature. A lack of a readily available dataset for priority ranking is due to priority generally not being recorded in hospital EHRs. Readily available in this instance refers to existence in a typical hospital database and not that it is easily and openly accessible.

Figure 2.  Visualisation of the studies that have been carried out regarding using machine learning to predict admissions and scheduling in the ED. Dashed lines indicate some studies opt to use these features.

Table 1.  Popularity of different methods and data availability for each of these problems.

5.3. Machine learning in elective admissions

We have primarily focused on machine learning applied to emergency admissions as this is the larger body of research in the field. The stochastic nature of these admissions in terms of number and type of admission means that these are the most disruptive to patient flow in a hospital. Elective patients are generally planned for and so resource is available to treat them.

There are however studies that also apply machine learning to the admission of elective patients. In the study conducted by Nelson et al ( 2019 ), the authors use machine learning to assess whether or not patients will actually attend their scheduled appointments in hospital. Despite the resource being prepared for these patients, a no-show will result in a waste of this resource and this work aims to provide a way to then re-direct that resource. The authors use information on the history of the patient with a gradient boosting machine to get a strong predictive accuracy. Srinivas and Ravindran ( 2018 ) carry out this same prediction of no-shows to elective appointments, however they then leverage the risk of no-show in order to update the scheduling system of the hospital.

With many health systems providing long waiting times for appointments (Xavier 2003 , Dimakou 2013 ), another important factor when it comes to elective patients is prioritising patients in the schedule. Yousefi et al ( 2019 ) approach this by first using a clustering algorithm to group patients into different priority categories. They then treat the schedule as a Markov decision process where waiting time for patients in the high priority clusters is to be minimised.

These approaches can be difficult to validate due to their direct impact on the scheduling of appointments. As a result, there is no chance to verify if the patients turn up or not once the schedule is changed. They also rely on historical behavioural data (such as how many times a patient has missed an appointment before) which are not stationary distributions and therefore limit how successful supervised learning can be in this domain in the long term.

5.4. Summary

Overall, the application of machine learning to predicting emergency patient admissions and scheduling is well-explored. Works are generally split between emergency and elective patients with further subdivisions according to the data used, the models used and what is being predicted (see figure 2 ). Very few works validate their models in hospitals in real-time, most using a retrospective test-set to assess performance. Furthermore, some models are difficult to validate due to being designed to intervene in the admission and scheduling process.

There is also very little connecting these studies. Most work is carried out with the data from the hospital that the authors are associated with and built around that. Due to hospitals being different, that leaves little scope for building on previous work or developing models that can be used universally. A public dataset that could be held as the gold-standard for patient flow would aid in this significantly as a benchmark for experiments.

6. The emergency-inpatient interface

The emergency-inpatient interface is an ill-defined area of many hospitals (Staib et al 2017 ). There is usually a lack of clarity on the ownership of this space of the hospital and who should manage the handover of patients from the ED to an inpatient setting. As a result of this lack of clarity, it should come as no surprise that there is much published on making predictions across this gap in the hospital. While it may seem like an obvious task to predict which patients need admission to hospital from the ED, it has been shown that this is not a trivial task (Beardsell and Robinson 2011 ). Whereas the works discussed in section 5 aim to provide predictions for planning (such as expected numbers or schedule planning), the predictions of the works found in this section are primarily designed for decision-support.

A natural question that can be asked is if admission to the hospital from the ED can be predicted. Hong et al ( 2018 ) and Graham et al ( 2018 ) show this can be done using multiple machine learning models including a logistic regression, XGBoost and a deep fully-connected neural network. They show this is possible using historic patient information as well as information from triage. This does however limit the potential use to patients who already have electronic health records. Leegon et al ( 2005 ) and Raita et al ( 2019 ) therefore also carried out this prediction but only using a few variables that are measured early in the ED admission process and showed using a Bayesian network that admission to hospital can still be accurately predicted. Sun et al ( 2011 ) echo this sentiment, setting up their classification such that the clinical staff may predict the risk of whether an inpatient bed is needed or not as soon as triage is complete in the ED. This prediction is then further augmented with the inclusion of using the free-text written by the triage clinical staff as features to improve the performance of the model (Zhang et al 2017 , Sterling et al 2019 ).

As was the case for prediction of admissions in section 5 , many authors find it useful to consider certain demographics of patients. An example is in Lucke et al ( 2018 ) where a multiple logistic regression is used to predict hospital admission from the ED for a cohort of patients over 70 years old and another below. This is due to older patients generally being more at risk of admission and so by creating a model conditioned on age, they are able to better predict those most at risk of admission. In Mowbray et al ( 2020 ), elderly patients are considered to be those aged 75 and over, however they also show that accurate predictions of admission can be made for an elderly cohort of patients.

Another demographic that is often targeted for prediction is that of paediatric patients (Walsh et al 2004 , Marlais et al 2011 ). In these studies, logistic regressions are used to predict whether a paediatric subset of patients will require admission to the hospital. Once again, by creating a separate cohort for these patients, they can make predictions comparing patients to other similar patients, rather than comparing with older patients who have different physiologies. This introduces a trade-off of improving model accuracy while reducing how generally the model can be applied.

Further subsets of paediatric patients have been made for example by considering those patients suffering from asthma exacerbation and predicting those most likely to be admitted to hospital for treatment (Patel et al 2018 ).

To augment the performance of a model predicting paediatric admissions to hospital from the ED, the textual data recorded during triage can also be used as features (Roquette et al 2020 ). Natural language processing techniques have been used in order to extract useful information which has been shown to improve predictability of admission.

6.1. Predicting inpatient resource utilisation

Many of the studies that are created in predicting admission to hospital focus on subsets of patients with certain conditions. As these patients will require the same treatments and specialist staff to treat them, this can be seen as resource prediction for patients being admitted to the hospital from the ED.

An example is in Ong et al ( 2012 ) where heart-rate variability in the ED is used alongside other demographic information on the patient as input features to a support vector machine. This is then used to create a score on the likeliness of cardiac arrest occurring in the next 72 h. While this is not strictly framed for patient flow, this prediction allows clinicians to plan for resource in the cardiac department. Predicting whether or not a patient is septic is also important for patient flow in terms of resource planning. As a result, models predicting whether or not ED patients are suffering from sepsis have been developed (Horng et al 2017 , McCoy et al 2017 , Delahanty et al 2019 ). The authors use a mixture of information available at triage, demographic information and free-text to make prediction of whether or not the patient is septic, which if accurate, could allow planning of their treatment before the patient becomes critically ill.

In fact, there have been many such studies predicting whether or not a patient is suffering with a certain condition in the ED which allows resource planning. These include predicting if a patient is suffering from acute kidney injury (Martinez et al 2020 ), requires intensive care (Fernandes et al 2020 , Finkelstein 2020 ), is suffering from a urinary tract infection (Taylor et al 2018 ), have bacterial infections (Ramgopal et al 2020 ) as well as predicting emergency hospitalisation of patients undergoing chemoradiation (Hong et al 2018 ).

While these predictions are useful for planning patient flow, they are not explicit predictions of admission. A more explicit approach is seen in Luo et al ( 2019 ) where the classifier is trained to predict admission to hospital of patients suffering from bronchiolitis.

While predicting admission to hospital from the ED is useful, a greater level of granularity, such as which departments in the hospital the patient will be admitted to, is more useful to clinical staff. An example is seen in Lee et al ( 2020 ) where rather than predicting admission, they predict the disposition of the admitted patient, choosing out of intensive care units, telemetry units, general practice units and observation units. As these 'ward types' tend to have separate resource, they are better able to adapt their resource according to the predictions made. This approach is also seen in El-Bouri et al ( 2020 ) where the authors also classify into 'ward types' to provide a similar level of granularity to the hospital admission prediction problem. However, in this case they use medical, cardiac, neuro, trauma, intensive care, surgical and general/obstetrics and gynaecology as their ward groupings. They develop a novel 'interpretable' layer for their deep neural network to guide information collection at triage and train the model using curriculum learning. El-Bouri et al ( 2020 ) further augment their model by using reinforcement learning to allow an agent to carry out the curriculum learning that maximises the performance of predicting where in the hospital patients will be admitted to. In order to make as general a model as possible, these studies of patient disposition do not consider subsets of patients but rather the entire population of the ED to replicate daily working conditions.

Figure 3 shows the general structure of works that have been conducted on predicting flow from the ED to hospital. It should be noted that as all of these works consider flow from the ED, all patients considered are emergency patients. Table 2 shows how readily available labelled datasets are for the EDii prediction problems and the popular approaches to tackling them. It should be noted that readily available here means data that would generally be saved on a hospital EHR and not data that would be easily accessible on a public dataset.

Figure 3.  A decision tree showing how the studies that have been conducted on predicting movement from the ED to hospital are structured. Dashed lines indicate that these features are used in some works but not all.

Table 2.  Popularity of different methods and data availability for each of these problems.

6.2. Summary

We have seen in this section that the emergency-inpatient interface in hospitals, while being ill-defined in practice, is well researched using machine learning. Authors predict admission from the ED in order to provide information for clinical staff to prepare space should it be needed. To improve the performance of the classifiers, many authors condition their models on the demographics of the patients (e.g. elderly or young patients) or on the patient disease (patients suffering from the same ailment in the ED). In order to provide greater granularity on which resource will be used in the hospital, some authors also predict which 'ward type' will be used by the patient to be admitted to the hospital.

However, once again there is little connecting these studies. None of the studies reviewed build off each other or use the same dataset for comparison. Furthermore, the definitions used to categorise patients vary by paper. As was seen when categorising elderly patients some studies use 70 and over and some use 75 and over. Clearly it would be beneficial to have an agreed range to make models more comparable. This further emphasises the need for a shared, publicly available dataset for use when creating machine learning models for patient flow. All definitions of demographics should be included in the dataset so that researchers make valid comparisons to models. It will also be beneficial in allowing researchers to compare their methodologies and validate them on the same dataset as others as well as apply them to their own hospital's data. This will also make research more consistent, allowing researchers to build and improve upon each others models instead of applying similar models to similar problems using different data.

7. Intra-hospital resource management

Once patients have been admitted to hospital, there is yet another layer of resource flows that need to be considered. Patients can be transferred between wards, need tests carried out and must be moved to use certain equipment such as MRI scanners. These all require staff to carry out the movements and therefore place a demand on the resource of the hospital. As this resource is part of that needed to deliver the patients through hospital to discharge, it is relevant to patient flow.

7.1. Ward transfer

The most common way in which machine learning is used to provide predictions for 'inpatient flow' is through predicting if patients will be transferred to another ward. Note that while in section 6.1 we considered studies which investigated patient degradation as a signal for resource preparation, we will not consider degradation for inpatients as a signal for resource prediction. This is due to hospitalised patients generally being admitted to wards that are capable of handling patients in their condition. It is also due to the fact that using machine learning for the monitoring of inpatients for degradation has a very rich literature and would require a review of it is own (Clifton et al 2015 ). As a result, we only focus on works that explicitly predict admissions or transferrals of patients.

7.1.1. ICU transfer

By far the most popular type of prediction to make in the inpatient setting is predicting admission of a patient to the ICU. This is due to the fact that the ICU is a resource intensive area of the hospital and any way of informing the planning of this unit is beneficial to the running of the hospital (Skowronski 2001 ).

Wellner et al ( 2017 ) use a logistic regression to show that it can be predicted that a patient will need admission to the ICU 16 h ahead of time. Furthermore they demonstrate this using data from three separate institutions, helping validate their model. Desautels et al ( 2017 ) carry out the same investigation in a tertiary care hospital but consider readmissions to the ICU in 48 h. This is also explored by Yoon et al ( 2016 ) who develop a 'Bayesian belief system' to predict admission to the ICU, but this time 9 h before it is requested by the clinician in charge. An NLP approach has also been investigated in Khattak et al ( 2019 ) where the online messages of doctors and nurses to each other are used in order to predict transferral of a patient to ICU 3 d prior to the event taking place. It should be noted that for all of these studies, the outcome being predicted is different and so the studies cannot be compared.

Echoing the narrative presented in section 5 , many researchers have also considered predicting readmissions of inpatients to the ICU. This is seen in Rojas et al ( 2018 ) where the authors investigate which patients, who were previously in the ICU, will be readmitted from their inpatient ward. To predict this they use a gradient boosting machine with features derived from the electronic health record of the patient as well as various blood tests that were taken. A time-series approach to this prediction was investigated by Lin et al ( 2019 ) where an LSTM was used and trained on the ICD-9 embeddings of the patients who had previously been admitted to the ICU, their demographics and the chart event features of the patients. They show a strong prediction accuracy when considering if a patient will be readmitted to the ICU within 30 d of their discharge.

Once again, conditioning the dataset on the demographic in question is utilised for the inpatient setting. Rubin et al ( 2018 ) demonstrate using adaptive and gradient-tree boosting that they can predict the transfer of a child to the paediatric ICU 8 h preceding the transfer. The prediction of transfer to paediatric ICU is also carried out in Zhai et al ( 2014 ) where a logistic regression is used to predict their transfer within the first 24 h of their inpatient status.

We again see works where the datasets (and therefore the models) are conditioned on the co-morbidities of the patients. Lee et al ( 2019 ) condition their dataset on patients who have undergone cardiac surgery and predict whether these patients will be readmitted to the ICU. They use a logistic regression with L1 regularisation to provide interpretability to their model, but also use a causal inference method to compare their findings. They find that there is little agreement between the two methods of feature importance ranking.

7.2. Resource management

During a patient's stay in hospital, various tests may be requested to help clinicians gain a better understanding of the patient's condition. These tests are also an important part of the patient flow process and timely testing helps to improve flow through the hospital. An example is seen in Molaei et al ( 2016 ) where the authors investigate whether or not they can predict if inpatients with traumatic brain injury require a CT scan using 'cost sensitive' random forests. In doing so, they aim to create a prioritisation system for scanning, which will allow faster treatment of patients and therefore a better patient throughput.

Another way in which resource management has been tackled with machine learning is in the scheduling of laboratory samples that need to be processed (Williams et al 2019 ). Again, by scheduling these samples in an efficient way, this allows patients to be treated more quickly in the hospital, and in some cases prevents the unnecessary hospitalisation of a patient.

These examples can be seen as assessing the risk of resource utilisation on a patient-by-patient basis. A more high-level view is used in Vieira and Hollmén ( 2016 ) where all resource is pooled together (anything including staff or use of machinery). Random forests are used to perform regression on the expected resource use in the next 30 d. While this has limited use to clinical staff due to the lack of granularity, it may be useful for budgeting purposes.

7.3. Hospital-wide flow

There are very few works that seek to predict the full patient journey through a hospital using machine learning. This may be due to the fact that transfers of inpatients is generally quite rare due to most inpatients being admitted to a ward that is capable of providing the appropriate care for them. Xu et al ( 2017 ) treat the hospital journey as a point process. They use a generalised linear model to predict the next location a patient will be transferred to as well as the dwell-time in that unit. They utilise the MIMIC-III dataset (Johnson et al 2016 ), which is an ICU based dataset and so the transitions they predict are between various types of intensive care unit. However, in terms of predicting the inpatient journey, this is a promising direction. Expanding to the entire hospital, it is possible to predict movement of patients between wards as well as for the use of machinery. Also predicting the dwell-time will allow for better planning of the flow of patients.

7.4. Summary

Of the four parts of the hospitalisation process that we have defined, the inpatient setting is the one in which machine learning has been used the least. The majority of studies investigate the transferral of inpatients to the ICU due to the resource-intense nature of ICUs. There have also been limited attempts at utilising machine learning to predict the expected resource that will be required by a hospital, either as a whole, or on a patient-by-patient basis. Very few works again have attempted to predict the whole hospital journey using machine learning. A common inconsistency throughout the literature is the prediction lookahead time that is considered. Standardising the lookahead time will allow studies to be more comparable and again, crucially, build upon previous work to further improve and integrate the field.

As the vast majority of studies are conducted with a clinical need in mind, this may reflect that the inpatient journey is not seen as a very important part of the patient flow problem. Figure 4 shows the structure of the studies that have been carried out in this area of patient flow. Table 3 shows the data availability for these prediction problems and popular methods used to tackle them.

Figure 4.  Visualisation of the studies carried out on using machine learning to aid in the inpatient journey. Dashed lines indicate some studies opt to use these features.

Table 3.  Popularity of different methods and data availability for each of these problems.

8. Discharge prediction

The importance of discharging patients in a timely fashion for patient flow cannot be over-stated. Long patient stays incur greater cost to the healthcare institution and reduce capacity for new patients to be admitted (Rotter et al 2008 , 2010 ). As a result, a standard metric of the quality of care being provided is the patient length-of-stay (LOS) (Brasel et al 2007 ). Patients who are admitted for long periods of time (either due to condition or due to having no appropriate discharge destination) are commonly referred to as 'bed-blockers' and can constitute a significant proportion of the hospital population (Coid and Crome 1986 , Styrborn and Thorslund 1993 , Mustafee et al 2012 ). Early recognition of the patients likely to have a long LOS should therefore allow for the planning of their treatment by the hospital, such as their admission to long-stay wards and beginning preparations for their discharge.

It should therefore be unsurprising that many researchers have seeked to employ machine learning in order to predict the LOS of patients in order to provide hospitals with a better idea of how much resource will be required for patient stays. Note that the prediction of LOS or of discharge are essentially the same as they both aim to predict when a patient is able to leave the hospital. We will refer to both types of prediction simply as 'discharge prediction'.

Discharge prediction can be separated into two separate subcategories for emergency and inpatient settings. In the emergency context, predicting the LOS of patients helps to understand whether the ED is at risk of overcrowding or not. In the inpatient setting, predicting the LOS is useful for the planning of patient admissions and preparation of post-discharge care should it be needed.

8.1. Discharge in the emergency department

Discharge from ED has been treated as a classification as seen in Rahman et al ( 2020 ). The authors predict if a patient will be in the ED for longer than 4 h or not. They use features that are available early in the ED process to train a decision tree binary classifier. This approach is mimicked in Sariyer et al ( 2019 ) where various learning algorithms are experimented with to classify patients according to their length of stay in the ED. Azari et al ( 2015 ) acknowledge the large imbalance there tends to be in LOS datasets (with far fewer patients having long LOS), and present an ensemble method combined with multiple logistic regression to overcome this imbalance. However in this work they define a long stay as patients in the ED for longer than 14 h.

Rather than classify patients according to their likely LOS category, some authors prefer to use regression to predict each patient's LOS in the ED. Combes et al ( 2014 ) use linear regression model to predict the likely LOS of each patient presented to the ED. Ding et al ( 2010 ) instead use quantile regression but once again for the prediction of LOS in the ED. Feedforward neural networks have also been used for regressing the likely LOS of patients (Gül and Güneri 2015 ). One advantage to this approach of regressing the probable LOS is that there are no longer inconsistencies between studies on what is defined as a long-stay. However, this approach is also more difficult to train and achieve an accurate model in practice.

8.2. The inpatient setting

Predicting the LOS of patients in the inpatient setting is significantly more popular as a research area than in the emergency setting. This may be due to a prediction of LOS in the ED being less actionable than in the hospital where preparations can be made to ready a patient for discharge.

A hospital-wide approach is adopted in Pendharkar and Khurana ( 2014 ) where a regression tree is used to predict the LOS of patients admitted to hospitals in Pennsylvania using data that is available at the time of admission. This approach is also applied in Tanuja et al ( 2011 ), this time using a feedforward neural network to regress the LOS. These predictions are carried out at the time of admission. An alternative approach is to implement a classifier every day before discharge and predict the patients who can be prioritised for discharge as seen in Barnes et al ( 2016 ). In framing the problem in this way the authors exploit a static model for a dynamic problem by repeatedly applying the algorithm prior to discharge sessions at the hospital. They use a classification decision tree to prioritise patients ready for discharge.

Predicting discharge has also been approached as a time-series problem. In McCoy et al ( 2018 ) an autoregressive integrated moving average model is used to incorporate a time-series of seasonal data to predict hospital discharge volume. They compare this with using Prophet (Taylor and Letham 2018 ), an additive regression model developed by Facebook Research for forecasting seasonal trends, for the same task. An NLP approach has also been used where the clinical notes from the ED are used in order to predict if a patient will be admitted to the hospital for more than 2 d (Bacchi et al 2020 ).

As has been a common theme throughout this review, discharge predictions are also conditioned on patient demographics. In other sections this is primarily to improve predictive performance amongst patient subgroups. However, in discharge prediction this is due to certain patient subgroups being more likely to be 'bed-blockers' such as elderly patients (Launay et al 2018 ). To maximise clinical utility it is more effective to condition the training dataset on these subgroups and apply the algorithms to these patients only. An example is in Elbattah and Molloy ( 2016 ) where a regression forest is used to predict the LOS of elderly patients in a hospital and a random forest is used to predict the location of discharge for these elderly patients. These predictions are used in conjunction with a discrete-event simulation in order to simulate the flow through an Irish hospital. Children are also a cohort of patients in which there can be great variability in LOS. To address this, Castiñeira et al ( 2020 ) use a gradient boosted tree to classify whether or not a child will be a long-stay patient in the paediatric ICU (with long-stay being defined as a stay of greater than 4 d). They also use the static model for a dynamic problem approach by extracting features from the time-series of the patient's vital signs and repeatedly feeding these to the classifier. Note that this prediction concerns the LOS within a ward and not the hospital stay as a whole.

As with conditioning on demographics, conditioning on co-morbidities is also done in discharge prediction. In fact, this tends to be the most popular form of setting the problem due to patients with different ailments and treatments generally requiring different recovery times.

One such prediction is carried out for patients with congestive heart failure in which the authors apply a static cubist model (Quinlan 1998 ) dynamically as data is updated during the patient stay (Turgeman et al 2017 ). The model is used to regress the likely LOS in hospital of the patient.

Further discharge predictions have been carried out on patient cohorts who have suffered from stroke (Al Taleb et al 2017 ), patients who have suffered hip-fracture (Elbattah and Molloy 2016 ), patients suffering from schizophrenia (Kirchebner et al 2020 ), patients admitted for cardiac care (Daghistani et al 2019 ), patients post-brain tumour surgery (Muhlestein et al 2019 ), patients who have undergone total hip-arthroplasty (Ramkumar et al 2019 ) and patients who have undergone surgery due to colorectal cancer (Stoean et al 2015 ). In all of these studies, there is no consensus for defining a 'long-stay' patient.

8.3. Summary

Discharge prediction is one of the more popular areas of patient flow for researchers to apply machine learning. Discharge prediction has been carried out by either predicting whether a patient is likely to be long-stay or by directly regressing the expected LOS of the patient. It has been applied to both emergency and inpatient settings. In the inpatient setting, studies have conditioned their datasets according to demographic. There have also been studies that condition their dataset according to the comorbidity or treatment that the patients of interest have undergone.

A clear inconsistency between studies is the definition of a long-stay patient. Having a common dataset with pre-defined long-stay patients will improve the ability of researchers to compare models and build upon previous work. Figure 5 shows the structure of the literature published in this field. Table 4 shows data availability and popular methods used to tackle the discharge problems. It should be noted that the difficulty with a labelled dataset for discharge readiness is that generally it is not recorded when a patient is ready for discharge but when they actually are discharged.

Figure 5.  Visualisation of the studies carried out on discharge prediction. Dashed lines indicate some studies opt to use these features.

Table 4.  Popularity of different methods and data availability for each of these problems.

9. The future of machine learning in patient flow

The current research efforts in the discipline of machine learning in patient flow have demonstrated the feasibility and potential of machine learning to optimise patient flow in all of the four subcategories outlined our study. However, due to the difficulty of expanding and scaling machine learning models across different healthcare contexts and institutions, the current research efforts are still removed from delivering value in the routine and daily management of patient flow in healthcare institutions. In this section, we outline the future research opportunities to advance the applicability of machine learning in patient flow.

9.1. Priorities in patient flow

While all of the problems outlined in the above review are important for clinical practice, solving some of these problems is more urgent than solving others. An example of a high-priority problem to solve is predicting readiness for discharge. One of the greatest problems dealt with in patient flow is the 'bed-blocker' phenomenon whereby patients do not have appropriate destinations to be discharged to. Predictions of readiness for discharge will not solve the lack of space in care homes, however it will allow for more effective allocation of the time and attention of clinical staff.

An equally important task is the prediction of ED admissions. This represents the front-end of the hospital with the discharge readiness representing the back-end. Being able to accurately predict patient admissions numbers in the ED would allow for accurate planning of staffing rotas thereby reducing costs and time wasted. It would also greatly improve the care provided for each individual patient.

Following on from this, should these predictions not be accurate enough, solving the ED-inpatient interface problem would be the next most important. This prediction would prevent the filling up of the ED due to inability to transfer patients into the hospital. Having an accurate model here would create a more streamlined flow of patients into the hospital, but naturally would depend on there being enough flow out as well.

Finally, the problem that should be least prioritised is inpatient transfer prediction. Despite being important, inpatient transfers are generally quite rare due to patients being admitted to appropriate wards from the outset. However, there is value in predicting resource flow and patient movement in order to plan that resource.

9.2. Current challenges

9.2.1. data challenges.

Throughout this review, we have emphasised the need for a common dataset that all researchers can use to benchmark their models and experiments on, as well as have agreed definitions of what age ranges 'elderly' patient fit amongst other definitions. However, creating a publicly available dataset does not come without its own challenges. The first issue is that of patient privacy. While there are many data anonymisation methods that can be used to remove association of the data with individuals, prior information such as the source hospital can be used to reconstruct the identities of the patients. There then exists a trade-off between how much information is hidden and how useful the data is to machine learning practitioners. A potential solution for this is sourcing data from multiple medical centres and compiling them together in a dataset. This brings us onto the second challenge which is a lack of standardisation in the recording of health data. In order to take advantage of the data from the EHRs from multiple hospitals, we must first stipulate that these hospitals record data in an agreed fashion.

One example of a publicly available healthcare dataset used for benchmarking is MIMIC-III (Johnson et al 2016 ). The success of this dataset can be seen through the volume of works that have used it for model comparison. However, for the purposes of patient flow, this dataset is difficult to use due to its focus on intensive care patients. It therefore does not include the data from the EHRs on the key resource utilisation and patient flows in the hospital (unless they are between intensive care units). A dataset built in a similar fashion to MIMIC-III but with the appropriate patient flow data would benefit the research community greatly.

9.2.2. Technical challenges

Currently the majority of patient flow models use a specific dataset from a hospital that can be derived from a certain subset of patients. The model is then applied to aid that hospital in prediction with very few researchers extending their models beyond their own hospitals. This approach is limited due to the variable and dynamic nature of healthcare datasets. Distributions from the same source hospital are subject to issues such as covariate shift whereby the underlying distributions of the features change with time. Examples are the changes to the distributions that can be found in the EHRs of hospitals during flu season or during the COVID-19 pandemic that has swept the world.

Variability also exists across health care delivery institutions and organisations ranging from small primary care centres to large tertiary hospitals. These organisations are different in their resources, organisational structure, staff training, and culture. These differences create variability in healthcare delivery practices, organisational processes, and patient flow across these different institutions as well as variability in what data is recorded and in what format it is recorded.

Differences also exist in the distributions recorded by healthcare institutions due to the differences in populations across the world. Examples include the prevalence of different diseases across different communities and geographical contexts (e.g. the presence of type II diabetes mellitus can vary from 3.5% to over 20% across different populations) (World Health Organization 2016 , James et al 2018 ).

Another issue that is faced is the lack of complete information delivered by the majority of prediction algorithms. While it is useful to know that a patient will be admitted to a certain location in the hospital, having some knowledge of their severity or the likely medications that will be needed for them will further help with the planning of their stay.

These issues faced during deployment create challenges in the applications of machine learning, particularly in the generalisation of models to other hospitals and for their continual use over long periods of time.

In the face of these challenges, we believe that certain research directions will aid future researchers to prepare models that will better serve hospitals to improve patient flow. These research directions should address the issues discussed above, as well as ensure that they integrate seamlessly into the running of the hospital.

9.3. Feature engineering

The majority of studies discussed in this review take advantage of the fact that there exist EHR systems in many modern hospitals which allow data extraction and dataset creation. However, there remain challenges in terms of data collection for the different tasks at hand.

The ED admission prediction relies on seasonal information which can be correlated with admissions but is generally a difficult prediction to make. Wearable sensors could benefit this prediction greatly, providing more granular information to the hospital. The sensors could also be provided to patients who need them most (and are most likely to be brought to the ED in an emergency such as elderly patients in care homes).

We believe that further improvements to data collection could be made in the inpatient journey as well as in discharge prediction. Currently, while scans in the hospital are logged on the EHR, the movement of patients to scans are not and nor is the resource associated in moving that patient. These data would be very helpful to provide a more complete picture of what resource each hospitalised patient utilises and thereby helping machine learning scientists create more accurate predictions of the likely resource needed.

In discharge prediction, one of the challenges is that it is generally not recorded when a patient is medically ready to leave the hospital but when they actually do. Augmenting a dataset with this information could help predict when a patient is ready to leave hospital and in doing so, allow the team looking after them to move their resource to more vital care, with a more generalised team looking after the patient thereafter until discharge.

9.4. Multitask learning

The first research direction to be considered is multitask learning, a machine learning method that allows multiple tasks to be learned at the same time. One of the aims is to exploit the learning signals generated by training on one task to create an inductive bias in the model that will allow the effective learning of another task by the same model (Caruana 1997 ).

Multitask learning can be applied to different problems across the four domains of patient flow to both related (e.g. predicting risks of various in-hospital complications) or unrelated tasks (e.g. predicting length of stay in the ED and predicting hospital admission destination). Once again this relates to the usefulness of having more granular information for clinicians to work with. An example may be when predicting the location of admission of a patient to hospital, also having some prediction of whether the patient is likely to deteriorate or not. This gives better indications of the likely resource requirement of the patient as well as their likely trajectory within the hospital. While this could be done using separate models for each prediction, a single model that can embed an accurate representation of the patient will be more informative and useful to clinical staff. As a result, a key component of this work will be in the development of representation learning algorithms (Bengio et al 2013 , Van Den Oord et al 2017 ) that are capable of representing patient conditions upon presentation to the ED or admission to hospital.

Multitask learning has been applied in many healthcare applications to leverage the shared information across different tasks. Huang and Dong ( 2018 ) have used multitask learning to predict major adverse cardiac events, identifying each type of adverse event as a single task as opposed to having a multiclass classification. Xia et al ( 2019 ) have also used this approach to predict prescription patterns for various drugs that are given to similar patients. Multi-task learning has also been used in medical imaging. Khosravan et al ( 2019 ) have used multitask learning in the detection of abnormal nodules on chest CT scans for lung cancer screening. They jointly train their model to segmenting potential abnormalities and identify the presence of a nodule in the region of interest. This is further evidence of how more granular information from the model can provide clinicians with better insights into the condition of the patient.

9.5. Transfer learning

Transfer learning is based on the principle of knowledge transfer across different machine learning tasks and models. It is based on the notion that knowledge gained by the algorithm when trained to solve a particular problem can be stored and applied to solve another related problem, which means it is closely related to multitask learning. This approach includes transferring knowledge from the source domain, D S , to the target domain, D T , to help improve the learning of the target-domain task, T T .

Transfer learning can provide significant advantages in the applications of machine learning in patient flow. It can enable (a) the transfer of knowledge across different tasks and (b) the transfer of knowledge across different populations. The former can help overcome the lack of clinical data for certain problems. For example, one of the barriers to developing effective machine learning tools for COVID-19 patients is the lack of data on COVID-19 patients. A transfer learning approach can provide a solution by using a model that is pretrained on a large non-COVID-19 dataset and adapting it to perform the task of interest in COVID-19 patients. Transfer learning has been used to overcome the lack of COVID-19 imaging data by Mahmud et al ( 2020 ). They trained a convolutional neural network by using a pretrained a neural network (pretrained on a dataset of bacterial and viral pneumonia chest x-ray scans) and fine-tuned this using scans from COVID-19 patients. This was done due to the scarce availability of chest x-rays from these patients.

Transfer learning can also help us transfer knowledge across different populations. This is valuable clinically given the diversity and differences in the genetic predispositions, prevalence of diseases, lifestyles, and risk factors across different populations. Mao et al ( 2018 ) used transfer learning to generalise their sepsis prediction algorithm to a new healthcare setting. They trained their prediction model using data from the MIMIC-III dataset (data from ICU patients) and transferred the model to a dataset from the University of California, San Francisco (UCSF) Medical Centre (a dataset of in-hospital patients from a variety of specialty wards). Their transfer learning approach was based on adding incremental amounts of data from the UCSF dataset to the MIMIC training dataset, resulting in better generalisation of the model to the dataset being introduced.

Transfer learning represents an interesting target for future research in patient flow machine learning applications. Transfer learning can be used to generalise models across different healthcare contexts and to overcome a lack of recorded data.

9.6. Continual learning

Continual or lifelong learning refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experiences. This approach has the potential to enable machine learning models in the healthcare space to adapt and adjust automatically to new context and settings like a new healthcare context, new patient population, or a new and emerging disease. This has the potential to enable the creation of dynamic clinical AI models that optimise clinical management decision in real time and learn from the continuous influx of information in real world healthcare context. A continually learning algorithm should be an adaptive algorithm capable of learning from a continuous stream of information, with such information becoming progressively available over time. The accommodation of new information should occur without catastrophic forgetting or interference (Parisi et al 2019 ).

However, continual learning represents a long-standing challenge due to the susceptibility of machine learning models to catastrophic forgetting. This phenomenon refers to the decrease in model performance or the complete overwriting of the previously learned information when new knowledge is introduced.

A paper published in the Lancet in 2020 (Lee and Lee 2020b ) highlights the promise of continual learning in revolutionising the applications of clinical AI and leveraging the continuous influx of clinical information to improve patient care. Shah et al ( 2019 ) highlight that machine learning algorithms that are capable of continuous learning are a critical future research and translational direction in healthcare AI. They also report that the FDA is considering widening its regulatory framework to include AI-based Software as Medical Device (SaMD) systems that are capable of continuously learning and optimising performance in real-time to improve patient care.

Continual learning promises considerable value in patient flow as it would enable machine learning models to adjust to different healthcare settings continuously and automatically. Therefore machine learning algorithms would be able to absorb the variation across different healthcare institutions and patient populations. Moreover, continual learning may enable machine learning algorithms to continuously learn after deployment to clinical settings gradually improving their performance through use.

10. Conclusion

We have seen in this review that machine learning in patient flow is a vast if disjoint field. There are many works published with the majority focused on the hospital associated with the authors and little by way of comparison to other hospitals or works. We therefore propose the introduction of a publicly available dataset based on the electronic health records of a given hospital. This should include enough information on all four subcategories of the patient flow process (as highlighted previously) and crucially, must have strict definitions for patient types. The dataset should include:

• Seasonal information such as the weather, national holidays and ideally EHR data from multiple hospitals.
• Strict definitions of what age ranges 'elderly' or 'young' patients fall into for reproducibility and model validation.
• Pre-defined tasks such as 'prediction of patient transfer in 3 h from time of measurement'. By creating these pre-defined tasks we improve the ability of researchers to benchmark against each others work and develop upon each others models.
• A standardised definition of co-morbidities in patients.

We believe that in creating this dataset, a culture of benchmarking on the dataset can be created thereby encouraging researchers to compare their models, build more sophisticated models based on previously published work and crucially provide some external validation to the trained models.

Acknowledgments

ReB is supported by the EPSRC industrial strategy award. T T is supported by The Wellcome Trust [200205/Z/15/Z]. A Y is supported by the Frontier of Development seed funding from the Royal Academy of Engineering (FoD2021424). T Z is supported by the RAEng Engineering for Development Research Fellowship. This research was supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Appendix: Models used for the prediction problems

In order to provide a more complete picture of the works that have been conducted in the space of machine learning in patient flow we here provide flow charts including the models and datasets that have been used to make the predictions. Figure A1 corresponds to ED admissions, figure A2 corresponds to the ED-inpatient interface, figure A3 corresponds to inpatient transfers and figure A4 corresponds to discharge.

Figure A1.  A flowchart showing the models that have been used for the separate prediction problems for predicting ED admissions. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

Figure A2.  A flowchart showing the models that have been used for the separate prediction problems for predicting ED to inpatient admissions. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

Figure A3.  A flowchart showing the models that have been used for the separate prediction problems for predicting inpatient transfers. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

Figure A4.  A flowchart showing the models that have been used for the separate prediction problems for predicting discharges. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

Self-attention Enhanced Patient Journey Understanding in Healthcare System

• Conference paper
• First Online: 25 February 2021
• Cite this conference paper

• Xueping Peng 12 ,
• Guodong Long 12 ,
• Tao Shen 12 ,
• Sen Wang 13 &
• Jing Jiang 12  

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12459))

Included in the following conference series:

• Joint European Conference on Machine Learning and Knowledge Discovery in Databases

1950 Accesses

5 Citations

Understanding patients’ journeys in healthcare system is a fundamental prepositive task for a broad range of AI-based healthcare applications. This task aims to learn an informative representation that can comprehensively encode hidden dependencies among medical events and its inner entities, and then the use of encoding outputs can greatly benefit the downstream application-driven tasks. A patient journey is a sequence of electronic health records (EHRs) over time that is organized at multiple levels: patient, visits and medical codes. The key challenge of patient journey understanding is to design an effective encoding mechanism which can properly tackle the aforementioned multi-level structured patient journey data with temporal sequential visits and a set of medical codes. This paper proposes a novel self-attention mechanism that can simultaneously capture the contextual and temporal relationships hidden in patient journeys. A multi-level self-attention network (MusaNet) is specifically designed to learn the representations of patient journeys that is used to be a long sequence of activities. We evaluated the efficacy of our method on two medical application tasks with real-world benchmark datasets. The results have demonstrated the proposed MusaNet produces higher-quality representations than state-of-the-art baseline methods. The source code is available in https://github.com/xueping/MusaNet .

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Continuous patient state attention model for addressing irregularity in electronic health records

Interpretable time-aware and co-occurrence-aware network for medical prediction

Multilevel Asynchronous Time Network for Medication Recommendation

GRAM [ 3 ] and KAME [ 18 ] and MMORE [ 31 ] are not baselines as they use external knowledge to learn the medical code representations.

http://www.icd9data.com .

Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 (2014)

Choi, E., et al.: Multi-layer representation learning for medical concepts. In: SIGKDD, pp. 1495–1504. ACM (2016)

Google Scholar  

Choi, E., Bahadori, M.T., Song, L., Stewart, W.F., Sun, J.: Gram: graph-based attention model for healthcare representation learning. In: SIGKDD, pp. 787–795. ACM (2017)

Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: Retain: an interpretable predictive model for healthcare using reverse time attention mechanism. In: NeurIPS, pp. 3504–3512 (2016)

Choi, E., Xiao, C., Stewart, W., Sun, J.: Mime: multilevel medical embedding of electronic health records for predictive healthcare. In: NeurIPS, pp. 4552–4562 (2018)

Choi, Y., Chiu, C.Y.I., Sontag, D.: Learning low-dimensional representations of medical concepts. AMIA Summits on Translational Science Proceedings 2016, p. 41 (2016)

Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: ICML, pp. 233–240. ACM (2006)

De Vine, L., Zuccon, G., Koopman, B., Sitbon, L., Bruza, P.: Medical semantic similarity with a neural language model. In: CIKM, pp. 1819–1822. ACM (2014)

Hu, M., Peng, Y., Huang, Z., Qiu, X., Wei, F., Zhou, M.: Reinforced mnemonic reader for machine reading comprehension. arXiv:1705.02798 (2017)

Jha, K., Wang, Y., Xun, G., Zhang, A.: Interpretable word embeddings for medical domain. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 1061–1066. IEEE (2018)

Johnson, A.E., Pollard, T.J., Shen, L., Li-wei, H.L., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Celi, L.A., Mark, R.G.: Mimic-iii, a freely accessible critical care database. Sci. Data 3 , 160035 (2016)

Article   Google Scholar  

Kho, A.N., Yu, J., Bryan, M.S., Gladfelter, C., Gordon, H.S., Grannis, S., Madden, M., Mendonca, E., Mitrovic, V., Shah, R., Tachinardi, U., Taylor, B.: Privacy-preserving record linkage to identify fragmented electronic medical records in the all of us research program. In: Cellier, P., Driessens, K. (eds.) ECML PKDD 2019. CCIS, vol. 1168, pp. 79–87. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43887-6_7

Chapter   Google Scholar  

Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP, pp. 1746–1751, October 2014

Lin, Z., et al.: A structured self-attentive sentence embedding. arXiv:1703.03130 (2017)

Liu, Y., Sun, C., Lin, L., Wang, X.: Learning natural language inference using bidirectional lstm model and inner-attention. arXiv:1605.09090 (2016)

Ma, F., Chitta, R., Zhou, J., You, Q., Sun, T., Gao, J.: Dipole: diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks. In: SIGKDD, pp. 1903–1911. ACM (2017)

Ma, F., Gao, J., Suo, Q., You, Q., Zhou, J., Zhang, A.: Risk prediction on electronic health records with prior medical knowledge. In: SIGKDD, pp. 1910–1919. ACM, July 2018

Ma, F., You, Q., Xiao, H., Chitta, R., Zhou, J., Gao, J.: KAME: knowledge-based attention model for diagnosis prediction in healthcare. In: CIKM, pp. 743–752. ACM, October 2018

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)

Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NeurIPS, pp. 3111–3119 (2013)

Minarro-Giménez, J.A., Marin-Alonso, O., Samwald, M.: Exploring the application of deep learning techniques on medical text corpora. Stud. Health Technol. Inform. 205 , 584–588 (2014)

Nguyen, P., Tran, T., Wickramasinghe, N., Venkatesh, S.: Deepr: a convolutional net for medical records. arXiv:1607.07519 (2016)

Peng, X., Long, G., Pan, S., Jiang, J., Niu, Z.: Attentive dual embedding for understanding medical concepts in electronic health records. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)

Peng, X., Long, G., Shen, T., Wang, S., Jiang, J., Blumenstein, M.: Temporal self-attention network for medical concept embedding. In: 2019 IEEE International Conference on Data Mining (ICDM), pp. 498–507 (2019)

Qiao, Z., Zhao, S., Xiao, C., Li, X., Qin, Y., Wang, F.: Pairwise-ranking based collaborative recurrent neural networks for clinical event prediction. In: IJCAI, pp. 3520–3526 (2018)

Rajkomar, A., et al.: Scalable and accurate deep learning with electronic health records. NPJ Digital Med. 1 (1), 18 (2018)

Shen, T., Zhou, T., Long, G., Jiang, J., Pan, S., Zhang, C.: Disan: directional self-attention network for RNN/CNN-free language understanding. In: AAAI (2018)

Shen, T., Zhou, T., Long, G., Jiang, J., Zhang, C.: Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv:1804.00857 (2018)

Shickel, B., Tighe, P.J., Bihorac, A., Rashidi, P.: Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J. Biomed. Health Inform. 22 (5), 1589–1604 (2018)

Song, H., Rajan, D., Thiagarajan, J.J., Spanias, A.: Attend and diagnose: clinical time series analysis using attention models. In: AAAI (2018)

Song, L., Cheong, C.W., Yin, K., Cheung, W.K., Cm, B.: Medical concept embedding with multiple ontological representations. In: IJCAI, pp. 4613–4619 (2019)

Vaswani, A., et al.: Attention is all you need. In: NeurIPS, pp. 5998–6008 (2017)

Wang, S., Li, X., Chang, X., Yao, L., Sheng, Q.Z., Long, G.: Learning multiple diagnosis codes for ICU patients with local disease correlation mining. ACM Trans. Knowl. Discovery Data (TKDD) 11 (3), 1–21 (2017)

Zhang, Q., Wu, J., Zhang, P., Long, G., Zhang, C.: Salient subsequence learning for time series clustering. IEEE Trans. Pattern Anal. Mach. Intell. 41 (9), 2193–2207 (2018)

Download references

Acknowledgments

This work was supported in part by the Australian Research Council (ARC) under Grant LP160100630, LP180100654 and DE190100626.

Author information

Authors and affiliations.

FEIT, AAII, University of Technology Sydney, Ultimo, Australia

Xueping Peng, Guodong Long, Tao Shen & Jing Jiang

School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, Australia

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Xueping Peng .

Editor information

Editors and affiliations.

Albert-Ludwigs-Universität, Freiburg, Germany

Frank Hutter

TU Darmstadt, Darmstadt, Germany

Kristian Kersting

Ghent University, Ghent, Belgium

Jefrey Lijffijt

Saarland University, Saarbrücken, Germany

Isabel Valera

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Cite this paper.

Peng, X., Long, G., Shen, T., Wang, S., Jiang, J. (2021). Self-attention Enhanced Patient Journey Understanding in Healthcare System. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12459. Springer, Cham. https://doi.org/10.1007/978-3-030-67664-3_43

Download citation

DOI : https://doi.org/10.1007/978-3-030-67664-3_43

Published : 25 February 2021

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-67663-6

Online ISBN : 978-3-030-67664-3

eBook Packages : Computer Science Computer Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

Societies and partnerships

• Find a journal
• Track your research

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Patient journey through cases of depression from claims database using machine learning algorithms

Roles Conceptualization, Investigation, Methodology, Project administration, Validation, Writing – original draft, Writing – review & editing

Affiliation Data Science Office, Shionogi & Co. Ltd., Osaka, Japan

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Roles Conceptualization, Methodology, Writing – review & editing

Affiliation Biometrics, Shionogi Inc, Florham Park, NJ, United States of America

• Yoshitake Kitanishi, 
• Masakazu Fujiwara, 
• Bruce Binkowitz

• Published: February 16, 2021
• https://doi.org/10.1371/journal.pone.0247059
• Reader Comments

Health insurance and acute hospital-based claims have recently become available as real-world data after marketing in Japan and, thus, classification and prediction using the machine learning approach can be applied to them. However, the methodology used for the analysis of real-world data has been hitherto under debate and research on visualizing the patient journey is still inconclusive. So far, to classify diseases based on medical histories and patient demographic background and to predict the patient prognosis for each disease, the correlation structure of real-world data has been estimated by machine learning. Therefore, we applied association analysis to real-world data to consider a combination of disease events as the patient journey for depression diagnoses. However, association analysis makes it difficult to interpret multiple outcome measures simultaneously and comprehensively. To address this issue, we applied the Topological Data Analysis (TDA) Mapper to sequentially interpret multiple indices, thus obtaining a visual classification of the diseases commonly associated with depression. Under this approach, the visual and continuous classification of related diseases may contribute to precision medicine research and can help pharmaceutical companies provide appropriate personalized medical care.

Citation: Kitanishi Y, Fujiwara M, Binkowitz B (2021) Patient journey through cases of depression from claims database using machine learning algorithms. PLoS ONE 16(2): e0247059. https://doi.org/10.1371/journal.pone.0247059

Editor: Kevin Lu, University of South Carolina College of Pharmacy, UNITED STATES

Received: July 29, 2020; Accepted: January 30, 2021; Published: February 16, 2021

Copyright: © 2021 Kitanishi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data used in this study are from the 'Diagnosis' (including ICD10 code information) and 'Enrollment Information' (including age and gender) datasets in the JMDC database. Future data inquiries should be addressed to JMDC ( https://www.jmdc.co.jp/en/jmdc-claims-database ).

Funding: YK and MF are employees of Shionogi Pharmaceutical Company (Shionogi & Co.), and BB is an employee of Shionogi Inc. The funder (Shionogi & Co. (Japan) and Shionogi Inc. (US)) provided support in the form of salaries for authors [YK and MF in Shionogi & Co., and BB in Shionogi Inc.] and research materials only, but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the 'author contributions' section.

Competing interests: YK and MF are salaried employees of Shionogi & Co. and BB is a salaried employee of Shionogi Inc. This does not alter our adherence to all PLOS ONE policies on sharing data and materials.

Introduction

In the pharmaceutical industry, various machine learning methods have been applied to big data for each value chain or across multiple value chains, such as research, development, manufacturing, and post-marketing [ 1 , 2 ]. As examples, drug discovery target molecules are searched by single nucleotide polymorphisms (SNPs) and genotoxicity is predicted by the quantitative structure-activity relationship (QSAR) in the research phase. Further, the efficacy of drugs is predicted by modeling and simulation of the relationship between pharmacokinetics and pharmacodynamics (PK/PD information) in the development phase, and there are also indirect efficacy comparisons between drugs or signal detections of safety in the post-marketing phase. The information from the research, development, manufacturing, and post-marketing stages is comprehensively integrated for the application of machine learning techniques, for example, for drug repositioning [ 3 , 4 ]. So far, machine learning has mainly been advanced for improving the efficiency of drug research and development and for determining the differentiation point with other drugs.

However, recent advances in digital technologies have allowed the pharmaceutical industry to deal with new categories of big data [ 5 , 6 ], for example, from social media networks, wearable devices, smartphone data (application, status of utilization such as tapping and scrolling), disease registries, and claims databases. As their use in clinical trials or other value chain phases can lead to new drug values, we focus on a claims database.

The objectives of a claims database may include understanding treatment practices (daily dose, rates of treatments continuation, drug adherences, post-marketing trends, and patient journeys) and the epidemiology of diseases (onset timing of drug related events, comorbidities, prevalence), as well as exploring the post-marketing safety of drugs (incidence of drug related events). Various studies using claims databases have been carried out from a variety of perspectives, including calculating the incidence of psychosis in ADHD patients [ 7 ], investigating the time course of treatment for glaucoma [ 8 ], estimating the number of patients with liver disease associated with hepatitis B/C virus infection [ 9 ], understanding treatment patterns in patients with hyperlipidemia [ 10 ], investigating the prevalence of current treatment practices for rheumatoid arthritis [ 11 ], understanding the duration and rate of antipsychotic drugs in outpatients with schizophrenia [ 12 ], or exploring the prescribing patterns of antiparkinsonian drugs [ 13 ].

However, although machine learning has been used to analyze claims databases, including the identification of subgroups of patients with type 2 diabetes [ 14 ] and detecting suicidality in patients with fibromyalgia [ 15 ], machine learning should rather be utilized from the viewpoint of the amount and complexity of information contained in claims databases.

Therefore, this paper proposes a novel approach, applying combinations of machine learning methods to claims databases to promote the understanding of a patient’s journey. Specifically, machine learning methods are applied to a claims database to classify a disease by several factors (e.g., medical histories, drugs, background factors) prior to the onset of the disease and the prognoses (e.g., complications) after the onset of the disease. To this end, the Japan Medical Data Center (JMDC) database is used. Association analysis, which is a method of data mining, is used for combination-based disease classification [ 16 – 19 ], as it is efficient for extracting combinations of data that meet certain relevant rules (combinations) from the big data. Additionally, the Topological Data Analysis (TDA) Mapper is a robust and stable method of visualizing data updates, which increases in multidimensional large-scale data by spatially grasping data characteristics [ 20 – 23 ]. As such, we use it to visualize the multi-dimensional indices obtained by association analysis.

To interpret the association analysis results, it is necessary to compare the results of each index and there exist no established methods for understanding such results to date. Therefore, by applying the TDA Mapper to the indices obtained from the association analysis, the relevance of the results can be visually understood. This new approach combines machine learning methods and can help researchers interpret the information and derive deeper insights by visualizing useful information from the claims database.

As one of the diseases for which this new approach is suitable, we focus on the depression which is a typical mental illness as the target disease. The exact cause and prognosis of depression are unclear, and it is reasonable to visualize their diverse relationships by this new approach. Various clinical studies and surveys confirm the usefulness or safety of antidepressant drugs but are carried out after they are marketed. For example, Jick et al. [ 24 ] investigated the association between the use of antidepressant drugs and suicidal behavior and reported the possibility of increased risks at the beginning of the treatment, while Weeke et al. [ 25 ] investigated the association between antidepressant drug use and out-of-hospital cardiopulmonary arrest and reported how selective serotonin reuptake inhibitors (SSRIs) or tricyclic antidepressant drugs are associated with out-of-hospital cardiopulmonary arrest. The main symptoms of depression could be described on a broad spectrum (low mood, loss of interests, lack of drive, and so on) and the changes in drugs or their concomitant use with other drugs may be appropriate, depending on the symptoms of patients [ 26 ]. Therefore, the adverse drug reactions induced by antidepressant drugs after marketing are diverse. These concerns about patient safety for patients with depression have led to several clinical studies and surveys of antidepressants after they have been marketed. Therefore, more safety information should be available prior to prescribing a drug that targets depression, and it is valuable to understand the variety of patient journeys, including whether the drug is likely to work for a patient and whether drug safety will be maintained for a patient based on his/her background.

Materials and methods

Jmdc database.

This study is a retrospective database analysis using the JMDC claims database, which is commercially available in Japan ( https://www.jmdc.co.jp/en/index ). Since the database consists of unlinkable anonymized data collected for secondary use, ethical approval and informed consent were not required according to Ethical Guidelines for Medical and Health Research Involving Human Subjects by the Ministry of Education, Culture, Sports, Science and Technology, and the Ministry of Health, Labour and Welfare, Japan.

Specifically, the JMDC claims database is an epidemiological receipt database that has accumulated receipt (inpatient, outpatient, dispensing) and medical examination data from multiple health insurance associations since 2005. The cumulative dataset includes approximately 5.6 million subjects (as of June 2018) and it is possible to follow the prevalence rate and incidence for a general population, including healthy individuals. Even if a subject transfers hospitals or uses multiple facilities, the data can still be tracked ( https://www.jmdc.co.jp/en/jmdc-claims-database ).

We used JMDC data from January 2005 to February 2018 as we started this study from February 2018, and focused on 2-year medical histories (diagnosis information) before the onset of depression as per the F32 (major depressive disorder) code of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) and the 2-year complications after the onset of depression. We extracted data on around 10,000 patients with depression from the JMDC database.

Association analysis

Medical histories were used to divide the patient journey leading to depression. Association analysis based on the Apriori algorithm which is searching frequent itemset and devising association rules from a database [ 16 – 19 ] was used to extract the combinations of medical histories in the 2 years before the onset of depression from the JMDC database. We used three indices to evaluate the strength of the association between X and Y: support, confidence, and lift, where X and Y are medical histories for the same patients. We used the medical histories 2 years before the onset of depression and did not consider the order or onset time of medical histories.

In the JMDC database, if there exists medical history X for a patient, the variable is set to “Yes,” and if there is no medical history X for a patient, it is set to “No.” X and Y can be represented by a contingency table ( Fig 1 ) for all patients in the JMDC database.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0247059.g001

When considering the combinations of medical histories, “A” indicates the number of cases in which both medical histories X and Y are reported as claim data before the onset of depression. Similarly, “B” represents the number of patients with medical history X reported before the onset of depression and no reported medical history Y and “C” represents the number of cases without medical histories X or Y reported before the onset of depression. Finally, “D” represents the number of reported patients without medical histories X or Y before the onset of depression. In the JMDC database, the number of patients reported with depression as F32 (major depressive disorder) is denoted as t. In this case, the support of the combination of X and Y is defined as a/t, confidence as a/(a+b), and lift as {a/(a+b)}/{(a+c)/t}. The lowercase means a realization or an observed value, and the uppercase means an event. These three indices represent the strength of the association between the combination of medical histories X and Y.

Additionally, association analysis using complication data after the onset of depression was also performed for each subset comprising the combinations of medical histories before the onset of the disease. SAS Viya was used for analysis.

The TDA Mapper algorithm makes it easy to visually understand data features, as opposed to the persistent homology that captures the details of data shapes as numerical values. For example, the TDA Mapper is suitable for scenarios wherein features are extracted from big data and comprises four steps: inputting the target data, adjusting the distance function by using the filter function, clustering, and visualizing each parameter.

First, the data is inputted in Step 1. The input is N data with M-dimensional features. Next, in Step 2, the distance between any two points in the input data is obtained and an N-by-N distance matrix, X, is created. Using a filter function for this distance matrix, mapping can be performed in a low dimension. The value obtained by mapping in a low dimension is called the filter value. Next, adjust the parameters of the filter function used at this time. In Step 3, the dataset is divided into intervals based on this filter value and clustering is performed at each interval. An overlap is also set when dividing the intervals. By setting the overlap, during visualization, it is possible to draw a line between the clusters with overlapping nodes and show the connections between them. In Step 4, clustering is performed for each interval and a line is drawn between the clusters containing the same nodes to represent the connection. Fig 2 shows the analysis process [ 20 , 27 ], with specific parameters as an example. Python Mapper [ 28 ] was used for the analysis.

The filtering parameters are as follows: range = 0–10; interval length = 2; intervals = 5; overlap = 50%.

https://doi.org/10.1371/journal.pone.0247059.g002

Association analysis for medical histories

There were 10,188 new depression patients in the JMDC database from January 2005 to February 2018. These patients had a total of 114,691 medical histories in the 2 years before the onset of depression, and 1,051 distinct medical histories. And these patients had a total of 138,549 complications in the 2 years after the onset of depression, and 1,089 distinct complications.

The results of the association analysis for each patient’s medical history data are shown in Table 1 . The combinations of medical histories up to the 10th in the descending order of support are shown. The combinations of medical histories with the highest support were X = other anxiety disorders, Y = sleeping disorder, and support (%), confidence (%), and lift were 9.3, 50.1, and 1.1, respectively.

https://doi.org/10.1371/journal.pone.0247059.t001

Among the combinations of medical histories in the top 10 in terms of support, there were five combinations, including sleeping disorder, which was the most frequent. The combination of medical histories without sleeping disorder was X = other causes of gastroenteritis and colitis, infections, and unknown causes; Y = gastritis and duodenal inflammation, with support (%), confidence (%), and lift of 8.2, 50.5, and 1.5, respectively.

Additionally, combinations of medical histories with 50 or less occurrences were infrequent and thus excluded, and the TDA Mapper was applied to the values of support (%), confidence (%), and lift for all 109 medical history combinations obtained by applying association analysis. Fig 3 shows the result of applying the Mapper.

https://doi.org/10.1371/journal.pone.0247059.g003

The classification was obtained by using support (%), confidence (%), and lift, creating successive combinations of the magnitudes of their respective values. As shown in Fig 3 , they can be classified into groups A–H. Looking at the composition of each group, a characteristic medical history is included. The tendency of the magnitude of the three variables in each group is also shown in Fig 3 . From these results, it is possible to identify strongly related combinations, even if their frequencies are not high.

Association analysis for complications

By the combinations of medical histories in Table 1 , we examined the combinations of complications in the first 2 years after the onset of depression. Among patients with medical histories X = other anxiety disorders and Y = sleeping disorder, which is the most common combination of medical histories for patients with depression, the combination of the most common complications in support was X = multiple and unspecified acute upper respiratory tract infections, Y = gastritis and duodenal inflammation, where support (%), confidence (%), and lift were 16.6, 59.3, and 1.6, respectively ( Table 2 ). Fig 4 shows the results of applying TDA Mapper to the results of the association analysis.

https://doi.org/10.1371/journal.pone.0247059.g004

https://doi.org/10.1371/journal.pone.0247059.t002

From Fig 4 , the classification is approximately carried out for each disease by the index of the three association rules.

Table 3 shows the results of applying association analysis to complications for 2 years in patients with a medical history of the combination with the highest support among the medical history combinations that did not include sleeping disorder (i.e., X = other causes of gastroenteritis and colitis, infections and unknown causes, Y = gastritis and duodenal inflammation). The combination of complications with the highest support was X = keratitis, Y = refraction and accommodation disorders, where support (%), confidence (%), and lift were 4.4, 60.3, and 3.7, respectively. TDA Mapper was not applied in this case because the number of combinations was small.

https://doi.org/10.1371/journal.pone.0247059.t003

We proposed a new approach for understanding patient journeys by classifying depression through combining medical histories before onset and confirming prognoses such as complications after onset for each medical history type. To this end, we applied machine learning methods (i.e., association analysis and TDA Mapper) to a claims database. This approach leads to a deeper, more diverse, and comprehensive understanding of patient journeys and a better understanding of the patients who should be prescribed such drugs and their prognoses (e.g., occurrence of adverse drug reactions), thus allowing for better personalized patient care. However, the main purpose of this paper is not to provide medical evidence, but to determine the usefulness of analyzing claims database data using machine learning methods. The support of the combinations of medical histories before the onset of depression in Table 1 is slightly lower. That is, the combinations of medical histories were diversified and the rate of each combination was low. The confidence values, which were chosen to be higher to denote a stronger association between X and Y, ranged between 0.5 and 1. Additionally, lift values were greater than 1 in nearly all instances, suggesting a strong association. These data suggest each considered association was moderately strong, although expression rates were low.

Many of the medical histories for which support was superior included medical histories related to gastrointestinal and sleeping disorders. Mental health disturbances have mainly manifested in terms of digestive system and sleep issues, which may have triggered depression [ 29 ]. Although these are well known causes of depression, their combination through the claims database is valuable in demonstrating the usefulness of the database.

The results of the association analysis were obtained using several multidimensional indicators, such as support, confidence, and lift. Therefore, it is difficult to intuitively understand the relationship between each combination of indicators. However, by using the TDA Mapper, as in Fig 3 , it is possible to visualize relationships. The TDA Mapper can help with two‐dimensional visualization by integrating the three‐dimensional indices of combinations, while maintaining the relationships of the combinations seems to have a good compatibility with the association analyses. Further, the TDA Mapper is more robust than hierarchical clustering because it allows overlap. In other words, the claims database can be interpreted in a robust manner in relation to the changes in the values of the parameters in the association analysis.

Tables 2 and 3 also show that the complications after the onset of depression for patients with medical histories that included sleeping disorders differed substantially from those after the onset of depression in patients with combinations of medical histories that did not include sleeping disorders. Therefore, the complications may differ according to the medical histories before onset and could thus promote a better understanding of patient journeys and lead to better personalized medical care. The data volume in the claims database is expected to increase in the future, but as association analysis is a method for extracting combinations from big data, the results are robust.

We focus on the depression which is a typical mental illness as the target disease. Although the main symptoms of depression could be described on a broad spectrum [ 26 ] and there may be the variety of patient journeys for the depression, our approach could detect medical histories and complications from various aspects as shown in Figs 3 and 4 . Therefore, our approach seems to have certain applicability to other diseases.

However, this paper is not without limitations. For instance, only the information on medical histories and complications in the claims database was used, but information on backgrounds, drugs, and dates is also available and patient journeys can be better understood by using this additional information. Our approach is also applicable to other diseases and databases including similar information and we would like to validate these results by combining interpretations in clinical practice. The JMDC is a database based on claims received from multiple health insurance societies. Therefore, the percentage of those aged 65 and over is small compared to the actual demographic distribution in Japan and, thus, we cannot deny the possibility that the results are biased in this respect. Additionally, since control patients could not be set, it is not possible to determine whether medical histories and complications were specific to the patients with depression or not. And, we don’t use sequential information on the order and onset time of medical histories, so our method could be developed using this information. This analysis pertains to future work.

Supporting information

https://doi.org/10.1371/journal.pone.0247059.s001

Acknowledgments

We are grateful to the anonymous referees for their constructive reviews.

• View Article
• PubMed/NCBI
• Google Scholar
• 16. Agrawal R, Imieliński T, Swami A. Mining association rules between sets of items in large databases. In Proceedings of the 1993 ACM SIGMOD International Conference on Management of Data; 1993. pp. 207–216.
• 17. Agrawal R, Srikant R. Fast algorithms for mining association rules. In Proceedings of the 20th International Conference of Very Large Databases, VLDB; 1994. pp. 487–499.
• 22. Torres-Tramón P, Hromic H, Heravi BR. Topic detection in Twitter using topology data analysis. In International Conference on Web Engineering; 2015. pp. 186–197. Springer, Cham.
• 23. Kitanishi Y, Ishioka F, Iizuka M, Kurihara K. Spatial perception for structured and unstructured data in topological data analysis. Congress Proceedings of IFCS 2019, Thessaloniki, Greece.
• 26. Rey JM, Bella-Awusah TT, Liu J. Depression in children and adolescents. In Rey JM, editor. IACAPAP e-textbook of child and adolescent mental health. Geneva: International Association for Child and Adolescent Psychiatry and Allied Professions; 2015.
• 28. Müllner D, Babu A. Python mapper: An open-source toolchain for data exploration, analysis and visualization. 2013. Available from: http://danifold . net/mapper

• Help & FAQ

Advancing Design Approaches through Data-Driven Techniques: Patient Community Journey Mapping Using Online Stories and Machine Learning

• Methodologie en Organisatie van Design
• Design, Organisation and Strategy

Research output : Contribution to journal › Article › Scientific › peer-review

• Patient Journey Mapping
• Machine Learning
• Hybrid Intelligence
• Patient Stories
• Healthcare Design

Access to Document

• 10.57698/v17i2.02
• 4671-15228-3-PB Final published version, 6.13 MB Licence: CC BY

Other files and links

• Link to publication in Scopus

Fingerprint

• design INIS 100%
• patients INIS 100%
• mapping INIS 100%
• communities INIS 100%
• machine learning INIS 100%
• data INIS 100%
• Mapping Engineering 100%
• Machine Learning Computer Science 100%

T1 - Advancing Design Approaches through Data-Driven Techniques

T2 - Patient Community Journey Mapping Using Online Stories and Machine Learning

AU - Jung, Jiwon

AU - Kim, K.

AU - Peters, Tess

AU - Snelders, Dirk

AU - Kleinsmann, Maaike

N2 - Designers are increasingly collaborating with data scientists to apply smart data technologies to understand large-scale user behavior during their design research. This is useful in specific impact domains with vulnerable users and unfamiliar contexts, such as healthcare design. Patient journey mapping is the most common design tool for developing and communicating patient-centred perspectives in healthcare design. However, creating a traditional patient journey map is labor intensive. Consequently, they often represent the experiences of a limited number of patients and, therefore, have limitations in including an extensive group patient experience. To overcome these challenges, we present a new data-driven and hybrid intelligent design approach that utilizes tens of thousands of online patient stories and machine-learning techniques through collaboration with data scientists. We set up two studies in the field of oncology and demonstrate that combining the two machine-learning techniques allows for quantifying the experiences of a wide range of patients, detecting relationships between co-occurring experiences within the journey, and detecting new design opportunities/directions. In these studies, designers gained a large-scale, yet qualitative and inspiring, understanding of a complex context in healthcare with reduced time and cost investments

AB - Designers are increasingly collaborating with data scientists to apply smart data technologies to understand large-scale user behavior during their design research. This is useful in specific impact domains with vulnerable users and unfamiliar contexts, such as healthcare design. Patient journey mapping is the most common design tool for developing and communicating patient-centred perspectives in healthcare design. However, creating a traditional patient journey map is labor intensive. Consequently, they often represent the experiences of a limited number of patients and, therefore, have limitations in including an extensive group patient experience. To overcome these challenges, we present a new data-driven and hybrid intelligent design approach that utilizes tens of thousands of online patient stories and machine-learning techniques through collaboration with data scientists. We set up two studies in the field of oncology and demonstrate that combining the two machine-learning techniques allows for quantifying the experiences of a wide range of patients, detecting relationships between co-occurring experiences within the journey, and detecting new design opportunities/directions. In these studies, designers gained a large-scale, yet qualitative and inspiring, understanding of a complex context in healthcare with reduced time and cost investments

KW - Patient Journey Mapping

KW - Machine Learning

KW - Hybrid Intelligence

KW - Patient Stories

KW - Healthcare Design

UR - http://www.scopus.com/inward/record.url?scp=85171795838&partnerID=8YFLogxK

U2 - 10.57698/v17i2.02

DO - 10.57698/v17i2.02

M3 - Article

SN - 1991-3761

JO - International Journal of Design

JF - International Journal of Design

Gaining Insights Into Patient Satisfaction Through Interpretable Machine Learning

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Patient journey through cases of depression from claims database using machine learning algorithms

Affiliations.

• 1 Data Science Office, Shionogi & Co. Ltd., Osaka, Japan.
• 2 Biometrics, Shionogi Inc, Florham Park, NJ, United States of America.
• PMID: 33592062
• PMCID: PMC7886120
• DOI: 10.1371/journal.pone.0247059

Health insurance and acute hospital-based claims have recently become available as real-world data after marketing in Japan and, thus, classification and prediction using the machine learning approach can be applied to them. However, the methodology used for the analysis of real-world data has been hitherto under debate and research on visualizing the patient journey is still inconclusive. So far, to classify diseases based on medical histories and patient demographic background and to predict the patient prognosis for each disease, the correlation structure of real-world data has been estimated by machine learning. Therefore, we applied association analysis to real-world data to consider a combination of disease events as the patient journey for depression diagnoses. However, association analysis makes it difficult to interpret multiple outcome measures simultaneously and comprehensively. To address this issue, we applied the Topological Data Analysis (TDA) Mapper to sequentially interpret multiple indices, thus obtaining a visual classification of the diseases commonly associated with depression. Under this approach, the visual and continuous classification of related diseases may contribute to precision medicine research and can help pharmaceutical companies provide appropriate personalized medical care.

Publication types

• Research Support, Non-U.S. Gov't
• Data Management / methods*
• Machine Learning*
• Precision Medicine / methods

Grants and funding

Artificial intelligence  is being used in healthcare for everything from answering patient questions to assisting with surgeries and developing new pharmaceuticals.

According to  Statista , the artificial intelligence (AI) healthcare market, which is valued at $11 billion in 2021, is projected to be worth $187 billion in 2030. That massive increase means we will likely continue to see considerable changes in how medical providers, hospitals, pharmaceutical and biotechnology companies, and others in the healthcare industry operate.

Better  machine learning (ML)  algorithms, more access to data, cheaper hardware, and the availability of 5G have contributed to the increasing application of AI in the healthcare industry, accelerating the pace of change. AI and ML technologies can sift through enormous volumes of health data—from health records and clinical studies to genetic information—and analyze it much faster than humans.

Healthcare organizations are using AI to improve the efficiency of all kinds of processes, from back-office tasks to patient care. The following are some examples of how AI might be used to benefit staff and patients:

• Administrative workflow:  Healthcare workers spend a lot of time doing paperwork and other administrative tasks. AI and automation can help perform many of those mundane tasks, freeing up employee time for other activities and giving them more face-to-face time with patients. For example, generative AI can help clinicians with note-taking and content summarization that can help keep medical records as thoroughly as possible. AI might also help with accurate coding and sharing of information between departments and billing.
• Virtual nursing assistants:  One study found that  64% of patients  are comfortable with the use of AI for around-the-clock access to answers that support nurses provide. AI virtual nurse assistants—which are AI-powered chatbots, apps, or other interfaces—can be used to help answer questions about medications, forward reports to doctors or surgeons and help patients schedule a visit with a physician. These sorts of routine tasks can help take work off the hands of clinical staff, who can then spend more time directly on patient care, where human judgment and interaction matter most.
• Dosage error reduction:  AI can be used to help identify errors in how a patient self-administers medication. One example comes from a study in  Nature Medicine , which found that up to 70% of patients don’t take insulin as prescribed. An AI-powered tool that sits in the patient’s background (much like a wifi router) might be used to flag errors in how the patient administers an insulin pen or inhaler.
• Less invasive surgeries:  AI-enabled robots might be used to work around sensitive organs and tissues to help reduce blood loss, infection risk and post-surgery pain.
• Fraud prevention:  Fraud in the healthcare industry is enormous, at $380 billion/year, and raises the cost of consumers’ medical premiums and out-of-pocket expenses. Implementing AI can help recognize unusual or suspicious patterns in insurance claims, such as billing for costly services or procedures that are not performed, unbundling (which is billing for the individual steps of a procedure as though they were separate procedures), and performing unnecessary tests to take advantage of insurance payments.

A recent study found that  83% of patients  report poor communication as the worst part of their experience, demonstrating a strong need for clearer communication between patients and providers. AI technologies like  natural language processing  (NLP), predictive analytics, and  speech recognition  might help healthcare providers have more effective communication with patients. AI might, for instance, deliver more specific information about a patient’s treatment options, allowing the healthcare provider to have more meaningful conversations with the patient for shared decision-making.

According to  Harvard’s School of Public Health , although it’s early days for this use, using AI to make diagnoses may reduce treatment costs by up to 50% and improve health outcomes by 40%.

One use case example is out of the  University of Hawaii , where a research team found that deploying  deep learning  AI technology can improve breast cancer risk prediction. More research is needed, but the lead researcher pointed out that an AI algorithm can be trained on a much larger set of images than a radiologist—as many as a million or more radiology images. Also, that algorithm can be replicated at no cost except for hardware.

An  MIT group  developed an ML algorithm to determine when a human expert is needed. In some instances, such as identifying cardiomegaly in chest X-rays, they found that a hybrid human-AI model produced the best results.

Another  published study  found that AI recognized skin cancer better than experienced doctors.  US, German and French researchers used deep learning on more than 100,000 images to identify skin cancer. Comparing the results of AI to those of 58 international dermatologists, they found AI did better.

As health and fitness monitors become more popular and more people use apps that track and analyze details about their health. They can share these real-time data sets with their doctors to monitor health issues and provide alerts in case of problems.

AI solutions—such as big data applications, machine learning algorithms and deep learning algorithms—might also be used to help humans analyze large data sets to help clinical and other decision-making. AI might also be used to help detect and track infectious diseases, such as COVID-19, tuberculosis, and malaria.

One benefit the use of AI brings to health systems is making gathering and sharing information easier. AI can help providers keep track of patient data more efficiently.

One example is diabetes. According to the  Centers for Disease Control and Prevention , 10% of the US population has diabetes. Patients can now use wearable and other monitoring devices that provide feedback about their glucose levels to themselves and their medical team. AI can help providers gather that information, store, and analyze it, and provide data-driven insights from vast numbers of people. Using this information can help healthcare professionals determine how to better treat and manage diseases.

Organizations are also starting to use AI to help improve drug safety. The company SELTA SQUARE, for example, is  innovating the pharmacovigilance (PV) process , a legally mandated discipline for detecting and reporting adverse effects from drugs, then assessing, understanding, and preventing those effects. PV demands significant effort and diligence from pharma producers because it’s performed from the clinical trials phase all the way through the drug’s lifetime availability. Selta Square uses a combination of AI and automation to make the PV process faster and more accurate, which helps make medicines safer for people worldwide.

Sometimes, AI might reduce the need to test potential drug compounds physically, which is an enormous cost-savings.  High-fidelity molecular simulations  can run on computers without incurring the high costs of traditional discovery methods.

AI also has the potential to help humans predict toxicity, bioactivity, and other characteristics of molecules or create previously unknown drug molecules from scratch.

As AI becomes more important in healthcare delivery and more AI medical applications are developed, ethical, and regulatory governance must be established. Issues that raise concern include the possibility of bias, lack of transparency, privacy concerns regarding data used for training AI models, and safety and liability issues.

“AI governance is necessary, especially for clinical applications of the technology,” said Laura Craft, VP Analyst at  Gartner . “However, because new AI techniques are largely new territory for most [health delivery organizations], there is a lack of common rules, processes, and guidelines for eager entrepreneurs to follow as they design their pilots.”

The World Health Organization (WHO) spent 18 months deliberating with leading experts in ethics, digital technology, law, and human rights and various Ministries of Health members to produce a report that is called  Ethics & Governance of Artificial Intelligence for Health . This report identifies ethical challenges to using AI in healthcare, identifies risks, and outlines six  consensus principles  to ensure AI works for the public’s benefit:

• Protecting autonomy
• Promoting human safety and well-being
• Ensuring transparency
• Fostering accountability
• Ensuring equity
• Promoting tools that are responsive and sustainable

The WHO report also provides recommendations that ensure governing AI for healthcare both maximizes the technology’s promise and holds healthcare workers accountable and responsive to the communities and people they work with.

AI provides opportunities to help reduce human error, assist medical professionals and staff, and provide patient services 24/7. As AI tools continue to develop, there is potential to use AI even more in reading medical images, X-rays and scans, diagnosing medical problems and creating treatment plans.

AI applications continue to help streamline various tasks, from answering phones to analyzing population health trends (and likely, applications yet to be considered). For instance, future AI tools may automate or augment more of the work of clinicians and staff members. That will free up humans to spend more time on more effective and compassionate face-to-face professional care.

When patients need help, they don’t want to (or can’t) wait on hold. Healthcare facilities’ resources are finite, so help isn’t always available instantaneously or 24/7—and even slight delays can create frustration and feelings of isolation or cause certain conditions to worsen.

IBM® watsonx Assistant™ AI healthcare chatbots  can help providers do two things: keep their time focused where it needs to be and empower patients who call in to get quick answers to simple questions.

IBM watsonx Assistant  is built on deep learning, machine learning and natural language processing (NLP) models to understand questions, search for the best answers and complete transactions by using conversational AI.

Get email updates about AI advancements, strategies, how-tos, expert perspective and more.

See IBM watsonx Assistant in action and request a demo

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.

• Reference Manager
• Simple TEXT file

People also looked at

Mini review article, artificial intelligence assisted acute patient journey.

• 1 Research Fellow, NeuroCare.AI Neuroscience Academy, Dallas, TX, United States
• 2 NeuroCare.AI, Dallas, TX, United States
• 3 Neurologypocketbook.com, Dallas, TX, United States

Artificial intelligence is taking the world by storm and soon will be aiding patients in their journey at the hospital. The trials and tribulations of the healthcare system during the COVID-19 pandemic have set the stage for shifting healthcare from a physical to a cyber-physical space. A physician can now remotely monitor a patient, admitting them only if they meet certain thresholds, thereby reducing the total number of admissions at the hospital. Coordination, communication, and resource management have been core issues for any industry. However, it is most accurate in healthcare. Both systems and providers are exhausted under the burden of increasing data and complexity of care delivery, increasing costs, and financial burden. Simultaneously, there is a digital transformation of healthcare in the making. This transformation provides an opportunity to create systems of care that are artificial intelligence-enabled. Healthcare resources can be utilized more justly. The wastage of financial and intellectual resources in an overcrowded healthcare system can be avoided by implementing IoT, telehealth, and AI/ML-based algorithms. It is imperative to consider the design principles of the patient's journey while simultaneously prioritizing a better user experience to alleviate physician concerns. This paper discusses the entire blueprint of the AI/ML-assisted patient journey and its impact on healthcare provision.

Introduction

Artificial intelligence is being used in the industry to leverage data for logistics. Previously, decisions were made at a human level to shelf individual products in a supermarket. Now we see that data enhances human decisions to find the right product and make seasonal recommendations. Similarly, AI will be helping in the complete journey of patients in terms of pre-hospital alert and in-hospital stay, and eventually, creating a pathway for post-hospital care. The healthcare industry is shifting its focus from decreasing readmissions to reducing admissions ( Kang et al., 2020 ). Telehealth, health IoT, and other medical devices are being introduced every day. The FDA has recently published a general framework to streamline the integration of medical devices into the healthcare system ( FDA, 2021 ).

Almost 25% of the US healthcare budget is being wasted due to multiple factors such as lack of coordination, over-treatment or low-value care, complex administrative procedures, and failure to provide care. Also, the healthcare system is getting crowded due to fewer human resources available ( Shrank et al., 2019 ). We have observed the collapse of the healthcare system during the COVID-19 pandemic, and many patients with chronic diseases were left without a physician's consultation. We can improve healthcare provision by leveraging IoT, telehealth, and AI/ML-based algorithms with clinical decision support systems. AI/ML-based applications help apply the 4p model of healthcare (predictive, preventive, personalized, and participatory) ( Briganti and Olivier, 2020 ).

Unfortunately, the current available solutions are isolated, and the true power of digital generation to create value in medicine comes from an ecosystem approach. Acute care is the most expensive portion of the United States healthcare system. This paper presents a narrative review of artificial intelligence technologies that can provide value to the community and insight to patients for better healthcare management. It aims to explain how a network of small digital solutions working together in coherence can impact healthcare provision throughout the patient's journey ( Figure 1 ). Furthermore, it provides a broader picture of what the digitization of healthcare means for the future.

Figure 1 . Al-Assisted Acute Patient Journey Mapping.

AI/ML has made a significant contribution to a patient's journey from their homes to being released from the hospital. It can aid in gathering data from wearables and earlier health records and, in the end, ingeniously compile a patient's medical history. Furthermore, E-triage and smart clinical decision support systems can help in the allocation of resources more effectively, which in turn will aid in reducing healthcare spending without compromising the standard of care.

Digital tools are now ubiquitous in healthcare, whether in wearable devices or smart monitors in emergency rooms. While digital tools can help patients in a variety of ways, we have focused on the journey of acute patients and highlighted the impact of AI/ML tools on care delivery.

Pre-hospital

One of the critical issues is the identification of patients that require a higher level of care in the pre-hospital settings. There have been many systems based on statistical modeling to generate automated emergency alerts and predict the necessary levels of care, but they lack continuous learning and contextual reinforcement.

Automated emergency response/automated alert system

Automated alert systems may play an essential role in individualized emergency and disaster management. We have limited systems for acute emergencies with connected EMS systems ( Pulsara, 2014 ). These systems need to be expanded, and more intelligent systems that link to wearable devices need to be developed.

The interest in applications such as fall and lost detection has risen over the past few years. Wang and the team reviewed the fall detection devices and concluded that these devices focus on offline analytics instead of automated healthcare monitoring. In addition, Apple has already implemented an emergency alert system and an automated fall detection system in the Apple Watch ( Apple., 2022 ). We need to increase the sensitivity and specificity of fall detection and integrate it into our existing healthcare system to generate accurate and automated alerts. Furthermore, we can replicate the same process for other problems ( Wang et al., 2020 ).

Healthcare data

We live in a digital world, and digital health as a reality relies on data. Machine learning (ML) modules require large amounts of high-quality data as training data to produce “ground truth.” We need to make high-quality real-time data collection a priority. This also creates a double-edged sword, opening the door to privacy and security concerns. Governance and management of data are central to creating secure data warehouses that provide access with differential privacy. Newly developed ML models like federated learning and swarm learning have brought new methods of decentralized learning. These models of decentralized learning will be crucial in healthcare as they provide privacy in learning, minimizing the need for a centralized data warehouse, and are more resourceful as they can push learning to the edge, decreasing cost and latency ( Herresthal et al., 2021 ).

In these times of information explosion, it is difficult for healthcare workers to sift through a large amount of data to find pertinent information. AI-based applications are crucial to separating signal from noise. With modern data visualization techniques, these results, like pertinent labs, and imaging, can be presented in an interactive format. Physicians can rapidly identify and act on collected data with contextual information to make better-informed decisions ( Stanfill and Marc, 2019 ). Additionally, by analyzing general population behavior and discovering new research avenues ( Karan et al., 2022 ) through user-generated content ( Saura et al., 2020 ) on social media, we can effectively fill any gaps in healthcare provision. This is the need of time to revolutionize the patient care paradigms and establish best practices to allocate financial resources, as the healthcare industry faces a heavy financial burden ( Cai et al., 2020 ).

Wearable devices and eHealth integration

The adoption of wearables is increasing each year. More importantly, these are increasingly being integrated into healthcare. Important patient data like blood pressure, pulse, oxygen saturation, temperature, ambulatory ECG, seizure, and stroke alerts ( Figure 2 ) collected from various devices is being integrated using a software development kit (SDK) by Google Fit™ and Apple HealthKit™ ( Henriksen et al., 2018 ). As an example of wearables in healthcare, Embrace 2 detects seizures and notifies users ( Embrace, 2018 ). This provides an opportunity for a collaborative and participatory healthcare environment between the patient and provider. When linked to emergency response services, these systems can significantly decrease latency and cost of care. More importantly, it will enable clinical decision support tools for diagnosis and management ( Dinh-Le et al., 2019 ).

Figure 2 . Health Internet of Things (IoT).

Rather than functioning as a standalone entity, mobile healthcare applications should be integrated into the existing healthcare system. mHealth applications can assist us in deciphering data from wearables and other smart devices. The lack of a standard framework for innovators to work on these avenues is the main barrier to innovation. Labrique and the team proposed a common framework for health-based that addressed all the essential components of the healthcare system, starting with communication, data collection and going all the way to financial transactions and incentives ( Labrique et al., 2013 ). In 2019, the FDA also announced the pre-certification program, establishing a framework for regulating digital health goods ( FDA, 2019 ). The development of a common framework will incentivize the process of innovation in this field.

Electronic-triage and severity index

The capabilities and availability of an electronic triage system (ETS) are improving. Currently, it is only available for a limited number of diseases and in a limited number of facilities. The core problem is the accuracy of determining disease severity. The Emergency Severity Index (ESI) is a commonly used severity indexing system in the USA. However, most patients fall into level 3 on the ESI. Level 3 is a middle ground between high severity requiring extensive resources and low severity with a decreased resource requirement. This, in essence, does not guide physicians in terms of resource allocation. Levin et al. (2018) applied a machine learning model to improve the accuracy of level 3 on ESI. They applied the random forest model to the ESI level 3 patient triage data (65% of 172,726 ED visits). ETS tagged almost 22.9% as level 1, compared to 16.9% by ESI. Patients who were up-triaged to level 2 or 1 by ETS were prone to a critical outcome by almost five times and two times more likely to get hospitalized, and similarly decreased the risk of a critical event in patients who were down-triaged to level 4 or 5. Also, the study showed that the detection rate of secondary clinical outcomes was similar to or better than ESI ( Levin et al., 2018 ).

Similarly, Kang et al. (2020) developed an AI algorithm to predict the need for critical care and compared it to existing triage systems. The combination of the AI algorithm and ESI outdid all other scoring systems ( Kang et al., 2020 ). Integration of AI/ML-based tools into existing EMS can help streamline the care of patients in the ED.

Resource allocation

Machine learning can be used for appropriate resource allocation to alleviate the overcrowding of ERs. Predicting the length of stay in the ER can help streamline the workflow and prevent resource waste. Yousaf and the team have developed a novel algorithm to decrease the length of stay in ED. By combining a chaotic genetic algorithm and Adaboot (meta-learning), they managed to reduce the length of stay from 5.47 to 4.75 h in public testing at the emergency department of the Recoleta Tolentino Neves Hospital, Brazil. By actively monitoring vitals and labs remotely during the transfer and on-site, AI/ML algorithms can reduce ED resource usage by providing clinical decision support and appropriate allocation of resources ( Yousefi et al., 2018 ).

Clinical decision support systems

AI-based clinical decision support system (AI-CDSS) is another avenue that needs to be explored. The poor integration of the existing clinical decision support system can lead to alarm fatigue and physician burnout. However, there are examples to follow in AI-CDSS implementation to improve patient care and provider satisfaction. The Canadian Association of Radiologists has explained in their recent paper that AI-based analysis of imaging will be more sophisticated and easier to integrate into our workflow. The diagnostic accuracy of lung nodules and congenital cataracts has been proven to be comparable to that of a trained physician ( Stanfill and Marc, 2019 ).

Duke University Hospital has implemented a sepsis watch tool to detect early signs of sepsis in patients. Despite the lack of previous experience with integrating algorithms based on AI/ML, they could incorporate the tool into their workflow. An algorithm was trained to create an alert almost 12 h before the presentation. Early analysis showed that the median detection time was 5 h before the patient's deterioration, providing ample time for the physician to confirm the diagnosis and intervene ( Sendak et al., 2020 ).

Clinical monitoring

Medical monitoring devices are based on the threshold alarming principle and do not contain analytical functions. AI/ML-based applications can help analyze large amounts of data and detect subtle clinical anomalies that humans may overlook ( Rush et al., 2019 ). Remote clinical monitoring is vital in monitoring chronic diseases. COVID-19 pandemic has dramatically pushed the healthcare industry toward telehealth and remote monitoring devices. The remote monitoring of blood glucose levels, atrial fibrillation, epilepsy, BP, pulse, temperature, and oxygen saturation can significantly help physicians intervene timely. Also, patients per day visiting can be reduced by resolving minor problems with the help of telehealth and remote monitoring.

Clinical documentation

Documenting clinical journeys with contextual information is helpful for patient pathology accounts and is critical to maintaining quality standards. There is a significant burden for patients and allied health professionals (clinical coordinators, nurse educators, EMS responders) in terms of time to ensure proper time-stamped documentation. Several technologies have recently been introduced to create an ambient intelligence environment. These technologies can improve the quality of documentation while simultaneously reducing the burden on clinicians. (1) Automated Voice Transcription, (2) Digital Voice Assistant for providers, and (3) Voice assistant for patients. However, these technologies need to be incorporated and interoperable with deep integration at a system level to achieve accurate ambient intelligence ( Microsoft, 2021 ).

Accurate clinical documentation leads to accurate medical coding, which is essential for (1) reimbursement, (2) quality improvement, and (3) future resource allocation planning. Implementing a common framework is a crucial requirement to allow the interoperability of AI/ML algorithms in the healthcare industry. This standardization and common framework will allow us to have resource planning at the level of a state, country, or even continents compared to county and hospital systems ( Stanfill and Marc, 2019 ).

Automated EMR summary

The current EHR applications are hindering the provision of healthcare rather than helping it. The process of finding relevant information is usually manual. EHR systems can be improved by introducing customized EHR systems, using open-source software and customizing it, and incorporating AI/ML-driven applications. Some famous companies such as Epic, Allscripts, Cerner, and Athena are introducing AI/ML-based EHR tools and decision support systems to tackle the problem of data explosion. Many other startups and big names, such as Amazon Web Services and Google, have also introduced cutting-edge AI/ML-based tools ( Davenport et al., 2018 ).

Rajkumar and the team published an article discussing the role of deep learning in developing predictive models and recommending that the Fast Healthcare Interoperability Resource (FHIR) be utilized. They have validated their hypothesis by using medical records from two hospitals. Results showed that the deep learning predictive model had outperformed the conventional prediction models in predicting mortality, prolonged length of stay, readmission, and discharge diagnosis. Deep learning predictive models can eliminate many variables that are the main hurdle in conventional predictive models. In addition to presenting relevant charts, history, and labs, to the physicians and paramedical staff, AI/ML integration can also help develop future learning aids ( Rajkomar et al., 2018 ).

Disposition and continuity of care

As mentioned above, AI can be crucial for judicious, value-based resource allocation. However, it can be instrumental in automating patient continuity of care. One of the critical issues in mapping the patient journey is quickly moving the patient toward rehabilitation and a home environment. Patients with continuous monitoring will give us the data to make decisions over a longitudinal period compared to brief clinical visits by providers, even in an inpatient setting.

For better health outcomes, it is crucial to avoid unnecessary hospital stays and timely discharge of patients. AI/ML-based models can predict the patients who should be discharged based on their medical records and detect the barriers to discharge ( Safavi et al., 2019 ). Mitigating those barriers can significantly reduce the anxiety of extended stays of patients on one hand while optimizing resource allocation and healthcare workers' efficiency on the other. As shown by a TEND Model study ( van Walraven and Forster, 2018 ), AI models can predict the number of patients discharged per day. As data becomes more accessible, it will reduce costs and avoid repeating labs. Providing progressive summaries should be an integral part of the patient journey, and AI can aid providers. It can help transfer medical records along with patients without any loss of information.

As we noted above, we can use wearables not only to detect events but also to improve follow-up visitation frequency both in-person and via telehealth. Bian et al. (2020) conducted research at Peking Union Medical College Hospital. They concluded that AI-assisted follow-up of individual patients is comparable to manual follow-up by phone calls but in a much lesser time, i.e., 0 h compared to 9.3 h per 100 patients. Such interventions can be made sooner via notification. Physicians can provide care by telepresence or in person, depending on the patient's situation, rather than assigning arbitrary 2, 4, 6-week visits ( Bian et al., 2020 ).

Health education

Education of the patient and the patient's family is essential to recovery, and many chronic conditions require a higher level of care. Relevant, just-in-time, and mixed media education can significantly improve patients' motivation, adherence, and compliance with medication and rehab therapy. Also, given the electronic nature and reproducibility, the patient can receive education about specific conditions, further increasing the chance of compliance.

Point-of-care education enhanced by AI/ML. Monitoring from various devices can be integrated into one application that can show notifications to educate patients according to their physical needs and current medical status. This kind of tailored education can improve the health outcomes of patients significantly.

Challenges and limitations

Though the advent of digital health tools will improve healthcare, it comes with challenges such as the reliability and validity of mHealth devices, access of the third party to patients' data, and lack of patient data management. Similarly, continuous monitoring might increase stress and raise concerns about patients' health ( Volpato et al., 2021 ). Explainability is another critical challenge. Occasionally, the involved doctor may find it difficult or impossible to explain the algorithms responsible for the diagnosis to the patient ( Davenport and Kalakota, 2019 ). Patients' privacy is another big concern as we propose more integration of healthcare data to produce wisdom through AI. More integrated systems are more vulnerable to cyberattacks and data theft.

There are substantial ethical challenges in AI implementation in healthcare. Accessibility and inequality have already profoundly impacted healthcare. AI can be expensive, and large portions of humanity may not have access to these tools because companies are charging premium prices for their services. Hence, it has the potential to deepen information inequality as well.

The bidirectional flow of information between healthcare providers and patients is crucial for continuously monitoring patients at risk for recurrent episodes of life-threatening disease. Therefore, it is critical to have a certain level of consistency among digital health tools in predicting the severity of symptoms in out-of-hospital patients. The significant challenges that hinder the implementation of these tools range from their reliability to their integration into existing health systems.

Digital healthcare, like digital banking, will be the future norm. We need to use a design process that enables intelligent communication, coordination, and resource management during the whole journey of acute patients instead of introducing isolated solutions. By integrating smart solutions into the healthcare system as a whole, we can save a tremendous amount of money and enhance healthcare delivery. This type of solution will not be limited to acute settings but can be implemented in outpatient, mobile clinics, and other touchpoints in a patient's journey. We will continue to see further convergence of technologies, enabling new experiences for both patients and providers.

Author contributions

TN had worked on literature research and writing the manuscript along with MM. MA was tasked on researching the literature and keep us updated regarding new modalities and publications and later on worked on limitations section. JK was our mentor who had supervised and helped us in formulating the table of content, editing, writing certain sections of manuscript etc. All authors contributed to the article and approved the submitted version.

JK personally paid for this research article.

Conflict of interest

Authors TN, MM and MA were employed by company NeuroCare.AI Neuroscience Academy. Author JK was employed by company NeuroCare.AI and Neurologypocketbook.com.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Apple. (2022). Use fall detection with Apple Watch. Apple Support . Available online at: https://support.apple.com/en-us/HT208944 (accessed August 2, 2022).

Google Scholar

Bian, Y., Xiang, Y., Tong, B., Feng, B., and Weng, X. (2020). Artificial intelligence-assisted system in postoperative follow-up of orthopedic patients: exploratory quantitative and qualitative study. J. Med. Internet Res. 22:e16896. doi: 10.2196/16896

PubMed Abstract | CrossRef Full Text | Google Scholar

Briganti, G., and Olivier, L. M. (2020). Artificial intelligence in medicine: today and tomorrow. Front. Med. 7, 27–27. doi: 10.3389/fmed.2020.00027

Cai, C., Runte, J., Ostrer, I., Berry, K., Ponce, N., Rodriguez, M., et al. (2020). Projected costs of single-payer healthcare financing in the United States: a systematic review of economic analyses. PLoS Med. 17:e1003013. doi: 10.1371/journal.pmed.1003013

Davenport, T., and Kalakota, R. (2019). The potential for artificial intelligence in healthcare. Future Healthc. J. 6, 94–98. doi: 10.7861/futurehosp.6-2-94

Davenport, T. H., Hongsermeier, T. M., and Mc Cord, K. A. (2018). Using AI to Improve Electronic Health Records . Available online at: https://hbr.org/2018/12/using-ai-to-improve-electronic-health-records (accessed March 16, 2022).

Dinh-Le, C., Chuang, R., Chokshi, S., and Mann, D. (2019). Wearable health technology and electronic health record integration: scoping review and future directions. JMIR Mhealth Uhealth 7:e12861. doi: 10.2196/12861

Embrace, W. (2018). Embrace2 Seizure Monitoring | Smarter Epilepsy Management | Embrace Watch | Empatica. Empatica . Available online at: https://www.empatica.com/embrace2/ (accessed April 5, 2022).

FDA (2019). Developing a Software Pre-certification Program: A Working Model. U.S. Food Drug version 1.0 . Available online at: https://www.fda.gov/media/119722/download (accessed February 26, 2022).

FDA (2021). Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD). Discussion Paper and Request for Feedback . Available online at: https://www.fda.gov/news-events/press-announcements/fda-releases-artificial-intelligencemachine-learning-action-plan (accessed February 26, 2022).

Henriksen, A., Haugen Mikalsen, M., Woldaregay, A. Z., Muzny, M., Hartvigsen, G., Hopstock, L. A., et al. (2018). Using fitness trackers and smartwatches to measure physical activity in research: analysis of consumer wrist-worn wearables. J. Med. Internet Res. 20:e110. doi: 10.2196/jmir.9157

Herresthal, W. S., Schultze, H., Shastry, K. L., Manamohan, S., Mukherjee, S., Garg, V., et al. (2021). Swarm learning for decentralized and confidential clinical machine learning. Nature 594, 265–270. doi: 10.1038/s41586-021-03583-3

Kang, D. Y., Cho, K. J., Kwon, O., Kwon, J. M., Jeon, K. H., Park, H., et al. (2020). Artificial intelligence algorithm to predict the need for critical care in prehospital emergency medical services. Scand. J. Trauma Resusc. Emerg. Med. 28:17. doi: 10.1186/s13049-020-0713-4

Karan, A., Mijwil, M. M., Sonia Al-Mistarehi, A-H., Safwan, A., Gök, M., Zein, A. M. A., and Safaa, H. A. (2022). Has the future started? The current growth of artificial intelligence, machine learning, and deep learning. Iraqi J. Comput. Sci. Math. 3, 115–123. doi: 10.52866/ijcsm.2022.01.01.013

Labrique, A. B., Vasudevan, L., Kochi, E., Fabricant, R., and Mehl, G. (2013). mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob. Health Sci. Pract. 1, 160–171. doi: 10.9745/GHSP-D-13-00031

Levin, S., Toerper, M., Hamrock, E., Hinson, J. S., Barnes, S., Gardner, H., et al. (2018). Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann. Emerg. Med. 71, 565–574.e562. doi: 10.1016/j.annemergmed.2017.08.005

Microsoft (2021). Project EmpowerMD: Medical Conversations to Medical Intelligence. Microsoft Research . Available online at: https://www.microsoft.com/en-us/research/project/empowermd/ (accessed April 5, 2022).

Pulsara (2014). Pulsara| Real-Time Team Communication Across Healthcare Entities . Available online at: https://www.pulsara.com/ (accessed April 5, 2022).

Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., et al. (2018). Scalable and accurate deep learning with electronic health records. NPJ Digit. Med. 1:18. doi: 10.1038/s41746-018-0029-1

Rush, B., Celi, L. A., and Stone, D. J. (2019). Applying machine learning to continuously monitored physiological data. J. Clin. Monit. Comput. 33, 887–893. doi: 10.1007/s10877-018-0219-z

Safavi, K. C., Khaniyev, T., Copenhaver, M., Seelen, M., Zenteno Langle, A. C., Zanger, J., et al. (2019). Development and validation of a machine learning model to aid discharge processes for inpatient surgical care. JAMA Netw. Open 2:e1917221. doi: 10.1001/jamanetworkopen.2019.17221

Saura, J. R., Reyes-Menendez, A., and Thomas, S. B. (2020). Gaining a deeper understanding of nutrition using social networks and user-generated content. Internet Intervent. 20:100312. doi: 10.1016/j.invent.2020.100312

Sendak, M. P., Ratliff, W., Sarro, D., Alderton, E., Futoma, J., Gao, M., et al. (2020). Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR Med. Inform. 8:e15182. doi: 10.2196/15182

Shrank, W. H., Rogstad, T. L., and Parekh, N. (2019). Waste in the US health care system: estimated costs and potential for savings. JAMA 322, 1501–1509. doi: 10.1001/jama.2019.13978

Stanfill, M. H., and Marc, D. T. (2019). Health information management: implications of artificial intelligence on healthcare data and information management. Yearb. Med. Inform. 28, 56–64. doi: 10.1055/s-0039-1677913

van Walraven, C., and Forster, A. J. (2018). The TEND (Tomorrow's Expected Number of Discharges) model accurately predicted the number of patients who were discharged from the hospital the next day. J. Hosp. Med. 13, 158–163. doi: 10.12788/jhm.2802

Volpato, L., Del Río Carral, M., Senn, N., and Santiago Delefosse, M. (2021). General practitioners' perceptions of the use of wearable electronic health monitoring devices: qualitative analysis of risks and benefits. JMIR Mhealth Uhealth 9:e23896. doi: 10.2196/23896

Wang, X., Ellul, J., and Azzopardi, G. (2020). Elderly fall detection systems: a literature survey. Front. Robot. AI 7:71. doi: 10.3389/frobt.2020.00071

Yousefi, M., Yousefi, M., Ferreira, R. P. M., Kim, J. H., and Fogliatto, F. S. (2018). Chaotic genetic algorithm and Adaboost ensemble metamodeling approach for optimum resource planning in emergency departments. Artif. Intell. Med. 84, 23–33. doi: 10.1016/j.artmed.2017.10.002

Keywords: artificial intelligence, acute patient journey, electronic-triage, health IoT, Automated EMR summary, AI-based clinical decision support system

Citation: Nazir T, Mushhood Ur Rehman M, Asghar MR and Kalia JS (2022) Artificial intelligence assisted acute patient journey. Front. Artif. Intell. 5:962165. doi: 10.3389/frai.2022.962165

Received: 06 June 2022; Accepted: 12 September 2022; Published: 04 October 2022.

Reviewed by:

Copyright © 2022 Nazir, Mushhood Ur Rehman, Asghar and Kalia. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Talha Nazir, talhanazir44@gmail.com ; Junaid S. Kalia, junaidkalia@aineurocare.com

This article is part of the Research Topic

Insights in AI: Medicine and Public Health 2022

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 20 May 2024

Development of a long noncoding RNA-based machine learning model to predict COVID-19 in-hospital mortality

• Yvan Devaux   ORCID: orcid.org/0000-0002-5321-8543 1 ,
• Lu Zhang 2 ,
• Andrew I. Lumley   ORCID: orcid.org/0000-0001-5935-3327 1 ,
• Kanita Karaduzovic-Hadziabdic 3 ,
• Vincent Mooser   ORCID: orcid.org/0000-0002-8632-0448 4 ,
• Simon Rousseau   ORCID: orcid.org/0000-0002-8773-575X 5 ,
• Muhammad Shoaib   ORCID: orcid.org/0000-0002-4854-4635 6 ,
• Venkata Satagopam   ORCID: orcid.org/0000-0002-6532-5880 6 ,
• Muhamed Adilovic   ORCID: orcid.org/0000-0002-5326-0944 3 ,
• Prashant Kumar Srivastava 7 ,
• Costanza Emanueli   ORCID: orcid.org/0000-0002-2392-0702 7 ,
• Fabio Martelli   ORCID: orcid.org/0000-0002-8624-7738 8 ,
• Simona Greco 8 ,
• Lina Badimon   ORCID: orcid.org/0000-0002-9162-2459 9 ,
• Teresa Padro   ORCID: orcid.org/0000-0003-1921-954X 9 ,
• Mitja Lustrek   ORCID: orcid.org/0000-0003-3219-2935 10 ,
• Markus Scholz   ORCID: orcid.org/0000-0002-4059-1779 11 ,
• Maciej Rosolowski 11 ,
• Marko Jordan 10 ,
• Timo Brandenburger 12 ,
• Bettina Benczik   ORCID: orcid.org/0000-0003-0379-2181 13 ,
• Bence Agg   ORCID: orcid.org/0000-0002-6492-0426 13 ,
• Peter Ferdinandy 13 ,
• Jörg Janne Vehreschild   ORCID: orcid.org/0000-0002-5446-7170 14 , 15 , 16 , 17 ,
• Bettina Lorenz-Depiereux 18 ,
• Marcus Dörr 19 ,
• Oliver Witzke 20 ,
• Gabriel Sanchez 21 ,
• Seval Kul 21 ,
• Andy H. Baker   ORCID: orcid.org/0000-0003-1441-5576 22 , 23 ,
• Guy Fagherazzi 24 ,
• Markus Ollert   ORCID: orcid.org/0000-0002-8055-0103 25 , 26 ,
• Ryan Wereski 27 ,
• Nicholas L. Mills   ORCID: orcid.org/0000-0003-0533-7991 27 , 28 &
• Hüseyin Firat 21  

Nature Communications volume  15 , Article number:  4259 ( 2024 ) Cite this article

Metrics details

• Epidemiology
• Predictive markers
• Prognostic markers

Tools for predicting COVID-19 outcomes enable personalized healthcare, potentially easing the disease burden. This collaborative study by 15 institutions across Europe aimed to develop a machine learning model for predicting the risk of in-hospital mortality post-SARS-CoV-2 infection. Blood samples and clinical data from 1286 COVID-19 patients collected from 2020 to 2023 across four cohorts in Europe and Canada were analyzed, with 2906 long non-coding RNAs profiled using targeted sequencing. From a discovery cohort combining three European cohorts and 804 patients, age and the long non-coding RNA LEF1-AS1 were identified as predictive features, yielding an AUC of 0.83 (95% CI 0.82–0.84) and a balanced accuracy of 0.78 (95% CI 0.77–0.79) with a feedforward neural network classifier. Validation in an independent Canadian cohort of 482 patients showed consistent performance. Cox regression analysis indicated that higher levels of LEF1-AS1 correlated with reduced mortality risk (age-adjusted hazard ratio 0.54, 95% CI 0.40–0.74). Quantitative PCR validated LEF1-AS1’s adaptability to be measured in hospital settings. Here, we demonstrate a promising predictive model for enhancing COVID-19 patient management.

Introduction

On October 2nd, 2023, the Nobel Assembly at Karolinska Institute awarded the 2023 Nobel Prize in Physiology or Medicine to Professors Katalin Karikó and Drew Weissman for their discovery that modifying the uridine nucleoside to pseudouridine blocks the inflammatory response consecutive to cell delivery of messenger RNA (mRNA) molecules, thereby increasing the production of proteins encoded by the mRNA 1 . This discovery 15 years ago revolutionized the therapeutic potential of mRNA and allowed the rapid development of mRNA vaccines against SARS-CoV-2. RNAs have come of age, not only for vaccines, but for diagnosing and treating disease 2 .

On March 2020, partners of the EU-CardioRNA COST Action network 3 , 4 gathered forces to develop a RNA-based diagnostic test based on artificial intelligence (AI) to predict clinical outcomes after COVID-19 5 . The rationale for this endeavor was that leveraging the power of non-coding RNAs may help reduce the devastating consequences of COVID-19 pandemic 6 . Indeed, risk prediction models could inform about clinical management of patients. Non-coding RNAs, unable to encode proteins like the better-known mRNAs, are regulated in virtually all pathological conditions and, since they are detectable in the blood, they have emerged in recent years as a new reservoir of non-invasive candidate biomarkers and therapeutic targets. Our consortium previously characterized a panel of 2906 cardiac-enriched or heart failure-associated long non-coding RNAs (lncRNAs) (FIMICS panel) 7 which, together with an in-house developed bioinformatics pipeline to maximize the benefit of targeted sequencing (Firalink pipeline 8 ), provides a new tool to discover disease-associated lncRNAs with potential to help in diagnosis and risk stratification. Since the FIMICS panel contains many inflammation-related lncRNAs and inflammation is a hallmark of host response to infection by SARS-CoV-2, we thought that it may be of usefulness to identify predictors of COVID-19 outcome.

In the H2020-funded FastTrack COVIRNA project, we aimed to apply the FIMICS panel to identify lncRNAs predictive of COVID-19 outcome. We used blood samples and clinical data from four cohorts of COVID-19 patients totaling 1286 patients. Three cohorts with 804 patients were merged as a discovery cohort for feature selection and choice of best performing machine learning (ML) models. The fourth cohort of 482 patients was used for validation purposes. Here, we have built a model based on one lncRNA and age able to predict in-hospital mortality with an area under the receiver operating characteristic curve (AUC) of 0.83 (0.82–0.84).

Study design

The study design is illustrated in Fig.  1 . The study population consisted of a total of 1329 patients with COVID-19, shared between a discovery cohort ( n  = 818) and a validation cohort ( n  = 511) used for ML model selection and evaluation, respectively. Three European cohorts were included in the discovery cohort (PrediCOVID from Luxembourg, n  = 141; NAPKON from Germany, n  = 557; and ISARIC4C from United Kingdom, n  = 120) and one cohort from Canada constituted the validation cohort (BQC19, n  = 511). Whole blood samples collected in PAXgene RNA tubes at baseline in all patients were centrally stored at −80 °C in a NF S96-900 certified biobank. RNA extraction, quality check, library preparation and RNA sequencing using the FIMICS panel were performed in our core lab. Raw sequencing data were normalized and merged with clinical data of patients in our central database. Data were curated and made available for analysis using ML/AI. Patients with RNAseq datasets that did not meet the quality criteria described in the Materials and Methods section, or with blood samples not collected at the time of enrolment in the study, or for which survival data were not available, were excluded from the analysis. After curation and quality checks, combined RNAseq datasets and clinical data from 136 PrediCOVID, 556 NAPKON, 112 ISARIC4C (804 patients for the discovery cohort) and 482 BQC19 patients for the validation cohort were available for ML analysis. Overall, a total of 1286 full datasets representing each a unique patient were available for analysis. After lncRNA selection by ML, a translational study was conducted by qPCR in a subgroup of 86 patients from the NAPKON cohort for which leftover RNA was available.

Study design.

Baseline characteristics of patients in the analysis are reported in Table  1 , in which the three merged European cohorts used for discovery are compared to the Canadian cohort used for validation of the selected features and ML models. Missing data are indicated and were imputed using missForest. The median number of days in hospital was 9 (Q1 = 5, Q3 = 19) and 8 (Q1 = 4, Q3 = 19) for the ISARIC4 and BQC19 cohort, respectively. In all cohorts, patients who died in hospital were older than survivors, more often had cardiovascular disease, and more often received oxygen therapy. Being a male was associated with a higher risk of death in the merged European cohorts. Diabetes and chronic lung disease were also risk factors in this cohort. Patients in the Canadian cohort were older, were more often females and were less often smokers than patients in the merged European cohorts. Supplementary Table  1 shows the characteristics of the three European cohorts individually, together with the nature of common COVID-19 symptoms across cohorts. The PrediCOVID cohort had younger patients than the two other cohorts and none of them died during the follow-up period. There were more smokers at the time of enrolment in the PrediCOVID cohort than in the NAPKON and ISARIC4C cohorts. Common baseline symptoms across the European cohorts included fever, headache, cough and dyspnea, which were less frequent in survivors (Supplementary Table  1 ). Ethnicity data was available in the NAPKON cohort, in which most patients were Caucasian and no apparent association between ethnicity and survival was found (Supplementary Table  1 ). In this cohort, vaccinated people had a lower risk of death as compared to non-vaccinated people (Supplementary Table  1 ). Vaccination data was unavailable in other cohorts.

Machine learning model building and characterization

We performed feature selection on the training set and evaluated five different ML classifiers (RF, kNN, Logit, MLP, SVM, XGB) on the discovery cohort derived from the 3 combined European cohorts ( n  = 804) in each of the 100 iterations as described in the Materials and Methods section and in Supplementary Fig.  1 . The median number of features selected in each iteration was 21 (Q1 = 16, Q3 = 25). The performance of each model to predict in-hospital mortality is shown in Table  2 . The logistic regression model (Logit) with the selected features in each iteration provided the most accurate prediction of in-hospital mortality with an AUC of 0.83 (95% CI 0.81–0.84), an accuracy of 0.74 (95% CI 0.73–0.76), a sensitivity of 0.77 (95% CI 0.74–0.79), and a specificity of 0.72 (95% CI 0.69–0.75).

The analysis yielded the selection of two features, age and the lncRNA LEF1-AS1, which appeared in 82 and 63 iterations out of the 100 iterations performed, respectively (Fig.  2A ). LEF1-AS1 is a lncRNA of 3,360 nucleotides transcribed from the lymphoid enhancer binding factor 1 (LEF1) locus located on chromosome 4. In the merged European cohorts (discovery cohort, n  = 804), patients who survived were younger and had higher expression levels of LEF1-AS1 than patients who died (Fig.  2B, C ). There was a significant albeit moderate negative correlation between age and LEF1-AS1 in this cohort (Fig.  2D ), as well as in the Validation cohort ( r  = −0.35, p  < 0.01). Also, LEF1-AS1 was differentially expressed between males and females in the Discovery (Fig.  2E ) and in the Validation cohort ( p  < 0.01 and p  = 0.02, respectively). The expression of LEF1-AS1 was associated with cancer diagnosis with an odds ratio of 0.71 [0.55–0.90] and 0.66 [0.52–0.84] in the NAPKON and BQC19 cohorts, respectively. The Shapley beeswarm plots shown in Supplementary Fig.  2 attest that higher age and lower expression of LEF1-AS1 led to positive SHAP values and thus had positive impacts on model output.

A Line plot of the selected times of the 10 most selected features. X -axis: the name of the features. SEQXXXX are the codes of the probes of the FIMICS panel. SEQ0235 probe recognizes the lncRNA LEF1-AS1. Y -axis: the number of times a feature appeared in the 100 iterations of the feature selection process. B , C Box/violin plots of age and LEF1-AS1 expression, which were significantly increased and decreased in the non-survivors group ( n  = 62 patients) of the European cohorts, respectively. P -value is from 2 sided Student’s t test. FDR (false discovery rate) is from DESeq2 algorithm. D Correlation between age and LEF1-AS1. A Pearson Correlation coefficient and a two-sided t -test p -value are indicated. E Comparison between expression levels of LEF1-AS1 in males ( n  = 480 patients) and females ( n  = 324 patients). P -value is from a two-sided Student’s t test. In B , C and E , the box is drawn from Q1 (25th percentile) to Q3 (75th percentile) with a horizontal line inside it to denote the median. The length of the whiskers indicate 1.5 times of IQR (Interquartile range Q3–Q1).

The five different ML classifiers with the two selected features (age and LEF1-AS1) were then evaluated on the discovery cohort in 100 iterations, using the same data splits as for feature selection. The model MLP exhibited the highest AUC of 0.82 (95% CI: 0.80–0.84) (Table  3 and Supplementary Fig.  3 ). There was no significant difference in performance between the models with the features from each iteration and the model with age and LEF1-AS1 (Tables  2 and 3 ). Adding the third best predictor selected during the feature selection step did not improve the performance of the prediction model in the balanced (AUC 0.84 [0.82–0.86]. p  = 0.11 for comparison with the model without oxygen therapy) and imbalanced (AUC = 0.83 [0.82–0.84], p  = 0.91 for comparison with the model without oxygen therapy) discovery dataset.

When predicting in-hospital mortality for the balanced datasets from the validation cohort (i.e., same number of survivors and deceased patients, Supplementary Fig.  1 ), the MLP model achieved an AUC of 0.84 (95% CI 0.82–0.86), an accuracy of 0.76 (95% CI 0.74–0.78), a sensitivity of 0.77 (95% CI 0.75–0.79), and a specificity of 0.75 (95% CI 0.72–0.78) (Table  4 ). We extended the testing to the original imbalanced datasets using the 2 selected features, yielding the following metrics for the discovery cohort: AUC 0.83 (95% CI 0.82–0.84), balanced accuracy 0.78 (95% CI 0.77–0.79), sensitivity 0.86 (95% CI 0.84–0.88), and specificity 0.71 (95% CI 0.70–0.71); for the validation cohort, the metrics were AUC 0.83 (95% CI 0.82–0.84), balanced accuracy 0.75 (95% CI 0.74–0.77), sensitivity 0.79 (95% CI 0.76–0.82), and specificity 0.72 (95% CI 0.71–0.73). The model with age alone yielded AUC 0.78 (95% CI 0.77–0.80), 0.79 (95% CI 0.78–0.80), 0.78 (95% CI 0.76–0.79), and 0.78 (95% CI 0.76–0.79) in balanced/imbalanced discovery and balanced/imbalanced validation cohort, respectively. Adding LEF1-AS1 significantly improved the model performance (Fig.  3 , Supplementary Fig.  4 ). Adding sex and/or the other 2 features which were selected more than 40 times in the feature selection iterations (oxygen therapy and SEQ0986, Fig.  2A ) did not significantly improve the model performance (Supplementary Fig.  5 ). Missing data imputation did not significantly influence results since the MLP model run without prior imputation of missing age data (concerning only 4 patients of the Discovery cohort, Table  1 ) reached an AUC of 0.83 (95% CI 0.81–0.85) and a balanced accuracy of 0.78 (95% CI 0.76–0.80) (Supplementary Table  3 ).

The evaluation was performed on the imbalanced data with 20 repeated 5-fold cross-validation. The error bars display the confidence interval. We indicated the significant ( p  < 0.05) difference compared to the model with 2 features using a two-sided Student’s t test.

We compared the predictive performance of the MLP model with age and LEF1-AS1 to a previously published model involving age, sex, C-reactive protein (CRP) and lactate dehydrogenase (LDH). As shown in Supplementary Fig.  6 , our MLP model and the four-parameter model had similar capacity to predict mortality in the BQC19 cohort (AUC 0.83 [0.82–0.84] vs 0.85 [0.84–0.86], respectively). Our MLP model outperformed the four-parameter model in the NAPKON cohort (AUC 0.82 [0.81–0.83] vs 0.78 [0.76–0.79], respectively). Brier score analysis was used to assess the calibration of our MLP model, where a lower Brier score indicates a more calibrated model. This analysis revealed a similar (for BQC-19 data) and a lower score (for NAPKON data) for our MLP model compared to the previously published four-parameter model (Supplementary Fig.  6 ).

Survival analysis

We then evaluated the association between the lncRNA LEF1-AS1 and in-hospital mortality using survival analysis. Patients with high levels of LEF1-AS1 were at low risk of death (age-adjusted HR 0.59, 95% CI 0.36–0.96) in the ISARIC4C subgroup of the discovery cohort (Fig.  4A ). In the validation cohort, the HR was 0.54 (95% CI 0.40–0.74) (Fig.  5A ). Kaplan–Meier curves using different cut-offs for LEF1-AS1 expression demonstrate the observed association of high expression levels of LEF1-AS1 with low risk of death (Figs.  4B and 5B ).

A Forest plot of the Hazard Ratio (HR) from Cox regression analysis shows a higher risk of death for older patients and a lower risk for patients with higher LEF1-AS1 expression level. The dots and the error bars display the HR and the confidence interval, respectively. The p values are from a two-sided Wald test. B Kaplan–Meier curves using the stratified LEF1-AS1 expression with the first quartile (Q1), the median and the third quartile (Q3), respectively. Patients with LEF1-AS1 expression levels below or equal to the first quartile (Q1) are at a high risk of death. The p values are from a two-sided log-rank test.

Note that 44 patients of the 482 patients of the BQC19 cohort did not have information on the number of hospitalized days, yet had information on in-hospital mortality. A Forest plot of the Hazard Ratio (HR) from Cox regression analysis shows a higher risk of death for older patients and a lower risk for patients with higher LEF1-AS1 expression level. The dots and the error bars display the HR and the confidence interval, respectively. The p values are from a two-sided Wald test. B Kaplan–Meier curves using the stratified LEF1-AS1 expression with the first quartile (Q1), the median and the third quartile (Q3), respectively. Patients with LEF1-AS1 expression levels below or equal to the first quartile (Q1) are at a high risk of death. The p values are from a two-sided log-rank test.

Translational perspective

To gain further insights into the feasibility of LEF1-AS1 testing in the hospital environment, e.g., for the development of a molecular diagnostic assay, we set-up a quantitative PCR protocol to measure blood levels of LEF1-AS1 in a subgroup of 84 patients of the NAPKON cohort. Patient characteristics are shown in Supplementary Table  2 . 41 patients survived and 43 died in hospital. The two groups were age-matched, sex-balanced and had similar average body mass index (BMI). We first validated that expression levels of LEF1-AS1 as assessed by quantitative PCR were correlated with the levels obtained by RNAseq using the FIMICS panel (Fig.  6A ). Moreover, as shown in Fig.  6B , patients who died during their hospital stay had a lower expression of LEF1-AS1 compared to survivors ( p  = 0.003). A patient was ~1.4 times as likely to survive at hospital discharge for every 1 unit (log2 transformed expression) increase in LEF1-AS1 (OR 1.39 95% CI 1.10–1.76). When we dichotomized the log2-transformed expression levels of LEF1-AS1 using a cut-off determined by the Youden’s index (to maximize specificity and sensitivity), patients who had LEF1-AS1 levels above 0.043 were 5 times more likely to survive after hospital discharge (OR 5.08 95% CI 2.02–12.73).

A Correlation between qPCR and RNAseq data obtained with the FIMICS panel. The gray area displays the confidence interval. A Spearman’s rank correlation coefficient and a two-sided t -test p value are indicated. B Box/violin plots of LEF1-AS1 expression, which was decreased in deceased patients ( n  = 43 patients). The box is drawn from Q1 (25th percentile) to Q3 (75th percentile) with a horizontal line inside it to denote the median. The length of the whiskers indicate 1.5 times IQR (Interquartile range Q3–Q1). P -value is from a two-sided Student’s t -test.

We hereby report the characterization of a machine learning model based on age and the lncRNA LEF1-AS1 able to predict in-hospital mortality of COVID-19 patients with clinically relevant accuracy.

COVID-19 pandemic has impacted peoples’ lives in many different ways. Healthcare management during the pandemic has been challenging, partly due to lack of preparedness and ability to triage the large numbers of people with infection arriving at the Emergency Department. Methods to help triage and risk stratify patients would have greatly facilitated the work of healthcare providers. Being able to identify patients at high-risk of poor outcome or death, or on the other hand patients with a high chance of survival, would have allowed a more personalized approach to the use of healthcare that could have improved outcomes overall.

Initiated in March 2020 during the first phase of the pandemic, this study aimed to cope with the above issue and design a new method to identify patients at high risk of poor outcome after being infected with SARS-CoV-2. We applied our previously developed FIMICS panel of lncRNAs 7 to whole blood samples of COVID-19 patients collected from four different European cohorts and a Canadian cohort. This panel allows for targeted sequencing, which is about 70 times more sensitive than whole genome sequencing, and therefore more suitable to detect and quantify potentially weakly expressed lncRNAs. Other studies have identified biomarkers of disease severity and outcome of COVID-19 9 , 10 , 11 . We previously reported that LEF1-AS1 expression in peripheral blood cells was negatively associated with disease severity and mortality in a modestly sized cohort of COVID-19 patients 12 , which is consistent with our present investigation in whole blood samples. Models to predict mortality of COVID-19 patients have been previously developed, yet they suffer from a high risk of bias 13 . The MLP model reported in the present study with only two features (age and LEF1-AS1) showed similar predictive performance in the BQC-19 cohort and higher performance in the NAPKON cohort compared to a model including age, sex, CRP and LDH. As compared to previous reports 13 , the strength of our study relies on its methodological aspects which reduce the risks of bias. We conducted a multi-center and well powered study, with patient numbers well above previous studies. We have used a machine learning pipeline including feature selection and testing of multiple machine learning models with Discovery and Validation cohorts, each split into training and testing subgroups. In each cohort, we have evaluated models on the imbalanced datasets using twenty times repeated 5-fold cross validation.

Even though we observed a consistently low expression of LEF1-AS1 in patients with high risk of death, a functional role of LEF1-AS1 in COVID-19 outcome has still to be demonstrated. LEF1-AS1 is an antisense RNA to the lymphoid enhancer binding factor 1 (LEF1) gene encoding a transcription factor expressed in pre-B and T cells which is involved in proliferation, activation of genes in the Wnt/β-catenin pathway and in regulating systemic inflammation. Consistent with our observed lower expression of LEF1-AS1 in severe patients, recent studies have illustrated that B cells undergo significant depletion following SARS-CoV-2 infection 14 . Additionally, pulmonary fibrosis stems from damage to alveoli and is a hallmark of SARS-CoV-2 infection. Recent work has demonstrated that alveolar damage can be suppressed through activation of LEF1, which is mediated by the transcription factor krüppel-like factor 4, thus hinting at a possible protective role of LEF1 following alveolar injury and SARS-CoV-2 infection 15 . These studies suggest a link between LEF1/LEF1-AS1, T or B cell proliferation, alveolar protection and COVID-19 severity which warrants further investigation.

The machine learning protocol used in the present study was inspired by the method from ref. 16 , which used Boruta, a random forest-based algorithm, to select features from electronic health records and evaluate a quantitative marker of coronary artery disease. We adapted their design to suit RNAseq data by adding DESeq2 for differential expression analysis. Many conventional statistical methods, such as t-tests and ANOVA, assume normal data distributions, which is often not the case for data generated by high throughput platforms, such as sequencing. New machine learning methods are able to deal with scale, diverse data distributions, and non-linearity, such as large omics datasets 17 . Multiple machine learning algorithms, including deep learning algorithms, have been developed to build powerful predictive models linking omics data to prediction of clinical outcomes 18 , 19 . While benefiting from the modeling flexibility and robustness, these models often suffer from difficulty in interpreting the role of each individual feature. Identifying biomarkers functionally associated with disease progression could help establish novel hypotheses regarding prevention, diagnosis, and treatment of complex human diseases 20 .

Translational perspectives

The present investigation was conducted using patient’s whole blood samples collected in PAXgene RNA tubes, which are certified for in vitro diagnostics. Other matrices could also be used and we do not exclude that other biomarkers may be found with relevant predictive value. Using quantitative PCR, a technique available in most hospital labs and cost-effective, we confirmed that LEF1-AS1 was readily and reliably detected. Furthermore, we validated that low levels of LEF1-AS1 were associated with a high risk of death. These data support the potential translation of our findings to clinical application.

With the current excitement for the use of RNA as both biomarker and therapeutic targets, lncRNAs may constitute a novel generation of actionable disease-monitoring biomarkers and drugs. Our data showing that lncRNAs are associated with mortality of COVID-19 patients support their potential as theranostic drugs, usable for both risk assessment and treatment of COVID-19. Circular RNAs particularly raised interest for future drug development since these closed RNA molecules are not only able to more stably induce therapeutic protein production compared to linear RNAs, they also have potential to capture and sequester unwanted molecules and thereby function as antisense RNAs, or they can regulate RNA editing 21 . Whether lncRNAs find utility for COVID-19 remains to be determined, as well as whether circRNAs hold similar or superior value to reduce disease burden. It will be interesting in such endeavors to develop multimodal approaches taking into account not only baseline clinical characteristics and biomarkers but also mental health indicators, considering the importance of pre-existing health problems and especially psychological problems in the development of post-COVID condition 22 . It would be interesting to apply a similar approach to see whether lncRNAs are associated with the long term impact of COVID-19, such as long COVID 23 . Considering the prevalence and devastating consequences of this novel disease 24 , setting-up methods to predict the risk of developing long COVID symptoms would have a significant impact on the enormous burden of long COVID or post-COVID symptoms.

Limitations

This work has some limitations. First, since patients enrolled in this study were from the first phase of the pandemic, we assume that most if not all patients were infected by the original SARS-CoV-2 variant. Also, there was limited information on vaccination status due to the fact that there were no widely available vaccines at the time of study enrolment. However, we cannot exclude that some patients were infected by other variants. Hence, we could not test the performance of the model in patients infected by different viral variants. Second, only limited clinical descriptions of the patients enrolled in the study could be provided due to heterogeneity of cohorts and difficulty to merge the clinical data from different cohorts. Third, none of the participants of the Luxembourg PrediCOVID cohort died in hospital, most probably due to the nationwide mass screening program, which allowed an improved control of the virus and an earlier hospitalization of patients 25 . Since this cohort was included at project inception and despite that the main aim of this study was to predict in-hospital mortality, it was kept in analyses and we verified that its removal does not affect study findings. Fourth, survival analysis using Cox regression and Kaplan–Meier curves could be conducted only in the ISARIC4C and BQC19 subgroups for which we had data on time to death. Fifth, even though we tested five different ML classifiers, others could provide stronger predictive value. Lastly, a full functional characterization of the role of LEF1-AS1 in post COVID-19 outcome remains to be done. We identified a machine learning-supported model combining age and the lncRNA LEF1-AS1 predictive of COVID-19 in-hospital mortality. This model may find utility for the management of COVID-19 patients. Its usefulness for long COVID patients remains to be tested.

Patient cohorts

This study was performed in full compliance with the Declaration of Helsinki. Involved cohorts comprise COVID-19-positive patients aged 18 years and older from Luxembourg (PrediCOVID study), Germany (NAPKON study), United Kingdom (ISARIC4C study), and Canada (BQC19 study). The Luxembourg PrediCOVID study was approved by the National Research Ethics Committee of Luxembourg (study Number 202003/07) and was registered under ClinicalTrials.gov (NCT04380987) 26 . The ISARIC-4C study was approved by the Oxford C Research Ethics Committee (Reference 13/SC/0149) (details on study design, registration and approvals are available in the online supplement). For the NAPKON Cross-Sectoral Platform, a primary ethics vote was obtained at the Ethics Committee of the Department of Medicine at Goethe University Frankfurt, Germany (local ethics ID approval 20-924). All further study sites received their local ethics votes at the respective ethics committees. The NAPKON Cross-Sectoral Platform is registered at ClinicalTrials.gov (Identifier: NCT04768998) 27 . The Biobanque québécoise de la COVID-19 (BQC19) study has been approved by the Research Ethics Board of the Center Hospitalier de l’Université de Montréal (CHUM) (#13.389) 28 . Periods of patient enrolment and biological samples collection were as follows: May 2020 - Present for PrediCOVID, July 2020 - Present for NAPKON, February 2020 - September 2020 for ISARIC4C, March 2020 - Present for BQC19. Informed consent was signed by all patients enrolled in these studies. Legal agreements for material and data sharing have been signed between each cohort and COVIRNA project coordinator Luxembourg Institute of Health (LIH).

Sample storage and RNA extraction

All procedures were performed in the ISO 17025, ISO 9001, and CAP accredited facility of Firalis. Whole blood samples collected in PAXgene™ Blood RNA tubes (PreAnalytiX, Cat. #762165; BD Biosciences, Aalst, Belgium) were shipped from the different patient cohorts to our central NF S96-900 certified Biobank and were stored at −80 °C. Whole blood samples were randomized according to age and sex in batches of 64 prior to RNA extraction. Total RNA was extracted with the KingFisher Apex instrument (Cat. #5400930 P, Thermo Scientific, Waltham, MA, USA) using the MagMAX™ for Stabilized Blood Tubes RNA Isolation Kit (Cat. #4451894, Invitrogen, Thermo Scientific). Extracted RNA samples were quantified using the Qubit 3.0 fluorometer (Cat. #Q33216, Invitrogen, Thermo Fisher Scientific) with the RNA high sensitivity Assay kit. Sample quality was assessed using a TapeStation 4150 electrophoresis platform (Cat. #G2992AA, Agilent, Santa Clara, CA, USA).

Library preparation, targeted RNA sequencing and raw data analysis

An extended version of this section is available in the Supplementary Material . Briefly, a second stratified randomization by age and sex was performed in batches of 46 samples prior to library preparation. The libraries were generated by the EpMotion 5075t NGS solution (Cat. #5075000962, Eppendorf, Hamburg, Germany) using the KAPA Stranded RNAseq Kit with RiboErase (HMR; Cat. #634444, Roche diagnostics, Basel, Switzerland) for ribosomal RNA (rRNA) depletion and total RNA libraries construction. The clean-ups were performed with Celemag clean-up beads (Cat. #CMCB57.6, Celemics, Seoul, Korea) and the purified libraries were dual indexed during a 13-cycle PCR using the library preparation box #2 (Cat. # LI20D96, Celemics).

The indexed libraries were then captured using the FIMICS panel targeting 2906 lncRNAs 7 (Cat. #BO5096, Celemics) and purified using Celemag streptavidin coated magnetic beads (Cat. #CMSB5.76, Celemics) and Celemics wash buffer (Cat. #TC4096, Celemics). The on-beads captured sequences were enriched by a 14 cycle PCR and purified using Celemag clean-up beads before quality assessment and quantification. The libraries were then normalized and pooled prior to being sequenced on the NextSeq 2000 platform (Cat. #20038897, Illumina Inc., San Diego, CA, USA) using the NextSeq 2000 P2 kit (Cat. #20046811, Illumina Inc.). Raw sequencing data were analysed using the Firalink pipeline 8 .

Data management and curation

RNA sequencing (RNAseq) datasets with a relative standard deviation <0.46 and with a number of lncRNAs detected with more than 10 reads in less than 10% of the total FIMICS lncRNAs were excluded. LncRNA data were merged with age, sex, and smoking status for the feature selection process. The missing values of these clinical data were imputed using the missForest function from the missForest R package 29 . Voom-transformed RNAseq data was used for ML analysis 30 .

Machine learning models

The three European cohorts (PrediCOVID, NAPKON, ISARIC4C) were combined and used as a discovery cohort, on which a machine learning procedure was iterated 100 times (Supplementary Fig.  1 ), following these steps: (1) random selection of 80% of deceased patients and a balanced set of living patients to construct a training dataset; (2) use of the remaining 20% of deceased patients along with a balanced set of the remaining living patients to form a test dataset; (3) identification of differentially expressed lncRNAs in the training dataset with a false discovery rate (FDR) < 0.00001 using the DESeq2 algorithm 31 ; (4) feature selection in R on clinical variables (age, sex, and smoking status) and differentially expressed lncRNAs from the training dataset using the Boruta function from the Boruta package 32 and the vif function from the rms package ( https://CRAN.R-project.org/package=rms ) with a cut-off of 5 to avoid multi-collinearity; (5) use of repeated (2x) 5-fold cross-validation to fine-tune various machine learning models, including random forest (RF), k-nearest neighbor (kNN), logistic regression (logit), multilayer perceptron (MLP), XGBoost (XGB) and support vector machine (SVM) model in the training dataset using scikit-learn package in Python; (6) evaluation of the model in the test dataset. Features that appeared more than 70 times during the 100 iterations were retained as the selected features that were used to train and evaluate ML models by repeating steps 1, 2, 5 and 6 within 100 iterations with the same seed. The algorithm yielding the model with the highest AUC with the selected features in the test cohort was retained for use in the validation cohort.

The BQC19 cohort was used as the validation cohort. We repeated steps 1 and 2 described above 100 times to split the validation cohort into training and test datasets. In each iteration, a model was trained with the algorithm selected in the discovery cohort using with the features selected there, and evaluated. We also evaluated the selected model on the original imbalanced datasets from the discovery and validation cohort respectively using repeated (20x) 5-fold cross-validation. To test the model robustness, we compared the selected model to the model after adding the top 4 ranked but not selected lncRNAs. The performance metrics, including the AUC, balanced accuracy (accuracy for balanced dataset), sensitivity, and specificity, were reported for the mean and 95% CIs across 100 iterations or the cross-validation. The sensitivity and specificity were determined using 0.5 as the threshold for the predicted class probability.

Quantitative PCR (qPCR)

RNA samples extracted from whole blood samples collected in PAXgene tubes were used to assess the expression levels of LEF1-AS1. 200 ng of each RNA sample were reverse transcribed with the High-capacity cDNA reverse transcription kit (ThermoFisher Scientific, Cat # 4368814). To avoid any batch effect, cDNA samples were then randomized in 3 different batches prior to being assessed by quantitative PCR using the CFX-OPUS-96 Dx qPCR device (Biorad, Temse, Belgium) with IQ SYBR Green Supermix (Biorad). Each sample was quantified in duplicate. The following primer sequences designed with the Beacon Designer software (Premier Biosoft) were used for LEF1-AS1: forward 5′- GTCCATGCTATGACCATCTCCA −3′, reverse 5′- ACACGAGTTAAGGCACATTCA −3′; and for SF3A1 which was used as normalizer: forward 5′- GATTGGCCCCAGCAAGCC-3′, reverse 5′- TGCGGAGACAACTGTAGTACG-3′. Splicing Factor 3a Subunit 1 (SF3A1) was chosen as a housekeeping gene for normalization. Expression levels were calculated by the relative quantification method (ΔΔCt) using the CFX Manager 2.1 software (Bio-Rad).

Statistical analysis

Continuous and categorical variables were compared with two-sided unpaired Student’s t -test and Fisher’s exact test, respectively. A Mann–Whitney test was used to compare non-normally distributed datasets, as assessed by the Shapiro–Wilk test. Correlation between qPCR and RNAseq data was evaluated using the Spearman’s rank test. Cox proportional hazards regression was used to test the association of lncRNAs with survival using the coxph function from the survival R package ( https://cran.r-project.org/web/packages/survival/index.html ). For survival time, the start date was the date of admission, and the end date was the date of death or the date of discharge. Association between lncRNAs and survival is reported as hazard ratio (HR), along with a measure of precision (95% confidence interval, CI). The significance level was set at 0.05. Kaplan–Meier curves stratified by lncRNA quartile were generated for survival analyses using the ggsurvplot function from the survminer R package ( https://cran.r-project.org/web/packages/survminer/index.html ).

Reporting summary

Further information on research design is available in the  Nature Portfolio Reporting Summary linked to this article.

Data availability

Due to legal and ethical issues related to General Data Protection Regulation guidelines, the data used in this study is available upon request to the COVIRNA consortium. Please email the corresponding author for more details and information about data access ([email protected]).

Code availability

Code accompanying the paper “Development of a long noncoding RNA-based machine learning model to predict COVID-19 in-hospital mortality” is available here: https://doi.org/10.24433/CO.6166592.v1 33 .

2023, N. P. O. A. The Nobel Prize in Physiology or Medicine 2023 , https://www.nobelprize.org/prizes/medicine/2023/summary/ (2023).

Winkle, M., El-Daly, S. M., Fabbri, M. & Calin, G. A. Noncoding RNA therapeutics—challenges and potential solutions. Nat. Rev. Drug Discov. 20 , 629–651 (2021).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Badimon, L. & Devaux, Y. Transcriptomics research to improve cardiovascular healthcare. Eur. Heart J. 41 , 3296–3298 (2020).

Article   PubMed   Google Scholar  

Gomes, C. P. C. et al. Catalyzing transcriptomics research in cardiovascular disease: the CardioRNA COST action CA17129. Noncoding RNA 5 , 31 (2019).

PubMed   PubMed Central   Google Scholar  

Badimon, L. et al. Cardiovascular RNA markers and artificial intelligence may improve COVID-19 outcome: a position paper from the EU-CardioRNA COST Action CA17129. Cardiovasc. Res. 117 , 1823–1840 (2021).

Robinson, E. L., Emanueli, C., Martelli, F. & Devaux, Y. Leveraging non-coding RNAs to fight cardiovascular disease: the EU-CardioRNA network. Eur. Heart J. 42 , 4881–4883 (2021).

Article   CAS   PubMed   Google Scholar  

Firat, H. et al. FIMICS: a panel of long noncoding RNAs for cardiovascular conditions. Heliyon 9 , e13087 (2023).

Chauviere, L. et al. Firalink: a bioinformatics pipeline for long non-coding RNA data analysis. Noncoding RNA Res. 8 , 602–604 (2023).

Mohamed, H. A. et al. MicroRNAs and cytokines as potential predictive biomarkers for COVID-19 disease progression. Sci. Rep. 13 , 3531 (2023).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Gelzo, M. et al. Matrix metalloproteinases (MMP) 3 and 9 as biomarkers of severity in COVID-19 patients. Sci. Rep. 12 , 1212 (2022).

Fu, Z. et al. A virus-derived microRNA-like small RNA serves as a serum biomarker to prioritize the COVID-19 patients at high risk of developing severe disease. Cell Discov. 7 , 48 (2021).

Greco, S. et al. HCG18, LEF1AS1 and lncCEACAM21 as biomarkers of disease severity in the peripheral blood mononuclear cells of COVID-19 patients. J. Transl. Med. 21 , 758 (2023).

Wynants, L. et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ 369 , m1328 (2020).

Article   PubMed   PubMed Central   Google Scholar  

Hausburg, M. A., Banton, K. L., Roshon, M. & Bar-Or, D. Clinically distinct COVID-19 cases share notably similar immune response progression: a follow-up analysis. Heliyon 7 , e05877 (2021).

Yang, M. et al. Lef1 is transcriptionally activated by Klf4 and suppresses hyperoxia-induced alveolar epithelial cell injury. Exp. Lung Res. 48 , 213–223 (2022).

Forrest, I. S. et al. Machine learning-based marker for coronary artery disease: derivation and validation in two longitudinal cohorts. Lancet 401 , 215–225 (2023).

Ng, S., Masarone, S., Watson, D. & Barnes, M. R. The benefits and pitfalls of machine learning for biomarker discovery. Cell Tissue Res. 394 , 17–31 (2023).

Sammut, S. J. et al. Multi-omic machine learning predictor of breast cancer therapy response. Nature 601 , 623–629 (2022).

Article   ADS   CAS   PubMed   Google Scholar  

Wang, C., Lue, W., Kaalia, R., Kumar, P. & Rajapakse, J. C. Network-based integration of multi-omics data for clinical outcome prediction in neuroblastoma. Sci. Rep. 12 , 15425 (2022).

Mi, X., Zou, B., Zou, F. & Hu, J. Permutation-based identification of important biomarkers for complex diseases via machine learning models. Nat. Commun. 12 , 3008 (2021).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Dolgin, E. Why rings of RNA could be the next blockbuster drug. Nature 622 , 22–24 (2023).

Reme, B. A., Gjesvik, J. & Magnusson, K. Predictors of the post-COVID condition following mild SARS-CoV-2 infection. Nat. Commun. 14 , 5839 (2023).

Thaweethai, T. et al. Development of a definition of postacute sequelae of SARS-CoV-2 infection. JAMA 329 , 1934–1946 (2023).

Global Burden of Disease Long, C. C. et al. Estimated global proportions of individuals with persistent fatigue, cognitive, and respiratory symptom clusters following symptomatic COVID-19 in 2020 and 2021. JAMA 328 , 1604–1615. https://doi.org/10.1001/jama.2022.18931 (2022).

Wilmes, P. et al. SARS-CoV-2 transmission risk from asymptomatic carriers: Results from a mass screening programme in Luxembourg. Lancet Reg. Health Eur. 4 , 100056 (2021).

Fagherazzi, G. et al. Protocol for a prospective, longitudinal cohort of people with COVID-19 and their household members to study factors associated with disease severity: the Predi-COVID study. BMJ Open 10 , e041834 (2020).

Schons, M. et al. The German National Pandemic Cohort Network (NAPKON): rationale, study design and baseline characteristics. Eur. J. Epidemiol. 37 , 849–870 (2022).

Tremblay, K. et al. The Biobanque quebecoise de la COVID-19 (BQC19)-A cohort to prospectively study the clinical and biological determinants of COVID-19 clinical trajectories. PLoS One 16 , e0245031 (2021).

Stekhoven, D. J. & Bühlmann, P. MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics 28 , 112–118 (2011).

Law, C. W., Chen, Y., Shi, W. & Smyth, G. K. voom: precision weights unlock linear model analysis tools for RNA-seq read counts. Genome Biol. 15 , R29 (2014).

Love, M. I., Huber, W. & Anders, S. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol. 15 , 550 (2014).

Kursa, M. B. & Rudnicki, W. R. Feature selection with the Boruta package. J. Stat. Softw. 36 , 1–13 (2010).

Article   Google Scholar  

Code accompanying the paper “Development of a long noncoding RNA-based machine learning model to predict COVID-19 in-hospital mortality” . https://doi.org/10.24433/CO.6166592.v1

Download references

Acknowledgements

The authors thank all members of COVIRNA project for their contribution: Claude Pelletier, Petr Nazarov, Adriana Voicu, Irina Carpusca, Eric Schordan, Rodwell Mkhwananzi, Stephanie Boutillier, Louis Chauviere, Joanna Michel, Florent Tessier, Reinhard Schneider, Irina Belaur, Wei Gu, Enrico Petretto, Michaela Noseda, Verena Zuber, Pranay Shah, Leonardo Bottolo, Leon de Windt, Emma Robinson, George Valiotis, Tina Hadzic, Federica Margheri, Chiara Gonzi, Detlef Kindgen-Milles, Christian Vollmer, Thomas Dimski, Emin Tahirovic. Further information on the COVIRNA project can be found at https://covirna.eu/ . We dedicate this paper to Claude Pelletier who passed away during the timeframe of the COVIRNA project. His invaluable contribution to data analysis is highly recognized and acknowledged. We are thankful to all the participants of the Predi-COVID study. We also acknowledge the involvement of the interdisciplinary and inter-institutional study team that contributed to Predi-COVID. The full list of the Predi-COVID team can be found here: https://sites.lih.lu/the-predi-covid-study/about-us/project-team/ . We would like to thank University of Edinburgh DataLoch ( https://dataloch.org ) and NHS Lothian Bioresource for their support and assistance with this study. This work uses data provided by patients and collected by the NHS as part of their care and support #DataSavesLives. We are extremely grateful to the 2,648 frontline NHS clinical and research staff and volunteer medical students, who collected data in challenging circumstances; and the generosity of the participants and their families for their individual contributions in these difficult times. We also acknowledge the support of Jeremy J Farrar and Nahoko Shindo. The study was carried out using the clinical-scientific infrastructure of NAPKON (Nationales Pandemie Kohorten Netz, German National Pandemic Cohort Network) and NUKLEUS (NUM Klinische Epidemiologie- und Studienplattform, NUM Clinical Epidemiology and Study Platform) of the Network University Medicine (NUM). We gratefully thank all NAPKON sites who contributed patient data and/or biosamples for this analysis. The representatives of NAPKON sites contributing at least 5 per mille to this analysis are (alphabetical order): Bielefeld University, Medical School and University Medical Center OWL, Bielefeld (Alsaad K, Hamelmann E, Heidenreich H, Hornberg C, Kulamadayil-Heidenreich NSA, Maasjosthusmann P, Muna A, Ruwe M, Stellbrink C, Tebbe J), Saarland University, Homburg (Keller A, Walter J), Saarland University Hospital, Homburg (Bals R, Herr C, Krawczyk M, Lensch C, Lepper PM, Riemenschneider M, Smola S, Zemlin M), University Hospital Augsburg, Augsburg (Bader S, Engelmann M, Fuchs A, Langer A, Maerkl B, Messmann H, Muzalyova A, Roemmele C), University Hospital Erlangen, Erlangen (Kraska D, Kremer AE, Leppkes M, Mang J, Neurath MF, Prokosch HU, Schmid J, Vetter M, Willam C, Wolf K), University Hospital Frankfurt, Frankfurt am Main (Arendt C, Bellinghausen C, Cremer S, Groh A, Gruenewaldt A, Khodamoradi Y, Klinsing S, Rohde G, Vehreschild M, Vogl T), University Hospital Hamburg-Eppendorf, Hamburg (Addo M, Almahfoud M, Engels ALF, Jarczak D, Kerinn M, Kluge S, Kobbe R, Petereit S, Schlesner C, Zeller T), University Hospital RWTH Aachen, Aachen (Dahl E, Dreher M, Marx N, Mueller-Wieland D, Wipperfuerth J), University Hospital Regensburg, Regensburg (Brochhausen-Delius C, Burkhardt R, Feustel M, Haag O, Hansch S, Hanses F, Malfertheiner M, Niedermair T, Schuster P, Wallner S), University Hospital Technical University Munich, Munich (Barkey W, Erber J, Fricke L, Lieb J, Michler T, Mueller L, Schneider J, Spinner C, Voit F, Winter C), University Hospital Tuebingen, Tuebingen (Bitzer M, Bunk S, Göpel S, Haeberle H, Kienzle K, Mahrhofer H, Malek N, Rosenberger P, Struemper C, Trauner F), University Hospitals of the Ruhr University Bochum, Bochum (Brinkmann F, Brueggemann Y, Gambichler T, Hellwig K, Luecke T, Reinacher-Schick A, Schmidt WE, Schuette C, Steinmann E, Torres Reyes C), University Medical Center Goettingen, Emergency Department, Goettingen (Blaschke S, Hermanns G, Santibanez-Santana M, Zeh S), University Medical Center Goettingen, Central Biobank, Goettingen (Nussbeck SY), University Medical Center Goettingen, Central Laboratory, Goettingen (Hafke A), University Medicine Essen, Essen (Brochhagen L, Dolff S, Elsner C, Krawczyk A, Madel RJ, Otte M, Witzke O), University Medicine Greifswald, Greifswald (Becker K, Doerr M, Lehnert K, Nauck M, Piasta N, Schaefer C, Schaefer E, Schattschneider M, Scheer C, Stahl D), University Medicine Oldenburg, Oldenburg (Arlt A, Griesinger F, Guenther U, Hamprecht A, Juergens K, Kluge A, Meinhardt C, Meinhardt K, Petersmann A, Prenzel R). This research was supported by the ACBB, the Augsburg Central BioBank ( www.biobank-augsburg.de ), the CCS Biobank at the University Heart and Vascular Center Hamburg ( https://www.uke.de/kliniken-institute/kliniken/kardiologie/forschung/ ), the Central Biobank Erlangen as a core unit of the Friedrich-Alexander-University Erlangen-Nürnberg in cooperation with the Department of Medicine 1, University Hospital Erlangen, the Central Biobank UMG as a core facility of the University Medical Center Goettingen (Germany), the HOM.BMB (Biobank Internal Medicine V, Saarland University, Homburg), the Institute of Clinical Chemistry and Laboratory Medicine - Integrated Research Biobank, University Medicine Greifswald, the RWTH centralized Biomaterial Bank (RWTH cBMB) of the Medical Faculty of RWTH Aachen University ( https://www.cbmb.ukaachen.de/ ), the West German Biobank Essen ( https://www.uni-due.de/med/biobank/ ), the ZBR, the Central Biobank Regensburg, the iBioTUM, the Central Interdisciplinary Biomaterial Bank as a Core Unit of the TUM School of Medicine and the University Hospital of the Technical University of Munich, the interdisciplinary Biobank and Database Frankfurt ( https://ibdf-frankfurt.de/ ). We gratefully thank all participating NAPKON infrastructures that contributed to this analysis. The representatives of these NAPKON infrastructures are (alphabetical order): Hannover Unified Biobank, Hannover Medical School, Hannover (Bernemann I, Illig T, Kersting M, Klopp N, Kopfnagel V, Muecke S), Institute of Epidemiology, Helmholtz Center Munich, Munich (Anton G, Kuehn-Steven A, Kunze S, Tauchert MK), University Hospital Frankfurt, Frankfurt (Appel K, Geisler R, Hagen M, Scherer M, Schneider J, Sikdar S, Weirauch T, Wolf L), University Hospital Cologne, Cologne (Brechtel M, Broehl I, Fiedler K, Hopff SM, Laugwitz M, Lee C, Mitrov L, Nunes de Miranda S, Sauer G, Schulze N, Seibel K, Stecher M, Wagner P), University Hospital Wuerzburg, Wuerzburg (Günther K, Haug J, Haug F,), University Hospital Wuerzburg and University of Wuerzburg, Wuerzburg (Fiessler C, Heuschmann PU, Miljukov O, Nürnberger C, Reese JP, Schmidbauer L), University of Wuerzburg, Wuerzburg (Jiru-Hillmann S), University Medicine Greifswald, Greifswald (Bahls T, Hoffmann W, Nauck M, Schaefer C, Schattschneider M, Stahl D, Valentin H), University Medicine Goettingen, Goettingen (Chaplinskaya I, Hans S, Krefting D, Pape C, Rainers M, Schoneberg A, Weinert N), Helmholtz Center Munich, Munich (Kraus M), Charite - Universitaetsmedizin Berlin, Berlin (Lorbeer R, Schaller J). We gratefully thank the NAPKON Steering Committee: University Hospital Giessen and Marburg, Giessen (Herold S), University of Wuerzburg, Wuerzburg (Heuschmann P), Charité - Universitaetsmedizin Berlin, Berlin (Heyder R), University Medicine Greifswald, Greifswald (Hoffmann W), Hannover Unified Biobank, Hannover Medical School, Hannover (Illig T), University Hospital Schleswig-Holstein, Kiel (Schreiber S), University Hospital Cologne and University Hospital Frankfurt, Cologne and Frankfurt (Vehreschild JJ), Charité - Universitaetsmedizin Berlin, Berlin (Witzenrath M). We greatly thank all ISARIC4C investigators. The full ISARIC4C Investigator list can be found at this link.This work uses Data / Material provided by patients and collected by the NHS as part of their care and support #DataSavesLives. The Data / Material used for this research were obtained from ISARIC4C. The COVID-19 Clinical Information Network (CO-CIN) data was collated by ISARIC4C Investigators. This work was made possible through open sharing of data and samples from the Biobanque québécoise de la COVID-19 (BQC19). We thank all participants of the BQC19 for their contribution. This work was supported by the EU Horizon 2020 project COVIRNA (grant agreement # 101016072) The Predi-COVID study was supported by the Luxembourg National Research Fund (FNR) (Predi-COVID, grant number 14716273), the André Losch Foundation and by European Regional Development Fund (FEDER, convention 2018-04-026-21). The project National Pandemic Cohort Network (NAPKON) is part of the Network University Medicine (NUM), funded by the German Federal Ministry of Education and Research (BMBF) (FKZ: 01KX2121). Parts of the infrastructure of the Würzburg study site were supported by the Bavarian Ministry of Research and Art to support Corona research projects. Parts of the NAPKON project suite and study protocols of the Cross-Sectoral Platform are based on projects funded by the German Center for Infection Research (DZIF). Data and Material provision for ISARIC4C was supported by grants from: the National Institute for Health Research (NIHR; award CO-CIN-01), the Medical Research Council (MRC; grant MC_PC_19059), and by the NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liverpool in partnership with Public Health England (PHE), (award 200907), NIHR HPRU in Respiratory Infections at Imperial College London with PHE (award 200927), Liverpool Experimental Cancer Medicine Center (grant C18616/A25153), NIHR Biomedical Research Center at Imperial College London (award IS-BRC-1215-20013), and NIHR Clinical Research Network providing infrastructure support. The Biobanque québécoise de la COVID-19 was funded by the Fonds de recherche du Québec - Santé, Génome Québec, the Public Health Agency of Canada and, as of March 2022, the Ministère de la Santé et des Services Sociaux. Project no. RRF-2.3.1-21-2022-00003 has been implemented with the support provided by the European Union. The 2020-1.1.5-GYORSÍTÓSÁV-2021-00011 project was funded by the Ministry for Innovation and Technology with support from the National Research Development and Innovation Fund under the 2020-1.1.5-GYORSÍTÓSÁV call program. The project was supported by grants from the National Research, Development and Innovation Office (NKFIH) of Hungary, 2020-1.1.6-JÖVŐ−2021-00013. Y.D. is funded by the EU Horizon 2020 project COVIRNA (grant agreement # 101016072), the National Research Fund (grants # C14/BM/8225223, C17/BM/11613033 and COVID-19/2020-1/14719577/miRCOVID), the Ministry of Higher Education and Research, and the Heart Foundation-Daniel Wagner of Luxembourg. F.M. is supported by the Italian Ministry of Health projects “Ricerca Corrente 2023”, and POS T4 CAL.HUB.RIA, cod. T4-AN-09.

Author information

Authors and affiliations.

Cardiovascular Research Unit, Department of Precision Health, Luxembourg Institute of Health, Strassen, Luxembourg

Yvan Devaux & Andrew I. Lumley

Bioinformatics Platform, Luxembourg Institute of Health, Strassen, Luxembourg

Faculty of Engineering and Natural Sciences, International University of Sarajevo, Sarajevo, Bosnia and Herzegovina

Kanita Karaduzovic-Hadziabdic & Muhamed Adilovic

Department of Human Genetics, McGill University, Montréal, QC, Canada

Vincent Mooser

The Meakins-Christie Laboratories at the Research Institute of the McGill University Heath Centre Research Institute, & Department of Medicine, Faculty of Medicine, McGill University, Montréal, QC, Canada

Simon Rousseau

Luxembourg Center for Systems Biomedicine, University of Luxembourg, Belval, Luxembourg

Muhammad Shoaib & Venkata Satagopam

National Heart and Lung Institute, Imperial College London, London, England, UK

Prashant Kumar Srivastava & Costanza Emanueli

Molecular Cardiology Laboratory, IRCCS Policlinico San Donato, Milan, Italy

Fabio Martelli & Simona Greco

Cardiovascular Program-ICCC, Institut d’Investigació Biomèdica Sant Pau (IIB SANT PAU); CIBERCV, Autonomous University of Barcelona, Barcelona, Spain

Lina Badimon & Teresa Padro

Department of Intelligent Systems, Jozef Stefan Institute, Ljubljana, Slovenia

Mitja Lustrek & Marko Jordan

Group Genetical Statistics and Biomathematical Modelling, Institute for Medical Informatics, Statistics and Epidemiology, University of Leipzig, Leipzig, Germany

Markus Scholz & Maciej Rosolowski

Medical University of Dusseldorf, Dusseldorf, Germany

Timo Brandenburger

HUN-REN–SU System Pharmacology Research Group, Department of Pharmacology and Pharmacotherapy, Semmelweis University, Budapest, Hungary; Pharmahungary Group, Szeged, Hungary

Bettina Benczik, Bence Agg & Peter Ferdinandy

Medical Department 2 (Hematology/Oncology and Infectious Diseases), Center for Internal Medicine, Goethe University Frankfurt, University Hospital, Frankfurt, Germany

Jörg Janne Vehreschild

University of Cologne, Faculty of Medicine and University Hospital Cologne, Cologne, Germany

Department I of Internal Medicine, Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf, Cologne, Germany

German Centre for Infection Research (DZIF), partner site Bonn-Cologne, Cologne, Germany

Institute of Epidemiology, Helmholtz Center Munich, Munich, Germany

Bettina Lorenz-Depiereux

Department of Internal Medicine B, University Medicine Greifswald, Greifswald, Germany; German Centre of Cardiovascular Research (DZHK), Greifswald, Germany

Marcus Dörr

Department of Infectious Diseases, West German Centre of Infectious Diseases, University Hospital Essen, University of Duisburg-Essen, Essen, Germany

Oliver Witzke

Firalis SA, Huningue, France

Gabriel Sanchez, Seval Kul & Hüseyin Firat

Centre for Cardiovascular Science, The Queen’s Medical Research Institute, University of Edinburgh, Edinburgh, Scotland

Andy H. Baker

CARIM Institute and Department of Pathology, University of Maastricht, Maastricht, The Netherlands

Deep Digital Phenotyping Research Unit, Department of Precision Health, Luxembourg Institute of Health, Strassen, Luxembourg

Guy Fagherazzi

Department of Infection and Immunity, Luxembourg Institute of Health, Esch-Sur-Alzette, Luxembourg

Markus Ollert

Department of Dermatology and Allergy Center, Odense Research Center for Anaphylaxis (ORCA), University of Southern Denmark, Odense, Denmark

Centre for Cardiovascular Science, University of Edinburgh, Edinburgh, UK

Ryan Wereski & Nicholas L. Mills

Usher Institute, University of Edinburgh, Edinburgh, UK

Nicholas L. Mills

You can also search for this author in PubMed   Google Scholar

Contributions

Y.D., L.Z. and A.I.L. led the writing and editing. Y.D., L.Z., K.H. analyzed the data. A.B., V.M., S.R., J.J.V., B.L.D., M.D., G.F., M.O. provided human samples. All others including M.Sh, V.S., M.A., P.K.S., C.E., F.M., S.G., L.B., T.P., M.L., M.Sc, M.R., M.J., T.B., B.A., P.F., B.B., O.W., G.S., S.K., R.W., N.L.M., H.F. provided background information, intellectual contributions, editing, and/or writing of the manuscript.

Corresponding author

Correspondence to Yvan Devaux .

Ethics declarations

Competing interests.

Y.D. holds patents and licensing agreements related to the use of RNAs for diagnostic and therapeutic purposes, and is SAB member of Firalis SA. P.F. is the founder and CEO of Pharmahungary Group, a group of R&D companies. J.J.V. has personal fees from Merck / MSD, Gilead, Pfizer, Astellas Pharma, Basilea, German Center for Infection Research (DZIF), University Hospital Freiburg/ Congress and Communication, Academy for Infectious Medicine, University Manchester, German Society for Infectious Diseases (DGI), Ärztekammer Nordrhein, University Hospital Aachen, Back Bay Strategies, German Society for Internal Medicine (DGIM), Shionogi, Molecular Health, Netzwerk Universitätsmedizin, Janssen, NordForsk, Biontech, APOGEPHA. J.J.V. has grants from Merck / MSD, Gilead, Pfizer, Astellas Pharma, Basilea, German Center for Infection Research (DZIF), German Federal Ministry of Education and Research (BMBF), Deutsches Zetrum für Luft- und Raumfahrt (DLR), University of Bristol, Rigshospitalet Copenhagen, Network University Medicine. L.B. declares to have acted as a SAB member of Sanofi, Ionnis, MSD and NovoNordisk; to have received speaker fees from Sanofi, Bayer and AB-Biotics SA and to have founded the spin-off Ivastatin Therapeutics S.L. (all unrelated to this work). T.P. declares to have received speaker fees from AB-Biotics SA, and to be a co-founder of the Spin-off Ivastatin Therapeutics S.L. (all unrelated to this work). M.S. received funding from Pfizer Inc. and from Owkin for projects not related to this research. N.L.M. has received honoraria from Abbott Diagnostics, Siemens Healthineers, Roche Diagnostics and LumiraDx. The University of Edinburgh has received research grants from Abbott Diagnostics, Novo Nordisk, and Siemens Healthineers. H.F. is the founder and owner of Firalis SA, a company commercializing the FIMICS panel. He holds patents and licenses for the use of RNAs as biomarkers and therapeutic targets. The remaining authors declare no competing interests

Peer review

Peer review information.

Nature Communications thanks Stephen Knight and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, reporting summary, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Devaux, Y., Zhang, L., Lumley, A.I. et al. Development of a long noncoding RNA-based machine learning model to predict COVID-19 in-hospital mortality. Nat Commun 15 , 4259 (2024). https://doi.org/10.1038/s41467-024-47557-1

Download citation

Received : 11 December 2023

Accepted : 03 April 2024

Published : 20 May 2024

DOI : https://doi.org/10.1038/s41467-024-47557-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Integrated patient journey

Drive better outcomes with multi-layered insights and analytics

One journey, many lenses

Gaining a granular view of the patient journey data is critical to life sciences commercialization in today’s patient-centric world, informing everything from product development to market access and post-launch management.

Unfortunately, patient journey efforts are often conducted in a fragmented fashion, with different teams employing different methodologies and data sources, depriving companies of a more comprehensive view of the barriers and drivers impacting their brands.

At Clarivate, we’re rethinking the way we trace the patient journey touchpoints with the aim of capturing the larger picture, in greater detail, so that our customers can better serve their customers.

“Our previous patient journey work was conducted in typical fashion – qualitative interviews with patients, maybe a quantitative study, but we really needed to understand the emotions patients were experiencing at each stage in order to identify messaging that resonates in order to better support patients and keep them on that journey longer.

Having that voice-of-the-patient lexicon was huge – a lot of what our marketing team puts out now utilizes the language patients are using online, as opposed to the more clinical language encouraged by medical, legal and regulatory. Patients don’t want to read language that they’re not using in the context of their day-to-day.”

• Head of Market Intelligence
• Mid-sized Pharma

How we help

At Clarivate, we examine the patient journey through multiple lenses in order to provide a granular and insights-rich view of the patient journey research:

Clinical journey

• Treatment and referral patterns
• Time to diagnosis
• Sites of care
• Line of therapy progression

Cost Journey

• Impact of cost burden and market access factors on treatment decisions and behaviors

Attitudinal Journey

• What patients are feeling at each stage in the treatment journey
• Connections with patient behaviors such as adoption, adherence, discontinuation and switching
• HCP attitudes

Informational Journey

• Channels, devices and sites sourced for health and medical info

We partner with clients on patient journey analyses to help enhance commercial activities, including:

• Sales force optimization
• Patient engagement
• Physician engagement
• Payer value communication
• Segmentation and targeting
• Adherence and switching management
• Opportunity assessment

Integrated patient journey assessment - a Clarivate solution

At Clarivate, we’re rethinking the way we trace the patient journey with the aim of capturing the larger picture, in greater detail, so that our customers can better serve their customers.

Diverse data sources

The power of the integrated patient journey approach lies in the combination of quantitative and qualitative data sources to reveal key intervention points and opportunities for differentiating products across multiple stakeholder audiences.

• 26 billion claims
• Visibility into 300+ million patients (over 3 years)
• 2+ million healthcare providers
• 98% of 750 U.S. payers, real-time updates
• Hospitals, physician offices, ASCs, pharmacies, long-term care, nursing facilities

Electronic health records

• Visibility into 100+ million patients (over 3 years)
• Diagnoses, diagnostic tests, assessments, vaccines, vitals, Rx

Social data

• Anonymized social conversation data from public social platforms, online communities and message boards

Online behavioral data

• Search data
• Clickstream data

Primary patient journey market research

• 25+ countries
• 25+ specialities
• 50+ therapeutic areas

Secondary patient journey market research

• Public health and demographic data sets
• Formulary coverage

Integrated Patient Journey: One Journey, Many Lenses

Integrated patient journey: Leveraging real world data and analytics to understand patients’ unmet needs

Integrated patient journey assessment – A Clarivate solution

Want to learn more?

Contact us to connect with a specialist on how we can enhance your healthcare patient journey mapping

Speak to our team

We've updated our Privacy Policy to make it clearer how we use your personal data. We use cookies to provide you with a better experience. You can read our Cookie Policy here.

Drug Discovery

Stay up to date on the topics that matter to you

Strengthening a Commitment to Diversity and Inclusion in R&D With the Help of Novel Machine Learning Approaches

Discover how a deep learning framework can enhance site selection while seeking to improve trial diversity..

Complete the form below to unlock access to ALL audio articles.

Historically , clinical trials have relied heavily on white male participants, but today we know that such practices create knowledge gaps in how we understand the natural progression and treatment of acute and chronic conditions across different patient populations. That’s why enrolling sufficient patients from all gender, racial and ethnic groups in numbers that reflect current population proportions is essential for ensuring a treatment’s efficacy within all groups.

Despite many efforts over decades to address disparities, the underrepresentation of gender, racial and ethnic minorities in research and development persists. For example, African Americans make up 13.4% of the US population, but only 5% of trial participants. Hispanics represent 18.1% of the US population, but less than 1% of trial participants. These enrollment disparities have real consequences in terms of improved health outcomes for traditionally underrepresented communities, and industry stakeholders are recognizing the need to achieve meaningful change.

For example, in 2022, the US Food and Drug Administration issued draft guidance for trial sponsors to develop and submit a “Race and Ethnicity Diversity Plan” for their trial programs prior to finalizing study designs. To ensure that drug developers take tangible steps for progress with early planning and proactive thought, plans should include specific enrollment goals and an operational strategy to reach underrepresented and underserved racial and ethnic patient populations.

As they explore transformative approaches to improve diversity and inclusion in clinical trials, industry stakeholders are leveraging advanced technologies and the wealth of available data sources to better understand patient needs, burdens and more so they may increase interest and participation in trials.

This has allowed us to see how machine learning (ML) programs can effectively rank a list of trial sites according to which may yield higher patient enrollment based on trial protocols, including eligibility criteria, previous trial performance, claims data and patient demographics. However, by constantly fine-tuning ML capabilities, we see how a deep reinforcement learning framework designed with intent can effectively learn to better prioritize inclusion while optimizing trial site selection.

In this article, we discuss how a deep learning framework can specifically address real-world challenges to enhance site selection while seeking to improve trial diversity.

Site selection’s real-world challenges

To address enrollment disparity, clinical trial sponsors are using data-driven methodologies to calculate trial protocol burdens according to race and ethnicity, participating in community events (e.g., health fairs) to create local awareness, providing culturally relevant communications to patients for stronger engagement and much more. With early planning, multi-pronged approaches have been shown to be impactful.

To maximize the potential for sites to have adequate representative samples of participants with diverse backgrounds, sponsors have to dig deeper into the nuances of site identification and evaluation methodologies. But there are two notable barriers to site identification that impact improving equity in clinical trial participation:

Missing data

Site identification often starts with an analysis of claims and specialty data, including patient and public involvement, site visits and clinical research coordinator participation, past enrollment performance and recruitment rates. These data sets will best inform the likelihood of local patient enrollment.

However, the issue to keep in mind is that trial sites with a greater minority population are potentially more likely to lack data due to insufficient data collection and reporting. Though race and ethnicity data reporting from trials is increasing , it is still a work in progress. Existing tools fail to cope with missing data, meaning this issue only exacerbates the underlying unfairness when sites in minority-rich locations are overlooked.

The enrollment-diversity trade-off

Adding diversity as a qualifier for target sites to maximize enrollment can be challenging. We cannot simply impose fairness by setting quotas for each racial or ethnic group because the fewer minority-population participants selected by existing approaches would effectively set enrollment caps. We must balance the trade-off between enrollment needs and fairness, and as such, we need to optimize simultaneously for both objectives.

Where deep reinforcement learning models can help

Given what must be considered to meet the challenges discussed above, clinical trial sponsors need to determine how best to optimize multiple site parameters to ensure better enrollment rates with diverse patient populations.

The use of ML solutions to improve clinical trial design and execution is broadening with time as skilled data scientists gather more practice-based insights and apply them to further fine-tune ML-based models for the task at hand. Currently, ML is helping to validate assumptions about trial feasibility, extract meaningful patterns of patient outcomes to drive trial design, predict trial outcomes and more.  

Flexible Clinical Modeling: How Advanced Analytics and AI/ML Can Help Ensure Effective Patient-Centered Drug Development

Going a step further, the ML subfield of deep learning is being used to predict the optimal physicians to run studies and maximize patient recruitment . However, patient diversity is not being considered.

To improve site selection with diversity and enrollment in mind, data scientists have tested a specific deep reinforcement learning model using data points from nearly 4,400 real-world clinical trials from 2016 through 2021. Results show this framework accounts for several key variables that can better address the challenges of missing site-specific data and trading off enrollment for diversity and vice versa.

Modality encoder for missing but needed data

While most ML-based research assumes datasets are complete and well-cleaned, that is not feasible within most real-world applications where data is often incomplete, which skews outcomes. In recognition of the need for a more uniform view among sites regarding missing or insufficient data insights, data scientists have tested this framework to bypass what is not available by taking data from multiple sources and then combining, enriching and enhancing it to provide a more holistic picture of each site. In addition, where data is missing, its content can be accurately inferred once the holistic overview is available.

Other existing ML-based strategies, such as modality dropout and cascaded residual autoencoders, do not directly model missing data. But, in this framework, it is possible to build a more accurate representation of a clinical trial site without complete site data.

Efficiently trading-off: a “reward system”  

Since trial-site representations no longer have “data holes,” this function puts a value on individual sites’ contributions as they relate to the “reward” given to their features. By using a reward system, where the final reward is being selected as a target site for the trial, this model uses an encoding layer to allow each site’s ranking/score to be impacted by other sites’ features. As seen in Figure 1 below, this emphasizes which sites may be both ideal for overall enrollment and able to reach diverse populations.

Figure 1: A visualization of a deep reinforcement learning model that considers fair ranking with missing modalities. This framework uses multi-modal site features and the trial representation to generate scores for rank and selection of a subset of prospective trial sites. The pipeline used to do so consists of modality encoders, a missing data handling mechanism, a scoring network and a reinforcement learning-based ranking approach. Credit: Theodorou B, Glass L, Xiao C, Sun J. 2024 . CC BY 4.0.

Consistently building cases for use

ML-driven solutions are one part of a more holistic approach to optimizing site selection that prioritizes sufficient representation in trials from diverse patient populations. As such, it is critical that data scientists and other stakeholders constantly finesse techniques to better uncover insights of interest in an unbiased and accurate way. There must be assurance that the ML approach used is embedded in the correct science and guided by the right group of subject-matter experts for clinical trials, including medical, clinical and data science experts.

Since deep reinforcement learning is based on insights gathered from trial and error of use, these models will continuously evolve to meet the needs at hand. For current use, the novel model discussed above helps trial sponsors bypass the need for complete data and limit or eliminate biases within its inputs to better tackle the longstanding industry challenge of selecting sites that can help improve the diversity of the enrolled patient group while also protecting enrollment rates.

Opportunities for this model and other deep learning tools to help drive smarter decisions in R&D to enhance patient care will come with time and a growing collection of insights to examine.

About the author: Greg Lever is director of AI solutions delivery at IQVIA. With more than 14 years of life sciences and technology experience, Greg currently helps clients discover innovative ways to bring life-changing therapies to patients faster within IQVIA’s Applied Data Science Center’s consulting sales team. Previously, he led a team of machine learning engineers within IQVIA’s Analytics Center of Excellence. 

Greg has worked with several technology startup companies in London and helped see Genomics England’s 100,000 Genomes Project through project completion. He received his PhD at the University of Cambridge, combining quantum physics and ML to develop new approaches for small-molecule drug discovery, and has worked as a postdoctoral associate at MIT.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• IOP Publishing

Machine learning in patient flow: a review

Rasheed el-bouri.

1 Institute of Biomedical Engineering, University of Oxford, Oxford, United Kingdom

Thomas Taylor

Alexey youssef, tingting zhu, david a clifton.

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

1. Introduction

When a country’s population and average age increase every year, it is inevitable that a strain is placed upon its healthcare system. This is due to the clinical attention that is generally required by older people and the increasing size of the ageing population. This is the situation faced by many countries in the world today (Andrews 2001 , Tinker 2002 , Oliver et al 2014 ). National media outlets can be particularly vocal about the performance of healthcare systems which makes the desire for a solution to poor efficiency in healthcare systems not only technically and economically desirable, but also politically important. The ability to cope with the demand for efficient healthcare has recently further been compromised due to the coronavirus pandemic that has swept the world which has shut down the normal operation of many healthcare institutions and reduced their capacity to treat patients significantly in many cases (Chen et al 2020 , Hick et al 2020 , Janbabai et al 2020 ). This has consequently increased the pressure placed on healthcare institutions as well as extending the waiting times faced by patients (Propper et al 2020 ). Despite numerous attempts by governments and hospitals to apply traditional management techniques and lean practices to improve the throughput of patients through hospitals, very little has proven effective in the long-term running of the hospital (Hall 2013 , Rutman et al 2015 ). Even fewer techniques developed have proved easily extendable to multiple hospitals as a simple solution to maximising flow throughput.

It is common today for hospitals today to have digital systems in which all patient data is recorded. These are called the electronic health records and store information on the patients passing through the hospital as well as the state of the hospital at a given time. With the abundance of this data, it has become increasingly feasible to adopt algorithmic approaches to the running of hospitals. As a result, many researchers have turned to utilising machine learning amongst other algorithmic approaches in order to tackle the issue of maximising patient flow through hospitals. In using this algorithmic approach, researchers hope to create solutions which can extend to any hospital which has an electronic health record system, thereby making their solutions ‘generalisable’ to the rest of the industry. In this review we aim to provide an understanding of the landscape of research that has been developed in the field of machine learning applied to the patient flow problem.

1.1. What is patient flow?

Patient Flow is a term used within healthcare services to refer to the way in which patients are moved through a healthcare facility. It involves the medical care, resources, and internal systems needed to get patients from admission to discharge while maintaining a standard of quality of care and satisfaction for the patient (Hall 2013 ). Many works have shown that patient flow can be predictable using machine learning techniques. These works aim to use these predictions to improve the flow of patients and resources in order to provide a faster and better service to patients.

2. Motivation

Patient flow is a topic that has been studied extensively by various researchers of differing backgrounds. As a result the literature associated with the improvement of patient flow is vast and a diverse range of techniques from different disciplines are employed in an attempt to tackle the problem. In this review, we will primarily focus on the history of how patient flow has been handled, as well as techniques that involve the use of machine learning methods. This is, however, by no means an exhaustive review of all methods used for the improvement of patient flow. It should also be noted that this review is not intended to summarise the machine learning methods that have been applied to patient flow or the best performing models for each task (as seen in Chen et al 2020 ) and so the performances of the models will not be included. Rather, it is to provide some structure to the field of machine learning applied to patient flow, to allow researchers to see how machine learning has already been applied to the patient flow problem and where there are (to the best of our knowledge) gaps in the literature.

While some authors have attempted to tackle patient flow as a single system through a hospital, most researchers break the problem into smaller constituent problems to tackle. These constituent parts are usually associated with the key flow bottlenecks in hospitals and these are: (a) prediction of patient admissions and demand on emergency departments (EDs), (b) prediction of flow through the emergency-to-inpatient interface (i.e. handover from ED to the hospital), (c) prediction of movement of patients (and associated resource) within the hospital and (d) prediction of length-of-stay. In this review we will discuss the work published in all of these topics and how they have been used to improve patient flow through hospitals.

Figure ​ Figure1 1 shows the process of hospitalisation for many hospitals with an ED (although many hospitals may also receive patients from different EDs). Hospital visits can be decomposed into two overarching types of admission: elective (planned) and emergency (unplanned). It is generally the unplanned emergency admissions which cause the greatest disruptions to patient flow through hospitals (Tancrez et al 2009 ).

A visualisation of the process of hospitalisation and the main considerations at each stage from a patient flow perspective.

Elective admissions are planned prior to their admission. As a result, the resource for these patients has been planned and there is bed space should it be needed. Elective patients have also been shown to have consistent lengths of stay in hospitals meaning they cause minimal disruption to the flow of the hospital (Kelly et al 2012 ).

Due to each emergency case being different there can be no estimate of the resource required or how long each patient will stay in hospital prior to their arrival. These therefore have become popular topics for the use of machine learning for prediction. Should these patients need hospitalisation, there is again little warning and so adapting the planning of the hospital becomes difficult.

In the following sections we will look at the work that has been carried out in applying machine learning to all of these sections of the hospitalisation process, the techniques that have been used and where we believe researchers should focus their attention on in the future to further improve patient flow.

4. How patient flow is currently managed

The effect of poor resource management on patient flow within the hospital is well known. Conceptually, high patient flow can be achieved by the effective balance of supply and demand within the system. If the supply of beds, staff and equipment is readily available to meet the needs of patients arriving at the door, then few perceivable barriers exist to prevent their immediate usage. However, studies of waiting lists have long shown that increasing supply in fact leads to a proportional stimulation of demand, highlighting the inadequacies of using relative need for services solely for the basis of resource provision (Feldstein and Severson 1964 ). If increasing supply cannot satiate demand, the optimisation of existing resources is an obvious and necessary strategy. Oredsson et al ( 2011 ) reviewed modern triage-based interventions designed to improve patient flow in emergency departments, demonstrating that the most significant improvements are observed through the use of fast-track and team triage approaches, indicating the importance of casemix as a fundamental consideration.

Current approaches to the management of patient flow in hospitals are typically driven by the need to report and improve upon key performance objectives. Within the United Kingdom National Health Service (NHS), the introduction of the Patient’s Charter allowed providers greater flexibility to curate local operational policies, whilst imposing stricter performance and reporting structures across the system (NHS England 2015 ). By specifying the metrics required to deliver an adequate level of care, the identification and treatment of bottlenecks in the system naturally become a focus of attention. Such metrics are often objective and time-based, such as the time taken for acute arrivals to be admitted or discharged. Perhaps the most significant of these targets introduced within the NHS was that of the 4 h waiting limit for ED arrivals, stipulating the need to admit, transfer or discharge a patient within this timeframe (Stevens 2004 ). The most widely used approach to fulfil this target in the UK is the use of the ‘See and Treat’ framework, which encourages rapid on-arrival assessment of the patients needs by an individual clinician, and allows full autonomy to that clinician to decide the treatments, referrals and investigations necessary to facilitate their care, or be discharged as appropriate. Saint Lamont ( 2005 ) discussed the benefits and limitations of this approach, including the barriers to adoption observed when additional resources or suitably trained staff are unavailable.

Anecdotally, a lack of efficiency and poor patient flow is typically perceived to correlate with a reduction in staff availability. This observation is particularly valid where patient satisfaction is concerned. A study by Thompson et al ( 1996 ) showed positive overall satisfaction was associated with the perception of short waiting times and accurate information delivery, rather than actual waiting times. Whilst increasing staff within the emergency department may improve turnaround times for rapid triage and discharge of non-urgent cases, it is less likely to result in an improvement for patients requiring admission, as shown by Bucheli and Martina ( 2004 ), indicating that the true bottlenecks exist further in the pathway beyond the emergency department. This fact has been clearly recognised in recent guidance, where the focus on enabling patient flow has shifted away from the performance of the ED and towards acute networks and support services (Ham 2017 ). At the one end, Clinical Streaming has been introduced as the process by which patients are assigned to one of several parallel pathways, according to their care requirements, allowing for more structured and reliable coordination of support services within the hospital. At the other, Discharge to Assess (D2A) models emphasise the need to address unnecessary delays in discharging clinically optimised patients from hospital, due to a lack of funding or support within the community (Hyslop 2020 ).

5. Machine learning for patient admissions

5.1. prediction of emergency admissions.

The number of patient admissions to the hospital is arguably one of the most important aspects of patient flow. This determines the demand that is placed upon the hospital and therefore affects how patients can be treated. The importance of predicting patient admissions is reflected by the number of publications in this area. However, with little information on patients prior to their arrival it is also one of the most difficult areas of patient flow to create accurate predictions.

Boyle et al ( 2008 , 2012 ) and Batal et al ( 2001 ) predict the number of emergency admissions using multiple regression. They frame the problem such that they forecast for daily admissions as well as weekly and monthly admissions. The use of regression is for interpretability of the predictions as well as the development of a simple model to improve the chance of being able to generalise to other hospitals. As mentioned previously, due to limited information on these patients prior to arrival, the authors use the days of the week and national holidays as features.

Whereas the aforementioned studies approach the problem as a static prediction (i.e. using information from a snapshot in time to make predictions), Tandberg and Qualls ( 1994 ), Au-Yeung et al ( 2009 ), and Schweigler et al ( 2009 ) treat the problem as a time-series. They use autoregressive models to account for the trajectory of the numbers of patients. This approach is more likely to be successful than a static approach due to the incorporation of data close to the event of interest. However, the benefit of a static approach (if the model is accurate) is that a prediction can be made at an early stage and action can be taken based on that prediction without needing to wait for the time-series to unfold. These time-series approaches also perform regressions to predict patient volumes in the coming days, weeks and months.

While the seasonal features such as weather and time of the year have been shown to be helpful with predicting patient numbers, they are not patient-specific and therefore are limited in their use for predicting when a patient will be admitted to hospital. As a result, LaMantia et al ( 2010 ) and Artetxe et al ( 2020 ) consider predicting patient readmissions to the ED instead of predicting any given admission. In doing so they are able to utilise the wealth of data already recorded by the hospitals on individual patients and identify markers that indicate high risk of readmission in an emergency. Hosseinzadeh et al ( 2013 ) use Naive Bayes and a decision tree in order to classify patients who are going to be readmitted to hospital using their health records as features. These works generally pose the readmission problem as predicting readmission within the next 30 d as this has the most impact on the health and welfare of the patient, as well as the scheduling of the hospital (Leppin et al 2014 ).

A problem that can arise due to these readmission predictions is that patients can be readmitted for various issues (for example a patient who was hospitalised for cardiac issues might need rehospitalisation for breaking their leg). Considering this type of readmission is not very useful for the hospital or the patient, as it is not indicative of an underlying condition and so the health records of the patient will not be useful for this prediction. To get around this issue, many authors have conditioned their prediction of admission on subsets of patients with certain underlying conditions. Shameer et al ( 2017 ) use a naive Bayes classifier to predict readmission and only considers a subset of patients with heart failure. They only consider a readmission to be valid if the patients are readmitted with heart failure within 30 d. Kalagara et al ( 2018 ) also condition their problem on a subset of patients who have had a neurosurgical procedure carried out and compare the performance of their model (trained used gradient boosted trees) using features available during the patients stay versus features that were obtained after the patients discharge. Naturally the model with access to features after the patient discharge performed better, however it is very difficult in most situations to obtain features post-discharge. Min et al ( 2019 ) carry out a similar study but consider patients suffering from COPD. They investigate various machine learning methods and find that gradient boosted trees offer the best prediction of readmission accuracy for their dataset. They also utilise recurrent neural networks in order to treat the problem as a time-series problem but the performance is significantly worse. In fact there are very few works that treat the prediction of readmission as a time-series due to the difficulty of obtaining data on patients post-discharge (Arora et al 2010 ).

5.2. Scheduling instead of admissions

The ultimate aim of all of the works mentioned in section 5.1 is to provide the hospital with an understanding of the volumes of patients that may be attending the ED. By forecasting this (and if the model is accurate) the hospitals may then plan the appropriate resource (including staff, tests and making equipment available) in order to be able to cope with the demand placed on them. For low numbers forecast, hospitals may also then reduce the required resource that is on standby which can lead to cost savings (Thungjaroenkul et al 2007 ).

Some authors however approach the problem from the scheduling perspective. This is different in that whereas predicting admissions makes the assumption that resource can be altered to meet demand, the scheduling approach does not. With this approach authors assume that there is fixed resource and how it is used can be optimised with varying patient numbers.

Rosemarin et al ( 2019 ) define the ED scheduling problem as needing to satisfy the following constraints: the schedule must minimise the risk of adverse consequences, minimise patient waiting time, minimise patient length-of-stay, minimise ED crowding and minimise interruption to caregivers. They use a mixture of health record data of the patients and data on the status the ED to reconstruct the state of the ED when the patients were there. They then use a mixed integer linear program to optimise these scenarios, maximising throughput while being constrained by the aforementioned constraints. They then train a deep learning architecture on this optimised data and use it as a ranking system to predict the optimal patient-caregiver pair in the ED.

Some authors prefer to allow the machine learning algorithms to discover the optimal policies instead of optimising the problem themselves to learn from. This is seen in Lee and Lee ( 2020a ) where a deep Q network (a reinforcement learning algorithm) is used to learn the optimal policy of treating patients in the ED. In order to do this a simulation is made of the ED which will allow the agent to take exploratory moves essential for reinforcement learning. The state of the model is defined as the distribution of acuity (sum of patients at each acuity level) within the ED as well as the distribution of needed treatment type. The action of the agent is to rank the next patient that needs to be seen meaning it is also a patient priority-ranking system. Krämer et al ( 2019 ) also present a priority ranking system based on severity prediction, but go as far as predicting whether patient presentations to the ED should be treated as elective visits given their low severity. They do this using the primary diagnosis code of the patient, however there may be difficulties in expanding this tool to other hospitals given that many hospitals assign diagnosis codes after the patient is discharged from hospital and not at admission.

Yeh and Lin ( 2007 ) and Arisha and Abo-Hamad ( 2013 ) approach the scheduling problem slightly differently in that instead of ranking the priority patients in the ED, they instead aim to design the staffing schedules. They do this using genetic algorithms and allowing the staffing schedules to be updated and ‘evolve’ to a point where they are suitable for the demand placed on the ED. This approach makes the assumption that should the staffing level be predicted accurately, then there will be no need to prioritise patients in the ED as there will be enough staff (and resource) to process them.

Figure ​ Figure2 2 shows the works that have been conducted so far on the prediction of admissions and scheduling in the ED. This is by no means an exhaustive summary but we aim to provide some structure to help other researchers understand what work has been conducted in the field of machine learning for patient flow through the ED. Table ​ Table1 1 further outlines the problems that have readily available datasets for prediction, and what models are popularly used to tackle the prediction problem in the literature. A lack of a readily available dataset for priority ranking is due to priority generally not being recorded in hospital EHRs. Readily available in this instance refers to existence in a typical hospital database and not that it is easily and openly accessible.

Visualisation of the studies that have been carried out regarding using machine learning to predict admissions and scheduling in the ED. Dashed lines indicate some studies opt to use these features.

Popularity of different methods and data availability for each of these problems.

5.3. Machine learning in elective admissions

We have primarily focused on machine learning applied to emergency admissions as this is the larger body of research in the field. The stochastic nature of these admissions in terms of number and type of admission means that these are the most disruptive to patient flow in a hospital. Elective patients are generally planned for and so resource is available to treat them.

There are however studies that also apply machine learning to the admission of elective patients. In the study conducted by Nelson et al ( 2019 ), the authors use machine learning to assess whether or not patients will actually attend their scheduled appointments in hospital. Despite the resource being prepared for these patients, a no-show will result in a waste of this resource and this work aims to provide a way to then re-direct that resource. The authors use information on the history of the patient with a gradient boosting machine to get a strong predictive accuracy. Srinivas and Ravindran ( 2018 ) carry out this same prediction of no-shows to elective appointments, however they then leverage the risk of no-show in order to update the scheduling system of the hospital.

With many health systems providing long waiting times for appointments (Xavier 2003 , Dimakou 2013 ), another important factor when it comes to elective patients is prioritising patients in the schedule. Yousefi et al ( 2019 ) approach this by first using a clustering algorithm to group patients into different priority categories. They then treat the schedule as a Markov decision process where waiting time for patients in the high priority clusters is to be minimised.

These approaches can be difficult to validate due to their direct impact on the scheduling of appointments. As a result, there is no chance to verify if the patients turn up or not once the schedule is changed. They also rely on historical behavioural data (such as how many times a patient has missed an appointment before) which are not stationary distributions and therefore limit how successful supervised learning can be in this domain in the long term.

5.4. Summary

Overall, the application of machine learning to predicting emergency patient admissions and scheduling is well-explored. Works are generally split between emergency and elective patients with further subdivisions according to the data used, the models used and what is being predicted (see figure ​ figure2). 2 ). Very few works validate their models in hospitals in real-time, most using a retrospective test-set to assess performance. Furthermore, some models are difficult to validate due to being designed to intervene in the admission and scheduling process.

There is also very little connecting these studies. Most work is carried out with the data from the hospital that the authors are associated with and built around that. Due to hospitals being different, that leaves little scope for building on previous work or developing models that can be used universally. A public dataset that could be held as the gold-standard for patient flow would aid in this significantly as a benchmark for experiments.

6. The emergency-inpatient interface

The emergency-inpatient interface is an ill-defined area of many hospitals (Staib et al 2017 ). There is usually a lack of clarity on the ownership of this space of the hospital and who should manage the handover of patients from the ED to an inpatient setting. As a result of this lack of clarity, it should come as no surprise that there is much published on making predictions across this gap in the hospital. While it may seem like an obvious task to predict which patients need admission to hospital from the ED, it has been shown that this is not a trivial task (Beardsell and Robinson 2011 ). Whereas the works discussed in section 5 aim to provide predictions for planning (such as expected numbers or schedule planning), the predictions of the works found in this section are primarily designed for decision-support.

A natural question that can be asked is if admission to the hospital from the ED can be predicted. Hong et al ( 2018 ) and Graham et al ( 2018 ) show this can be done using multiple machine learning models including a logistic regression, XGBoost and a deep fully-connected neural network. They show this is possible using historic patient information as well as information from triage. This does however limit the potential use to patients who already have electronic health records. Leegon et al ( 2005 ) and Raita et al ( 2019 ) therefore also carried out this prediction but only using a few variables that are measured early in the ED admission process and showed using a Bayesian network that admission to hospital can still be accurately predicted. Sun et al ( 2011 ) echo this sentiment, setting up their classification such that the clinical staff may predict the risk of whether an inpatient bed is needed or not as soon as triage is complete in the ED. This prediction is then further augmented with the inclusion of using the free-text written by the triage clinical staff as features to improve the performance of the model (Zhang et al 2017 , Sterling et al 2019 ).

As was the case for prediction of admissions in section 5 , many authors find it useful to consider certain demographics of patients. An example is in Lucke et al ( 2018 ) where a multiple logistic regression is used to predict hospital admission from the ED for a cohort of patients over 70 years old and another below. This is due to older patients generally being more at risk of admission and so by creating a model conditioned on age, they are able to better predict those most at risk of admission. In Mowbray et al ( 2020 ), elderly patients are considered to be those aged 75 and over, however they also show that accurate predictions of admission can be made for an elderly cohort of patients.

Another demographic that is often targeted for prediction is that of paediatric patients (Walsh et al 2004 , Marlais et al 2011 ). In these studies, logistic regressions are used to predict whether a paediatric subset of patients will require admission to the hospital. Once again, by creating a separate cohort for these patients, they can make predictions comparing patients to other similar patients, rather than comparing with older patients who have different physiologies. This introduces a trade-off of improving model accuracy while reducing how generally the model can be applied.

Further subsets of paediatric patients have been made for example by considering those patients suffering from asthma exacerbation and predicting those most likely to be admitted to hospital for treatment (Patel et al 2018 ).

To augment the performance of a model predicting paediatric admissions to hospital from the ED, the textual data recorded during triage can also be used as features (Roquette et al 2020 ). Natural language processing techniques have been used in order to extract useful information which has been shown to improve predictability of admission.

6.1. Predicting inpatient resource utilisation

Many of the studies that are created in predicting admission to hospital focus on subsets of patients with certain conditions. As these patients will require the same treatments and specialist staff to treat them, this can be seen as resource prediction for patients being admitted to the hospital from the ED.

An example is in Ong et al ( 2012 ) where heart-rate variability in the ED is used alongside other demographic information on the patient as input features to a support vector machine. This is then used to create a score on the likeliness of cardiac arrest occurring in the next 72 h. While this is not strictly framed for patient flow, this prediction allows clinicians to plan for resource in the cardiac department. Predicting whether or not a patient is septic is also important for patient flow in terms of resource planning. As a result, models predicting whether or not ED patients are suffering from sepsis have been developed (Horng et al 2017 , McCoy et al 2017 , Delahanty et al 2019 ). The authors use a mixture of information available at triage, demographic information and free-text to make prediction of whether or not the patient is septic, which if accurate, could allow planning of their treatment before the patient becomes critically ill.

In fact, there have been many such studies predicting whether or not a patient is suffering with a certain condition in the ED which allows resource planning. These include predicting if a patient is suffering from acute kidney injury (Martinez et al 2020 ), requires intensive care (Fernandes et al 2020 , Finkelstein 2020 ), is suffering from a urinary tract infection (Taylor et al 2018 ), have bacterial infections (Ramgopal et al 2020 ) as well as predicting emergency hospitalisation of patients undergoing chemoradiation (Hong et al 2018 ).

While these predictions are useful for planning patient flow, they are not explicit predictions of admission. A more explicit approach is seen in Luo et al ( 2019 ) where the classifier is trained to predict admission to hospital of patients suffering from bronchiolitis.

While predicting admission to hospital from the ED is useful, a greater level of granularity, such as which departments in the hospital the patient will be admitted to, is more useful to clinical staff. An example is seen in Lee et al ( 2020 ) where rather than predicting admission, they predict the disposition of the admitted patient, choosing out of intensive care units, telemetry units, general practice units and observation units. As these ‘ward types’ tend to have separate resource, they are better able to adapt their resource according to the predictions made. This approach is also seen in El-Bouri et al ( 2020 ) where the authors also classify into ‘ward types’ to provide a similar level of granularity to the hospital admission prediction problem. However, in this case they use medical, cardiac, neuro, trauma, intensive care, surgical and general/obstetrics and gynaecology as their ward groupings. They develop a novel ‘interpretable’ layer for their deep neural network to guide information collection at triage and train the model using curriculum learning. El-Bouri et al ( 2020 ) further augment their model by using reinforcement learning to allow an agent to carry out the curriculum learning that maximises the performance of predicting where in the hospital patients will be admitted to. In order to make as general a model as possible, these studies of patient disposition do not consider subsets of patients but rather the entire population of the ED to replicate daily working conditions.

Figure ​ Figure3 3 shows the general structure of works that have been conducted on predicting flow from the ED to hospital. It should be noted that as all of these works consider flow from the ED, all patients considered are emergency patients. Table ​ Table2 2 shows how readily available labelled datasets are for the EDii prediction problems and the popular approaches to tackling them. It should be noted that readily available here means data that would generally be saved on a hospital EHR and not data that would be easily accessible on a public dataset.

A decision tree showing how the studies that have been conducted on predicting movement from the ED to hospital are structured. Dashed lines indicate that these features are used in some works but not all.

6.2. Summary

We have seen in this section that the emergency-inpatient interface in hospitals, while being ill-defined in practice, is well researched using machine learning. Authors predict admission from the ED in order to provide information for clinical staff to prepare space should it be needed. To improve the performance of the classifiers, many authors condition their models on the demographics of the patients (e.g. elderly or young patients) or on the patient disease (patients suffering from the same ailment in the ED). In order to provide greater granularity on which resource will be used in the hospital, some authors also predict which ‘ward type’ will be used by the patient to be admitted to the hospital.

However, once again there is little connecting these studies. None of the studies reviewed build off each other or use the same dataset for comparison. Furthermore, the definitions used to categorise patients vary by paper. As was seen when categorising elderly patients some studies use 70 and over and some use 75 and over. Clearly it would be beneficial to have an agreed range to make models more comparable. This further emphasises the need for a shared, publicly available dataset for use when creating machine learning models for patient flow. All definitions of demographics should be included in the dataset so that researchers make valid comparisons to models. It will also be beneficial in allowing researchers to compare their methodologies and validate them on the same dataset as others as well as apply them to their own hospital’s data. This will also make research more consistent, allowing researchers to build and improve upon each others models instead of applying similar models to similar problems using different data.

7. Intra-hospital resource management

Once patients have been admitted to hospital, there is yet another layer of resource flows that need to be considered. Patients can be transferred between wards, need tests carried out and must be moved to use certain equipment such as MRI scanners. These all require staff to carry out the movements and therefore place a demand on the resource of the hospital. As this resource is part of that needed to deliver the patients through hospital to discharge, it is relevant to patient flow.

7.1. Ward transfer

The most common way in which machine learning is used to provide predictions for ‘inpatient flow’ is through predicting if patients will be transferred to another ward. Note that while in section 6.1 we considered studies which investigated patient degradation as a signal for resource preparation, we will not consider degradation for inpatients as a signal for resource prediction. This is due to hospitalised patients generally being admitted to wards that are capable of handling patients in their condition. It is also due to the fact that using machine learning for the monitoring of inpatients for degradation has a very rich literature and would require a review of it is own (Clifton et al 2015 ). As a result, we only focus on works that explicitly predict admissions or transferrals of patients.

7.1.1. ICU transfer

By far the most popular type of prediction to make in the inpatient setting is predicting admission of a patient to the ICU. This is due to the fact that the ICU is a resource intensive area of the hospital and any way of informing the planning of this unit is beneficial to the running of the hospital (Skowronski 2001 ).

Wellner et al ( 2017 ) use a logistic regression to show that it can be predicted that a patient will need admission to the ICU 16 h ahead of time. Furthermore they demonstrate this using data from three separate institutions, helping validate their model. Desautels et al ( 2017 ) carry out the same investigation in a tertiary care hospital but consider readmissions to the ICU in 48 h. This is also explored by Yoon et al ( 2016 ) who develop a ‘Bayesian belief system’ to predict admission to the ICU, but this time 9 h before it is requested by the clinician in charge. An NLP approach has also been investigated in Khattak et al ( 2019 ) where the online messages of doctors and nurses to each other are used in order to predict transferral of a patient to ICU 3 d prior to the event taking place. It should be noted that for all of these studies, the outcome being predicted is different and so the studies cannot be compared.

Echoing the narrative presented in section 5 , many researchers have also considered predicting readmissions of inpatients to the ICU. This is seen in Rojas et al ( 2018 ) where the authors investigate which patients, who were previously in the ICU, will be readmitted from their inpatient ward. To predict this they use a gradient boosting machine with features derived from the electronic health record of the patient as well as various blood tests that were taken. A time-series approach to this prediction was investigated by Lin et al ( 2019 ) where an LSTM was used and trained on the ICD-9 embeddings of the patients who had previously been admitted to the ICU, their demographics and the chart event features of the patients. They show a strong prediction accuracy when considering if a patient will be readmitted to the ICU within 30 d of their discharge.

Once again, conditioning the dataset on the demographic in question is utilised for the inpatient setting. Rubin et al ( 2018 ) demonstrate using adaptive and gradient-tree boosting that they can predict the transfer of a child to the paediatric ICU 8 h preceding the transfer. The prediction of transfer to paediatric ICU is also carried out in Zhai et al ( 2014 ) where a logistic regression is used to predict their transfer within the first 24 h of their inpatient status.

We again see works where the datasets (and therefore the models) are conditioned on the co-morbidities of the patients. Lee et al ( 2019 ) condition their dataset on patients who have undergone cardiac surgery and predict whether these patients will be readmitted to the ICU. They use a logistic regression with L1 regularisation to provide interpretability to their model, but also use a causal inference method to compare their findings. They find that there is little agreement between the two methods of feature importance ranking.

7.2. Resource management

During a patient’s stay in hospital, various tests may be requested to help clinicians gain a better understanding of the patient’s condition. These tests are also an important part of the patient flow process and timely testing helps to improve flow through the hospital. An example is seen in Molaei et al ( 2016 ) where the authors investigate whether or not they can predict if inpatients with traumatic brain injury require a CT scan using ‘cost sensitive’ random forests. In doing so, they aim to create a prioritisation system for scanning, which will allow faster treatment of patients and therefore a better patient throughput.

Another way in which resource management has been tackled with machine learning is in the scheduling of laboratory samples that need to be processed (Williams et al 2019 ). Again, by scheduling these samples in an efficient way, this allows patients to be treated more quickly in the hospital, and in some cases prevents the unnecessary hospitalisation of a patient.

These examples can be seen as assessing the risk of resource utilisation on a patient-by-patient basis. A more high-level view is used in Vieira and Hollmén ( 2016 ) where all resource is pooled together (anything including staff or use of machinery). Random forests are used to perform regression on the expected resource use in the next 30 d. While this has limited use to clinical staff due to the lack of granularity, it may be useful for budgeting purposes.

7.3. Hospital-wide flow

There are very few works that seek to predict the full patient journey through a hospital using machine learning. This may be due to the fact that transfers of inpatients is generally quite rare due to most inpatients being admitted to a ward that is capable of providing the appropriate care for them. Xu et al ( 2017 ) treat the hospital journey as a point process. They use a generalised linear model to predict the next location a patient will be transferred to as well as the dwell-time in that unit. They utilise the MIMIC-III dataset (Johnson et al 2016 ), which is an ICU based dataset and so the transitions they predict are between various types of intensive care unit. However, in terms of predicting the inpatient journey, this is a promising direction. Expanding to the entire hospital, it is possible to predict movement of patients between wards as well as for the use of machinery. Also predicting the dwell-time will allow for better planning of the flow of patients.

7.4. Summary

Of the four parts of the hospitalisation process that we have defined, the inpatient setting is the one in which machine learning has been used the least. The majority of studies investigate the transferral of inpatients to the ICU due to the resource-intense nature of ICUs. There have also been limited attempts at utilising machine learning to predict the expected resource that will be required by a hospital, either as a whole, or on a patient-by-patient basis. Very few works again have attempted to predict the whole hospital journey using machine learning. A common inconsistency throughout the literature is the prediction lookahead time that is considered. Standardising the lookahead time will allow studies to be more comparable and again, crucially, build upon previous work to further improve and integrate the field.

As the vast majority of studies are conducted with a clinical need in mind, this may reflect that the inpatient journey is not seen as a very important part of the patient flow problem. Figure ​ Figure4 4 shows the structure of the studies that have been carried out in this area of patient flow. Table ​ Table3 3 shows the data availability for these prediction problems and popular methods used to tackle them.

Visualisation of the studies carried out on using machine learning to aid in the inpatient journey. Dashed lines indicate some studies opt to use these features.

8. Discharge prediction

The importance of discharging patients in a timely fashion for patient flow cannot be over-stated. Long patient stays incur greater cost to the healthcare institution and reduce capacity for new patients to be admitted (Rotter et al 2008 , 2010 ). As a result, a standard metric of the quality of care being provided is the patient length-of-stay (LOS) (Brasel et al 2007 ). Patients who are admitted for long periods of time (either due to condition or due to having no appropriate discharge destination) are commonly referred to as ‘bed-blockers’ and can constitute a significant proportion of the hospital population (Coid and Crome 1986 , Styrborn and Thorslund 1993 , Mustafee et al 2012 ). Early recognition of the patients likely to have a long LOS should therefore allow for the planning of their treatment by the hospital, such as their admission to long-stay wards and beginning preparations for their discharge.

It should therefore be unsurprising that many researchers have seeked to employ machine learning in order to predict the LOS of patients in order to provide hospitals with a better idea of how much resource will be required for patient stays. Note that the prediction of LOS or of discharge are essentially the same as they both aim to predict when a patient is able to leave the hospital. We will refer to both types of prediction simply as ‘discharge prediction’.

Discharge prediction can be separated into two separate subcategories for emergency and inpatient settings. In the emergency context, predicting the LOS of patients helps to understand whether the ED is at risk of overcrowding or not. In the inpatient setting, predicting the LOS is useful for the planning of patient admissions and preparation of post-discharge care should it be needed.

8.1. Discharge in the emergency department

Discharge from ED has been treated as a classification as seen in Rahman et al ( 2020 ). The authors predict if a patient will be in the ED for longer than 4 h or not. They use features that are available early in the ED process to train a decision tree binary classifier. This approach is mimicked in Sariyer et al ( 2019 ) where various learning algorithms are experimented with to classify patients according to their length of stay in the ED. Azari et al ( 2015 ) acknowledge the large imbalance there tends to be in LOS datasets (with far fewer patients having long LOS), and present an ensemble method combined with multiple logistic regression to overcome this imbalance. However in this work they define a long stay as patients in the ED for longer than 14 h.

Rather than classify patients according to their likely LOS category, some authors prefer to use regression to predict each patient’s LOS in the ED. Combes et al ( 2014 ) use linear regression model to predict the likely LOS of each patient presented to the ED. Ding et al ( 2010 ) instead use quantile regression but once again for the prediction of LOS in the ED. Feedforward neural networks have also been used for regressing the likely LOS of patients (Gül and Güneri 2015 ). One advantage to this approach of regressing the probable LOS is that there are no longer inconsistencies between studies on what is defined as a long-stay. However, this approach is also more difficult to train and achieve an accurate model in practice.

8.2. The inpatient setting

Predicting the LOS of patients in the inpatient setting is significantly more popular as a research area than in the emergency setting. This may be due to a prediction of LOS in the ED being less actionable than in the hospital where preparations can be made to ready a patient for discharge.

A hospital-wide approach is adopted in Pendharkar and Khurana ( 2014 ) where a regression tree is used to predict the LOS of patients admitted to hospitals in Pennsylvania using data that is available at the time of admission. This approach is also applied in Tanuja et al ( 2011 ), this time using a feedforward neural network to regress the LOS. These predictions are carried out at the time of admission. An alternative approach is to implement a classifier every day before discharge and predict the patients who can be prioritised for discharge as seen in Barnes et al ( 2016 ). In framing the problem in this way the authors exploit a static model for a dynamic problem by repeatedly applying the algorithm prior to discharge sessions at the hospital. They use a classification decision tree to prioritise patients ready for discharge.

Predicting discharge has also been approached as a time-series problem. In McCoy et al ( 2018 ) an autoregressive integrated moving average model is used to incorporate a time-series of seasonal data to predict hospital discharge volume. They compare this with using Prophet (Taylor and Letham 2018 ), an additive regression model developed by Facebook Research for forecasting seasonal trends, for the same task. An NLP approach has also been used where the clinical notes from the ED are used in order to predict if a patient will be admitted to the hospital for more than 2 d (Bacchi et al 2020 ).

As has been a common theme throughout this review, discharge predictions are also conditioned on patient demographics. In other sections this is primarily to improve predictive performance amongst patient subgroups. However, in discharge prediction this is due to certain patient subgroups being more likely to be ‘bed-blockers’ such as elderly patients (Launay et al 2018 ). To maximise clinical utility it is more effective to condition the training dataset on these subgroups and apply the algorithms to these patients only. An example is in Elbattah and Molloy ( 2016 ) where a regression forest is used to predict the LOS of elderly patients in a hospital and a random forest is used to predict the location of discharge for these elderly patients. These predictions are used in conjunction with a discrete-event simulation in order to simulate the flow through an Irish hospital. Children are also a cohort of patients in which there can be great variability in LOS. To address this, Castiñeira et al ( 2020 ) use a gradient boosted tree to classify whether or not a child will be a long-stay patient in the paediatric ICU (with long-stay being defined as a stay of greater than 4 d). They also use the static model for a dynamic problem approach by extracting features from the time-series of the patient’s vital signs and repeatedly feeding these to the classifier. Note that this prediction concerns the LOS within a ward and not the hospital stay as a whole.

As with conditioning on demographics, conditioning on co-morbidities is also done in discharge prediction. In fact, this tends to be the most popular form of setting the problem due to patients with different ailments and treatments generally requiring different recovery times.

One such prediction is carried out for patients with congestive heart failure in which the authors apply a static cubist model (Quinlan 1998 ) dynamically as data is updated during the patient stay (Turgeman et al 2017 ). The model is used to regress the likely LOS in hospital of the patient.

Further discharge predictions have been carried out on patient cohorts who have suffered from stroke (Al Taleb et al 2017 ), patients who have suffered hip-fracture (Elbattah and Molloy 2016 ), patients suffering from schizophrenia (Kirchebner et al 2020 ), patients admitted for cardiac care (Daghistani et al 2019 ), patients post-brain tumour surgery (Muhlestein et al 2019 ), patients who have undergone total hip-arthroplasty (Ramkumar et al 2019 ) and patients who have undergone surgery due to colorectal cancer (Stoean et al 2015 ). In all of these studies, there is no consensus for defining a ‘long-stay’ patient.

8.3. Summary

Discharge prediction is one of the more popular areas of patient flow for researchers to apply machine learning. Discharge prediction has been carried out by either predicting whether a patient is likely to be long-stay or by directly regressing the expected LOS of the patient. It has been applied to both emergency and inpatient settings. In the inpatient setting, studies have conditioned their datasets according to demographic. There have also been studies that condition their dataset according to the comorbidity or treatment that the patients of interest have undergone.

A clear inconsistency between studies is the definition of a long-stay patient. Having a common dataset with pre-defined long-stay patients will improve the ability of researchers to compare models and build upon previous work. Figure ​ Figure5 5 shows the structure of the literature published in this field. Table ​ Table4 4 shows data availability and popular methods used to tackle the discharge problems. It should be noted that the difficulty with a labelled dataset for discharge readiness is that generally it is not recorded when a patient is ready for discharge but when they actually are discharged.

Visualisation of the studies carried out on discharge prediction. Dashed lines indicate some studies opt to use these features.

9. The future of machine learning in patient flow

The current research efforts in the discipline of machine learning in patient flow have demonstrated the feasibility and potential of machine learning to optimise patient flow in all of the four subcategories outlined our study. However, due to the difficulty of expanding and scaling machine learning models across different healthcare contexts and institutions, the current research efforts are still removed from delivering value in the routine and daily management of patient flow in healthcare institutions. In this section, we outline the future research opportunities to advance the applicability of machine learning in patient flow.

9.1. Priorities in patient flow

While all of the problems outlined in the above review are important for clinical practice, solving some of these problems is more urgent than solving others. An example of a high-priority problem to solve is predicting readiness for discharge. One of the greatest problems dealt with in patient flow is the ‘bed-blocker’ phenomenon whereby patients do not have appropriate destinations to be discharged to. Predictions of readiness for discharge will not solve the lack of space in care homes, however it will allow for more effective allocation of the time and attention of clinical staff.

An equally important task is the prediction of ED admissions. This represents the front-end of the hospital with the discharge readiness representing the back-end. Being able to accurately predict patient admissions numbers in the ED would allow for accurate planning of staffing rotas thereby reducing costs and time wasted. It would also greatly improve the care provided for each individual patient.

Following on from this, should these predictions not be accurate enough, solving the ED-inpatient interface problem would be the next most important. This prediction would prevent the filling up of the ED due to inability to transfer patients into the hospital. Having an accurate model here would create a more streamlined flow of patients into the hospital, but naturally would depend on there being enough flow out as well.

Finally, the problem that should be least prioritised is inpatient transfer prediction. Despite being important, inpatient transfers are generally quite rare due to patients being admitted to appropriate wards from the outset. However, there is value in predicting resource flow and patient movement in order to plan that resource.

9.2. Current challenges

9.2.1. data challenges.

Throughout this review, we have emphasised the need for a common dataset that all researchers can use to benchmark their models and experiments on, as well as have agreed definitions of what age ranges ‘elderly’ patient fit amongst other definitions. However, creating a publicly available dataset does not come without its own challenges. The first issue is that of patient privacy. While there are many data anonymisation methods that can be used to remove association of the data with individuals, prior information such as the source hospital can be used to reconstruct the identities of the patients. There then exists a trade-off between how much information is hidden and how useful the data is to machine learning practitioners. A potential solution for this is sourcing data from multiple medical centres and compiling them together in a dataset. This brings us onto the second challenge which is a lack of standardisation in the recording of health data. In order to take advantage of the data from the EHRs from multiple hospitals, we must first stipulate that these hospitals record data in an agreed fashion.

One example of a publicly available healthcare dataset used for benchmarking is MIMIC-III (Johnson et al 2016 ). The success of this dataset can be seen through the volume of works that have used it for model comparison. However, for the purposes of patient flow, this dataset is difficult to use due to its focus on intensive care patients. It therefore does not include the data from the EHRs on the key resource utilisation and patient flows in the hospital (unless they are between intensive care units). A dataset built in a similar fashion to MIMIC-III but with the appropriate patient flow data would benefit the research community greatly.

9.2.2. Technical challenges

Currently the majority of patient flow models use a specific dataset from a hospital that can be derived from a certain subset of patients. The model is then applied to aid that hospital in prediction with very few researchers extending their models beyond their own hospitals. This approach is limited due to the variable and dynamic nature of healthcare datasets. Distributions from the same source hospital are subject to issues such as covariate shift whereby the underlying distributions of the features change with time. Examples are the changes to the distributions that can be found in the EHRs of hospitals during flu season or during the COVID-19 pandemic that has swept the world.

Variability also exists across health care delivery institutions and organisations ranging from small primary care centres to large tertiary hospitals. These organisations are different in their resources, organisational structure, staff training, and culture. These differences create variability in healthcare delivery practices, organisational processes, and patient flow across these different institutions as well as variability in what data is recorded and in what format it is recorded.

Differences also exist in the distributions recorded by healthcare institutions due to the differences in populations across the world. Examples include the prevalence of different diseases across different communities and geographical contexts (e.g. the presence of type II diabetes mellitus can vary from 3.5% to over 20% across different populations) (World Health Organization 2016 , James et al 2018 ).

Another issue that is faced is the lack of complete information delivered by the majority of prediction algorithms. While it is useful to know that a patient will be admitted to a certain location in the hospital, having some knowledge of their severity or the likely medications that will be needed for them will further help with the planning of their stay.

These issues faced during deployment create challenges in the applications of machine learning, particularly in the generalisation of models to other hospitals and for their continual use over long periods of time.

In the face of these challenges, we believe that certain research directions will aid future researchers to prepare models that will better serve hospitals to improve patient flow. These research directions should address the issues discussed above, as well as ensure that they integrate seamlessly into the running of the hospital.

9.3. Feature engineering

The majority of studies discussed in this review take advantage of the fact that there exist EHR systems in many modern hospitals which allow data extraction and dataset creation. However, there remain challenges in terms of data collection for the different tasks at hand.

The ED admission prediction relies on seasonal information which can be correlated with admissions but is generally a difficult prediction to make. Wearable sensors could benefit this prediction greatly, providing more granular information to the hospital. The sensors could also be provided to patients who need them most (and are most likely to be brought to the ED in an emergency such as elderly patients in care homes).

We believe that further improvements to data collection could be made in the inpatient journey as well as in discharge prediction. Currently, while scans in the hospital are logged on the EHR, the movement of patients to scans are not and nor is the resource associated in moving that patient. These data would be very helpful to provide a more complete picture of what resource each hospitalised patient utilises and thereby helping machine learning scientists create more accurate predictions of the likely resource needed.

In discharge prediction, one of the challenges is that it is generally not recorded when a patient is medically ready to leave the hospital but when they actually do. Augmenting a dataset with this information could help predict when a patient is ready to leave hospital and in doing so, allow the team looking after them to move their resource to more vital care, with a more generalised team looking after the patient thereafter until discharge.

9.4. Multitask learning

The first research direction to be considered is multitask learning, a machine learning method that allows multiple tasks to be learned at the same time. One of the aims is to exploit the learning signals generated by training on one task to create an inductive bias in the model that will allow the effective learning of another task by the same model (Caruana 1997 ).

Multitask learning can be applied to different problems across the four domains of patient flow to both related (e.g. predicting risks of various in-hospital complications) or unrelated tasks (e.g. predicting length of stay in the ED and predicting hospital admission destination). Once again this relates to the usefulness of having more granular information for clinicians to work with. An example may be when predicting the location of admission of a patient to hospital, also having some prediction of whether the patient is likely to deteriorate or not. This gives better indications of the likely resource requirement of the patient as well as their likely trajectory within the hospital. While this could be done using separate models for each prediction, a single model that can embed an accurate representation of the patient will be more informative and useful to clinical staff. As a result, a key component of this work will be in the development of representation learning algorithms (Bengio et al 2013 , Van Den Oord et al 2017 ) that are capable of representing patient conditions upon presentation to the ED or admission to hospital.

Multitask learning has been applied in many healthcare applications to leverage the shared information across different tasks. Huang and Dong ( 2018 ) have used multitask learning to predict major adverse cardiac events, identifying each type of adverse event as a single task as opposed to having a multiclass classification. Xia et al ( 2019 ) have also used this approach to predict prescription patterns for various drugs that are given to similar patients. Multi-task learning has also been used in medical imaging. Khosravan et al ( 2019 ) have used multitask learning in the detection of abnormal nodules on chest CT scans for lung cancer screening. They jointly train their model to segmenting potential abnormalities and identify the presence of a nodule in the region of interest. This is further evidence of how more granular information from the model can provide clinicians with better insights into the condition of the patient.

9.5. Transfer learning

Transfer learning is based on the principle of knowledge transfer across different machine learning tasks and models. It is based on the notion that knowledge gained by the algorithm when trained to solve a particular problem can be stored and applied to solve another related problem, which means it is closely related to multitask learning. This approach includes transferring knowledge from the source domain, D S , to the target domain, D T , to help improve the learning of the target-domain task, T T .

Transfer learning can provide significant advantages in the applications of machine learning in patient flow. It can enable (a) the transfer of knowledge across different tasks and (b) the transfer of knowledge across different populations. The former can help overcome the lack of clinical data for certain problems. For example, one of the barriers to developing effective machine learning tools for COVID-19 patients is the lack of data on COVID-19 patients. A transfer learning approach can provide a solution by using a model that is pretrained on a large non-COVID-19 dataset and adapting it to perform the task of interest in COVID-19 patients. Transfer learning has been used to overcome the lack of COVID-19 imaging data by Mahmud et al ( 2020 ). They trained a convolutional neural network by using a pretrained a neural network (pretrained on a dataset of bacterial and viral pneumonia chest x-ray scans) and fine-tuned this using scans from COVID-19 patients. This was done due to the scarce availability of chest x-rays from these patients.

Transfer learning can also help us transfer knowledge across different populations. This is valuable clinically given the diversity and differences in the genetic predispositions, prevalence of diseases, lifestyles, and risk factors across different populations. Mao et al ( 2018 ) used transfer learning to generalise their sepsis prediction algorithm to a new healthcare setting. They trained their prediction model using data from the MIMIC-III dataset (data from ICU patients) and transferred the model to a dataset from the University of California, San Francisco (UCSF) Medical Centre (a dataset of in-hospital patients from a variety of specialty wards). Their transfer learning approach was based on adding incremental amounts of data from the UCSF dataset to the MIMIC training dataset, resulting in better generalisation of the model to the dataset being introduced.

Transfer learning represents an interesting target for future research in patient flow machine learning applications. Transfer learning can be used to generalise models across different healthcare contexts and to overcome a lack of recorded data.

9.6. Continual learning

Continual or lifelong learning refers to the ability to continually learn over time by accommodating new knowledge while retaining previously learned experiences. This approach has the potential to enable machine learning models in the healthcare space to adapt and adjust automatically to new context and settings like a new healthcare context, new patient population, or a new and emerging disease. This has the potential to enable the creation of dynamic clinical AI models that optimise clinical management decision in real time and learn from the continuous influx of information in real world healthcare context. A continually learning algorithm should be an adaptive algorithm capable of learning from a continuous stream of information, with such information becoming progressively available over time. The accommodation of new information should occur without catastrophic forgetting or interference (Parisi et al 2019 ).

However, continual learning represents a long-standing challenge due to the susceptibility of machine learning models to catastrophic forgetting. This phenomenon refers to the decrease in model performance or the complete overwriting of the previously learned information when new knowledge is introduced.

A paper published in the Lancet in 2020 (Lee and Lee 2020b ) highlights the promise of continual learning in revolutionising the applications of clinical AI and leveraging the continuous influx of clinical information to improve patient care. Shah et al ( 2019 ) highlight that machine learning algorithms that are capable of continuous learning are a critical future research and translational direction in healthcare AI. They also report that the FDA is considering widening its regulatory framework to include AI-based Software as Medical Device (SaMD) systems that are capable of continuously learning and optimising performance in real-time to improve patient care.

Continual learning promises considerable value in patient flow as it would enable machine learning models to adjust to different healthcare settings continuously and automatically. Therefore machine learning algorithms would be able to absorb the variation across different healthcare institutions and patient populations. Moreover, continual learning may enable machine learning algorithms to continuously learn after deployment to clinical settings gradually improving their performance through use.

10. Conclusion

We have seen in this review that machine learning in patient flow is a vast if disjoint field. There are many works published with the majority focused on the hospital associated with the authors and little by way of comparison to other hospitals or works. We therefore propose the introduction of a publicly available dataset based on the electronic health records of a given hospital. This should include enough information on all four subcategories of the patient flow process (as highlighted previously) and crucially, must have strict definitions for patient types. The dataset should include:

• • Seasonal information such as the weather, national holidays and ideally EHR data from multiple hospitals.
• • Strict definitions of what age ranges ‘elderly’ or ‘young’ patients fall into for reproducibility and model validation.
• • Pre-defined tasks such as ‘prediction of patient transfer in 3 h from time of measurement’. By creating these pre-defined tasks we improve the ability of researchers to benchmark against each others work and develop upon each others models.
• • A standardised definition of co-morbidities in patients.

We believe that in creating this dataset, a culture of benchmarking on the dataset can be created thereby encouraging researchers to compare their models, build more sophisticated models based on previously published work and crucially provide some external validation to the trained models.

Acknowledgments

ReB is supported by the EPSRC industrial strategy award. T T is supported by The Wellcome Trust [200205/Z/15/Z]. A Y is supported by the Frontier of Development seed funding from the Royal Academy of Engineering (FoD2021424). T Z is supported by the RAEng Engineering for Development Research Fellowship. This research was supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Appendix. Models used for the prediction problems

In order to provide a more complete picture of the works that have been conducted in the space of machine learning in patient flow we here provide flow charts including the models and datasets that have been used to make the predictions. Figure ​ FigureA1 A1 corresponds to ED admissions, figure ​ figureA2 A2 corresponds to the ED-inpatient interface, figure ​ figureA3 A3 corresponds to inpatient transfers and figure ​ figureA4 A4 corresponds to discharge.

A flowchart showing the models that have been used for the separate prediction problems for predicting ED admissions. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

A flowchart showing the models that have been used for the separate prediction problems for predicting ED to inpatient admissions. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

A flowchart showing the models that have been used for the separate prediction problems for predicting inpatient transfers. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.

A flowchart showing the models that have been used for the separate prediction problems for predicting discharges. The top row shows the sources of data used, the row below shows the models that have been used in various different works for the prediction problem, below this is the problem being tackled and below each of these problems is one of the datasets used in a study to train the model.