Issue: 2020 > December > original article

Prediction admission in the older population in the Emergency Department: the CLEARED tool



ORIGINAL ARTICLE
A. Brink, J. Alsma, H.S. Brink, J. de Gelder, J.A. Lucke, S.P. Mooijaart, R. Zietse, S.C.E. Schuit, H.F. Lingsma
AbstractFull textPDF

Full text

KEYWORDS

Aging patient population, crowding, emergency department, prediction model, triage


INTRODUCTION

Elderly patients represent a growing population in the Emergency Department (ED).1-3 Older patients, defined as aged 70 years and over, are overall more vulnerable than the general adult population. They have less physical endurance, are more likely to have multiple co-morbidities,4 and are also more susceptible to polypharmacy and associated risks.5 In the ED, elderly patients have a longer length of stay (LOS) compared to younger patients.6-9 This can be partly explained by an atypical and non-specific presentation of illnesses in the ED.10-12 Moreover, symptom-based triage classifications tend to underestimate the severity of disease,13-15 whilst elderly patients visits in general have a greater level of urgency.6,16 As a consequence, care for elderly is more complex and elderly patients are more often assessed by multiple specialists.17 LOS in the ED is associated with poorer quality of care and may have negative effects on outcomes for the individual patient. Time spent in the ED correlates strongly to the total LOS in the hospital,18 and periods of longer LOS due to ED crowding are associated with increased inpatient mortality.19 Predicting which patients should be admitted directly after presentation to the ED may reduce waiting times and time to treatment whilst improving diagnostic trajectories and quality of patient care. It has been suggested that new strategies to decrease ED LOS can decrease patient morbidity and healthcare expenditure.20,21 Certain patient characteristics, such as aberrant vital parameters, have been shown to be predictive for admission,22,23 yet it is unknown which set of predictors contributes most to admission. We aimed to develop and validate a prediction model for admission, using non-invasive and readily available variables, applicable to the general elderly population at the ED.  


MATERIALS AND METHODS

Study design
We performed a retrospective cohort study. Data from one hospital was used for model development and data from two other hospitals were used for external validation.

Setting and participants
For model development, data were acquired from a large consecutive ED cohort in the Erasmus University Medical Centre, Rotterdam (Erasmus MC), the Netherlands, including all patient visits in the ED from January 1st, 2012 until June 30th, 2014. This ED is a level 3 trauma centre and is situated in the largest hospital in the Netherlands, with 30,000 patient visits annually. Elderly patients, defined as people aged 70 years and over, were selected from this database. Both the first visit of a patient as well as repeat visits were included. Patients were excluded when they died on presentation or died during the ED visit. For external validation, data from the Leiden University Medical Centre (LUMC), which is the academic hospital situated in Leiden, the Netherlands, with approximately 27,000 ED visits annually, and the Medisch Spectrum Twente (MST), a large teaching hospital in Enschede, the Netherlands, with approximately 26,500 ED visits annually, were used. In the LUMC, data were used from an existing database with patients visits in the ED in 2012. For external validation in MST, data were collected from all patients visiting the ED within the first three days of each month in 2015.

Variables and measurement
Outcome was defined as admission or transfer to another hospital for admission, and was collected from the patient records. Basic characteristics including information on sex and age were retrieved from the patient records. Additional ED arrival information was extracted from the patient charts, containing time of arrival and discharge from the ED, triage classification based on the Manchester Triage System,24 vital parameters at arrival (blood pressure (in mmHg), heart rate (per minute), respiratory rate (RR) (per minute), body temperature (in degrees centigrade), peripheral oxygen saturation (SpO2, in percentage), state of consciousness using the AVPU25 or Glasgow Coma Scale scoring system26 (AVPU/GCS), laboratory testing (yes/no), radiology testing (yes/no), and referral status to the ED (i.e., by ambulance, self-referral, by general practitioner).
After merging all the different variables from the patient record, the patients were coded in order to anonymise the collected data. Only contributors to the study had access to the database. This study was evaluated and approved by the Medical Research Ethics Committee of the Erasmus MC. Potential predictors were categorised based on to their normal values. Body temperature was categorized in four groups (≤ 35.9, 36.0-37.0, 37.1-38.4, ≥ 38.5 ̊C). Heart rate was classified in three categories based on the categories used in the Modified Early Warning Score (MEWS). The MEWS is a guide to aid in recognition of deteriorating patients and is based on physiological parameters.27 In order to facilitate a clear model, the original five MEWS categories were reduced to three (≤ 50, 51-100, > 100 bpm). Both systolic and diastolic blood pressure were coded according to the current definition of hypotension (< 90 vs. < 60 mmHg) and hypertension (> 140 vs. > 90 mmHg) for systolic and diastolic blood pressure, respectively. RR was categorised according to the definition of bradypnoea (< 12 times per minute), normopnoea (12-20 times per minute), and tachypnoea (> 20 times per minute). SpO2 was classified in three groups (≤ 92, 93-97, ≥ 98%). Finally, referral status in the ED was coded in three classes based on whether patients were (1) referred by any specialist or general practitioner; (2) arrived by ambulance, in which case the ambulance nurse decided to present the patient at the ED; or (3) were self-referred. In every parameter a missing value category was created to make the model better applicable in daily practice. 

Statistical methods
Univariate logistic regression was used to assess the association between potential predictors and admission. The predictive value was assessed and quantified with the Akaike Information Criterion (AIC) based on the Likelihood Ratio χ2 (LRχ2 ). The AIC is a measure of the relative quality of a model or a parameter and can be used when the database is large and selection on p value will result in a large number of selected parameters. The selected predictors from the univariate analysis were combined in a multivariate model and selection of the final set of predictors was based on added values of each predictor (based on the AIC) and clinical knowledge. 
Statistical methods Univariate logistic regression was used to assess the association between potential predictors and admission. The predictive value was assessed and quantified with the Akaike Information Criterion (AIC) based on the Likelihood Ratio χ2 (LRχ2 ). The AIC is a measure of the relative quality of a model or a parameter and can be used when the database is large and selection on p value will result in a large number of selected parameters. The selected predictors from the univariate analysis were combined in a multivariate model and selection of the final set of predictors was based on added values of each predictor (based on the AIC) and clinical knowledge. 
The performance of the model was calculated using the Area Under the Receiver Operating Characteristic (AUC). During internal validation, the AUC was corrected for overfitting using the bootstrapping method28 on the dataset, with repetition of the procedure of 500 times. External validation was performed in the LUMC and MST databases. For validation, 100 events of ‘admission’ and at least 100 non-events were required to occur.29 Based on admission rates in both LUMC and MST, we considered a validation sample of at least 500 patient visits to be sufficient for external validation of our model. 
The external validity was examined by calibration and discrimination of the model in the validation samples, using calibration plots and the AUC. In the calibration plot, the calculated probability of admission is plotted against the observed admission. The calibration slope is the regression coefficient of the model in which the linear predictor (admission yes or no) is the only parameter. Ideally, the slope is 1.30,31 The intercept in the plot indicates whether predictions are systematically too high or too low and should ideally be zero.32 In the LUMC database, data on whether radiology tests had been performed were not recorded. Therefore, we developed a new model on the data following the same strategy for model development, however, leaving out radiology testing. This alternative model was validated in the LUMC sample. 
Subsequently, a score chart was developed based on the regression coefficients fitted on the combined data. Therefore, data on radiology testing were imputed for the LUMC database using multiple imputation. An application was built to calculate the chance on admission to facilitate accessibility of the tool in day-to-day practice.33 To aid in the decision whether preparations for admission should be started for a specific patient, a specific cut-off point of chance on admission should be determined to guide this decision. Such a cut-off should be based on sensitivity and specificity and the importance of avoiding false-negatives and false-positives (i.e., taking action in a patient that in the end does not need to be admitted versus taking no action in a patient that does need to be admitted). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for a range of possible cut-offs in the combined cohort of the MST and LUMC cohorts were calculated.
All analyses were undertaken using R statistics version 3.1.3 (March 9th, 2015).34 The foreign library was used to transfer the database from SPSS (version 21) to R.35 For model development, the lrm function of the rms package was used.36 Finally, the calibration plot was built using the val.prob.ci function, which is an adjustment to the val. prob function of the rms package. For the application, we used Rstudio.35


RESULTS

Participants
The derivation database consisted of 76,663 ED visits between January 2012 and June 2014. Selection on age of 70 and over reduced the number to 5,265 patients who visited the ED 7,606 times. The admission rate was 54%. In the derivation group, 55.8% of the patients were male and the median age was 76 years. The validation dataset consisted of 4,250 patient visits from LUMC, of whom 45% were admitted and 563 patient visits from MST, of whom 71% were admitted (table 1)

Predictors of admission
In the derivation cohort, the strongest predictors of admission were laboratory testing (OR [95% CI]: 13.202 [11.104-15.695], p: < 0.001) and arrival by ambulance (OR [95% CI]: 5.168 [4.389-6.085], p: < 0.001) (table 2). The categories ‘referred’ and ‘arrival by ambulance’ in the predictor referral status had similar odds ratios and were therefore combined. The ‘immediate’ (red) and ‘very urgent’ (orange) triage groups in the MTS classification were also combined. These adjustments did not alter the model performance. 

Model development
Based on the AIC, we included the following parameters in the final model: laboratory testing, body temperature, heart rate, diastolic blood pressure, systolic blood pressure, SpO2, respiratory rate, referral status, MTS category, and radiology testing, which yielded an AUC of 0.770. Bootstrap resampling decreased the performance by 0.004, resulting in an internally validated AUC of 0.766.

External validation
Patient characteristics in the derivation and validation cohort did not significantly differ, except for admission rates in MST. The univariate effects of the predictors within the validation and derivation cohort were comparable.
However, the OR for MTS classification ‘immediate’ in the LUMC was higher than in Erasmus MC and MST (OR [95% CI]: 64.526 [15.870-262.350] versus 1.861 [1.408-2.460] versus 3.898 [0.897-16.945], respectively). Discrimination in the LUMC data showed an AUC of 0.725. The calibration plot showed an intercept of -0.308, reflecting the fact that the overall admission rate was lower (45%) compared to the development cohort. The calibration slope was 0.826 (figure 1). Discrimination in the MST data showed an AUC of 0.797. However, MST had an admission rate of 71 per 100 patient visits, resulting in a calibration intercept of 1.018 and a calibration slope of 0.904 (figure 2). 


Score chart
The final model was named ‘CalcuLation of the Elderly Admission Risk in the Emergency Department’ (CLEARED) tool and can be used to calculate the probability of admission. We used the parameters from the derivation cohort and introduced a ‘hospital factor’ to correct for the differences between admission rates of MST (non-academic hospital) and Erasmus MC/LUMC (academic hospital). The formula is presented in table 3; the online application is accessible through the following link: http://bit.ly/clearedtool. 

We calculated predictive probabilities of the CLEARED tool of admission for decile cut-off points (table 4). The predictive probabilities for the separate MST and LUMC cohorts were similar, however PPV was higher for MST. This could be explained by the higher admission rate. The positive predictive value ranged from 0.57 to 0.91. This indicates that in the highest decile, 91% of the patients were correctly admitted. An admission cut-off point of 80% would result in the identification of 7.8% (n = 975) who are eligible for earlier admission. Of these patients, 86.9% (n = 847) were actually admitted, meaning that there were 13% unnecessary hospital admissions. Patients with a low admission risk (< 80%) had a similar ED LOS than patients who were admitted with a high admission risk (> 80%). However, patients who were eventually discharged despite a positive advice indicated by the CLEARED tool, had a significantly longer LOS, median 223 vs. 185 vs. 178 minutes (p < 0.001), respectively.


DISCUSSION

The aim of this study was to develop and validate a prediction model for admission for the general elderly population presenting to the ED. We successfully developed the CLEARED tool using data from our retrospective cohort study; a prompt that accurately predicts admission of elderly patients visiting the ED. External validation showed accurate performance. As outlined in the introduction, increased LOS in the ED has detrimental effects on elderly patients. This is not only because the time to adequate treatment is lengthened, but also due to the stay in the ED itself, which enhances the development of a delirious state.37,38 In our hospital after arrival at the ED, initial evaluation and triage of urgency is performed by a nurse followed by a primary survey of the patient by a physician, followed by diagnostics tests. The decision whether or not to admit the patient to the hospital is made only after the results of the diagnostics are known. Subsequently, a hospital bed request is made and the patient awaits transfer to the ward. In this way, decision-making about admission is late in the process, which causes a marked increase of LOS in the ED.39 The use of the CLEARED tool can detect elderly patients who are to be admitted to the hospital, shortly after their arrival in the ED. From that moment on, admission to the hospital can be organised without delaying the diagnostic and therapeutic processes. This can dramatically shorten the LOS in the ED for patients. Further reduction of LOS may be possible when using the CLEARED tool in combination with an acute medical unit.
An increasing number of Dutch hospitals have an acute medical unit. In these hospitals, patients who are identified as in need of admission, can be transferred to such a unit where, after the initial diagnostics and therapeutic interventions are performed in the ED, further diagnostics and treatment can be completed. The incorporation of an acute medical unit in addition to the CLEARED tool can further reduce LOS in the ED, which enhances the workflow in the ED and reduces crowding. An acute medical unit is not necessarily a unit, from which every patient gets admitted in-hospital. It can also function as an extension of the ED, allowing observation of patients for several hours, followed by admission or discharge. In our opinion, patients with a high chance of admission according to the CLEARED tool should be admitted to the acute medical unit as early as possible to optimise patient flow through the ED. If admission is not needed – even with a positive CLEARED – they can easily be discharged, which is in line with current practice. 
One of the key points of the CLEARED tool is the inclusion of vital parameters as predictors for admission. Using these parameters, we are able to form a better estimate of the severity of illness of a patient, which is the main reason for admission. These parameters are also readily available on arrival at the ED, and are measurements that are routinely performed when patients enter the ED. This makes the CLEARED tool easily applicable.
Another strength of our model is that it is developed using a large database, which reduces the chance of overfitting and limits the uncertainty of the model. Several other prediction models to predict admission have been previously developed. The identification of Seniors at Risk tool (ISAR) is the most well-known screening tool to identify elderly with a high chance of adverse outcomes.40 This screening tool is composed of six ‘yes-no’ questions on major topics like cognitive impairment, polypharmacy, and previous hospital admission. The main aim of the ISAR tool is to identify patients at risk of loss-offunctionality after a hospital stay, and not to identify patients at risk of clinical deterioration and death. The predictors used in the ISAR tool differ vastly from the predictors in our study. In contrast to the CLEARED tool, the ISAR tool focuses on the pre-existing situation before arriving at the ED, in contrast CLEARED, which focuses on vital parameters at presentation. Therefore, gathering information for the ISAR tool takes more time. The discrimination performance of the ISAR for admission ranges from 0.65 [0.62-0.68]41 to 0.68 [0.66-0.70],42 which is lower than our model.
Similar to the ISAR, is the Triage Risk Screening Tool (TRST), which also comprises of six ‘yes-no’ questions, such as cognitive and physical impairment, polypharmacy, previous ED visits, and hospitalisation.43 This model is tested for admission in two studies and performed with an AUC of 0.64 and 0.66, respectively.42,43 Other admission prediction tools for elderly are triage based (AUC = 0.73, 0.77, 0.741),44-46 the Silver Code (AUC = 0.63),41 and a tool derived by Yip et al. (AUC = 0.713).47 Unfortunately, many of these models have not been externally validated.
The higher AUC of our tool suggests that the CLEARED tool is superior in discrimination, although the AUC is, apart from a measure of model performance, also a reflection of the underlying population. Therefore, a prospective validation study should be performed comparing all the existing tools in one large database. A merit of the external validation of the CLEARED tool was that it was performed in both an academic and a large non-academic teaching hospital. The discrimination remained high in both centres, and calibration was good in the LUMC, but our model underestimated the chance of admission in MST. This was most likely due to the high admission rate (71%) in this centre and possibly a difference in population as it is located in different part of the country. A factor was introduced to account for the fact it is a non-academic teaching hospital, although we were unfortunately unable to confirm this with other non-academic hospitals. We expected that patient characteristics might differ between academic and non-academic hospitals; nonetheless, it is satisfactory that the CLEARED tool performs well in both settings. This makes it more likely that the model can be generalised to other EDs.
When the CLEARED tool is eventually implemented in clinical practice, we recommend that implementation takes place in a stepwise process. First, validation of the model should take place to establish its congruency with local protocol. Next, all caregivers in the ED, especially triage nurses, should be familiar with the model. To make the model better applicable in clinical practice, we developed an online application, which will automatically calculate the chance of admission after measurement of the parameters. To further determine practical applicability and predictive power, a prospective study should be performed.
Our study has several limitations. A general limitation of this study is that the prediction model is developed based on a retrospective database. As a result, not all parameters that are considered in the model are completely accurate, which could have resulted in biased estimates of the effect of certain predictors. For example, it was not assessed whether oxygen therapy was administered before collecting data on both SpO2 and respiratory rate. Therefore, the predictive value of these variables is probably underestimated in the CLEARED tool. The need for oxygen therapy on its own could also be an independent predictive variable, however, a prospective study is needed to confirm this. In addition, respiratory rate was only recorded in just more than half of patient entries. An explanation for this could be that the respiratory rate was only measured when the patient was already in a more severe condition. The predictive value of respiratory rate as an independent variable should be evaluated more in future study.
Inevitably, there were many missing values. In order to deal with missing values and prevent losing data, missing values were categorised in a separate group. Using this method, the model can always be applied, even when not all parameters are recorded. In conclusion, the CLEARED tool can accurately predict admission in the elderly. It proved to exceed the performance of comparable tools. However, further research is needed to implement the tool and to evaluate the effect of the CLEARED tool on LOS and crowding in the ED, length of hospitalisation, and mortality. 


ACKNOWLEDGEMENTS AND DISCLOSURES

The authors would like to thank Marcel Eitink for making the data available at MST. All authors confirm they have no conflicts of interest. No funding or financial support was received.


PRESENTATIONS

Dutch Acute Medicine Conference, June 10th, 2016, Enschede (winner, presentation award).


REFERENCES

  1. Centers for Disease Control and Prevention. National Hospital Ambulatory Medical Care Survey: 2011 Emergency Department Summary Tables [Internet]. 2014 [cited September 9th, 2016]. Available from: http:// www.cdc.gov/nchs/data/ahcd/nhamcs_emergency/2011_ed_web_tables. pdf.
  2. Roberts DC, McKay MP, Shaffer A. Increasing rates of emergency department visits for elderly patients in the United States, 1993 to 2003. Ann Emerg Med. 2008;51(6):769-74.
  3. Downing A, Wilson R. Older people’s use of Accident and Emergency services. Age Ageing. 2005;34(1):24-30.
  4. van den Akker M, Buntinx F, Metsemakers JF, Roos S, Knottnerus JA. Multimorbidity in general practice: prevalence, incidence, and determinants of co-occurring chronic and recurrent diseases. J Clin Epidemiol. 1998;51(5):367-75.
  5. Hajjar ER, Cafiero AC, Hanlon JT. Polypharmacy in elderly patients. Am J Geriatr Pharmacother. 2007;5(4):345-51.
  6. Aminzadeh F, Dalziel WB. Older adults in the emergency department: a systematic review of patterns of use, adverse outcomes, and effectiveness of interventions. Ann Emerg Med. 2002;39(3):238-47.
  7. Casalino E, Wargon M, Peroziello A, et al. Predictive factors for longer length of stay in an emergency department: a prospective multicentre study evaluating the impact of age, patient’s clinical acuity and complexity, and care pathways. Emerg Med J. 2014;31(5):361-8.
  8. Kreindler SA, Cui Y, Metge CJ, Raynard M. Patient characteristics associated with longer emergency department stay: a rapid review. Emerg Med J. 2016;33(3):194-9.
  9. Latham LP, Ackroyd-Stolarz S. Emergency department utilization by older adults: a descriptive study. Can Geriatr J. 2014;17(4):118-25.
  10. Vanpee D, Swine C, Vandenbossche P, Gillet JB. Epidemiological profile of geriatric patients admitted to the emergency department of a university hospital localized in a rural area. Eur J Emerg Med. 2001;8(4):301-4.
  11. Limpawattana P, Phungoen P, Mitsungnern T, Laosuangkoon W, Tansangworn N. Atypical presentations of older adults at the emergency department and associated factors. Arch Gerontol Geriatr. 2016;62:97-102.
  12. Salvi F, Morichi V, Grilli A, Giorgi R, De Tommaso G, Dessi-Fulgheri P. The elderly in the emergency department: a critical review of problems and solutions. Intern Emerg Med. 2007;2(4):292-301.
  13. Rutschmann OT, Chevalley T, Zumwald C, Luthy C, Vermeulen B, Sarasin FP. Pitfalls in the emergency department triage of frail elderly patients without specific complaints. Swiss Med Wkly. 2005;135(9-10):145-50.
  14. Phillips S, Rond PC, 3rd, Kelly SM, Swartz PD. The failure of triage criteria to identify geriatric patients with trauma: results from the Florida Trauma Triage Study. J Trauma. 1996;40(2):278-83.
  15. Chang DC, Bass RR, Cornwell EE, Mackenzie EJ. Undertriage of elderly trauma patients to state-designated trauma centers. Arch Surg. 2008;143(8):776-81; discussion 82.
  16. Baum SA, Rubenstein LZ. Old people in the emergency room: age-related differences in emergency department use and care. J Am Geriatr Soc. 1987;35(5):398-404.
  17. Schrijver EJ, Toppinga Q, de Vries OJ, Kramer MH, Nanayakkara PW. An observational cohort study on geriatric patient profile in an emergency department in the Netherlands. Neth J Med. 2013;71(6):324-30.
  18. Liew D, Liew D, Kennedy MP. Emergency department length of stay independently predicts excess inpatient length of stay. Med J Aust. 2003;179(10):524-6.
  19. Sun BC, Hsia RY, Weiss RE, et al. Effect of emergency department crowding on outcomes of admitted patients. Ann Emerg Med. 2013;61(6):605-11 e6.
  20. Hwang U, Morrison RS. The geriatric emergency department. J Am Geriatr Soc. 2007;55(11):1873-6.
  21. Shankar KN, Bhatia BK, Schuur JD. Toward patient-centered care: a systematic review of older adults’ views of quality emergency care. Ann Emerg Med. 2014;63(5):529-50 e1.
  22. Barfod C, Lauritzen MM, Danker JK, et al. Abnormal vital signs are strong predictors for intensive care unit admission and in-hospital mortality in adults triaged in the emergency department - a prospective cohort study. Scand J Trauma Resusc Emerg Med. 2012;20:28.
  23. Considine J, Thomas S, Potter R. Predictors of critical care admission in emergency department patients triaged as low to moderate urgency. J Adv Nurs. 2009;65(4):818-27.
  24. Mackway-Jones K, Marsden J, Windle J. Emergency Triage. 3rd edition. London: John Wiley & Sons Ltd.; 2014. 3: The triage method; 11-24.
  25. Alexander RH, Proctor HJ American College of Surgeons.  Advanced trauma life support program for physicians: ATLS. 5th edition. Chicago, IL: American College of Surgeons; 1993. Committee on Trauma.
  26. Teasdale G, Jennett B. Assessment of coma and impaired consciousness. A practical scale. Lancet. 1974;2(7872):81-4.
  27. Stenhouse C, Coates S, Tivey M, Allsop P, Parker T. Prospective evaluation of a modified Early Warning Score to aid earlier detection of patients developing critical illness on a general surgical ward. British Journal of Anaesthesia. 2000;84(5):663.
  28. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. 1st edition. New York: Chapman and Hall; 1994. 9: Regression models; 105-21.
  29. Steyerberg EW. Clinical Prediction Models. 1st edition. New York: Springer Science + Business Media; 2009. 19: Patterns of external validity; 356-7.
  30. Steyerberg EW, Harrell FE, Jr., Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54(8):774-81.
  31. Miller ME, Langefeld CD, Tierney WM, Hui SL, McDonald CJ. Validation of probabilistic predictions. Med Decis Making. 1993;13(1):49-58.
  32. Dijkland SA, Roozenbeek B, Brouwer PA, et al. Prediction of 60-Day Case Fatality After Aneurysmal Subarachnoid Hemorrhage: External Validation of a Prediction Model. Crit Care Med. 2016;44(8):1523-9.
  33. RStudio Team. RStudio: Integrated Development for R [Internet]. 2015 [cited September 9th, 2016]. Available from: http://www.rstudio.com/.
  34. R Core Team. R: A language and environment for statistical computing [Internet]. 2015 [cited September 9th, 2016]. Available from: http:// www.R-project.org/.
  35. R Core Team. foreign: Read Data Stored by Minitab, S, SAS, SPSS, Stata, Systat, Weka, dBase, ... [Internet]. 2014 [cited September 9th, 2016]. Available from: http://CRAN.R-project.org/package = foreign.
  36. Harrell FE, Jr. rms: Regression Modeling Strategies 2014, R package version 4.2-1 [Internet]. 2014 [cited September 9th, 2016]. Available from: http://CRAN.R-project.org/package=rms.
  37. Barron EA, Holmes J. Delirium within the emergency care setting, occurrence and detection: a systematic review. Emerg Med J. 2013;30(4):263-8.
  38. Bo M, Bonetto M, Bottignole G, et al. Length of Stay in the Emergency Department and Occurrence of Delirium in Older Medical Patients. Journal of the American Geriatrics Society. 2016;64(5):1114-9.
  39. Asplin BR, Magid DJ, Rhodes KV, Solberg LI, Lurie N, Camargo CA. A conceptual model of emergency department crowding. Ann Emerg Med 2003;42(2):173-80.
  40. McCusker J, Bellavance F, Cardin S, Trepanier S, Verdon J, Ardman O. Detection of older people at increased risk of adverse health outcomes after an emergency visit: the ISAR screening tool. J Am Geriatr Soc. 1999;47(10):1229-37.
  41. Di Bari M, Salvi F, Roberts AT, et al. Prognostic stratification of elderly patients in the emergency department: a comparison between the “Identification of Seniors at Risk” and the “Silver Code”. J Gerontol A Biol Sci Med Sci. 2012;67(5):544-50.
  42. Salvi F, Morichi V, Lorenzetti B, et al. Risk stratification of older patients in the emergency department: comparison between the Identification of Seniors at Risk and Triage Risk Screening Tool. Rejuvenation Res. 2012;15(3):288-94.
  43. Meldon SW, Mion LC, Palmer RM, et al. A brief risk-stratification tool to predict repeat emergency department visits and hospitalizations in older patients discharged from the emergency department. Acad Emerg Med. 2003;10(3):224-32
  44. Baumann MR, Strout TD. Triage of geriatric patients in the emergency department: validity and survival with the Emergency Severity Index. Ann Emerg Med. 2007;49(2):234-40.
  45. LaMantia MA, Platts-Mills TF, Biese K, et al. Predicting hospital admission and returns to the emergency department for elderly patients. Acad Emerg Med. 2010;17(3):252-9.
  46. Grossmann FF, Zumbrunn T, Frauchiger A, Delport K, Bingisser R, Nickel CH. At risk of undertriage? Testing the performance and accuracy of the emergency severity index in older emergency department patients. Ann Emerg Med. 2012;60(3):317-25 e3.
  47. Yip WL, Fan KL, Lui CT, Leung LP, Ng F, Tsui KL. Utilization of the Accident & Emergency Departments by Chinese elderly in Hong Kong. World J Emerg Med. 2015;6(4):283-8.