KEYWORDS
ISAR-HP, cut-off points, older patients, hospitalisation, functional decline
INTRODUCTION
In the Netherlands, at the suggestion of both the Health Care Inspectorate (IGZ) and insurance companies, the Identification of Seniors At Risk – Hospitalised Patients (ISAR-HP) screening instrument is currently being promoted for use as a frailty indicator, for example in older patients with an indication for colon surgery.1 Comprehensive geriatric assessment (CGA) is subsequently advised for ‘frail’ patients in order to prevent functional decline. Identification of patients at high risk for functional decline is essential to ensure that interventions are targeted effective at those who will benefit most.2
The ISAR-HP is a recently developed screening instrument to predict 90-day functional decline in older patients who were acutely admitted to the department of internal medicine.3 Test characteristics were reasonable with respect to discrimination (area under the receiver operating curve, AUC), but the positive predictive value was rather low. Using the conventional cut-off score of ≥ 2 classified more than half of all older patients as being at risk for functional decline.3,4 However, classification was incorrect for 57% of the internal medicine patients in the development cohort and 64% of older patients undergoing cardiac surgery in a validation cohort, because no functional decline was experienced.3,4 As a consequence it is questionable whether using intensive interventions, such as the relatively time-consuming CGA, can be cost-effective.
The aim of the present study was to evaluate the performance of the ISAR-HP in predicting adverse health outcomes in acutely hospitalised older patients in two hospitals in the Netherlands. Predictive performance was tested by using different cut-off points of the ISAR-HP for predicting functional decline or mortality.
METHODS
Study design and setting
The Acutely Presenting Older Patients (APOP) study is a prospective multicentre cohort study in older patients visiting the emergency department. Data were collected in the emergency departments of the Leiden University Medical Center from September 2014 until November 2014 and the Alrijne Hospital Leiderdorp from March 2015 until May 2015. In both hospitals patients were included 7 days a week for a period of 12 weeks. The inclusion criterion of the APOP study was all patients aged 70 years and older who visited the emergency department. Exclusion criteria were red on the Manchester Triage System (i.e. patients requiring acute medical attention, such as cardiopulmonary resuscitation), an unstable medical condition, refusal to participate by the patient, an impaired mental condition of patients in the absence of a proxy to provide informed consent, and presence of a language barrier. For the current analyses, all acutely hospitalised patients of the APOP cohort with an ISAR-HP score at baseline were included. The ISAR-HP scores were calculated afterwards and not noted in the patient records, to ensure that all patients received usual care. Written informed consent was obtained from all patients. The Medical Ethics Committee of the Leiden University Medical Center and Alrijne Hospital approved the study. A more detailed description of the study design can be found in a previously published paper.5
Characteristics
Baseline characteristics included age, gender, living situation, level of education, clinical specialism, number of medications, history of dementia, Katz ADL score and cognitive impairment. Independent living situation represents patients living independently on their own or with others, high education was defined as higher vocational training or university, and number of medications represents the number of medications used at home as reported by the patient. Clinical specialism corresponds to the responsible specialism on the ward patients were admitted to. The cognitive status was assessed with the six-item Cognitive Impairment Test (6CIT);6 this score ranges from 0 to 28, with a score of 11 or higher indicating moderate to severe cognitive impairment.7 Functionality two weeks prior to admission was evaluated by means of the Katz ADL score, which contains six items: bathing, dressing, toileting, transferring, eating and the use of incontinence material.8 Each item is scored as independent (0 points) or dependent (1 point), with higher scores corresponding to more dependency.
ISAR-HP
The ISAR-HP is a scorecard with four yes/no questions on needing assistance on a regular basis, use of a walking device, needing assistance for travelling and having received education after the age of 14 years (Appendix, figure 1). 3 The score ranges from 0 to 5 with a score of 2 or more indicating a high risk of functional decline. The originally developed regression model of the ISAR-HP was: 1 / 1 + exp (– ( –1.93 + 0.48 × ‘pre-admission need for assistance in IADL on a regular base’ + 0.81 x ‘use of a walking device’ + 0.57 x ‘need of assistance in travelling’ + 0.42 × ‘no education after age 14’)).
Outcomes
Originally the ISAR-HP was developed for predicting solely functional decline,3 but at the moment of obtaining an ISAR-HP score it is impossible to distinguish patients who will not die within 90 days of follow-up from those who will. Therefore, in the present study the ISAR-HP was validated for predicting the composite outcome of functional decline or mortality within 90 days of follow-up after hospital admission. Information on functional dependency was assessed by telephone. Functional decline was defined as either an increase of at least 1 point on the Katz ADL score 90 days after hospitalisation compared with two weeks prior to admission or moving from an independent living situation to a dependent living situation. Dates of death were obtained from the Dutch municipality records.
Additionally, the ISAR-HP was validated for solely functional decline, for which we used the same exclusion criteria as the development study.3 Patients with a maximum Katz ADL score at baseline (fully dependent patients) and patients living in a nursing home at baseline were excluded, because these patients could not decline further as defined in our study. Also patients who were lost to follow-up or died within 90 days were excluded.
Statistical analysis
The baseline characteristics are presented as numbers with percentages or medians with interquartile ranges (IQR). A minimum of 100 events was considered necessary to provide sufficient statistical power for external validation.9 Predictive performance of the ISAR-HP was assessed by examining measures of discrimination and calibration.
Discrimination of the ISAR-HP score was quantified by calculating the AUC. The sensitivity, specificity, positive and negative predictive values (PPV and NPV) and positive and negative likelihood ratio were calculated for using the conventional cut-off of ≥ 2 points, but also using other thresholds of the ISAR-HP score (≥ 1, ≥ 3 and ≥ 4). Calibration of the internally validated ISAR-HP regression equation was assessed by plotting observed versus predicted probabilities, calculating calibration slope and with a goodness-of-fit test (Hosmer and Lemeshow test10). Data were analysed using IBM SPSS Statistics version 23 (IBM Corp, Armonk, NY) and R Statistics version 3.3.0.11.
RESULTS
In the APOP study, 1965 consecutive older patients visiting the emergency department of the Leiden University Medical Center or Alrijne Hospital were eligible for participation. In total 1632 patients (83.1%) were included after informed consent, of whom 771 (42.2%) were subsequently hospitalised. After exclusion of three missing and three incomplete ISAR-HP scores, the study population for the present analyses contains 765 patients. Table 1 shows the baseline characteristics of the study population. The median age was 79 years (IQR 74-84), 374 patients (48.9%) were male and 698 patients (91.2%) were living independently either on their own or with others. Most patients were admitted for the clinical specialism internal medicine (242 patients, 36.1%), cardiology (168 patients, 22.0%) or surgery/orthopaedics (154 patients, 20.1%). The median Katz ADL score was 0 (IQR 0-2) and 172 patients (25.1%) had cognitive impairment.
Figure 1 displays the distribution of ISAR-HP scores. The median ISAR-HP score was 2 (IQR 0-4) and 432 patients (56.5%) were at high risk for functional decline or mortality when using the conventional cut-off of ≥ 2. In total 276 patients (36.1%) experienced functional decline or mortality within 90 days of follow-up. We first performed an external validation of the ISAR-HP. Figure 2 shows the calibration plot of the ISAR-HP for functional decline or mortality. Calibration was insufficient with a Hosmer and Lemeshow goodness-of-fit p-value of 0.007. Predicted probabilities were lower than the observed probabilities with a calibration slope of 0.877 and an intercept of 0.246, indicating an underestimation of the outcome.
The predictive performance of the ISAR-HP is shown in table 2. Accuracy of the ISAR-HP was reasonable, with an AUC of 0.69 (95%CI 0.65-0.73). External validation of the ISAR-HP for the conventional cut-off of ≥ 2 resulted in a sensitivity of 0.77 (95% CI 0.72-0.82), a specificity of 0.55 (95 %CI 0.51-0.60), a PPV of 0.49 (95% CI 0.45-0.54) which results in 45% incorrectly classified high-risk patients and 29% incorrectly classified low-risk patients. Additionally, although of limited clinical applicability, we validated the ISAR-HP for solely functional decline to allow comparison of performance compared with the original study (Appendix: table 1 and figure 2). After applying the exclusion criteria for functional decline (113 patients died, 30 patients were lost to follow-up, 17 patients were unable to demonstrate functional decline and 13 patients refused), 592 patients were included of whom 162 (27.4%) experienced functional decline. Discrimination was fair (AUC 0.72, 95% CI 0.67-0.76) and calibration satisfactory (Hosmer and Lemeshow goodness-of-fit p-value 0.068). The PPV and NPV were lower for all cut-offs, with a PPV of 0.42 (95%CI 0.37-0.48) and NPV of 0.89 (95%CI 0.84-0.92) for an ISAR-HP score of ≥ 2. 
DISCUSSION
The main finding of the present study is that the ISAR-HP with the conventional cut-off of ≥ 2 resulted in more than half of all acutely admitted patients to be considered at high risk for 90-day functional decline or mortality; of these patients 51% did not experience this outcome. Raising the ISAR-HP cut-off to ≥ 4 resulted in a quarter of all patients being classified at high risk and the predictive performance increased marginally.
The ISAR-HP was originally developed to predict 90-day functional decline in patients aged 65 years and older, who were acutely hospitalised for at least 48 hours on a general internal medicine ward.3 Age and gender were comparable, but compared with our cohort more patients were living dependently (24% in development cohort vs. 9% in APOP cohort) and more patients were cognitively impaired (34% in development cohort vs. 25% in APOP cohort). In the original development study 70% of all older patients were identified as high risk with a PPV of 0.43, implying that 57% of these high-risk patients did not suffer from functional decline. In the APOP cohort 57% were assigned as high risk with a PPV of 0.49, which means 51% showed no decline. The ISAR-HP was also validated for functional decline in cardiac surgery patients aged 65 years and older.4 Compared with the APOP cohort the patients were younger (mean age 73 years), more often male (64%) and had a better cognitive performance (14% with memory problems). In total 16% of the cardiac surgery patients suffered a functional decline and the AUC of the ISAR-HP was 0.72. The reported PPV of an ISAR-HP score of ≥ 2 was 0.36, which implies that approximately two out of three patients were incorrectly considered high risk. Results from a meta-analysis of screening instruments to predict functional decline and mortality in older patients visiting the emergency department were in line with the results of the ISAR-HP.12 High-risk groups could not accurately be distinguished from low-risk groups and a relatively high number of patients were incorrectly classified as high risk. In the Netherlands older hospitalised patients are systematically screened for undernutrition, ADL limitations, falls and delirium to prevent 90-day functional decline or mortality: the Safety Management System (VMS+) screening.13 A third of all VMS+ screened older patients were considered to be at high risk and predictive performance was comparable with our results, with a PPV ranging between 0.50 and 0.57. In summary, the results of our study were in line with ISAR-HP in older patients on both a general internal medicine ward and undergoing cardiac surgery and in line with the performance of the VMS+ screening.
The ISAR-HP was developed and validated to predict only functional decline. In order to replicate that analysis, we had to exclude almost a quarter of patients due to the exclusion criteria, including those who had died during follow-up. As a consequence, this selection no longer reflects clinical practice, where screening is implemented for all patients. A composite outcome of functional decline with mortality as ultimate decline in physical functioning was therefore used. In the present study the predictive performance of different cut-off points for predicting the composite outcome was studied in order to improve efficiency of ISAR-HP screening. Increasing the cut-off point to ≥ 4 resulted in selection of 26.8% of the patients at highest risk and the PPV improved to 55%. Although one in two patients would be inappropriately assigned to the ‘high-risk’ group, less patients are considered at high risk. To date, the ISAR-HP has been used twice in study settings as a screening instrument for CGA interventions to specifically prevent functional decline in older hospitalised patients. In the Prevention and Reactivation Care Program (PReCaP) older patients at risk of functional decline received supplementary multidisciplinary, goal-oriented care.14 In the Transitional Care Bridge Randomised Controlled Trial older patients at risk of functional decline received a systematic CGA, followed by a hospital visit of the community care registered nurse and subsequent multiple home visits after discharge.15 Although the interventions should be appropriate to prevent functional decline, both the PReCap and the Transitional Care Bridge Randomised Controlled Trial showed no effect on ADL functioning. Based on the findings of our study this may be explained by the fact that many patients were incorrectly selected for the intervention. Taken together, the predictive performance of the ISAR-HP is characterised by low PPVs. Using the ISAR-HP for identifying patients at high risk for functional decline could result in providing inefficient follow-up care.
In screening instruments there is a certain clinical threshold which is determined by the relative weight of false-negative versus false-positive errors. A conservative low cut-off is useful if missing a patient who will undergo a functional decline is more important than incorrect classification of a patient who will not.16 From the patient perspective a low cut-off is desirable, especially if it results in an intervention without side effects. However, using a stricter cut-off is more useful from another perspective. As an example, in 2012 in total 734,000 patients aged 65 years and over were admitted to hospital in the Netherlands (25% of all older adults)17 of whom 415,000 might have been at high risk according to our results with the conventional ISAR-HP cut-off of ≥ 2. If the recommended CGA had been performed in all patients, 212,000 CGAs would be carried out unnecessarily in order to prevent the occurrence of the outcome. Using a stricter ISAR-HP cut-off ≥ 4 would result in 197,000 high-risk patients, with 88,000 incorrectly classified as high risk. In patient groups with a lower incidence of functional decline or mortality, such as older cardiac surgery patients, even more interventions will be performed unnecessarily.4
Taking into account the limited efficacy and capacity to perform CGAs we therefore recommend to target interventions in a larger group of patients which are inexpensive, less time consuming and not a burden for the patient. A more specific screening instrument for hospitalised older patients is needed to be able to target resource intensive interventions in a smaller group of patients. In our publication on the development and validation of a new screening instrument for older patients visiting the emergency department, we were able to increase specificity compared with the widely accepted ISAR screening instrument.5 During hospital stay more patient data will become available, such as vital parameters and laboratory results, which may improve predictive performance. Therefore, we are currently developing a new dynamic predicting model for hospitalised older patients.
Several limitations need to be addressed. First, the performance of the ISAR-HP in non-acutely hospitalised older patients, such as elective admissions and via the outpatient clinic, has not been evaluated. Second, the follow-up data on functional decline were incomplete for 43 patients. From municipal records we know that these patients were alive, which might therefore have resulted in an underestimation of functional decline. The major strength of the study is the inclusion of a representative cohort of older patients. In total 83% of the eligible older patients from different specialisms were included, from both an academic and regional hospital. A second strength is that ISAR-HP was evaluated for the composite outcomes, which reflects predictive performance in clinical practice.
In conclusion, the ISAR-HP with the conventional cut-off of ≥2 incorrectly identifies a large group of patients as being at high risk for functional decline or mortality, and raising the cut-off to 4 only marginally improved performance. Caution is warranted to ensure efficient screening and follow-up interventions.
DISCLOSURES
The authors declare no conflict of interest. The Institute for Evidence-Based Medicine in Old Age (IEMO) is supported by the Dutch Ministry of Health, Welfare and Sport and supported by the Netherlands Organisation for Health Research and Development (ZonMW project number 62700.3002).
REFERENCES
