KEYWORDS
Patient safety, quality improvement, readmissions
INTRODUCTION
Hospital readmissions within 30 days are of interest to many policy makers internationally.1 They are used as a quality and safety indicator with financial penalties levied in many countries including the United States and United Kingdom.2 The main problem in using readmissions as a quality indicator is that the preventability of these readmissions is not properly defined and integrated in this indicator making it difficult to use as a genuine measure of quality of care.3,4 By not distinguishing between preventable and non-preventable readmissions this indicator might therefore result in distorted evaluation of hospital care. Furthermore, there is increasing evidence that the causes of mostly medical readmissions are often multifactorial and usually the result of natural disease progression, underlying comorbidity or socio-environmental factors beyond the control of the hospital and not solely caused by inadequate hospital care.5-9 The use of readmissions as a quality indicator necessitates that they reflect poor care, are preventable and that a consensus definition for these two aspects is agreed. Previous research has not yet been able to determine uniform factors related to preventable readmissions.10 To date, consensus definition of preventability has not been established. Many studies use the opinion of physicians as the gold standard to determine if readmissions are preventable, and derive factors that would predict preventable readmissions from these findings.11-13 However, to our knowledge, no study has yet been performed to examine the interobserver reliability of the physicians’ judgement on preventability.
Therefore, we performed an international study to assess if there is any consensus between physicians regarding the predictability and preventability of medical readmissions.
MATERIAL AND METHODS
This study is an initiative of the safer@home consortium, an international group founded in 2013 consisting of 13 acute medical physicians, emergency physicians and epidemiologists from Europe that focus on readmissions and safer discharge processes.
During the 3-month study period (1 September to 1 December 2015) a survey on eight cases based on common clinical scenarios (see appendix) was distributed to physicians throughout Europe.
Survey
The survey consisted of eight case-based medical scenarios (table 1 shows a summary of the case vignettes). The scenarios were generated using a Delphi-type methodology, whereby multiple scenarios were generated and then represented to the safer@home consortium in two rounds. In the first round underlying assumptions and information leading to different judgements was explored using current readmission literature. This round took place in a face-to-face half-yearly consortium meeting with all 13 members. After this, seven clinically active medical physicians in the group were asked to provide examples from their daily work in order to compose cases. In the second round, these cases were discussed in a conference call during which the cases that would be representable for all countries were selected through consensus. In addition we assessed if, in our ‘expert’ opinion, the cases could potentially be used to fulfil the purpose of our research question. Subsequently, a pilot was performed on a small group of physicians from all countries to ensure cases were understandable and varied sufficiently. Final case selection ensured that: a) the cases would be representative of patients requiring unselected medical admission in northern Europe; b) the scenarios covered the range of factors suggested by the literature to impact on readmissions; c) cases were not traceable to real-life patients.
The online survey consisted of two parts: 1) Physicians were asked about their opinion on predictability of medical readmissions; from four cases describing an index admission, physicians were asked to rate the chance of readmission within 30 days. 2) Physicians were asked to assess the preventability of four described medical readmission cases.
From the physicians’ assessment of predictability and preventability, the degree of consensus could be derived. For both parts of the survey a five-point Likert Scale was used as an answering model (part 1: Definitely not predictable (1) – Definitely predictable (5); part 2: Definitely not preventable (1) - Definitely preventable (5)).14
Data on the country and primary specialty of the responding physician filling out the survey and the number of years of clinical experience were collected in order to explore agreement within these subgroups. The survey was anonymised to ensure the researchers could not trace which physician filled out which survey. Finally, general comments concerning readmission could be made after completing the survey.
Distribution
The survey was distributed among physicians throughout Europe; they all worked solely in a medical specialty and not in any surgical specialty. Invitations were sent to the members of the Society for Acute Medicine in the UK, the Dutch Acute Medicine Society, the Danish Society for Emergency Medicine, physicians from Switzerland and Ireland using a common web-based platform SurveyMonkey®. In order to calculate an accurate response rate, each physician communicated the number of requests sent to one research member (LG), who was responsible for data processing and statistical analysis. The ethics committee of the VU University Medical Center, Amsterdam approved the study. No funding was received for this study.
Statistics
Descriptive characteristics and frequencies were calculated in SPSS version 22.0. Ratings of physicians are presented as frequencies and percentages. Using the intraclass correlation (ICC, a reliability coefficient) we assessed agreement among physicians regarding the predictability and subsequently, the preventability of the assessed medical readmissions. This coefficient (ICC) is used to assess the agreement of ratings made by multiple observers (in our study ‘physicians’) measuring the same outcome (in our study ‘the predictability and preventability of readmissions both based on four real-life readmission scenarios’). The ICC is a ratio ranging in value between 0 (representing no agreement) and 1 (implying agreement).
Calculating the variance components we constructed the ICC formulas from which the ICC could be calculated. For dependent variables we used the outcome ‘Likert scores’ and for random factors ‘physicians’ and ‘case numbers (1.1, 1.2, 1.3, 1.4 and 2.1, 2.2, 2.3, 2.4)’ were used. The variance among cases (case numbers 1.1, 1.2, 1.3, 1.4 and 2.1, 2.2, 2.3, 2.4) was analysed separately, among physicians, and the random error were calculated in SPSS using the VARCOMP procedure. From the variance components we calculated the ICC for absolute agreement as the variance among cases divided by the total variance of the cases, physicians and random error.15
RESULTS
Physician characteristics
During the three-month study period (1 September to 1 December 2015) the survey was distributed to physicians in Europe. In total 526 medical physicians filled out the survey. The overall response rate was 24.2%. Seventy-seven (14.6%) physicians did not complete all the questions in the survey. Table 2 shows physician characteristics. Dutch physicians were the largest group of respondents (46.2%), followed by Danish (25.1%) and physicians from the United Kingdom (23.6%). Most physicians had internal medicine (33.3%) as their primary specialty followed by acute medicine (24.5%) and geriatrics (12.5%). The median years of clinical experience was 10.75 (interquartile range: 5-20).
Agreement on predictability of readmission
For the first part of the survey physicians were asked if they could predict a readmission based on the four case descriptions of medical index admissions. Responses are shown in figure 1. The results show that there was substantial variation in the degree of predictability between the physicians’ judgements in all four cases. The cases were assessed with different degrees of predictability. To illustrate, for case number 1.2, about half of the physicians assessed the likelihood of readmission as ‘definitely not’ (score 1), while in case number 2.1 over 60% of the physicians predicted that the patient will definitely be readmitted (score 5).
The ICC for agreement of predictability was 0.67 (Var(Casenumber) 1,444, Var(Observer) 0.054, Var(error) 0.649) which indicates a moderate to strong interobserver agreement between the raters (physicians). These findings suggest that the surveyed doctors had a moderate to good degree of agreement about the patients that were prone to come back, they predicted the same patients as having a higher chance of a readmission occurring.
Agreement on preventability of readmission
In the second part of the survey the respondents were asked to rate the preventability of four medical readmission cases. The results in figure 2 show the distribution of answers by the physicians. It shows that the physicians rated the cases differently; there was a wide variety in assessment. In all four cases no clear majority seemed to rate the same readmissions with similar scores.
These findings were also reflected in the ICC for this part of the survey.
The ICC was calculated at 0.13 (Var(Casenumber) 0,194, Var(Observer) 0.168, Var(error) 1.076), which implies poor agreement. Doctors do not seem to agree on the preventability of readmissions. However, one must note that the variance among case numbers was relatively low which may indicate that the cases assessed were not sufficiently distinct enough to obtain a high reliability coefficient.
Subgroup analysis
To assess if there was any difference in agreement between subgroups of physicians we subdivided the doctors into years of clinical experience. They were grouped based on clinical experience up to 5 years (n = 151, 28.7%), from 5-15 years (n = 208, 39,.5%), and 15 years and higher (n = 167, 31.7%).
Results suggest that medical physicians with less clinical experience had a trend towards greater agreement than those with more clinical experience as to the likelihood of readmission but these differences are minimal ((ICC 0.70, 0.69, 0.63, respectively). Physicians with more clinical experience seemed to have more agreement about the preventability of a readmission compared with those with less clinical experience (ICC 0.08, 0.01, 0.19, respectively).
DISCUSSION
In this survey among 526 European physicians there was moderate agreement as to the predictability of medical readmissions but poor agreement about their preventability. These results suggest that doctors agree on the patients who have a higher risk of being readmitted, but the physicians differ on how preventable these readmissions are. To our knowledge, the current study is the first to investigate interobserver reliability on the evaluation of unplanned readmissions in such a large group of observers.
Unplanned readmissions are a complex phenomenon, which are influenced not only by medical factors but also by a range of social and political issues.5,16,17 Readmission risk is difficult to define and is less predictable than mortality.18 Nevertheless, there are a number of risk factors which are recognised as increasing the risk of readmission and multiple predictive scoring systems based on these factors have been designed.5,6,19
Although the risk factors for predominantly medical readmission are increasingly well recognised, the dynamic of how they interact and whether they can be influenced remains controversial.20 The poor consensus among physicians found in our study as to whether the readmissions were preventable underlines this issue. A US study of 17 hospitalists reviewing 300 consecutive readmissions also found a wide variation in their scoring of preventability; however, comparability might be limited since these were real-life readmissions.21 We can concur their findings of interobserver variability in a European setting.
The above findings illustrate the problem faced by policy makers trying to integrate preventability in the readmission indicator since doctors, who are supposed to be experts in the field, cannot even agree on the readmissions that are potentially preventable. Current literature, however, often uses the opinion of one or more physicians as the gold standard to gain insight into preventability and draw conclusions on factors predicting preventability. The results in this study, however, demonstrate that the assumptions derived from these studies might lead to misperception since physicians do not share similar ideas on the potential preventability of readmissions.22-24 Hence, it can be questioned whether conclusions drawn from these studies might not provide reliable conclusions to create an appropriate quality indicator.
Readmitted medical patients are a heterogeneous group; there is a wide variation in the age, comorbidities and social support of these patients. It remains unclear as to whether the factors which drive unplanned readmission, including medical, social, cultural and environmental, are modifiable.20 This is reflected by an increasing body of evidence that suggests readmissions do not always reflect poor care and preventability of these readmissions is poorly defined.3,4,21,25 More research studying ‘the preventability’ in a structured manner might help to improve the difficult task in creating a reliable indicator.
We used adapted real-life case scenarios in our study, which may be a limitation. This was also reflected in the comments section, where physicians mentioned they were missing information that would allow them to thoroughly assess the case, for example more details on the patients’ social situation. It would, however, be difficult to incorporate all the potentially relevant social and environmental factors into scenarios particularly in a pan-European study where there is a wide variety of political and health policies that influence readmissions. Furthermore, in calculating the ICC for the preventability part of the survey one could suggest that there was little variation in the preventability of the cases. This may reflect either that there was insufficient variation with regards to preventability within the scenarios, potentially caused by balancing between uniformity in the cases in a way they could be representable for all countries participating in the study and enough variation in the cases in order to create different opinions per case. It may also reflect an uncertainty among physicians regarding what comprises a preventable admission.
On a final note, our respondents were of high seniority with a median of 11 years of clinical experience. If clinicians with this level of experience cannot agree on the predictability of readmission, is it wise to use it as a marker of quality of care?
CONCLUSION
This study demonstrates that there is moderate agreement among experienced medical physicians about the predictability of readmissions but poor agreement about their preventability. Therefore, the conclusions derived from earlier studies on preventability, on the basis of physician consensus as the gold standard, are questionable. Hence, a good way of defining and integrating preventability into this quality indicator remains elusive.
ACKNOWLEDGEMENTS
A full list of membership of the safer@home Consortium is as follows: M. Braband, T. Cooksley, L. van Galen, H. Haak, R. Kidney, J. Kellet, H. Merten, P. Nanayakkara, C. Nickel, J. Soong, C. Subbe, L. Vaughan, I. Weichert.
DISCLOSURES
The authors have nothing to disclose.
REFERENCES
APPENDIX
Physician
Years of clinical experience:
Primary specialty:
Country:
Rationale and aims of this research project
Hospital readmissions within 30 days are highly prevalent and costly.1,2 Readmission rates are already commonly used as a quantity and safety measure to rate and reimburse units across hospitals internationally.3,4 As of 2016, the Dutch National Health Care Program readmission rates are also an official new indicator to assess quality in hospitals. The problem in using readmission as such a quality indicator is that this approach does not focus on readmissions that are preventable. This is noteworthy since it seems to be logical that penalties should only be attached to those readmissions that could have been prevented.
Current literature, however, has not been able to find reliable percentages of readmissions deemed preventable. More importantly, the definition of preventability has not yet been defined uniformly.5 The above mentioned leads to difficulties faced by the health inspection services globally in finding an integral way to get accurate ‘preventable’ data from electronic databases. Previous research performed on preventable readmissions is often based on some physicians randomly determining if readmissions are preventable.6
However, no study has ever been performed to look into the interobserver reliability of these observations.
Therefore the main aim of this study is to assess if there is any consensus between physicians internationally regarding the occurrence and preventability of readmissions. This study is effectuated by a survey based on real-life cases and spread over clinicians internationally. It is the initiative of the international group that focuses on readmission (‘safer@homeconsortium’).
References
Part I
Please assess on a scale from 1-5 if you think the following four admissions are followed by a readmission within 30 days:
Definitely not (1) - Definitely (5) (choose 1,2,3,4,5)
1. An 83-year-old female is admitted to the acute medical unit with a collapse secondary to atrial fibrillation (160 bpm) due to a urinary tract infection – she did not have any other complaints before collapsing. Blood counts show moderately elevated CRP and leucocytes, urine testing shows leukocytes and nitrite. Her medical history includes hypertension and severe epistaxis. She is a widow and lives alone with some additional help from her daughter when needed. She takes lisinopril 5 mg, but she does not use any blood thinners or other medication. Because of her epistaxis, treating doctors decide not to start oral anticoagulation or rhythm control but only aspirin as prophylaxis for her heart rhythm disorders. During admission she receives antibiotic treatment, and her heart rate returns to sinus rhythm without extra intervention. At discharge she has no complaints.
2. A 20-year-female presents with acute onset of occipital headache suggestive of a subarachnoid haemorrhage. She is investigated with a CT brain, which is normal and subsequently undergoes a lumbar puncture, which is also entirely normal. She is treated with IV fluids, paracetamol and NSAIDs. Her headache settles and following investigations she is discharged.
3. A 60-year-old lady with severe COPD on home oxygen and nebulisers is admitted with increasing dyspnoea and a productive cough. She is diagnosed as having an infective exacerbation of COPD and is treated with steroids and antibiotics alongside her usual nebulisers, inhalers and oxygen. She makes steady progress, her sputum culture is negative and she is discharged three days later.
4. A 94-year-old woman was initially admitted with pneumonia. She was treated with ceftriaxone 2g once a day, azithromycin 500 mg once a day and discharged on doxycycline 100 mg once a day. According to the patient file, the patient was clinically doing better. However, the night before discharge the patient experienced chest pain, dyspnoea and vomiting. Last measured CRP and leucocytes were 60 (< 8 mg/l) and 22.4(4-10 x 109/l) 2 days before discharge. 
Part II
Please assess on a scale from 1 to 5 if you think the following four readmissions within 30 days are:
Definitely not preventable (1) - Definitely preventable (5) (choose 1,2,3,4,5)
1. A 63-year-old lady with locally advanced pancreatic cancer presents with an episode of fever 12 hours after receiving her dose of gemcitabine. She has no other systemic symptoms and feels well in herself. Investigations are unremarkable. She is diagnosed with gemcitabine-induced fever and is discharged. Ten days later she presents with fever and diarrhoea. She is diagnosed with neutropenic sepsis.
2. A 40-year-old male with alcohol dependence presents with ascites. He consumes one bottle of whisky a day. At presentation he has Childs B cirrhosis. His undergoes abdominal paracentesis. He is commenced on diuretics, vitamin supplements and lactulose. He is offered support to reduce his alcohol intake but declines this. Three weeks later he is readmitted with haematemesis and melaena.
3. A 55-year-old male presents with an anterior STEMI. He is transferred straight to the cardiac catheter theatre and undergoes angioplasty to his LAD. He makes a good recovery. He is started on standard and appropriate secondary prevention treatment. His echo shows moderate LV dysfunction. Follow-up in the cardiac rehabilitation clinic is arranged. Three weeks later he is readmitted with dyspnoea. Clinical and radiological findings are consistent with pulmonary oedema.
4. A 32-year-old woman was admitted to the internal medicine ward under the suspicion of pyelonephritis. Her medical history mentioned an atonic bladder and hydronephrosis with ureter stenosis causing recurring pyelonephritis. The pyelonephritis was treated with iv antibiotics according to urine culture and antibiogram. Because the patient wanted to be discharged so badly and was fairly mobile it was decided to administer the iv antibiotics daily at the outpatient clinic according to the treatment plan as discussed with the nephrologist and the microbiologist. A little less than a month later the patient presented again with pyelonephritis. In between admissions the patient had been free of any UTI symptoms. Patient was treated in the same way as during the index admission and was once again discharged with a course of daily iv antibiotics at the outpatient clinic. 
Thank you very much for filling out this survey!
This survey was performed as a research project from the safer@home consortium. We are an international group that focuses on readmissions. In January 2016 we will start the first prospective multicentre study on readmission:
The CURIOS@ study (CaptUring Readmission InternatiOnally to prevent Readmission by Safer@home consortium)
In this study the main aims are: 1) to inventarise (preventable) risk factors for readmission, and 2) to verify the opinions of patients, their informal carers, nurses and physicians about the preventability of their readmissions.
We are still looking for centres to join! Your participation is much appreciated. Are you or do you know anyone that is interested? Please contact us at saferathomeconsortium@ gmail.com or at lo.vangalen@vumc.nl.
On behalf of the safer@home consortium:
Mikkel Brabrand, Denmark
Tim Cooksley, United Kingdom
Kristien Fluitman, the Netherlands
Louise van Galen, the Netherlands
John Kellet, Ireland Rachel Kidney, Ireland
Hanneke Merten, the Netherlands
Prabath Nanayakkara, the Netherlands
Christian Nickel, Switzerland
John Soong, United Kingdom
Christian Subbe, Wales
Louella Vaughan, United Kingdom
Immo Weichert, United Kingdom