KEYWORDS
FilmArray, PCR, rapid test, respiratory virus, validation
INTRODUCTION
Respiratory viruses predominate as causative pathogens in patients hospitalized with severe acute respiratory illness (SARI), accounting for up to 50% of microbial etiologies.1,2 Although antibiotics can be safely withheld in proven viral infections, viral SARI is often treated with antibiotics because viral and bacterial lower respiratory tract infections (RTI) cannot reliably be distinguished based on clinical presentation.1,3 This leads to unnecessary costs and higher risk of antimicrobial resistance.4-7 Furthermore, studies have shown reduced mortality when oseltamivir was administered as early as possible (e.g. within two days of symptom onset).8 Rapid syndromic sample-to-result (SS2R) diagnostics based on multiplex polymerase chain reaction (mPCR), generally generating results within two hours, are promising in accelerating virus and bacterial identification2 and consequently in targeting antibiotic and antiviral therapy.9 In addition, rapidly ruling out a viral cause of RTI could lead to more efficient use of in-hospital isolation facilities during the viral respiratory season. Currently, a wide range of molecular rapid tests is available. Promising commercially available SS2R techniques are Alere I® Influenza A&B Nucleic Acid Amplification Test (Abbott), Cobas® Liat® (Roche Diagnostics), eSensor® Respiratory Viral Panel (Genmark), FilmArray® Respiratory Panel (BioFire Diagnostics), GeneXpert® (Cepheid), and the Luminex® xTAG Respiratory Viral Panel (Luminex).2 This paper describes a prospective clinical study evaluating the accuracy, turnaround time (TAT) and logistical barriers for the implementation of one of these SS2R diagnostics, the FilmArray® Respiratory Panel (from now on SS2R), a FDA cleared /CE marked rapid mPCR for a panel of 17 viruses plus three bacteria commonly found in RTI. The choice for this specific test reflects the choice for this SS2R in our hospital. Apart from the analysis of accuracy and implementation issues, the applicability of the SS2R is evaluated by calculating the potential impact on use of in-hospital isolation facilities and oseltamivir.
MATERIALS AND METHODS
Patient inclusion and microbiological
testing During the peak of the 2016-2017 viral respiratory season (3 January – 4 February 2017),10 nasopharyngeal swabs were taken from patients (age ≥ 18) with symptoms of a RTI presenting on the Emergency Department (ED) of the University Medical Center Utrecht (UMCU), a 1042 bed tertiary hospital (The Netherlands). Symptoms of RTI were defined as upper or lower respiratory complaints with acute onset. To obtain informed consent from each patient and calculate TATs of the SS2R, patients were only included during lab opening hours (8 am - 5 pm). Two swabs were taken in parallel, one for the regular diagnostic pathway and one for the experimental diagnostic pathway with the SS2R. The first swab was tested for the most common respiratory viruses (table 2) using conventional real-time polymerase chain reaction (RT-PCR). Conventional RT-PCR for respiratory viruses was performed in nasopharyngeal swabs, nasal washes or bronchoalveolar lavage. Nucleic acids were extracted using the total nucleic acid protocol with the MagNA Pure LC nucleic acid isolation system (Roche Diagnostics, Basel, Switzerland). For detection of RNA viruses, cDNA was synthesized using MultiScribe RT and random hexamers (Applied Biosystems, Foster City, CA). Detection of viral pathogens was performed in parallel, using laboratory developed RT-PCR assays specific for the following viruses: respiratory syncytial virus A and B; influenza virus A and B; parainfluenza virus 1-4; rhinoviruses; adenoviruses; human coronaviruses OC43, NL63, and 229E; human metapneumovirus. RT-PCR procedures were performed as described in earlier literature.11 In brief, samples were assayed in duplicate in a 25-μL reaction mixture containing 10 μL of cDNA, 12.5 μL of TaqMan Universal PCR Master Mix (Applied Biosystems), 300-900 nmol/l of the forward and reverse primers, and 75-200 nmol/l of each probe. To monitor for inhibition, a fixed amount of an internal control virus (murine encephalomyocarditis virus [RNA virus] and porcine herpesvirus [DNA virus]) was added before extraction.12 The cycle of threshold (Ct) gives an impression of the quantity of the viral load (i.e. a semi quantitative value). The cut-off value for a positive RT-PCR result was a Ct value < 45.13 The second nasopharyngeal swab was tested with the SS2R (FilmArray® Respiratory Panel version 1.7). The FilmArray® contains all needed reagents in a freeze-dried format for extraction, amplification, and detection steps. Respiratory samples are collected in universal transport media. The FilmArray® test was performed according to the manufacturer’s instructions. In brief, prior to run 1 ml of hydration solution and 300 μl of respiratory sample was added to the reagent pouch. The pouch was then placed on the FilmArray® instrument and the test performed using the FilmArray® system. After extraction and purification of all nucleic acids from the sample, a nested multiplex PCR is performed followed by an individual singleplex second-stage PCR reactions to detect the products from the first-stage nested PCR. Both the in-house PCR and FilmArray® were performed by the virology laboratory and results were approved by a clinical virologist. Results of the SS2R were not used in routine care.
Data collection and statistical analysis
Standardized collection of clinical and virological data from ED-presentation and, if applicable, from the following hospital admission was performed manually from the electronic patient charts. Results are given as percentages or median with IQR. The accuracy analysis of the SS2R compared to RT-PCR and a comparison of Ct-values with a t-test, were performed with IBM SPSS Statistics® (Version 21.0). Accuracy was calculated per detected virus, e.g. influenza virus, respiratory syncytial virus (RSV), coronavirus, rhinovirus and human metapneumovirus (HMPV), and per sample. Test concordance of the SS2R compared to RT-PCR was presented using sensitivity, specificity and positive and negative predictive values for clinical purposes. Since RT-PCR alone might not be considered a ‘gold’ standard accuracy (or overall percentage agreement) and a Cohen’s Kappa statistic were calculated as well. For the accuracy analysis, final results of both diagnostics were used after retesting in case of invalid results. Samples with discrepant results were retested with both diagnostics, no additional sequencing was done. Ct-values of positive samples were measured by RT-PCR. TATs of both diagnostics were calculated for patients of whom the SS2R could be obtained on the day of sampling, in hours from sampling at the ED until the result was reported to the study team. During the clinical study only one SS2R system was available, precluding the possibility of parallel testing. Separately, potential barriers for implementation, advantages and disadvantages of the SS2R were explored by interviewing laboratory technicians working with the SS2R. Answers to the questions were presented descriptively.
RESULTS
From January 3rd till February 4th, a (differential) diagnosis of upper or lower RTI was made in 148 adults at the ED, in 104 of whom RT-PCR was performed. Of these, 56 presented during lab opening hours, making a SS2R possible. Additionally, nasopharyngeal swabs for both RT-PCR and SS2R were taken from six patients after leaving the ED, who had been referred there with respiratory complaints, but with a non-RTI working diagnosis, so that eventually 62 patients were included. Median age was 64.5 years (IQR 44.3-75.0). Twenty-five patients (40%) were immunocompromised at the time of presentation. Thirty-nine (63%) were admitted to the hospital, nine of whom to the Intensive Care Unit (table 1). Results of the RT-PCR and SS2R showed 58% and 60% samples with one virus detected and 8% and 2% with dual or triple viral pathogens respectively. Most frequented detected viruses were influenza A virus, coronavirus, rhinovirus and RSV (table 2). Viral-bacterial coinfection was present in 14 patients (Streptococcus pneumoniae (n = 4), Haemophilus influenza (n = 3), Pseudomonas aeruginosa (n = 4), Staphylococcus aureus (n = 2), Proteus mirabilis (n = 1)). Five patients (8%) died, in all of whom a virus was detected (influenza A virus (n = 2), RSV (n = 2), coronavirus (n = 1)).
Diagnostic accuracy
Sixty-two patients were included in the diagnostic accuracy analysis (table 3). Compared to the reference method, SS2R had a sensitivity of 82.9% [95% CI 67.9-92.9] and specificity of 95.2% [95% CI 76.2-99.9] for complete virus detection. In two samples the SS2R initially gave an invalid result (one sample negative, one with rhinovirus in the RT-PCR). After retesting with the SS2R, the results were similar to the reference standard. Discrepant results were found in nine samples. SS2R missed influenza A virus (n = 3), RSV (n = 3) and coronavirus (n = 3) in seven samples. The median Ct-value of 36.04 (SD ± 4.21) of these discrepant results (n = 9), with one Ct-value > 40 (Influenza A, Ct-value 40.89), was significantly higher than the median Ct-value of 26.03 (SD ± 7.05) of all concordant virus positive results (n = 37), with one Ct-value > 40 (coronavirus, Ct-value 42.23) (p = 0.001). SS2R had a rhinovirus positive result in one sample (RT-PCR sample negative) and an influenza A / influenza B virus equivocal result in one sample (RT-PCR sample influenza A virus, Ct-value 36.04), the latter being considered concordant in the accuracy analysis.
Logistics
The median TAT of the SS2R was 2:06 hours (IQR 1:44-3:16) for patients of whom the SS2R was performed on the day of sampling (n = 46). For 16 of these (35%), results became available during their ED-stay. The median RT-PCR sample TAT of the 46 patients was 32:00 hours (IQR 26:50-40:42). Based on the questionnaires, laboratory technicians (n = 5) were positive about ease-of-use, short hands-on time (≤ 10 minutes) and fast TAT of the SS2R. A low test capacity, due to the availability of only one SS2R system, resulting in less optimal TATs, was mentioned as a drawback.
DISCUSSION
This prospective clinical study focuses not only on accuracy and TATs of rapid SS2R testing, but also on applicability and implementation strategies. SS2R showed 90% sensitivity and 95% specificity in the detection of respiratory viruses compared to the current gold standard, RT-PCR. The poor diagnostic accuracy for some viruses, for example RSV, is due to a small number of patients and is reflected in the wide confidence intervals around the accuracy estimates. In our hospital, SS2R had a rapid TAT, even when used in a laboratory setting. The SS2R only had two out of 62 invalid initial results. This SS2R therefore has great potential in the improvement of clinical outcomes inpatients, for example in terms of targeted oseltamivir prescription and possibly also reduced antibiotic treatment. It may also benefit hospital management, for example by contributing to more adequate use of in-hospital isolation facilities. It should be noted that all swabs were taken during a month in the viral respiratory season with a high prevalence of influenza virus and RSV in particular. Not only the virological results, but also potential effects on clinical outcomes should be viewed from this perspective. Among published articles on the accuracy of the specific SS2R (FilmArray®)14-35 used in this study, only a few compared SS2R to RT-PCR,14-20 with a calculated pooled sensitivity of 87.6% [95%CI 84.6-90.1] and specificity of 91.1% [95%CI 87.5-93.7] (n = 945 samples, table 4). Differences between studies, including the current study, and the relatively low overall sensitivity might be explained by genetic variability of viruses, differences in sampling and analyzing methods, patient numbers and/or heterogeneity of the patients involved. It is hard to predict genetic variability but methods of sampling, data analysis and patient inclusion are influential. It is therefore useful to find out how accuracy can be optimized before implementing SS2R. First, the sampling site has a significant effect on sensitivity. Although lower respiratory tract samples like bronchoalveolar fluids, have highest sensitivity,17,28 nasopharyngeal swabs – which were used in the current study – are the most sensitive upper respiratory samples36,37 and are most feasible in an ED-setting. Second, in analyzing accuracy data for multiple viral pathogens, the number and choice of viruses and the cut-off Ct-value influence accuracy. In this study, accuracy was calculated both per virus and per sample, using initial results and results of repeated testing for invalid results to reflect clinical practice as close as possible. The high cut-off value of > 45 used in this study, leading to a somewhat lower sensitivity of the SS2R due to one influenza A virus (Ct-value of 40.89) that was missed by the SS2R, was chosen to reflect clinical practice, in which higher cut-off values are often used for RNA-viruses. Furthermore, choosing a reference standard, including the performance of a discrepancy analysis, either by repeated testing or by sequencing, greatly influences accuracy. Although RT-PCR is considered the best available reference standard, composite reference standards and discrepancy analyses may lead to higher numbers of agreement. In this study accuracy measures were given in a way the estimates are interpretable by clinicians. From a more epidemiologic point of view, accuracy or percentage agreement (87% [95% CI 76-94]) with Cohen’s Kappa statistic (0.73 [95% CI 0.56-0.90]) might be more suitable considering the imperfect reference standard (table 3). Third, significant differences have been shown in the sensitivity of PCR assays between different study populations, with higher sensitivities in patient groups with higher viral loads and lower viral clearance rate38 due to short symptom duration or specific characteristics such as being a child,39,40 being immunocompromised40 or having COPD.41 The current study is underpowered to compare patient groups, but confirms that discordant SS2R negative results had significantly lower viral loads (e.g. higher Ct-values) than concordant results. In daily practice this means that SS2R has reliable results for patients with high viral loads, being the patients at highest risk of complications and respiratory insufficiency.42
In our hospital, SS2R has a rapid TAT compared to that of previous studies (average TAT 2:30 hours (n = 5576)).34,35,43-48 Since patients were only included during lab opening hours, TATs may not be representative for sample testing during evenings, nights and weekends. A proposal to relocate the SS2R system to the ED raised objections among the staff. Objections included lack of laboratory skills, a highly variable workload of the ED-personnel and need for an isolated test room, making SS2R implementation at the ED unfeasible at our hospital. Also, the ISO-certification would have to be extended for point-of-care testing at the ED. However, the current study showed that TAT was rapid even when used in a laboratory setting. Apart from TATs, the effect of SS2R testing is also affected by early availability of results, which was delayed in this research setting due to the necessity of asking informed consent and taking a second swab. When swabs are taken shortly upon a patient’s arrival at the ED, it can be assumed that the percentage of available SS2R results during ED-stay will be much higher than 35%. Altogether, SS2R implementation in the laboratory setting has great potential in affecting clinical outcomes, with fewer practical issues to overcome than at the ED. The effect of rapid testing can be optimized by extending laboratory opening hours and using more than one SS2R system for parallel testing.
Even though the results of this study regarding diagnostic accuracy, TATs and applicability are promising with respect to clinical outcomes, no exact estimations can be made. A recent randomized controlled diagnostic intervention trial49 showed beneficial effects of rapid testing with the same SS2R on both oseltamivir prescription, duration of antibiotic treatment and use of in-hospital isolation facilities, strengthening our hypothesis. In this trial, as compared to conventional diagnostics, SS2R led to a 26% increase in correct oseltamivir prescription in influenza virus positive patients (91% versus 65%, p = 0.003) and an 8% decrease in the number of patients who received antibiotic therapy for > 48 hours.49 However, the percentage of antibiotic prescriptions was similar in both groups and the use of SS2R increased the use of in-hospital isolation facilities for patients with confirmed viral RTI by 8% (17% versus 9%, p = 0.02).49 The hypothesis that rapid viral SS2R testing in patients with RTI may also reduce antibiotic prescription is one of the most important issues in the current landscape of increasing antibiotic resistance. Nevertheless, since the decision to prescribe antibiotics is not based on the SS2R results alone, neither observational nor experimental studies have so far been able to show an advantageous effect of rapid SS2R testing on antibiotic prescription. The most plausible explanation for this disappointing effect of SS2R testing on antibiotic prescription is that many patients have already been started on antibiotics before the results of a rapid test become available,49 underlining the importance of an optimal implementation strategy.
In conclusion, rapid syndromic sample to result PCR like the FilmArray® Respiratory Panel are fast, easy to use and accurate, especially with high-risk patients. Implementation of rapid mPCR diagnostics in routine care, even when put in a laboratory setting, could further improve clinical management of patients suspected of respiratory tract infection presenting at the emergency department.
DISCLOSURES
All authors declare no conflict of interest. No funding or financial support was received.
REFERENCES