Using quality indicators to assess performance of endobronchial ultrasound in the staging and diagnosis of lung cancer: a pre/post study at a New Zealand centre

Lung cancer is the leading cause of cancer-related death in New Zealand and Australia.^1,2 Endobronchial ultrasound-guided transbronchial needle aspiration (EBUS-TBNA) is a bronchoscopic procedure that has become a key modality for the staging and diagnosis of lung cancer. EBUS-TBNA is generally performed by respiratory physicians in an outpatient setting and has significantly reduced the need for surgical sampling of the mediastinum, namely mediastinoscopy, and its associated risks.

At present, there is no routine requirement for the reporting of EBUS performance for lung cancer staging and diagnosis and there are no data regarding EBUS performance in New Zealand. Until recently, there have been no published quality indicators relating to EBUS performance monitoring. However, a national EBUS service specification was published in the United Kingdom (UK) in 2019, which outlines key quality indicators.³ Furthermore, care pathways and quality standards have been developed to provide a structured, multidisciplinary pathway for people with suspected lung cancer, including in New Zealand and Australia.^4–6 These pathways highlight key performance indicators and recommended timeframes for diagnosis and treatment, and while they are not specific to EBUS, they are clinically relevant and can be appraised when examining the use of EBUS in lung cancer.

EBUS-TBNA has a vital role in mediastinal nodal staging in people with lung cancer. Nodal stage is a predictor of prognosis, with a higher stage conveying a worse outcome.⁷ In cases of potentially radically treatable lung cancer, EBUS-TBNA can be performed as a staging procedure that improves the accuracy of mediastinal staging compared to radiological stage alone and is essential for determining optimal treatment.⁸ The radical management of a patient with lung cancer and mediastinal nodal involvement (N2/3 disease) requires a multimodality approach, with treatment choice being influenced by location, distribution and volume of involved nodes.⁹ A staging EBUS should therefore provide high sensitivity and negative predictive value (NPV) for the detection of N2/3 disease. Theoretically, sensitivity should not be dependent on prevalence of N2/3 disease; however, sensitivity and NPV have been shown to be correlated with the prevalence of N2/3 nodal disease, with prevalence being positively correlated with sensitivity and inversely correlated with NPV.^8,10,11 This likely reflects biological variation between patients in higher and lower prevalence populations. For example, patients with large, morphologically malignant, positron emission tomography (PET/CT) avid mediastinal nodes will have a higher prevalence of malignant nodal involvement, and therefore a lower false negative sampling rate, compared to those with small non-avid nodes when the prevalence of malignant nodal disease is likely to be lower or microscopic in nature, with an increased false negative sampling rate.¹¹ Therefore, when assessing the performance of EBUS in lung cancer staging, it is essential to present sensitivity and NPV alongside the prevalence of N2/3 disease in those undergoing EBUS. The specificity of EBUS-TBNA is generally reported at 100%—false positive results are very rare but have been described.¹²

In cases of advanced lung cancer, when radical treatment is not possible and detailed mediastinal staging is not required, EBUS can be utilised as a diagnostic procedure. A diagnostic EBUS involves targeted sampling of any abnormal node or tissue in order to provide adequate material for tumour subtyping and molecular analysis. Pathological confirmation rate and adequacy of sampling for molecular analysis are therefore key performance metrics for diagnostic EBUS.

In this study, performance against EBUS quality indicators was compared before and after the commencement of local performance monitoring in order to assess whether this resulted in improved performance of EBUS in the staging and diagnosis of lung cancer.

Methods

This performance review consisted of two phases. In September 2021, the UK EBUS service specification together with national lung cancer guidelines and clinical quality indicators from New Zealand and Australia were reviewed, with recommendations applicable to EBUS being extracted for use in this study. These recommendations and quality standards can be found in Table 1. For staging EBUS, the number of lymph node stations being sampled was also recorded, as there is evidence that this influences performance.¹³

In September 2021, a retrospective review of all lung cancer EBUS performed at our institution between September 2019 and September 2021 (phase 1) was performed. Phase 1 data were analysed and compared to the published quality standards, with findings being presented within our local lung cancer governance group, which determined the need for prospective data collection with regular reporting of performance. Prospective data collection was then performed for all lung cancer EBUS between October 2021 and October 2022 (phase 2). Performance of EBUS was then compared between phases 1 and 2 in order to determine the effect of performance monitoring and identify areas for service improvement.

Our centre has provided an EBUS service since 2014, and during the study period all procedures were performed by one of five respiratory physicians alongside a specialist trainee in respiratory medicine, using local anaesthesia with topical lignocaine to the airways and conscious sedation with intravenous fentanyl and midazolam. During the study period, patients with suspected lung cancer proceeded to EBUS following initial staging computed tomography (CT) and clinical review, with or without PET/CT, at the discretion of the treating clinician. Lung cancer diagnosis was confirmed from a review of medical records and confirmation of clinical-pathological diagnosis at 6 months post-procedure. This 6-month follow-up period was chosen based on the recommendation from the published EBUS service specification and expert opinion.^{3, 11} Initial staging CT, or PET/CT if performed, was reviewed to determine the radiologic stage and the American College of Chest Physicians (ACCP) radiologic group, a description of which can be found in the Appendix.⁸ Cases without evidence of metastatic disease on initial staging CT and/or PET/CT and meeting criteria for ACCP groups B, C or D were assigned to the staging EBUS group. Those with metastatic disease or those in ACCP group A were assigned to the diagnostic EBUS group.

Electronic health records were reviewed to assess demographics and lung cancer pathway metrics. EBUS-specific data were retrieved from electronic procedural reports. Final cytopathology reports were reviewed to categorise positive and negative results, adequacy of sampling for tumour subtyping and adequacy for molecular analysis. For staging EBUS, final clinical-pathological stage was reviewed at 6 months post-procedure, with procedures being classed as true positive, true negative, or false negative for N2/3 nodal involvement, with subsequent calculation of N2/3 prevalence, sensitivity and NPV. A false negative EBUS was defined as a negative EBUS-TBNA for N2/3 nodal disease that later proved positive, either at surgical resection or following 6 months of clinical-radiological follow-up, including for N2/3 nodes inaccessible by EBUS. Adverse events were categorised according to the British Thoracic Society (BTS) bronchoscopy guideline.¹⁴

Demographic and clinical characteristics were summarised using standard descriptive statistics depending on the type and distribution of data. Continuous variables were compared using an Independent Samples t-Test or Mann–Whitney U test, with categorical variables compared using Pearson Chi-squared test or Fisher’s exact test. All tests were two-tailed and statistical significance was set at p<0.05. The SPSS Statistics package (version 26.0, 2019; IBM) was used for data analysis. The study was approved by our institutional Research and Knowledge Department, and individual patient consent was not required given the nature of the study.

View Table 1–5.

Results

A total of 232 EBUS were performed for lung cancer staging and diagnosis during the study period (115 staging EBUS and 117 diagnostic EBUS). Age, sex and ethnicity were similar between groups. Demographics, clinical characteristics and procedural data are reported in Table 2.

In phase 1, four patients required a repeat staging EBUS: two following PET/CT that showed increased metabolic activity in an N2 node that was not sampled previously (both true negative); one following a negative EBUS prior to surgery (true negative); and one due to progressive nodal enlargement following a negative EBUS (false negative). Three patients required a repeat diagnostic EBUS, one due to insufficient material for tumour subtyping, and two due to insufficient tissue for molecular analysis. In phase 2, three patients required a repeat staging EBUS: one following PET/CT showing mild avidity in a previously sampled N2 node (true negative); one due to insufficient sampling of an enlarged N2 node (true negative); and one due to insufficient tissue for molecular analysis. One patient required repeat diagnostic EBUS due to insufficient tissue for molecular analysis.

There were no significant differences in ACCP group distribution or tumour subtype between phases 1 and 2, with adenocarcinoma being the most frequent diagnosis in both the staging and diagnostic EBUS groups.

Staging EBUS performance

Table 3 summarises the performance of EBUS in the staging of lung cancer. The overall prevalence of N2/3 nodal disease in phases 1 and 2 was 55% and 57% respectively, resulting in sensitivity and NPV targets of >85%.^3,11 Across both phases, sensitivity and NPV for the detection of N2/3 disease met the recommended targets and demonstrated a small improvement in phase 2 compared to phase 1; however, this was not statistically significant. Adequacy of sampling for molecular analysis was high across both phases, meeting recommended targets.

There were increases in the mean number of nodes sampled per procedure in phase 2 (1.9, standard deviation [SD] 0.85) compared to phase 1 (1.6, SD 0.7) (t=2.154, p=0.03), and an increase in the mean number of N2/3 nodes sampled per procedure in phase 2 (1.5, SD 0.72) compared to phase 1 (1.1, SD 0.71) (t=2.77, p=<0.01). In phase 2, patients were more likely to have two or more N2/3 nodes sampled compared to phase 1 (odds ratio [OR] 2.41, 95% confidence interval [CI] 1.1–5.28, p=0.03).

Diagnostic EBUS performance

Table 4 summarises the performance of EBUS in the diagnosis of lung cancer. Pathological confirmation rate was high in both phases and met the recommended target, although was numerically lower in phase 2. The rate of non-small cell lung cancer not otherwise specified (NSCLC-NOS) was higher in phase 2 compared to phase 1, falling above the recommended target. Adequate tissue for molecular analysis was obtained in 92% of procedures during both phases. The rate of repeat sampling was low, noting that not all patients with inadequate tissue for molecular analysis underwent repeat testing (due to declining performance status or patient wishes).

Lung cancer pathway performance indicators and safety data

Table 5 shows performance indicators for EBUS-specific lung cancer pathway metrics and associated targets. Waiting time from referral to staging EBUS was longer during phase 2 with an increase in median wait time from 4 to 5 days, associated with a decrease from 93% to 83% 7-day completion during phase 2—falling short of the New Zealand quality standard of 95% compliance. Time from EBUS to initial pathology reporting (excluding molecular analysis) was above the target set by both the UK EBUS service specification and the New Zealand quality standard. With reference to the Australian optimal care pathway, between 42% and 57% of cases had a pathology result available within 3 days of EBUS, which can act as a comparator for other services as there is no published compliance target for this metric. The time interval from EBUS to receipt of final pathology, including molecular analysis, did not meet the target set by the UK service specification. During the study period, this was only applicable to those patients with non-squamous NSCLC when testing for anaplastic lymphoma kinase (ALK) rearrangement and common mutations were required. In phase 2, the proportion of cases having final pathology results available within 10 or 14 days from staging EBUS improved by 17% and 12%, and for diagnostic EBUS by 18% and 9%, respectively, although these differences did not reach statistical significance. There was a move towards reduction in total pathway time (all results available within 14 days from referral) in both cohorts, not meeting statistical significance.

There was one serious adverse event resulting in a hospital admission due to presumed EBUS-related chest infection requiring a 3-day inpatient stay. Four procedures resulted in moderate bleeding requiring the endobronchial administration of cold saline or adrenaline, with all patients being discharged the same day with no further bleeding complications.

Discussion

This single-centre review of EBUS performance in the staging and diagnosis of lung cancer has provided evidence of a high-quality service across a number of quality indicators. While there was no significant improvement in the majority of quality indicators between phases 1 and 2, overall EBUS performance was generally excellent with both sensitivity and NPV for staging EBUS being above the targets set out in published quality standards, and pathological confirmation rate for diagnostic EBUS being high across both phases. Adequacy of material for molecular profiling was also above the published standards in both the staging and diagnostic cohorts. Pathway times varied across phases 1 and 2, with small improvements in time to final pathology reported being seen in phase 2. Importantly, there is evidence of a transition towards more guideline-concordant care evidenced by more detailed nodal sampling during staging EBUS, which is worthy of further discussion.

With regards to staging EBUS, thoracic surgical guidelines for mediastinal staging define a staging EBUS as either selective, with sampling of suspicious/radiologically abnormal nodes only, or systematic, involving assessment of all nodal stations aiming for sampling from ≥3 mediastinal nodal stations.^8,11,15–17 Hypothesising that the number of N2/3 nodes sampled is a surrogate for selective versus systematic staging, our data suggest that a selective staging approach was favoured with an average of 1.1 mediastinal nodal stations being sampled during phase 1, increasing to 1.5 mediastinal nodal stations during phase 2. In phase 2, fewer cases had only N1 nodes sampled compared to during phase 1 (7% versus 19%), and more cases had ≥2 mediastinal nodes sampled (48% versus 27%). Only a small number of cases had three mediastinal nodes sampled: one case in phase 1 and three cases in phase 2. While the average number of mediastinal nodes sampled per procedure increased between the two phases, this resulted in only small improvements in sensitivity and NPV in phase 2 (1.7% and 0.9%, respectively). However, there are data to support systematic rather than selective mediastinal sampling for lung cancer staging, with a randomised control trial comparing selective EBUS, systematic EBUS and systematic EBUS plus EUS-B (endoscopic ultrasound-guided sampling using the EBUS scope in the oesophagus) showing a 4% increase in sensitivity with systematic compared to selective EBUS, and an additional 5% improvement when EUS-B was added.¹³ A large meta-analysis by Korevaar et al. further demonstrated improved sensitivity when EUS-B is used alongside EBUS; however, the routine use of EUS-B is not commonplace due to lack of availability and expertise, and is not in use at our centre.^13,18 The meta-analysis reported a sensitivity of 72% for the detection of N2/3 nodal disease with EBUS across all available studies; however, direct comparison with our cohort is not possible given the varying prevalence of N2/3 disease in the meta-analysis population.¹⁸

Rapid onsite evaluation (ROSE) is utilised alongside EBUS at our centre as it can provide real-time feedback regarding the cellularity of a nodal specimen, having also been shown to increase the rate of successful lung cancer genotyping when compared with standard care, and this may influence sampling strategy.¹⁹ However, the benefits of ROSE alongside EBUS remain the subject of debate.^19–21 The use of ROSE may explain the relatively low number of N2/3 nodes being sampled in our staging cohort when compared to guideline recommendations, as the bronchoscopist may choose to stop sampling once adequate material has been obtained from an N2/3 node if it provides sufficient staging information to inform treatment. However, there are potential downsides to a ROSE-guided sampling approach, as there is evidence to suggest that the total number rather than just anatomical location of malignant nodes influences prognosis.²² Additionally, precise nodal characterisation aids in the planning of radical radiotherapy for those patients who do not undergo surgery.²³ Overall, despite a more selective staging approach being favoured at our centre, we provide assurance of high-quality mediastinal nodal staging evidenced by high sensitivity and NPV for the detection of N2/3 disease that meets published quality standards.

Importantly, across both phases, only around 20% of patients undergoing staging EBUS were clinical stage N0/N1 (ACCP groups C and D, or a radiologically normal mediastinum). A meta-analysis evaluating EBUS-TBNA for systematic mediastinal staging in clinical N0/N1 lung cancer demonstrated a mean prevalence of occult N2/3 disease of 15%, with a number needed to test to detect occult N2/3 disease of 14 (95% CI 10.8–16.3).²⁴ We may be missing opportunities to identify patients with occult mediastinal nodal involvement in patients with a radiologically normal mediastinum. This is particularly relevant in those who may receive ablative radiotherapy rather than surgical resection and lymph node dissection for complete pathologic staging. Further, the CheckMate 816 trial has shown significant improvements in outcomes following neoadjuvant chemotherapy-immunotherapy followed by surgery in those with lymph node metastases, with the National Institute for Health and Care Excellence (NICE) now recommending this treatment for patients with tumour size >4cm or nodal metastases at diagnosis.^25,26 While this treatment is not currently available in New Zealand, this study highlights the importance of precise mediastinal staging in order to optimise treatment.

During this study, the timing of PET/CT and staging EBUS was at the discretion of the treating clinician; however, the UK NICE lung cancer guideline recommends that PET/CT be performed prior to staging EBUS.²⁷ PET/CT was available for 45% and 46% of staging procedures in phases 1 and 2 respectively. Sensitivity for the detection of N2/3 disease was lower in those who underwent PET/CT prior to EBUS (83% versus 89%, p=0.68) and NPV was higher in those who underwent PET/CT prior to EBUS (92% versus 77%, p=0.14). However, the prevalence of N2/3 disease in those with and without prior PET/CT was significantly different, being 35% and 73% respectively. This reflects the relationship between N2/3 disease prevalence and sensitivity (positive correlation) and NPV (negative correlation), as described earlier.^8,11

Lung cancer pathways and service standards are endorsed in both Australia and New Zealand.^4,6 While cancer pathways aid in mapping the patient journey from diagnosis to treatment, the focus mainly relates to the overall timeliness of care, and many do not provide explicit recommendations regarding EBUS. Delays can occur at any part of the lung cancer pathway, so critical appraisal of each step may identify areas for improvement.

Access to EBUS was better during phase 1 compared to phase 2 (93% performed within 7 days of referral versus 83%). This is likely to be multifactorial, but increased demand on lung cancer services coupled with the effects of the SARS-CoV-2 pandemic on EBUS utilisation, particularly during phase 1, are likely to be contributory. Time to initial pathology reporting is excellent (98% reported in ≤5 days in Phase 2), but time to molecular pathology reporting is prolonged. The mean turnaround times from initial pathology report to final molecular pathology (for non-squamous NSCLC) in the staging and diagnostic groups across both phases were 8.4 and 9.0 days respectively. Total pathway time (from EBUS referral to receipt of all pathology results) was prolonged with <40% of patients receiving all results within 14 days. This may have a greater impact on those patients with more advanced disease, as delays in the reporting of a potential targetable oncogenic driver would influence first-line treatment. In our region, molecular pathology testing is outsourced to a regional provider, with inherent delays involved in the transport and processing of samples and publishing of results. During phase 1, mutation analysis was performed upfront, followed by ALK rearrangement by fluorescence in situ hybridisation (FISH) if the initial mutation panel was negative (limited to epidermal growth factor receptor mutations during this study period). During phase 2, ALK FISH was replaced by ALK immunohistochemistry (IHC), which can be performed upfront alongside standard IHC, and likely contributed to the reduction in time to final molecular diagnosis seen in phase 2.

While this performance review provides important data regarding the performance of EBUS-TBNA in a New Zealand context, there are important limitations to acknowledge. Given the retrospective data collection during phase 1, there is risk of investigator bias in the allocation of patients to either the staging or diagnostic EBUS groups. Based on the index CT scan of the chest, those with ACCP group B, C or D were automatically assumed to have undergone a staging EBUS. However, this does not take into account the likely treatment intent for individual patients, as some patients with ACCP group B, for instance, may not have been suitable for radical treatment and may have undergone a targeted EBUS procedure only. However, the same group allocation criteria were used during phase 2 in order to allow direct comparison. Further, this study does not account for potential variability in practice between different bronchoscopists, which may influence overall performance. Although procedures were performed by a group of five operators, over 85% of all procedures were performed by three of the group. For diagnostic EBUS, diagnostic confirmation rate and adequacy for molecular analysis were similar across all operators despite the differences in volume, being over 90% for both quality indicators. For staging EBUS, overall sensitivity and NPV ranged from 84% to 100% and 67% to 100%, respectively. Paradoxically, the highest sensitivity and NPV were from operators who performed the fewest number of cases (fewer than seven staging EBUS per year per operator). While it is reassuring that high sensitivity and NPV are provided by those performing low numbers of procedures, this may represent differing case selection and caution should be used when appraising these values. It is important to note, however, that procedural competence is not necessarily related to the overall number of cases performed,²⁸ and monitoring of procedural volume per clinician may form a useful part of a quality assurance process.

Importantly, since completion of this study, an international expert consensus statement on proposed EBUS quality indicators and recommended reporting has been published.²⁹ The statement expands on the quality indicators reported in our review, and may form the basis of local EBUS quality assurance programmes going forward.

Conclusion

EBUS-TBNA is an essential part of the lung cancer pathway and can be provided in a timely and safe manner. Monitoring and reporting of local performance allow critical assessment of current practice and can identify areas for quality improvement with a view to improving care. This review demonstrated good initial performance, but prompted a move towards more guideline-concordant practice with increased mediastinal nodal sampling during staging procedures. In an Australasian context, consideration should be given to the adoption of routine EBUS performance monitoring within local and/or regional networks, which could be incorporated alongside the newly proposed Lung Cancer Clinical Quality Registry and would allow EBUS centres to benchmark current practice and act as a driver for quality improvement.³⁰

View Appendix.