Using a randomised controlled trial to test the effectiveness of social norms feedback to reduce antibiotic prescribing without increasing inequities

In 2020, the World Health Organization (WHO) described antimicrobial resistance (AMR) accelerated by misuse and overuse of antibiotics as “one of the biggest threats to global health, food security, and development today.”¹ Overprescription of antibiotics is especially problematic in Aotearoa New Zealand, which in 2017 had the fourth highest community prescription rate of antibiotics in the OECD.²

Most antibiotic consumption (85–90%) in Aotearoa New Zealand occurs outside of hospital, predominantly on prescription from general practitioners (GPs).³ Local and international evidence suggests inappropriate prescribing of antibiotics exists^4,5 and is often driven by diagnostic uncertainty and other factors,^6,7 with consequences for not only selection for antibiotic-resistant bacteria, but also other effects on health.⁶

However, underprescribing of antibiotics is also an issue. Despite years of evidence showing the burden of disease for which antibiotics are indicated affects Māori and Pacific peoples more than other ethnicities,^8–13 there is longitudinal evidence that Māori do not receive antibiotics when it is important to receive them.^14,15 A 2019 report from PHARMAC found antibiotics lead all of the many classes of medications shown to have inequities in access.¹⁵

The many recent interventions to encourage better antibiotic stewardship include patient education to address demand, clinician education and communication skills, decision-support tools, monitoring, point-of-care diagnostic tests, accountable justification (where non-indicated antibiotic prescribing must be documented and explained) and delayed prescribing.^6,16–18

Increasingly, evidence for efficacy is accumulating around audit and feedback mechanisms that incorporate peer comparison, with adaptations informed by behavioural science.^6,19 Audit and feedback have been used worldwide to encourage quality improvement in healthcare,²⁰ including BPAC audit reports and Pegasus Small Group Rounds in New Zealand. Several studies have now shown the efficacy of a peer-comparison intervention (referred to as “descriptive social norms” in the social psychology literature, after the Theory of Planned Behaviour²¹) on antibiotic prescribing, whereby high prescribers in primary care are sent a letter from a trusted source that compares the GP’s prescribing patterns against anonymised peer-prescribing patterns.^22–27

We aimed to determine the impact on prescribing levels in the Aotearoa New Zealand context of such an audit and feedback intervention targeting the antibiotic prescribing of GPs identified as high prescribers, including the impact on antibiotic prescribing levels for Māori and Pacific populations.

We report the methodology and results of a randomised controlled trial of this intervention, conducted with GPs who were in the top 30% of antibiotic prescribers in the 2018 calendar year (n=1,260) in each of the 20 district health boards (DHBs). We assessed the effect on prescribing by these GPs from September to December 2019 after the letter was mailed in August 2019, using the 2018 calendar year as baseline, with special attention to the effects of the intervention on prescribing to Māori and Pacific peoples, and discuss the results.

Methods

The intervention was evaluated with a two-armed randomised controlled trial (RCT), with randomisation at the GP level and stratification by DHB.

Study design and participants

The top 30% of GPs prescribing antibiotics in each DHB for the 2018 calendar year were identified using data from the Pharmaceutical Collection, the Ministry of Health’s national collection of claim and payment information for subsidised dispensings.²⁸

This de-identified data described all dispensings of medicines from community pharmacies in Aotearoa New Zealand in the calendar year 2018, including the age and ethnicity of patients dispensed to. All but GP dispensings were then excluded (including nurse practitioners, dentists and all doctors except for GPs). Locums were excluded from the trial because it was unlikely that we would be able to send them a letter. Individual GPs were linked to a practice by ascertaining the practice enrolment of patients prescribed to by that GP and their DHB was identified. Codes for individual GPs from the Pharmaceutical Collection data were linked to GP names at the Ministry of Health and address for practices were ascertained via publicly available information.

This yielded a sample of 1,260 high-prescribing GPs, who were then randomised into intervention and control groups. Randomisation was conducted in R, using the “stratified” function in the splitstackshape package. High-prescribing GPs were stratified by DHB due to the substantial differences in prescribing by DHB (partly driven by differences in patient characteristics). Balance checks of these two cohorts showed close comparability (Appendix 1).

After excluding 15 intervention group GPs who could not be linked to a name (inclusion did not meaningfully change results) and who were lost to follow-up due to inactivity in the trial period, the final sample for analysis yielded 1,214 GPs: 602 GPs in the intervention arm and 612 GPs in the control arm. Power calculations showed we would be able to detect an effect size of around 0.75 at p=0.05 (a 4% change in the antibiotic prescribing rate). Figure 1 shows the cohort design.

Figure 1: Sample of GPs at assessment, allocation, follow-up and analysis.

GPs were excluded if:

1. The GP could not be assigned to a practice in the last quarter of 2018 (to ensure up-to-date information was used); or
2. The GP prescribed any medicine to fewer than 10 patients a month on average (to ensure statistical robustness) (there was no minimum time GPs must be active to be included); or\
3. 80% or more of the GP’s patients were 75 years or older, suggesting their caseload is largely limited to patients in rest homes (to avoid comparisons with GPs with a unique patient population).

The intervention

The design of the intervention letter (Appendix 2) drew on previous successful trials and the behavioural science literature, with adaptations for the Aotearoa New Zealand context.

The letter began with a clear call to action that emphasised the importance of appropriate antibiotic prescribing and how to tackle this issue collectively. The peer comparison drew on the known effects of social norms to shift behaviour.^22,29–31 These differences were presented visually in a graph, as did the most effective letter in the Australian trial.²⁵ The letter also provided three clear steps to change behaviour (Appendix 2).

A particular adaptation to the Aotearoa New Zealand context was the inclusion of detailed data of prescribing to at-risk groups, as a foreseeable unintended effect of the intervention might have been a reduction in appropriate prescribing to Māori and Pacific peoples. Each GP’s prescribing to Māori, Pacific and other peoples, and the GP’s place in the distribution relative to their peers, was visualised specifically in their letter (Figure 2). This was intended to prompt more nuanced behaviour change rather than a blanket reduction that might further restrict prescribing to groups who needed prescriptions. Ethnicity data were coded using the Level 1 approach as per the HISO 10001:2017 Ethnicity Data Protocols and prioritised in cases where an individual has multiple ethnicities reported. “Asian” ethnicity was included in the “other” category.

Figure 2: Example graphs from intervention letter showing individual GPs’ antibiotic prescribing to different demographic groups relative to their peers.

Data on prescribing of specific antibiotics was also provided²⁵ to provide more granular levers for change. The letters were addressed from Dr Janice Wilson, chief executive officer of the Health Quality and Safety Commission (the Commission), to draw on the “messenger effects”³² shown to be effective in other trials, whereby trust and respect felt for the signatory of the letter increases the likelihood the letter will be received positively and acted upon.

The intervention letters were sent to participants in the intervention arm in August 2019.

Analysis of the trial

Our primary and secondary analyses were registered on the American Economic Association’s registry for randomized controlled trials (AEARCTR-0005526).³³

Our two primary outcome measures were:

• the antibiotic prescribing rates of control versus intervention cohorts, averaged over the period September–December 2019, and
• the number of antibiotic scripts dispensed from the GPs prescribing between September and December 2019, after the letters were mailed in August 2019.

The secondary analyses registered examined effects of the intervention for all high-prescribing GPs in terms of:

• the prescribing rate of antibiotics to Māori, Pacific and non-Māori and non-Pacific patients, and
• the number of people prescribed paracetamol scripts per 100 prescribed any medicine.

The antibiotic prescribing rate chosen was:

(100 * number of patients dispensed an antibiotic) / (number of patients dispensed any medicine)

This prescribing rate was defined in consultation with the authors and representatives from the Commission, PHARMAC, the Ministry of Health, the Royal New Zealand College of General Practitioners, the New Zealand Medical Association and several practising GPs.

Antibiotics included in the count were nine commonly prescribed antibiotics whose use (trimethoprim aside) tends to increase in winter:

• Amoxicillin
• Amoxicillin with clavulanic acid
• Cefaclor
• Co-trimoxazole
• Doxycycline
• Erythromycin
• Penicillin V
• Roxithromycin
• Trimethoprim

This rate was calculated at a monthly level (ie, what proportion of patients were prescribed any of these nine antibiotics in a given month) and then averaged over the 12 months of 2018 to calculate one antibiotic rate for each GP. This definition, which is based on the number of people dispensed antibiotics rather than the number of scripts, showed more stability over time, with high prescribers in winter also being high prescribers in summer. This was then compared to the post-intervention antibiotic prescribing rate for September–December 2019.

Three regressions for each outcome variable were estimated: one with just the intervention variable included, one with the intervention variable and the DHB stratification variable included as a covariate, and one with the intervention variable, the DHB stratification variable, and the GP’s 2018 value of the prescribing outcome variable to increase precision. In all analyses below we only report the estimate from the third model with additional controls. (Though, as Appendix 3 shows, the results of the three models are similar for our analyses.)

Ethical approval

Ethical approval for this work was obtained from the New Zealand Ethics Committee on 26 July 2019 (ref: NZEC Application 2019_34).

The R statistics package was used to analyse data.

Results

Preliminary analysis

Raw data showing the average prescribing rate for intervention and control GPs from January 2018 to December 2019 were plotted to assess whether descriptive analyses would align with inferential analyses (Figure 3). These data suggested a divergence in antibiotic prescribing rates between the control and intervention groups may have occurred, with intervention groups showing a lower antibiotic rate in all months subsequent to the letters being sent in August 2019. We assessed this further with the pre-registered primary and secondary analyses discussed above.³⁶

Figure 3: Prescribing over time of intervention and control GPs (grey dotted line shows timing of intervention).

Effects on primary outcome measures

Figure 4 shows the effect of the intervention on the first primary outcome measure, where the grey bar shows the unadjusted average prescribing rate for the control group (17.88) and the blue bar represents the average prescribing rate for the intervention group (16.23), after controlling for DHB and past prescribing. The difference between the two bars shows the intervention effect: a reduction of 1.65 in the prescribing rate, reflecting a 9.2% relative difference (p<0.001). This means that for every 100 patients who are prescribed any medicine by a GP, 1.7 fewer patients will be prescribed an antibiotic. The corresponding regression output is presented in column (1) of Table 1.

Figure 4: Antibiotic prescribing rates, control and intervention groups, September–December 2019.

*** p<0.001.

Column (2) of Table 1 shows the intervention effect for the second primary outcome: the number of antibiotic scripts dispensed from the GP’s prescribing between September and December 2019. The intervention effect is 13.3, representing a 7.1% difference between the intervention and control arms (p<0.01). This represents 8,000 fewer courses of antibiotics dispensed over the four-month trial period subsequent to the intervention.

See Supplementary Material 1 for effects on other commonly dispensed drugs.

Table 1: Intervention effect on our primary and secondary outcome measures.

This table presents the coefficients from ordinary least squares regressions at the GP level where the outcome measure is described in column headers. The sample period is September–December 2019. In columns (1) and (3)–(5), the outcome measure is calculated as the average of the GP’s monthly rate from September–December 2019. In column (2), the outcome measure is the total number of scripts from September–December 2019. In each column, the covariate for past prescribing is the same as the outcome measure but calculated over the 2018 calendar year. In columns (3)–(5), GPs were excluded if they prescribed no medicine to the given ethnicity group in 2018 or over the trial period of 2019. This explains the slight variation in sample size across the columns, and it explains why these sample sizes are smaller than in columns (1)–(2). Standard errors, in parentheses, are heteroskedasticity-robust. Asterisks denote: *** p<0.001, ** p<0.01.

Effects on secondary outcome measures and exploratory analyses: prescribing to different ethnicity groups

Next, the effects of the intervention on prescribing to different ethnicity groups were analysed. GPs’ prescribing to Māori, Pacific peoples and all others were analysed separately. The antibiotic rate is calculated in the same way as the overall rate, but the numerator and denominator are limited to those who are of the ethnicity of interest.

Columns (3)–(5) of Table 1 show statistically significant changes (p<0.01) in prescribing to all three ethnicities. Column (3) shows that the prescribing rate to Māori patients reduced by 2.2, a 10.2% difference compared to the control arm. Column (2) shows the prescribing rate to Pacific peoples reduced by 3.1, a 13.3% difference compared to the control arm. Column (3) shows the prescribing rate to non-Māori and non-Pacific patients peoples reduced by 1.7, a 9.9% difference compared to the control arm. The intervention appears to have had an impact on all three groups, but this analysis does not test how the letters changed the behaviour of GPs who underprescribe to Māori and Pacific peoples. This question is explored in the following exploratory analyses.

Low prescribers to Māori were defined as those in the bottom 50% of all GPs in 2018 (not just among high-prescribing trial GPs). High prescribers to Māori were those in the top 50%. These calculations were only performed with a sample that included GPs who prescribe sufficiently to Māori to make the comparisons meaningful (at least 10 people of Māori ethnicity prescribed any medicine a month). Figure 5 shows GPs who were high prescribers overall, and therefore received the intervention, as above the orange horizontal line. A subsection of these GPs were also low prescribers to Māori patients are highlighted in blue. The following analyses investigate the effect of the intervention on these GPs highlighted in blue who are high prescribers overall but low prescribers to Māori and how this compares to the effect of the intervention on those GPs who are high prescribers to all peoples.

Figure 5: GPs’ prescribing percentile rank to all people and to Māori, January–December 2018.

Each dot represents an individual GP.

Columns (1) and (2) of Table 2 look at the impact on prescribing to Māori, using this split. Column (1), as expected, shows that the intervention reduced prescribing to Māori among past high prescribers (a statistically significant difference of 2.19, or 9.9%). Column (2) shows no statistically significant impact on past low prescribers.

Columns (3) and (4) replicate this analysis but look at the impact on prescribing to Pacific peoples. Column (3) shows that the intervention reduced prescribing to Pacific peoples among past high prescribers (a statistically significant difference of 2.59, or 10.9%). Column (4) shows no statistically significant impact on past low prescribers.

Table 2: Intervention effect on previous high and low prescribers to Māori and Pacific people.

This table presents the coefficients from ordinary least squares regressions at the GP level where the outcome measure is described in column header and the sample is limited to past high or low prescribers to Māori or Pacific people, as described for each column. GPs are only included if they prescribe sufficiently to the ethnicity of interest (at least 10 patients prescribed any medicine a month on average). See further notes to Table 1. Asterisks denote: *** p<0.001, ** p<0.01, + p<0.1

Discussion

Building on the research on interventions using social norms feedback to influence behaviour, the Behavioural Insights Team (BIT, a global social purpose company jointly owned by the UK Cabinet Office, innovation charity Nesta and their employees),³⁴ in partnership with the Health Quality and Safety Commission, PHARMAC and the authors, generated a letter intervention targeted at high-prescribing GPs in New Zealand.

Interventions to improve antibiotic stewardship have been evaluated in what is now a large and various body of evidence.⁶ This current study has highlighted that behavioural science adaptations of audit and feedback mechanisms with peer comparison (social norms) are likely to be effective in reducing antibiotic prescriptions in Aotearoa New Zealand. This builds on previous evidence showing that letters from a trusted source informing high prescribers of their rates of prescribing and providing comparison with their peers have positive effects on overprescribing in the UK²² (with effects reproduced in succeeding years²³), Ireland,²⁴ Australia²⁵ (reproduced again one year on²⁶) and the United States.²⁷

The results of this randomised controlled trial suggest that such an intervention was also effective in an Aotearoa New Zealand context. Our results show that after mailing such a letter to a sample of 1,214 high-prescribing GPs in all DHBs in the period September–December 2019, the average antibiotic prescribing rate in the control arm was 179 patients prescribed antibiotics per 1,000 patients prescribed any medicine. The average antibiotic prescribing rate in the intervention arm was 162, a relative difference of 9.5% (p<0.001).

The GPs who received the intervention arm of the trial were responsible for an average of 173.5 prescriptions of antibiotics in the four months subsequent to the intervention, whereas those in the control arm were responsible for an average of 186.8 prescriptions. This is a difference of 13.3, that is, a 7.1% difference between the intervention and control arms (p <0.01). This represents 8,000 fewer courses of antibiotics dispensed over the four-month trial period subsequent to the intervention. The results from this analysis indicated that the letters reduced both the overall number of antibiotic prescriptions and the number of people that were prescribed antibiotics.

If extrapolated to all high-prescribing GPs, this would translate to around 48,000 fewer scripts over a calendar year. This may be an overestimate if the intervention effect diminishes over time, or an underestimate if the intervention effect is larger in winter (our trial period excluded winter). Evidence from the follow-up to the Australian trial would indicate that this might be an underestimate.²⁶ This trial found the effect was bigger during winter months, that it attenuated during the summer months and that it reappeared during the next winter without a second letter being sent. This suggests that the social norms feedback letters had a long-term effect on prescribing, which is accentuated during periods of high prescribing (ie, flu season). We also do not know whether the effect would be larger or smaller if repeated over time, or if tied to ongoing quality improvement efforts by primary care organisations. This could be investigated in future work.

In investigating any potential spillover effects, the results included as Supplementary Material 1 show no significant change in paracetamol prescriptions (although, there was a trend suggesting a slight reduction, with p=0.06).

Effects for Māori and Pacific peoples

Critically, in an Aotearoa New Zealand context, the effects of the intervention must be evaluated in a context of well-described underprescribing of needed antibiotics to Māori and Pacific peoples.^14,15

Inequities in access to, quality of and outcomes of healthcare for Māori and Pacific peoples have been shown in decades of data and publications,³⁵ are clearly described in rulings of the Waitangi Tribunal as a violation of the Treaty of Waitangi³⁶ and violate the United Nations Declaration on the Rights of Indigenous Peoples, to which Aotearoa New Zealand is a (late) signatory.³⁷ Māori are proportionately more affected by conditions requiring proper antibiotic treatment,^8–13 and are proportionately less likely to be prescribed such treatment.^14,15 Any intervention targeting better antibiotic stewardship must be considered a failure if decreases in antibiotic prescribing lead to further reduced access to proper care for Māori.

Our results suggest that the intervention lowered the rate that intervention GPs prescribed to Māri, Pacific, and other-ethnicity patients. An Aotearoa New Zealand-specific adaptation of the intervention was the inclusion of visualisations of prescribing rates to all ethnicities, to ensure the intervention did not reduce access to needed antibiotics for Māori and Pacific patients. Exploratory analyses spurred by our findings investigated the effects of the intervention on those GPs who were high prescribers of antibiotics overall but low prescribers to Māori and Pacific patients. (Though note that our secondary analyses, which show the intervention reduced prescribing to Māori and Pacific patients overall, were pre-registered at https://www.socialscienceregistry.org/trials/5526.) Our results suggest high prescribers also reduced their prescribing to Māori and Pacific patients by a statistically significant amount, but that low prescribers of antibiotics overall did not reduce prescribing to Māori and Pacific patients (this latter analysis was not registered in the initial trial registration). Future work should examine the potential to target low prescribers to Māori and Pacific patients.

Strengths and limitations

Strengths of this study include the fact that this is the first study to our knowledge in which behaviourally informed interventions have been shown to be demonstrably effective in reducing GPs’ prescription behaviours in Aotearoa New Zealand, opening the door for other investigators to test whether or not other such interventions may be effective in Aotearoa New Zealand. The intervention is affordable, sustainable and iterable. The overall approach may also be useful in targeting other issues, such as underprescription or overprescription of other medicines.

This study was also the first globally, to our knowledge, that tested a behaviourally informed intervention that specifically sought to investigate health inequities in antibiotic prescription rates.

Limitations include the fact that, as in past trials, the investigators only had access to dispensing data and were thus compelled to use an instance of dispensing as a proxy for an instance of prescribing. Thus prescriptions that were not dispensed were excluded from analysis. However, any difference between prescribing and dispensing should be balanced across the intervention and control groups, so we are confident our results accurately capture changes in prescribing. We also lack information on whether dispensed antibiotics were taken appropriately after dispensing, a potentially important issue regarding the patient–clinician health-literacy dynamic and effective communication on the part of providers. These are issues which survey data have suggested may negatively affect Māori and Pacific health consumers’ ability to be effectively engaged in their treatment.³⁸

Analyses of the effects of the intervention on prescribing to Māori and Pacific peoples by past low prescribers were underpowered due to low numbers, which limits our ability to make strong inferential claims.

Finally, we could not measure the impact on health outcomes, meaning we could not directly test whether health outcomes worsened due to some doctors no longer prescribing necessary antibiotics. Nonetheless, the fact that our sample size was small when looking at past low prescribers to Māori and Pacific peoples suggests our intervention was well targeted, and the positive estimates in these analyses were promising. In addition, systematic reviews of interventions to reduce antibiotic overuse show no negative impact on treatment effectiveness or patient satisfaction.³⁸

Recommendations

This study has given strong evidence that letters that incorporate social norms feedback can reduce high prescribing, and indicative evidence that this intervention can simultaneously address low prescribing. The lack of power when looking at under-prescribers means that we cannot definitively claim that this occurred. However, we recommend that future work investigates how social norms feedback can be used to reduce underprescription.

It is also important to scale these insights to the control group and any new GPs that have been identified as high prescribers. Other medicines that are likely to be overprescribed should also be identified and the same intervention may be used to address this overprescription.

Conclusion

This study demonstrates that targeted interventions using social norms feedback can lead to a decrease in dispensing in high prescribers of antibiotics, and suggests that such an approach may be promising to address inequities in access to and use of antibiotics by Māori and Pacific peoples, who have historically been underserved by prescribers.

Supplementary material

• Supplementary Material 1: Paracetamol dispensed, control and intervention groups, September–December 2019