Patient experience surveys are vital in the twenty-first century: let’s put some myths to rest

In 2016, frustrated researchers at the RAND Corporation were moved to publish a debunking of “the most prevalent myths” riddling the critiques of patient experience survey data used to measure and improve healthcare quality.^1,2

In the Aotearoa New Zealand context, criticisms of patient-reported experience data are widespread, and equally frustrating. Few of these criticisms are, however, published in the peer-reviewed literature, possibly for the reasons we outline below in debunking these myths—they don’t hold sufficient water. However, this received wisdom has come to our attention, and tends to take the form of the following chestnuts, many common to other contexts, here deployed without evidence: that Aotearoa New Zealand’s patient experience survey response rates are too low to tell us anything of use, and that consequently measures are not robust, valid or reliable. That surveys undersample and are not valid or reliable for Māori and Pacific peoples.³ That surveys provide data, sometimes insights and opportunities, but no solutions.⁴ That none of this relates to improving outcomes.

Let’s put some of these myths to rest, but first: why involve patient experience and the patient voice in design and delivery of care in the first place?

A strong consumer and whānau voice are critical in a twenty-first century health system—and now a statutory requirement

The consumer health movement is not new, with its roots in the de-institutionalisation of mental health,^5,6 rise in disability rights movement^7,8 and special interest consumer groups.⁹ Similarly, the calls to better understand if and how these movements, and the involvement of consumers more generally, have led to a safer and higher quality health system remains a perennial subject.^10,11,12 In a 2018 narrative review in BMJ Quality & Safety, 11 leaders in patient safety from the US and the UK wrote, “overwhelming evidence indicates that collecting patient feedback and including patients as equal partners in their care supports improvement in both patient experience of care and clinical outcomes.”¹³ A systematic review of 55 studies published in the BMJ in 2013 found “consistent positive associations between patient experience, patient safety and clinical effectiveness for a wide range of disease areas, settings, outcome measures and study designs … patient experience is positively associated with clinical effectiveness and patient safety, and support the case for the inclusion of patient experience as one of the central pillars of quality in healthcare.”¹⁴ Another 2014 review found, “better patient-reported care experiences are often associated with other aspects of healthcare quality, specifically:

• higher levels of adherence to recommended prevention and treatment processes
• better clinical outcomes
• less unnecessary healthcare utilisation.”¹⁵

Yet despite evidence for the benefits of patient and public involvement in designing, planning and co-producing healthcare services, and a broadly supportive policy context in the UK and internationally, a 2016 BMJ Quality & Safety paper found that progress in improving consumer engagement in production of healthcare services has been “patchy and slow and often concentrated at the lowest levels of involvement.”¹⁶

Te Tāhū Hauora has championed consumer and whānau engagement since its inception in 2010, and this work has now culminated in the development and embedding as secondary legislation of the Code of Expectations for Engagement with Consumers and Whānau for health entities¹⁷ (the Code) in section 59 of the Pae Ora (Healthy Futures) Act 2022.¹⁸

The Code sets the expectations for how health entities must work with consumers, whānau and communities in the planning, design, delivery and evaluation of health services. Between 2021 and 2022, Te Tāhū Hauora alongside the Health Transition Unit developed the Code, and under the Act there is now a statutory requirement for health entities to give effect to it, and to report annually on how it has been applied.

Part 2.2 of the Code states that health entities must use “lived experience, including consumer experience data to inform improvements in health services with a focus on reducing health inequities, particularly for Māori, Pacific peoples and disabled people.”¹⁹

For health service providers, use of patient experience survey programme data is a critical element in fulfilling this statutory expectation. Te Tāhū Hauora provides an implementation guide²⁰ for health entities to learn how they can implement the Code, including use of survey programme data.

Satisfaction versus experience—there is a critical difference

The first critical distinction to make in understanding patient-reported survey data is the distinction between patient satisfaction and patient experience.

Both satisfaction and experience surveys include multi-choice items.

Patient satisfaction surveys ask questions related to “How did we do?” with responses from “very poor” to “very good”. However, patient experience surveys ask questions related to what happened, with responses such as “always” to “never”. From this, we can derive actionable knowledge of what did or did not actually happen in a healthcare experience for large groups of people, rather than a non-actionable measure of how people felt, in a given place at a given time, for in many ways unknowable reasons.

Consider your own personal mood when confronted with the HappyOrNot™ terminal at an airport baggage carousel. You have a choice ranging between a green smiley face and a red angry face. But how much is your satisfaction with your baggage handling mediated by your prior expectation of a terrible experience, for example, or a particularly awful flight, or some fantastic news you received when you turned airport mode off on your phone at disembarkation?

Patient satisfaction responses are considered subjective and biased because satisfaction is personal and related to expectations and other unknowable influences. It is not an objective reflection of the presence or quality of a service or aspect of that service.²¹ Patient satisfaction responses also tend to be overwhelmingly positive and are easy to manipulate with framing of questions.¹⁹

Thus, rather than a question like “How did we do?”, patient experience surveys ask questions such as “In the last 12 months, was there ever a time when you wanted healthcare from a GP or nurse, but you could not get it?” A large number of responses thus gives a sense of barriers of access to care, and who experiences these. Another example, from the hospital inpatient survey, is, “Were you told what the medicine (or prescription for medicine) you left the hospital with was for?” This question reflects quality of discharge and communication, and using the experience explorer (see below), and other questions around discharge, services can easily see if they are providing the same quality of discharge information to all people, and how well they are doing in comparison with other parts of the country.

Patient experience is a more objective construct and survey questions that address specific aspects of care can prompt specific actions to generate improvement. Patient experience is also an intrinsic element of most generally accepted frameworks of healthcare quality, including the Triple Aim, a framework developed by the Institute for Healthcare Improvement that has been adopted as a set of principles for health system reform in multiple countries and organisations worldwide,^22,23 the Quadruple Aim (building on the Triple Aim with the inclusion of workforce experience of providing care)²¹ and the National Academy of Medicine’s early and seminal six domains of quality developed in the US.²⁴ For these reasons we believe patient experience surveys to be superior to satisfaction surveys.

But what of the accusations that Aotearoa New Zealand’s patient experience survey programme’s response rates are too low and sample sizes are too small to make accurate judgments, and that these samples are skewed in composition due to the nature of those the survey reaches, and who chooses to respond? What of the accusation that the length of the survey causes people to give up on it?

The accusations don’t hold water.

Response rates, who responds and how much is enough? Scope and scale of the Aotearoa New Zealand patient experience survey programme

Te Tāhū Hauora’s patient-reported survey programme²⁵ is in fact the second largest government survey programme in Aotearoa New Zealand we are aware of (larger than the New Zealand Household Disability Survey and the New Zealand Health Survey, and second only to the New Zealand Census). The programme surveys the experiences of adult inpatients in hospital settings and adult primary care patients (an adult hospital outpatient survey has been more recently established). Combined, the former two surveys collected data from over 154,000 respondents between May 2022 and February 2023.

The adult hospital inpatient experience survey (established 2014) and the adult primary care patient experience survey (established 2016) were also reviewed and substantially refreshed in 2019 and 2020.²⁶ Cognitive testing was performed to understand and improve how patients understand and interpret questions and instructions in the survey and involved multiple in-depth interviews, with a particular focus on Māori and Pacific peoples’ responses.

The survey is a major repository of data. During the early period of the pandemic many national programmes, including survey rounds, were paused as workforce focussed on the pandemic response. A one-off COVID-19 specific survey to assess impacts of the pandemic response on patients’ experience of access to primary care during Level 3 restrictions was conducted,²⁷ and surveying began again with the refreshed instruments. The data repository for the hospital inpatient survey now contains about 35,000 responses from August 2020 (after the refresh during pandemic restrictions). The primary care patient experience survey now covers over 90% of general practices, with data from 315,000 responses since the 2020 refresh. On average, 3,000 hospital inpatients respond to the adult hospital inpatient survey every quarter, and 35,000 primary care patients respond to the adult primary care patient experience survey every quarter. Every quarter this data pool grows, keeping pace with a changing system and context.

So much for response size. But what about response rate? Booker and colleagues cite comprehensive reviews of comparable surveys in primary care internationally and found response rates vary between 10–61%.^28–31

The hospital inpatient survey’s response rate compares well, in the middle of the pack at 25%.

The primary care patient survey response rate, at 16–17%, is not so big, so we compensate for this by ensuring the large sample and comprehensive reach to practices (over 90%) mentioned above.

Response rates are important for three reasons, which we deal with in turn as follows:

1. Too few responders each time the survey is run can give unstable results stemming from natural variation over time alone. However, we know the survey is reliable from the consistent results we receive for the primary care survey quarter-on-quarter. As Figure 1 shows, results are consistent over time and natural variation is not creating unstable findings.
2. An unrepresentative group may be responding. We deal with this question in more detail below, but the results of focussed work in improving Māori response rates mean Māori response rates to the primary care survey now average 17% annually, while non-Māori/non-Pacific response rates now average 16% annually.
3. The group who responds may be attitudinally unrepresentative compared to those who choose not to respond. In order to understand if this was the case, we conducted a peer-reviewed study published in the New Zealand Medical Journal in 2018.³² We discuss the findings in more detail as follows.

View Figure 1–2.

Criticisms of so-called “opt-out” surveys draw attention to non-response bias, where those who choose to respond to a given survey are systematically different in some key way to those who choose not to. To understand if non-response bias existed in this large collection of survey data, a study was performed comparing the responses from a sample of initial hospital inpatient survey respondents (n=910) with the responses of a sample of non-responders who were followed up and invited again to take the survey (n=163). The study found no significant differences between the two groups, and that “responders to follow-up have similar experiences of inpatient care in New Zealand to initial responders.”³¹ There are of course limitations to this study, including potential bias in the follow-up responder group limiting the ability to ascertain the true extent of non-response bias.

So, there is evidence that non-response bias does not exist, and it is furthermore true that response rates per se are less important than representativeness—a large response rate from a group who are very different from non-responders will produced more biased results.³³ Hence the importance to the Aotearoa New Zealand patient experience survey data of Amhed et al.’s finding that “response rates are only weakly associated with non-response bias in surveys that adhere to high standards of survey methodology”.^33–35 Our patient experience survey adheres to best-practice survey methodology promulgated by Statistics NZ and international researchers^36,37 to minimise attitudinal response bias. Measures are tested and validated, and weighting is used to adjust for patient characteristics that are likely to affect individual perceptions and expectations to enable comparison across healthcare providers.³⁸ Representativeness we discuss below.

The last point of contention (again unevidenced), that the length of the survey causes many respondents to give up, is also regularly monitored. The completion rate of the surveys is over 90%, suggesting the majority of those who start the survey finish it, but the programme is constantly looking to improve. Completion time, drop-outs, item non-response and patient comments about the survey are regularly monitored. This helps identify questions that people find hard to answer (issues of cognitive load and response burden), which is a more important issue than the number of questions per se.³⁹ As part of the annual review process, we also analyse responses so we can remove questions that are tightly linked to others. Another way we are reducing length is to move questions into annual modules to be asked once a year rather than every quarter.

The primary focus of the review and a constant in the evolution of the programme has been on improving participation for Māori and Pacific peoples, including increasing their response rates. We look next at whether this has succeeded.

Representativeness in the patient experience survey of Māori and Pacific peoples’ experience

Some have claimed that patient experience surveys are not valid or reliable for Māori and Pacific peoples due to low response rates. The significant work invested into improvements in this area has meant that, in the four quarterly surveys administered between early 2022 and early 2023, the historical equity gap in primary care survey response rates between Māori and non-Māori/non-Pacific peoples has disappeared.

Māori response rates now average 17% annually, while non-Māori/non-Pacific response rates now average 16% annually. Implementation of the successful improvements seen in the primary care survey are now in process for the hospital inpatient survey.

There were two key aspects of increasing participation rates: increasing the number of patients who are invited to take part, and increasing the proportion of patients who take part once invited.

The primary care survey now uses representative sampling to ensure a sufficient count of Māori and Pacific people are invited to the survey. In practice, this means a two-week sample of people identifying as being of Māori or Pacific ethnicity, and one week of all other ethnicities.

A range of other methods to increase both the numbers invited and the proportion of those who subsequently take part have addressed the entire pathway of survey response, including email address collection rates, culturally appropriate invitations, testing of the survey instruments with focus groups of Māori and Pacific peoples, and using a zero-data rated website, which allows access without using respondents’ mobile data allowances.⁴⁰

Patient experience for quality improvement

Can patient experience in fact be acted on to improve the quality of services? It is clear to most that data help identify issues and areas that can benefit from further exploration and development of insights that prompt quality improvement action, but where do providers start in using these data for quality improvement? Te Tāhū Hauora provides resources specifically designed to assist providers in using survey results for quality improvement in both adult hospital inpatient and primary care settings⁴¹ to assist in the development of strategic, actionable insights and guide improvement.

At the national level

Results from the surveys are also currently used across the country in multiple ways. At the national level these data are used to publicly report on the quality and safety of services, for example, through the annual Window on the quality of health care reports published by Te Tāhū Hauora. The latest Window report published in June 2023⁴² provided an update on effects of the pandemic on our health system and included a focus on experience of care for disabled people during pandemic-affected periods.

Survey feedback also informs policy and monitoring of how health services are performing. The interim Health and Disability System Review drew on patient experience survey data to highlight areas for improvement in the health system restructure process, for example in reported barriers to accessing primary care.⁴³ Data from the surveys where respondents report prescription cost as a barrier was used in the debate on Budget 2023’s change to remove prescription costs. These insights to inform high-level policy can only be obtained through nationally consistent, robust and valid surveys.

At the local level

At local and regional levels, providers use survey data to monitor what is working well and what could be improved. For instance, during COVID-19 lockdowns, some hospitals were reporting their survey data to staff to show that, despite all the challenges associated with providing care in personal protective equipment and lockdown, patients were continuing to report positive experiences and were grateful for the care provided. Examples of specific quality improvement work drawing on survey findings are those which have been pursued at four district health boards (DHBs)—Northland, Waikato, Bay of Plenty, and Nelson Marlborough, addressing patient awareness of medication side effects and condition management post-discharge.^44,45 Survey data for quality improvement work has also been embedded in the annual System Level Measures Improvement Plans of alliances in every district from 2016 to 2022. Examples are available from Auckland, Waitematā and Counties Manukau alliances for the 2020–2021 and 2021–2022 financial years.^46,47

Health providers participating in the survey can access their own local results through a secure online portal. These dashboards enable people to access, cut and use the data in different ways, by local regions, demographic variables, question and domain, according to their needs and interests.

Providers and researchers interested in investigating the data are able to and encouraged to get in touch (contact, in the first instance, the corresponding author Catherine Gerard).

Both the adult primary care patient and adult hospital inpatient surveys now also have publicly available data explorers^48,49 to make access to and investigation of experience data public, easy and intuitive. The explorers also allow data to be filtered by survey topics and questions, locations, periods and demographic elements of responders such as age, gender, ethnicity and disability status.

See Figure 2 for an example screenshot of the adult primary care patient experience data explorer, showing most-positive responses to the question: “Did the healthcare professional involve you as much as you wanted to be in making decisions about your treatment and care?”

Conclusion

Te Tāhū Hauora has administered national patient experience surveys since 2014. Since its inception, the programme has continuously grown and evolved, delivering valid, reliable, relevant, systematic and practical patient experience surveys and resulting data reflecting patient experiences of specific aspects of care. The patient-reported data and trends are translated into relevant metrics and indicators, easily accessible, to give providers a guide to how well their services are being delivered to their populations.

Te Tāhū Hauora patient experience survey data supplies coordinates and a compass to guide more patient-centred care. These surveys place patient feedback at the heart of quality improvement and health system responsiveness. The myths debunked above, we suspect, derive simply from suspicion premised on unfamiliarity.

To this we would add: thanks to the survey, we have heard from a representative sample of New Zealanders about the experience of healthcare they have had. We have this data, and we have a statutory imperative. We are now legally and ethically^50,51 bound to listen and to use the results for improvement.