Implementation of artificial intelligence in healthcare in Aotearoa New Zealand: learnings from the diabetic retinal screening use case

People with diabetes are at risk of retinopathy that can lead to blindness if not detected and addressed early. Many countries have large-scale screening programmes for diabetic retinopathy for all people with diabetes. This generally involves retinal imaging every 2 years.¹

Due to low screening rates in some population groups in Aotearoa New Zealand, such as for Pacific peoples,² a new model of care was introduced in a pilot project in New Zealand. This involves taking “mobile” specialist ophthalmological cameras into seven Pacific primary care practices in Pacific communities and training kaiāwhina (lay healthcare workers) to take the retinal images, reducing the need for patients to travel to a secondary care centre during work hours for specialised screening.

To support the delivery of the new model of care, the potential for artificial intelligence (AI) tools to assist with streamlining the grading process and identifying those who need early follow-up by ophthalmology services was investigated. AI tools have been shown to be accurate in identifying diabetic retinopathy in research studies internationally^3–5 and in New Zealand.⁶ Additionally, AI tools have been introduced in a limited number of large-scale screening programmes overseas, such as in Singapore⁷ and Scotland.⁸

The initial process involved retrospective testing of two commercially available AI diabetic retinal screening (DRS) tools on existing New Zealand retinal images. Some challenges were identified during this initial testing, such as using human grading as the “gold standard” comparator (where human graders may, for example, be overly cautious in referring images for specialist review) and AI’s performance with differences in images. These issues could possibly be expected to some extent in any application of new AI tools in different contexts.

However, it has been in the implementation of the new model of care supported by AI that we have encountered substantial and sometimes unexpected challenges in this pilot that have delayed progress. Challenges with the implementation of technological innovations are not new, but the introduction of AI introduces further challenges that must be navigated. Examples include the risks of automation bias and algorithmic drift, as well as the potential for data biases, such as variations in retinal pigmentation, to impact performance for specific population groups. This paper outlines these challenges as they may relate to the implementation of any AI tools (not specifically about DRS) in the hope that our learnings may be useful for others going forward. The model of care itself was evaluated and reported separately to this viewpoint.

These challenges have been collected over an 18-month period as presented by the pilot project team for discussion in the steering group meetings. These have been collated and themed by the project evaluation lead (RD), AI lead (CKJ) and a co-chair of the steering group (RW). All three are members of the health system’s national Artificial Intelligence and Algorithm Expert Advisory Group, and digital health academics at The University of Auckland. Our findings have been shared and discussed with the remainder of the steering group, which includes Pacific Health team leads, primary care provider representatives, digital team members, DRS team leads and ophthalmology services.

Challenges

These challenges fall into four overarching areas: enabling digital systems, adjusting models of care, clinician readiness and AI appropriateness.

1.      Enabling digital systems

Many hospitals have had a myriad of older legacy software systems providing key parts or integration of parts of their digital system. Older technology may not support the addition of AI tools in ways that may not always be immediately obvious. In our case, this manifested in local standalone systems that did not allow image sharing with other services/parts of the organisation. This led to significantly more work than expected in two areas—one involving local “workarounds” by our digital team that required significant support and would not be scalable should the pilot project be successful and extended, and the other involving a new contract that required substantial privacy and security reviews and negotiations with a large multinational company that was not aligned with the size of the pilot but was required in case it should be successful and extended.

Ongoing frustrations arose from sharing the grading workload outside the local district and not being able to integrate the call back for the next screen back into local systems except through individual manual entry for each patient. These types of issues lead to failure due to legacy disconnected systems and are often only discovered once the project team starts analysing old and new process steps in detail. Where workarounds may have been considered acceptable in the past, in our forward-looking digital context staff expect that we will implement new tools in sustainable ways that do not add extra administrative burden.

A key benefit of AI is its scalability across the country. Although piloted in one district, the intention would be wider implementation if it were a success. However, disparate systems in operation across and within health services can pose challenges for scale-up following pilots. Across New Zealand there are a range of digital systems and vendor contracts including diverse patient management systems. Therefore, even if successful in a pilot, the processes developed may not be feasible in other systems or districts.

2.      Adjusting models of care

The real benefits of adding AI to our health services often come from allowing us to “do things differently”. We should be looking for those opportunities that do not just support current processes with the addition of AI but actually allow better patient and/or clinician processes and outcomes through the use of AI. In saying this, it then becomes obvious that the introduction of AI may also be a bigger change than just implementing the AI tool. Planning needs to allow for significant change management, including: mapping out the current process (from referral and image capture through to grading and referral or recall); consideration of the multiple systems involved (e.g., primary care, laboratory, DRS, ophthalmology, image storage, booking and scheduling, and recall systems); collaborating with all potentially impacted stakeholders on what the better process could be, including upstream and downstream impacts; applying good change management methods to plan and support staff through the change; and monitoring the impacts of change. For example, if the new model of care’s goal is increased screening rates, ensuring the capacity to grade images and treat the likely increase in discovered diabetic retinopathy is essential in the planning stage.

At every step of our pilot project, we uncovered further issues in trying to adjust processes to a new model of care. This will not be a surprise to those working in change management but may not be well known by those thinking they can switch on AI tools without fully considering the multiple clinical and administrative processes within which they are embedded. Addressing all these changes will be necessary and essential for success.

3.      Clinician readiness

Mixed reactions from clinicians and other staff to the introduction of AI tools are to be expected; however, it may not be for the reasons that you expect. Understanding the motivations behind the reactions is important to addressing them and bringing staff on board with the change required. There were those that worried about the security of their roles long term and the fear of “being replaced by AI”. Others may have felt threatened by change and the potential for more work or a greater burden being placed on them in their role, often due to previous experiences with new IT systems. There will always be those who feel a change is forced upon them without time for full consultation in this fast-paced technology environment, as opposed to where changes to clinical practice have traditionally taken many years. There was some degree of existing tensions in the relationships between groups for other unrelated reasons that may have impacted their ability to pull together in support of a change. These unrelated conflicts may even need to be resolved before the project can continue.

There are also likely to be concerns arising with the use of new tools and potentially new roles. Our kaiāwhina needed to be trained to use the new mobile cameras to take the images. But beyond just operating the cameras, it became clear that they also had to capture images of sufficiently high quality for the AI to work, and where the images were not exact they could not be used, and the patient may have had to be recalled for another session. In the early days, this was highly stressful for all involved and required more “at the elbow” support than initially intended across a number of disparate sites (community clinics) at the same time. This onsite support was required over several weeks before kaiāwhina felt fully self-sufficient and confident to instruct patients on how to position themselves in order to get the best images.

4.      AI appropriateness

Many around the world are finding that using AI tools in the real-world context of their health systems does not give the same results as the highly structured retrospective testing and research studies conducted. Our pilot was no different. Even though one of the tools was developed using New Zealand data, the testing still uncovered issues with the nuances of pigmentation and therefore clinical appropriateness for the target population. Evidence from recent clinical and diagnostic studies suggests that retinal pigmentation significantly impacts the performance of AI algorithms used in retinal screening.⁹ Even when AI models are trained on local data, they often get fine-tuned for specific populations. Further, the specific clinical processes, equipment and data acquisition can impact results and limit generalisability.

Another complexity around AI appropriateness arises from the low levels of AI literacy in the population and fears around AI and its use. Clear transparency around the use of AI in care and the use of patient data (images) in testing is needed, but how should we do this if patients do not understand AI to the level required to give their informed consent? Adding AI into health services and patient care needs to be done in a way that enhances patient trust and upholds the patient–clinician relationship. Beyond literacy we must also consider readiness, cultural views and preferences for human- versus machine-informed care. Understanding patient values and preferences is essential for appropriateness, particularly where AI recommendations may differ from traditional approaches. We have been attempting to conduct an ongoing programme of research into patient perspectives alongside our implementation work.

Discussion

There is significant potential for AI tools to support new models of care and deliver real benefits for the New Zealand health system and our population—but those preparing cases and implementation plans need to be realistic about the time and costs associated with these projects. The challenges we experienced are not unique to the AI context, with similar challenges experienced with the implementation of other digital technologies and innovations.¹⁰ Our experience with the DRS pilot project has shown that it is important to conduct “proof of value” evaluations within the real-world context the tools are intended to be used in. Challenges can be unexpected and time consuming, delaying outcomes and results, but unless these are addressed any further rollout will likely be impossible. While pilots and testing are often considered a preliminary step in AI implementation, in this case the pilot was essential. Without it, full-scale deployment would likely have failed, and, importantly, for populations already experiencing the greatest inequities in access to care.

Taking the time to follow a good process for testing, evaluating and ongoing monitoring of the performance of AI tools will be essential to ensure the beneficial impact of these tools is realised. But this means that implementations may take longer than expected and people should plan contingencies for this in their timeline and budget. Everything from staff training to equipment, AI tool procurement and sign-off can take considerable time and resources. This is especially the case with AI where expertise within the approval process may still be developing in this emerging field. One of the significant hurdles in implementing AI within DRS was the integration of AI tools into existing legacy systems. Many health organisations rely on outdated infrastructure and bespoke electronic health record platforms that were not designed with interoperability in mind. Therefore, a key early step in AI implementation involves evaluating existing infrastructure to ensure it can support real-time data flow between imaging devices, the AI tool and clinical records. In certain situations, upgrades to existing information technology (IT) platforms will be required for the AI to function, and therefore the project can end up being considerably more time consuming and costly than anticipated.

Our approach to testing and implementation of AI in DRS was impacted by the unique grading scale used in New Zealand. Most internationally available DRS AI tools are trained and validated against United States of America or United Kingdom–based standards, which do not map directly onto the New Zealand classification system. This limits the number of off-the-shelf solutions suitable for immediate use. Further, while AI algorithms may demonstrate excellent accuracy in controlled studies, success in real-world implementation depends heavily on how well the technology integrates into existing clinical workflows. In DRS, this means considering the entire patient journey from image capture to recall. As part of the project, multiple workarounds were required to address the paper-based grading process, recall management and return of results into the appropriate environment. Although the workarounds were able to achieve the desired goal, they were often manual and would prevent further scale-up. These implementation challenges highlight that successful AI deployment requires not just algorithmic accuracy but often requires workflow redesign and could require infrastructure investment to bridge the gap between new technology and established healthcare electronic systems. Such considerations should be taken into account when planning AI-related projects.

Organisations or project teams wishing to implement AI must foster a culture that acknowledges both the limitations of humans and the potential for AI to improve existing clinical practice. This requires moving beyond defensive attitudes about clinical expertise towards a collaborative approach where AI serves as a safety net and decision support tool. Research has demonstrated the ability of some AIs to meet or surpass human abilities.^5,11,12 Organisations or services must be willing to examine their current practices, review cases where there are discrepancies between human and AI and implement quality improvement processes that leverage both AI and human insights.

Staff hesitancy is expected with changes to practice or to the systems they operate within, but especially with new and emerging technologies where understanding and experience can be limited. But resistance may not be resulting from what you expect. Although there is still concern from many about the potential for AI to take jobs, there is also resistance arising from the change in already burdened workforces and fears for liability where AI may not get it right. Expecting our workforces, many of which are stretched to their limits, to accommodate and integrate AI into their service delivery is a big ask, let alone taking time for upskilling and training. We must prioritise supporting our workforce to embrace the changes that come with AI, increase their AI literacy and skills and confidence around AI use and provide clear governance processes providing reassurances. At the end of the day the successful implementation of AI in making a difference to the health outcomes of New Zealanders depends on the people using and supporting these tools.

This project is continuing to work through and address all of these issues. By working together with digital and clinical teams, as well as healthcare services and communities, we hope to ensure that not only the systems and technology support successful delivery of the AI-integrated DRS but that the workforce and communities have adequate support and resources to do so.

Conclusions

A proof of concept aimed at piloting a model of care for AI-integrated DRS to increase access to timely screening for Pacific peoples in New Zealand has faced a range of challenges. What appeared to many as a straightforward AI use case, currently in use overseas within public health systems and with growing evidence of tool effectiveness, proved not to be so in many ways. These challenges are not unique and need to be overcome to realise the benefits of AI across many use cases. Careful planning, adequate resourcing and organisational buy-in and support are all required for any AI implementation projects to succeed and for the benefits to the population and health system to be realised.