The impact of Individual Placement and Support on employment, health and social outcomes: quasi-experimental evidence from Aotearoa New Zealand

Mental health conditions and problematic substance use are the leading cause of health-related income loss among working-aged adults in Aotearoa New Zealand, accounting for 30% of the total.¹ For people most severely affected, income losses and employment penalties are large.^2,3 Many report a desire to work and a need for additional assistance to maintain or return to work.⁴ In an Australian national survey of people with a psychosis diagnosis, a third of all respondents and 45% of 18–34-year-olds reported lack of employment as one of their biggest challenges.⁵

Individual Placement and Support (IPS) is an internationally used, evidence-based, voluntary approach to helping people receiving mental health and addiction treatment who want to work into employment. The programme logic is to “place-and-then-support”—job search is rapid, and training and support is provided as needed once participants are in employment. This contrasts with commonly practiced vocational supports that assume that training, job preparation activities or sheltered work is needed before employment.³ IPS has been available in parts of Aotearoa New Zealand for nearly 2 decades, funded by health regions and the Ministry of Social Development (MSD), but does not have national coverage.^6,7

With IPS, an employment consultant is included in the clinical team and employment support is integrated with mental health and addiction treatment. Assistance the employment consultant provides includes: help finding jobs that fit a person’s preferences and skills; working with local employers to identify job opportunities; and “benefits counselling” to explain the impact of paid employment on income support payments and facilitate connections with local Work and Income services. Personalised supports continue for as long as the person wants, including support to keep their job, find another job or advance their career. Mental health and addiction practitioners may refer people to the employment consultant, or people can self-refer. Work experience, diagnosis, symptoms, current or previous substance use and convictions do not affect access.³

There is evidence from overseas randomised controlled trials (RCTs) that IPS participants are more likely to obtain competitive employment than those in control conditions (often train-first vocational programmes or non-integrated supported employment). Rate ratios across seven meta-analyses range between 1.6 and 2.5.³ Evidence on outcomes such as mental health and quality of life is still developing.^3,8 Recent Aotearoa New Zealand research has found that IPS programmes can achieve employment outcomes at and above international benchmarks, but the absence of an RCT means that Aotearoa New Zealand evidence on efficacy is limited.⁶

The aim of this paper is to examine effects of IPS on employment, income, health, education and justice outcomes in Aotearoa New Zealand.

Methods

Data were sourced from Stats NZ’s Integrated Data Infrastructure (IDI). The IDI is a database containing linked individual-level microdata.⁹ Data come from a range of government and non-government administrative and survey sources, and are probabilistically linked and de-identified. For more information about the IDI, see https://www.stats.govt.nz/integrated-data/integrated-data-infrastructure.

The base study population was adults who had face-to-face contact with a publicly funded secondary mental health or addiction team over the 3 years to 31 March 2018, as recorded in Programme for the Integration of Mental Health Data (PRIMHD). Face-to-face contact included all contacts except those coded as audiovisual, other social media/E-therapy, telephone, SMS text messaging, written correspondence or unknown/other location.

The study population was split into two cohorts: those who participated in IPS and those who did not. The IPS cohort was those who commenced IPS in the 3 years to 31 March 2018 in the former Taranaki, Lakes, Waikato, Auckland or Counties Manukau district health boards (DHBs). These areas had established IPS services at that time. A matched cohort of people who did not commence receiving IPS over the same period was identified from the same regions. The approach of matching within regions with IPS was chosen so that matched controls faced the same local labour market and mental health service delivery conditions and drew from the same location-specific demographic groups. Potential for overstatement of effects due to positive selection bias as a result of drawing IPS participants and matched controls from the same regions was assessed by examining sensitivity to drawing matched controls from areas with no IPS.

Matching methods are described below. To approximate study populations in RCTs, participants aged over 62, in employment, or receiving Accident Compensation Corporation (ACC) weekly compensation at referral were excluded. Effects were estimated for all participants, for Māori participants, and separately for males and females.

The following outcomes were examined over a 3-year follow-up from IPS referral.

• Employment and benefit receipt. Inland Revenue (IR) data were used to identify months with paid employment, defined as periods receiving income from wages or salaries or self-employment. MSD data were used to identify months in receipt of main working-age benefits (Jobseeker Support, Supported Living Payment and Sole Parent Support).
• Income and transfers. IR and MSD data were used to calculate real (NZ$ 2021) income. Total net income was examined, as well as net income from: employment (including self-employment); MSD benefits; and other transfer payments such as Working for Families tax credits and ACC weekly compensation. IR data were used to calculate taxes paid. Net government transfers were calculated as net income from benefits and other transfers, less taxes paid.
• Justice. Department of Corrections data were used to identify time spent serving any corrections sentence, and time spent serving custodial sentences.
• Study and qualifications. Time participants spent enrolled in education or training and highest qualification based on the National Qualifications Framework (NQF) were established using Ministry of Education data.
• Health service usage. PRIMHD data were used to identify face-to-face contact with mental health or addiction teams and IPS teams, and admis­sions to inpatient psychiatric facilities. Data from the Ministry of Health’s National Minimum Dataset (NMDS) were used to identify public hospital admissions. These were categorised into mental health-related, non-mental health-related, and hospitalisations for self-harm. National Non-Admitted Patient Collection (NNPAC) data were used to identify emergency department attendances.

IPS participants were matched to similar people who did not participate using propensity score matching (Appendix 1) and selected exact-match criteria.¹⁰ Matching variables included a range of variables that could influence selection into IPS and/or the outcomes of interest, including age, gender, level-1 ethnic groups, number of children and age of youngest, neighbourhood deprivation, rural–urban status, past mental health diagnoses and service use, co-occurring health conditions, having a private or commercial driver’s licence, previous participation in IPS before the start of the study period, type of benefit received (if any), the percentage of time since age 18 receiving benefit and time spent overseas (see Appendix 2). Matching variables also included the employment, income and transfer, education, justice and health service usage measures listed above derived for a 3-year look-back period (excluding the 2 months immediately prior to referral).

One-to-one nearest neighbour matching on propensity score was used (with replacement so that one non-participant could match with multiple participants), with exact matching on the calendar quarter of referral, broad benefit type, whether the person identified as Māori or Pacific peoples and whether the person’s DHB was in Auckland. IPS non-participants were considered a potential match in each calendar quarter they had a face-to-face meeting with a mental health or addiction service.

The “average treatment effect on the treated” was estimated comparing mean outcomes for the matched groups, using a weighted two-sample variance formula.¹¹ This method accounts for bias caused by repeated matches in matching with replacement. We tested different matching criteria before settling on a caliper width of 0.2 times the pooled standard deviation of the logit of the propensity score, recommended by Austin (2011).¹²

Because we examined multiple outcomes across multiple populations, some estimated effects could be statistically significant by chance. We accounted for this by assessing statistical significance using false discovery rate adjusted q-values.¹³

The study was reviewed by the MSD Research Ethics Panel. This included review of the use of de-identified data for the purposes of the research without specific consent. IDI data access was approved by Stats NZ. Data were accessed from the October 2022 refresh of the IDI. Data were extracted and analysed using SAS Enterprise Guide version 7.1.

Results

For the 1,839 IPS participants in the study population there were 28,797 people in the same former DHBs who could act as potential controls. Matches could not be found for 9.8% of participants (Appendix 3, Table 1). Those unmatched were dropped from the study population. In total, 1,659 participants were matched to 1,503 controls. Standardised mean differences in propensity scores were all less than 0.25 and variance ratios were between 0.5 and 2, within the range recommended by Rubin (2001).¹⁴

Appendix 3 Figure 1 shows the common support between the matched samples—propensity scores were almost identically distributed. Characteristics were also similar (Table 1; Appendix 3 Table 2–5), suggesting the matched samples were well balanced. Unmatched participants had higher propensity scores, indicating difficulty finding matches for participants with characteristics that made them very likely to participate. They also had characteristics suggesting higher employment barriers, including being more likely to have a diagnosis of schizophrenia or bipolar disorder, to have been prescribed anti-psychotic medication and to have had mental health-related hospitalisations.

Figure 1 shows monthly employment rates for the matched groups. Matched IPS participants had significantly higher rates of employment than matched controls. The gap was largest after around 1 year and narrowed towards the end of the 3-year follow-up. Effects on employment while on a main benefit were largest in the first year; effects on employment while independent of main benefits were largest in the second year (Appendix 3 Figure 2–3).

View Figure 1, Table 1–2.

Table 1 shows cumulative outcomes in the 3 years post-IPS referral for the matched groups. The IPS group had significantly greater time in employment. They spent almost three more months employed (2.78, 95% CI 1.85, 3.70). Around a third of this was increased employment while on main benefits, with the remainder increased employment while independent of main benefits. Average time on main benefits was not significantly different.

Income from all sources was NZ$4,221 higher for the IPS group (95% CI NZ$899, NZ$7,542). There was an increase of NZ$5,056 in employment income. IPS participants paid NZ$753 more in tax.

The IPS group was more likely to gain a qualification at NQF level 2 or above after starting IPS (2.17 percentage point increase, 95% CI 0.80, 3.54). There was no significant difference in the time spent enrolled in education and training. Differences in time serving corrections sentences were not statistically significant.

The IPS group had more face-to-face contacts with mental health and addiction teams and, as would be expected, more contacts with IPS teams, especially in the 12 months following referral to IPS (Appendix 3 Figure 4 and 5), and more mental health-related inpatient stays and mental health service crisis contacts. Emergency department visits, hospital discharges for self-harm and non-mental health hospitalisations were not significantly different.

Estimated effects for Māori IPS participants were similar in direction and scale to the overall results; while estimated effects on employment, months with face-to-face contacts with mental health and addiction teams, and the percentage with mental health service crisis contacts were significant, other estimates were non-significant, which may reflect the increased uncertainty in estimation due to smaller participant numbers (Table 2).

Effects differed for people who identify as male versus female (Appendix 3 Table 6 and 7). Females had no increase in income from all sources, despite larger estimated effects on employment and employment income than for males. This was due to reduction in net government transfers. Females, but not males, had increased likelihood of having mental health-related inpatient stays and reduced likelihood of having non-mental health-related inpatient stays.

In our sensitivity test, drawing matched controls from areas with no IPS service at the time (and therefore less potential for selection bias), matches could be found for 83% of IPS participants (Appendix 3 Table 1). Results were similar to the main analysis (Appendix 3 Table 8).

Discussion

This paper examined the differences between matched IPS participants and non-participants in employment, income, health, education and justice outcomes in Aotearoa New Zealand.

IPS participants had more employment income, longer employment duration and a higher rate of employment (which reduced over time as employment in the control cohort increased). This is consistent with international evidence.^3,15 IPS participants also had higher total income, after accounting for losses of benefits and other transfers and taxes, and gained more qualifications. Few previous studies have examined effects on these outcomes.^3,7 Total income was not higher for females, however, because higher employment income was offset by lower transfer income. This result is concerning because income support policy aims to ensure income is higher in employment if people receive the in-work benefits and other transfers they can qualify for. It suggests a need to strengthen benefits counselling, and/or improve design and delivery of income support.

Actively looking for work brings stresses that may increase the need for mental health support,¹⁶ as might trying out jobs to see if they fit.¹⁷ IPS is intentionally designed to support mental health and employment needs together, recognising these potential stressors. An increase in mental health support in the transition to employment is a function of the programme design. On average, we find that IPS participants had similar levels of previous face-to-face contacts with mental health and addiction teams. In the follow-up they had more face-to-face contacts, mental health-related inpatient stays and crisis contacts. One possible interpretation is that this reflects IPS operating to increase engagement with mental health and addiction treatment and care in the transition to employment, resulting in people being more readily able to access needed services, and/or clinicians engaging more proactively with IPS participants. That females but not males had a higher likelihood of having mental health-related inpatient stays may reflect the additional stressors that transitioning to employment brings when people have primary care of children and/or flow on effects of the lack of income gain for females suggested by our results.

Another possible explanation is that the results partly reflect uncontrolled selection effects whereby people with greater need for mental health and addiction services were more likely to be referred to and participate in IPS. Overall, our findings do not show strong support for this explanation, with the matched IPS and control groups having broadly similar levels of prior mental health and addiction service use.

A limitation of this study is that we were not able to measure mental health directly. Evidence on the effects of IPS participation on mental health symptoms and broader wellbeing is limited. One RCT reported no substantive effects on psychiatric symptoms or self-reported quality of life despite IPS participants having more contacts with mental health services than the control group, and more use of emergency care and psychiatric evaluation.¹⁸ Meta-analysis of the few studies with results for quality of life, global functioning and mental health suggests positive effects, but with confidence intervals that include the null, and heterogeneity between studies.⁸ However, maintaining employment is a good marker for functional recovery. Research to better understand the interactions between IPS, engagement with mental health and addiction services, and mental health and quality of life would enhance knowledge of recovery and broader wellbeing.^3,8,19

As far as we are aware, no previous studies have examined efficacy and effectiveness of IPS for Indigenous peoples. Despite increased uncertainty in the estimation due to smaller participant numbers, we find significant increases in employment and two measures showing increased mental health service engagement for Māori. Positive effects on employment are notable given the high levels of labour market and mental health disadvantage.^4,20,21 For Māori wellbeing, sustainable employment and economic prosperity and security sit alongside a range of culturally-valued aspirations, ways of working and outcomes.²² While our results suggest that IPS provides effective employment support for Māori, further research is needed to identify, and support strengthening of, the cultural principles underpinning implementation for Māori.^23,24 Estimated programme effects for IPS compare favourably with those for other employment assistance.²⁵ However, it is not possible to compare effectiveness of IPS with that of programmes for which impact evaluation evidence is sparse, including Kaupapa Māori initiatives.

Our results show that people with mental health conditions and problematic substance use who receive employment support made available together with mental health and addiction treatment have more employment, gains in qualifications and more independent income when compared with similar people who do not receive this support. These are outcomes that many people affected by mental health conditions and problematic substance say they want.⁴ Expanding access to evidence-based integrated employment support has been recommended in several reports,^4,22,26–27 including a 2023 framework that identifies integrated employment support as a core component to be offered through secondary mental health and addiction services.²⁷ Despite recent expansion, IPS is not available in all regions, and is not available at sufficient levels to meet demand in others, with limited availability in addiction services.^6,7 To achieve national scale-up, a sustainable cross-government funding stream for IPS programmes, national and local-level co-ordination, and implementation support systems would be needed.²⁸

A particular strength of our study was the ability to examine outcomes across a range of domains beyond employment using linked administrative data. These data allowed a longitudinal perspective, avoided non-response and recall bias, and provided a comprehensive sample of the population of interest. Despite this, sample size was not large enough to examine impacts for Pacific peoples or other policy-relevant sub-groups. Recent expansions of IPS means that numbers will be large enough to include these sub-groups in future, and to examine newer services. These have had more implementation support to improve fidelity to IPS evidence-based practices. Positive impacts on employment may be larger as a result.

Without an experimental design, this study is subject to potential bias from unobserved factors that influence selection into IPS. These could include caring responsibilities, motivation, employment preferences and experiences of colonisation, trauma and discrimination that may affect engagement with government programmes. Nonetheless, quasi-experimental designs are a useful tool when RCTs are not available, and results from propensity score matching can replicate RCTs.²⁹

A further limitation is that matches could not be found for 9.8% of IPS participants overall and 12.4% of Māori participants. Unmatched participants were more likely to have a diagnosis of schizophrenia or bipolar disorder, to have been prescribed anti-psychotic medication and to have had mental health-related hospitalisations than matched participants. Meta-analysis of RCTs shows that IPS is effective in increasing employment irrespective of diagnostic, clinical, functional and personal characteristics. The effect also appears to be greatest for populations with diagnoses of mental health conditions such as schizophrenia and bipolar disorder, and for those with lower symptom severity independent of diagnosis.³⁰ This suggests that IPS would have positive effects for unmatched participants, but it remains uncertain whether those effects would be larger or smaller than those for matched participants.

This investigation suggests IPS supports employment and improves income and qualifications for people in contact with Aotearoa New Zealand mental health and addiction services. Combined with international evidence, this suggests that expanded IPS availability would be beneficial. More research to understand the effects on mental health symptoms and broader wellbeing and support of cultural responsiveness is needed, alongside repeated impact evaluation.

View Appendices.