Comparison of two prognostic calculators predicting functional independence of patients with severe traumatic brain injury at discharge from rehabilitation services

Traumatic brain injury (TBI) is defined as altered brain function or other evidence of brain pathology caused by external force.¹ It is a leading cause of death and disability among individuals aged 40 years and under worldwide. The incidence rate in New Zealand is high, estimated at 790 per 100,000 people, based on a district study conducted in New Zealand from 2010 to 2011.² Each year, an average of 609 people are hospitalised with TBI, with 5% classified as severe TBI (sTBI).

TBI severity can be classified based on the functional Glasgow Coma Scale (GCS) or the anatomical Abbreviated Injury Scale (AIS). The GCS scores range from 3 to 15, with 3–8 indicating sTBI, 9–12 indicating moderate TBI and 13–15 indicating mild TBI.³ The Injury Severity Score (ISS) is a well-established medical score to determine trauma severity, ranging from 1 to 75.⁴ If an injury is untreatable (i.e., AIS of 6), the ISS score is automatically set at 75. There are 137 AIS codes that related to injuries of the brain (region 1, brain/cranium) including that of brain parenchymal injury. The severity scores are categorised as 1–2 for mild, 3–4 for moderate and 5–6 for sTBI.^4,5 A recent New Zealand cross-sectional study of 609 participants found that half of sTBI patients were admitted to intensive care, one-third required neurosurgical intervention and one-third died.⁵ Current TBI severity classification methods may not fully capture the multidimensional nature of injuries, limiting accurate outcome predictions.⁶

Recovering from sTBI is a continuous process of functional recovery.⁷ In a longitudinal database study of 17,470 patients enrolled with acute moderate or severe TBI, Kowalski et al. found that 80% of patients discharged in a comatose state regained consciousness during follow-up in inpatient rehabilitation and 40% achieved semi- or full-functional independence.⁸ This indicates that there is a chance for full recovery of function following admission with moderate-to-severe TBI. A multicentre study, the Transforming Research and Clinical Knowledge in Traumatic Brain Injury (TRACK-TBI) consortium, discovered that moderate-to-severe TBI patient outcomes continued improving from 2 weeks to 1 year post-trauma. Fifty-two-point-four percent of the sTBI patients recovered an independent level of functioning by 12 months post-injury.⁹

Identifying patients with a good recovery trajectory is a focus of clinical care and research trial design. Two recognised tools predict TBI outcomes based on presentation: Corticosteroid Randomisation after Significant Head injury (CRASH) and the International Mission for Prognosis and Analysis of Clinical Trials in TBI (IMPACT).^10,11 Clinical predictors of TBI outcomes include: age (>40 years, worsening with increasing age); initial GCS post-resuscitation; hypotension; hypoxia; pupil size and reaction to light; intracranial pressure; subdural, extradural or subarachnoid haematoma; and presence of comorbidities.¹² The predicted outcomes include: risk of 14-day mortality; percentage probability of 6-month mortality; and the percentage probability of 6-month unfavourable outcome, which is defined as death, vegetative state, or severe disability requiring dependence on others at 6 months following injury.

Patients’ clinical outcomes post-sTBI can be measured using the Functional Independence Measure (FIM) score, the Disability Rating Scale (DRS), the Glasgow Outcome Scale (GOS) or the Extended Glasgow Outcome Scale (GOS-E).^13–16 Both CRASH and IMPACT were developed using GOS-E as the outcome, and the FIM measures only a fraction of what the GOS-E measures.

Current literature highlights the need to evaluate whether these calculators have any impact on sTBI patient outcomes.¹² Formal outcome prediction of sTBI is not routinely performed in New Zealand. Prognostic calculators have been developed on an experimental basis using large international cohorts. Testing the level of agreement and bias of IMPACT and CRASH on a New Zealand regional population would help determine whether these calculators are capable of predicting outcomes accurately. The clinical care applications of such models include facilitating discussions by intensive care unit (ICU) and surgical teams with patients’ families regarding sTBI prognosis that supports treatment decisions and resource allocation, including rehabilitation.

The primary aim of this study is to assess the functional outcomes of sTBI after hospital discharge at rehabilitation within the Te Manawa Taki (TMT) health region of New Zealand. Additionally, we will compare prognosis parameters of the TMT cohort with published results of both CRASH and IMPACT. The secondary aim is to compare the level of agreement of the CRASH and IMPACT, using FIM as the outcome in the TMT cohort data, with the premise that there will be a good level of agreement between the CRASH and IMPACT in our study cohort.

Methods

Study design and population

This is an observational, retrospective cohort study over 10 years and 7 months, from 1 June 2012 to 31 December 2022, using data from the TMT trauma registry that includes records of 65,780 trauma events over this period. It focusses on sTBI patients who received treatment at a Level 1 trauma centre in the TMT Region, and were later transferred to, and subsequently discharged from, the Acquired Brain Injury (ABI) rehabilitation service. The TMT trauma registry collates data to evaluate demographics, hospital resource utilisation, injury categorisation and discharge destinations for acute care facilities within the region. Waikato Hospital, the region’s main healthcare facility, is a Level 1 trauma centre, officially verified by the Royal Australasian College of Surgeons. Eligible TBI patients are transferred from the hospital to the ABI rehabilitation service for a period of rehabilitation, and their stay is usually funded by the Accident Compensation Corporation (ACC). Data between the two services (i.e., TMT and ABI) are not linked and are held in separate databases. This study was approved by the Health and Disability Ethics Committees (2022EXP14029) and received locality approval (RD022040). A waiver of consent was granted for this research study.

The inclusion criteria included a total GCS score of ≤8, which is consistent with the definition of sTBI and a requirement for inpatient rehabilitation in the ABI service, and an age of ≥18 years upon admission to the emergency department (ED) following injury. The GCS score of ≤8 on admission to the ED, regardless of whether the patient is intubated or not, has been considered during the recruitment of patients to the study. The reason for the age of ≥18 years is that this is in keeping with the age criteria for adults in many countries, including the United States of America (USA) and United Kingdom, while not being the normal adult definition in New Zealand. Two hundred and forty-one patients met the inclusion criteria from the TMT trauma registry database. The data extracted from the electronic and paper clinical records include: age; sex; ethnicity; GCS total score; GCS motor score; pupillary size and reaction to light; major extracranial injury (defined as AIS ≥3 injuries in the face, chest, abdomen and pelvis/extremities); on admission hypoxia (defined as oxygen saturation less than 90% on admission); hypotension (defined as systolic blood pressure less than 90mmHg on admission); computed tomography (CT) brain findings including Marshall CT classification (I-VI), traumatic subarachnoid haemorrhage, epidural haematoma, petechial haemorrhage, obliteration of third ventricle or basal cisterns, midline shift >5mm, non-evacuated haematoma; and laboratory findings including glucose (mmol/l) and haemoglobin (g/dL). All of these data pertain to a single time point on admission to the ED, except imaging and blood results, and there are no missing data. Hypoxia and hypotension due to blood loss, which was subsequently replaced using a blood transfusion, are both factors contributing to poor outcomes from sTBI due to causing poor perfusion to the brain. Pre-hospital GCS, mechanism of injury, associated injuries and time to definitive care have been omitted from the data collection due to missing pre-hospital data, limited-to-no access to ambulance documentation and mechanism of injury, and associated injuries not being included in the prognostic calculator parameters. The mean length of stay in hospital and in the ABI service and the rate of neurosurgical intervention was also measured. As sTBI is an inclusion criterion, not all patients transferred to the ABI service were included, regardless of admission GCS score.

Exclusion criteria included death at any point from hospital admission to discharge from ABI service, trauma caused by asphyxiation or previous medical conditions, and absence of FIM score on discharge from ABI service. These were applied to the 241 patients, resulting in the exclusion of 162 patients. The final study population (i.e., TMT cohort) included 79 patients (Figure 1). The total trauma events in the TMT Region over the study period are 77,079. Total TBI and sTBI events over the same period are 11,434 and 241, respectively. There were no major changes to healthcare pathways or treatments over the study period.

View Figure 1–2, Table 1.

Prognostic models

IMPACT

This model was based on 8,509 patients of GCS score ≤12 (moderate-to-severe TBI) from three observational studies and eight randomised controlled trials (RCTs).¹¹ We used the full prediction model (i.e., Core+CT+Lab) to determine the likelihood of an unfavourable outcome at 6 months, which is defined as death, vegetative state or severe disability requiring dependence on others at 6 months following injury. The IMPACT score for unfavourable outcome was converted into a percentage risk for ease of analysis.

CRASH

This model was based on 10,006 TBI patients of GCS score ≤14 from one RCT.¹⁰ We used the CT model for the prediction of unfavourable outcome at 6 months, which is defined as death, vegetative state, or severe disability requiring dependence on others at 6 months following injury. The CRASH score for unfavourable outcome was converted into a percent risk for ease of analysis. Major extracranial injury was an AIS score ≥3 on at least one of the extracranial domains.

Functional outcomes

The FIM was used in our study because it was the only outcome measure available from ABI service. The FIM is an 18-item scale that is well validated with good inter-rater variability, used to quantify disability level in the first year after brain injury, and is commonly used in rehabilitation. The FIM total score (from 18 to 126) on discharge from ABI service was used.²⁰ Levels of disability were either mild (scores of 108–126), moderate (scores of 54–107) or severe (scores of 18–53). Higher scores (mild disability) indicated good recovery following sTBI, whereas lower scores (severe disability) indicated either a vegetative state or need for assistance to perform activities of daily living (ADLs). We used a cutoff FIM score of 81, due to a FIM total score <80 indicating complete dependence or a burden of care that cannot be provided at home, which is comparable to a GOS-E unfavourable cut-point set at ≤4 (Upper Severe Disability or can be left unsupervised in the home for more than 8 hours, but is dependent outside the home).¹⁸ The purpose of using a FIM cutoff score of <80 was to make the outcome measure close to and comparable to a GOS-E unfavourable cutoff point. Thus, FIM scores of 18–80 on discharge from ABI service represented an unfavourable sTBI outcome.

The FIM assessments are conducted at ABI service by an assigned lead therapist, and it is required that they be completed within the first 72 hours of admission, with collegial discussions. The discharge FIM is completed within 72 hours of discharge by the same therapist. The ABI rehabilitation service has staff who are certified as FIM trainers. They facilitate introduction and refresher courses within the facility. All nurses and allied health professionals who assess and complete FIM scoring are certified. The criteria for admission to the ABI unit include: age over 16 (or age 14 and over with ACC approval); medically stable; accepted ACC claim; and meets criteria of a moderate to severe brain injury (i.e., GCS less than 13 on admission to the ED or deemed in post-traumatic amnesia >24 hours). The timing of the FIM scoring is based on admission and discharge dates (i.e., not standardised to 6 months). The time frames related to the patient’s healthcare pathway vary among patients in the cohort, depending on their length of stay.

Statistical analysis

Descriptive statistics on demographic and clinical variables are presented either as percentages or medians, and interquartile range. To test and compare the CRASH and IMPACT tools and test the hypothesis that there will be a good level of agreement between them, a Bland–Altman analysis was conducted, and the agreement interval and the bias between the mean differences were evaluated. All statistical analyses were performed using R software (version 3.4.3).¹⁹

Results

Between 1 June 2012 and 31 December 2022, 241 sTBI patients met the inclusion criteria. After applying the exclusion criteria, we had a total of 79 patients in the TMT cohort who had FIM scores on discharge from ABI service. The participants’ characteristics are summarised in Table 1.

The TMT-cohort FIM scores on discharge from ABI service showed eight (10%) severe, 26 (33%) moderate and 45 (57%) mild disability. Fifty-seven had a GCS motor score of 1, 73 had hypoxia and 70 had hypotension. These results show that the TMT cohort had higher rates of GCS motor score of 1 (72% vs 16%), hypoxia (92% vs 20%), and hypotension (89% vs 18%) compared to IMPACT.¹¹ The rate of “no pupil reacted” was highest in IMPACT (20.9%), compared to CRASH (8.2%) or TMT (5%). Compared to the CRASH cohort,¹¹ the TMT cohort had higher rates of major extracranial injury (57% vs 22.7%), traumatic subarachnoid haemorrhage (66% vs 31.6%), petechial haemorrhage (71% vs 28.7%) and non-evacuated haematoma (85% vs 27.1%). The median glucose and haemoglobin values were similar between the TMT and IMPACT cohorts (8.6 vs 8.2mmol/l and 13.6 vs 12.7g/dL). The unfavourable outcomes for CRASH and IMPACT cohorts reported were greater (66.01% and 55.95%, respectively) when compared to the TMT cohort's actual outcomes of 17.7% with FIM 18–81 on discharge from ABI service.

Within the TMT cohort, the mean length of stay in hospital was 30.68 days (standard deviation 16.06), the rate of neurosurgical intervention (i.e., decompressive craniectomy) was 12 out of 79 patients or 15%, and the mean length of stay in ABI service was 81.6 days (standard deviation 79.14).

Bland–Altman analysis demonstrated a mean bias of −10.09 units, indicating that IMPACT systematically under-estimated CRASH. Limits of agreement showed 94.9% of observations fell within the 95% that indicates good agreement overall but limited interchangeability between the two measures.

Discussion

The Medical Research Council (MRC) CRASH trial collaborators found that in a prospective trial cohort of 8,509 TBI patients, external validation for unfavourable outcome at 6 months in high-income countries showed that basic and CT models had good discrimination but poor calibration. In this trial, Steyerberg et al.¹¹ found that in 6,681 patients with moderate and severe TBI from the MRC CRASH Trial, external validation confirmed adequate discriminative ability of the model (AUC) (0.80). Outcomes were systematically worse than predicted, but less so in 1,588 patients who were from high-income countries.^10,11 In a retrospective cohort study of 26,228 patients admitted with moderate-to-severe TBI in the USA, Camarano et al.²⁰ used external validation to show that IMPACT demonstrated a marginally greater AUC (0.863; 95% CI: 0.858–0.867) than CRASH (0.858; 0.854–0.863); p<0.001. On calibration, IMPACT over-predicted at lower scores and under-predicted at higher scores but had good calibration-in-the-large (indicating no systemic over- or under-prediction), while CRASH consistently under-predicted mortality. This indicates that both models perform well at differentiating between patients who died and those who survived. In a prospective, observational cohort study of 4,509 patients presenting to EDs in 18 countries with TBI, Dijkland et al.²¹ found that, on external validation for CRASH, AUCs ranged from 0.82 to 0.88 in 1,742 patients and from 0.66 to 0.80 in the stricter IMPACT selection (n=1,173). Calibration of the IMPACT and CRASH models was generally moderate, with calibration-in-the-large and calibration slopes ranging between −2.02 and 0.61 and between 0.48 and 1.39, respectively. This indicates that the IMPACT and CRASH models adequately identify patients at high risk for mortality or unfavourable outcomes.^20,21

In a systematic literature review of 20 observational cohort studies with 1,855 sTBI patients, Mostert et al.²² found that mortality was reported in 46% of patients (range 18–75%). Unfavourable outcome rates ranged from 29 to 100%, and full recovery was seen in 21 to 27% of patients. This indicates that there is significant heterogeneity of long-term (i.e., ≥2 years) sTBI outcomes.²² A large USA study of 4,624 patients over 5 years assessed the functional outcome trajectory following TBI, regardless of severity. It reported improved function within 1 to 2 years post-injury, followed by reduced function by 5 years post-injury. Factors such as older age, reduced level of function pre-injury, non-white race, longer inpatient rehabilitation stay and lower functional status on discharge reduced the recovery trajectory over time.²⁵ An Italian study had 309 patients admitted to a 6-month neurorehabilitation unit after mild-to-severe TBI, of whom 98 (31.7%) patients received a decompressive craniectomy. There was no significant association between decompressive craniectomy and poor FIM scores during rehabilitation or mortality.²⁶

The prognostic models only predicted 14% or less in outcome; a lot is still left unexplained by the models within our study. One of the main reasons that the calculators did not perform well is that they were not developed for the FIM, but for the GOS-E, and the FIM and GOS-E are not interchangeable outcome measures. The use of prognostic models may adjust clinicians’ practice regarding discussions on prognosticating sTBI when comparing observed versus expected outcomes. The high rate of misclassifying patients with an unfavourable prognosis (approximately 20%) may be attributed to prognostic models relying on early post-injury clinical characteristics.²⁷ Future studies should identify additional factors impacting the positive predictive value and false-positive rate of these models.

Other variables may play a key role in predicting functional recovery after sTBI. The risk of predicting an inability to regain lost function should be balanced against setting realistic rehabilitation goals and informing discharge planning. It is important to acknowledge and explore the level of understanding and concerns of the patient’s family before initiating a discussion regarding continuing or withdrawing treatment, especially in the ICU. This inclusive, compassionate and holistic approach may facilitate reaching a consensus with acceptance of a certain level of unpredictability regarding the trajectory of functional recovery. Predictive models depict probabilities across patient populations and do not make predictions on an individual, case-by-case basis.²⁸

Our study has several limitations. One significant weakness is the small sample size of the final study group, which includes only 79 patients. As elaborated in the methods section, 162 patients were excluded due to various reasons, such as death, discharge to home or to a local hospital or specialist centre, not being transferred to the ABI rehabilitation service, their trauma resulting from asphyxiation or a medical condition, or missing FIM data at discharge from the ABI rehabilitation service. The study, being a retrospective analysis, had single points of data and lacked information on pre-existing conditions and prehospital information. These limitations could also introduce selection bias, and the findings are not generalisable. The use of the FIM score itself is a limitation, as it is not a recommended measure for TBI outcome studies, which typically use GOS-E (the instrument on which the predictor tools were developed). A FIM cutoff score of <80 was used in our study to make the outcome measure close to and comparable to a GOS-E unfavourable cutoff point. Additional limitations may include data collection at single points in time, the absence of multi-trauma data (which may influence outcome) and the omission of prehospital data. Reasons that ABI services may not be available to all surviving sTBI patients are due to the need for pre-authorisation of funding by ACC, geographic location (before 2024, ABI centres were located in either Auckland or Wellington, which makes travel arrangements difficult for family members living in Hamilton) and limited facility availability (between 22 and 35 beds in each ABI service location).

The study is an observational, retrospective cohort study, which is suitable for the two prognostic models. The TMT cohort included only patients who survived and were discharged from the ABI rehabilitation service, potentially introducing survival bias, over-estimating positive outcomes, and under-estimating the severity of sTBI consequences. There are inherent limitations in calculating true GCS scores due to factors like sedation drugs, intubation, pharmacological paralysis, intoxication and the presence of previous physical disability or hearing impairment. With mitigation for this factor using a score of 1 for eye opening, verbal response and motor response, respectively, while intubated and sedated, most patients were intubated before admission to the ED, making a true total GCS score incalculable. Hypoxia or hypotension may support the true injury severity by indicating poor perfusion of vital organs, including the brain, which can affect both neurological function and subsequent recovery from neurological injury. In addition, using the FIM at discharge from ABI rehabilitation service is reasonable, but the timing of this measurement varies.

In addition to improving the measurement of indicators of injury severity, future studies could utilise larger sample sizes, longer follow-up periods and investigate long-term recovery trajectories (e.g., 1–2 years after injury). Further investigation of the factors that inhibit the delivery of specialised TBI rehabilitation post-discharge to all sTBI patients will help guide resource allocation. Future work could externally validate outcomes across multiple Level 1 trauma centres. Further studies could also consider comparing CRASH and IMPACT to not only FIM, but also to DRS, GOS and GOS-E. In a prospective cohort study of 598 patients presenting to EDs with sTBI, Eagle et al.²⁷ found that the CRASH and IMPACT models’ discriminative validity remained consistent over time and comparable to earlier recovery time points (AUC=0.77–0.83). Both models had a poor fit for unfavourable outcomes, explaining less than one-quarter of the variation in outcomes for sTBI patients. The CRASH model had significant values for the Hosmer–Lemeshow test at 12 and 24 months, indicating poor model fit past the previous validation point of 6 months post-injury. This indicates that using the CRASH and IMPACT models carries a word of caution when being applied to routine clinical practice, due to the model fit worsening over time, with large unexplained variance in outcomes. In a retrospective cohort study of 467 patients admitted with sTBI, Eagle et al.²⁷ found that sensitivity (35.3–50.0) and positive predictive values (66.7–69.2) were poor in the CRASH models, while specificity (52.3–53.1) and negative predictive values (58.1–63.6) were poor in IMPACT models. All models had unacceptable false-positive rates (20.8–33.3%). This indicates that both models have lower accuracy and higher false-positive rates.^27,28

Conclusion

This study highlighted functional outcomes of sTBI patients in the TMT cohort at follow-up, providing valuable insight into regional functional outcomes. Compared to the CRASH and IMPACT cohorts, the TMT cohort showed higher rates of poor prognostic indicators. Importantly, the two models show good agreement in predicting the functional outcome, but because of the systematic bias, we cannot compare the prediction outcome for a patient with CRASH against the prediction outcome of another patient with IMPACT. Future studies are needed with larger sample sizes, longer follow-up periods and to investigate long-term recovery trajectories (e.g., 1–2 years after injury), and they could externally validate outcomes across multiple Level 1 trauma centres. Further studies could also consider comparing CRASH and IMPACT to not only FIM, but also to DRS, GOS and GOS-E.