Consumer outcomes in mental health care

Summary

Key points

Information was recorded for 202,234 people in 2019–20, representing 43.3% of consumers of public mental health services.
During 2019–20, the most common clinically significant problems for consumers aged 11–17 at the start of an episode of care were Emotional and Family issues; for adults aged 18 and over these were Depressed mood and Other mental and behavioural problems; and for people aged 65 and over Physical/disability problems were also common.
Clinician-rated measures were completed at much higher numbers than consumer-rated measures for all age bands during 2019–20.
During 2019–20, completed consumer episodes showing Improvement outcomes on clinician’s ratings were higher for inpatient than ambulatory (non-admitted) care:
- inpatient care, at 57.9% (aged 11–17), 73.4% (aged 18–64) and 72.6% (65 and older) of episodes; consumers in this setting showed Deterioration in up to 9.5% of episodes.
- ambulatory (non-admitted) care, at 55.0% (aged 11–17), 51.6% (aged 18–64), and 45.9% (65 and older) of episodes; consumers in this setting showed Deterioration in up to 6.8% of episodes.

The National Outcomes and Casemix Collection (NOCC) collects information using measures completed by clinicians and consumers about a person’s clinical status during their mental health care. During 2019–20, NOCC measures were collected for 202,234 people nationally, which is 43.3% of people who received clinical care from public sector specialised mental health services.

During 2019–20 the most frequent mental health-related problems affecting consumers were Emotional problems (adolescents) and Depressed mood and Other mental and behavioural problems (adults). Mental disorder not otherwise specified and Depressive episode were among the most frequent principal diagnoses recorded at discharge for all age groups.

Clinical outcomes can be calculated on a subset of care episodes where clinical measures have been completed twice in a matched pair. This shows whether consumers show improvement, no change or deterioration from mental health care.

For all age bands and consumer groups, clinician-rated outcome measures showed:

Improvement for between 25.2% of episodes (for consumers aged 18–64 years in ongoing ambulatory care) and 73.4% of episodes (for consumers aged 18–64 years who completed acute inpatient care)
No change for between 22.1% of episodes (for consumers aged 18–64 years who completed acute inpatient care) and 59.2% of episodes (for consumer aged 65 years and over in ongoing ambulatory care)
Deterioration in up to 17.5% of episodes (for consumers aged 18–64 years in ongoing ambulatory care).

Overall, a higher proportion of consumer episodes showed Improvement compared to other outcomes if the consumer had completed acute inpatient care, and a higher proportion of consumer episodes showed No change if the consumer was still receiving ongoing ambulatory care.

Data obtained using the NOCC protocol provide valuable information about the outcomes of people accessing Australian public mental health care and enables reporting under the Key Performance Indicators for Australian Public Mental Health Services. Data collection, reporting and development under the NOCC has been in place for two decades and continue due to significant ongoing collaboration between state/territory governments, the Australian Government Department of Health, the Australian Institute of Health and Welfare, and the Australian Mental Health Outcomes and Classification Network.

Spotlight data

Do people get better, get worse, or stay the same from their public mental health care?

Introduction

Mental health treatment and support services have an important role in the recovery of people with mental health issues. This section presents information about the mental health-related problems experienced by consumers of public sector specialised mental health services and whether there is improvement after receiving mental health care, as measured by a set of clinically-derived indicators.

Data are available for public sector specialised mental health services. A range of other mental health services are not included here – for example, clinical measures may be collected to aid consumers’ recovery in private hospitals, private clinicians’ practices, non-government organisations, primary health care networks, and other services. This is due to the fact that outcomes data from those services are not currently routinely collected under national agreements and thus are not available for reporting.

Clinical measures are particular surveys or forms that are used to gather information about a person's clinical mental health status and functioning. These measures can be completed by clinicians about the consumer (known as clinician-rated), completed by the consumer (consumer-rated), and completed by families and carers about the consumer (carer-rated). When the same clinical measures are completed more than once, they can be used to determine whether a person shows improvement, no change, or deterioration from mental health care.

Data reported in this section are gathered under the National Outcomes and Casemix Collection (NOCC), which was specified in 2003 to guide states and territories in the implementation of routine consumer outcomes measurement in public mental health services. All consumers who receive clinical care in public sector specialised mental health services should be included in the NOCC, including psychiatric inpatient, residential and ambulatory (non-admitted) service settings. More information about the NOCC is in the data source section.

This section provides an overview of the NOCC and key national findings. More detailed data are available via the National Outcomes and Casemix Collection Web Decision Support Tool and Reports Portal.

What are outcomes and casemix?

The NOCC collects information about a person's clinical mental health status and functioning during their episode of mental health care. Measures completed by clinicians about the consumer (known as clinician-rated) and measures completed by the consumer (consumer-rated) are used. These measures are completed at multiple collection occasions during an episode of care to monitor changes in consumers' clinical status and functioning.

Ratings information is used to report on consumers’ outcomes of care – that is, whether consumers of mental health services show improvement, no change, or deterioration from receiving mental health care. Clinical outcomes such as these are just one aspect of a consumer’s treatment and recovery.

Change in mental health consumers’ clinical outcomes is included in the Key Performance Indicators for Australian Public Mental Health Services. These indicators contribute to measuring the performance and progress of mental health services in Australia. The indicators are also reported on Mental health services in Australia. Refer to the data source section for more information.

In addition to outcomes, data items in the NOCC gather information about other factors that together are known as casemix. In this section, the reported casemix items are the consumer’s mental health legal status and diagnosis.

The collection of measures for the NOCC is guided by a set of rules on what measures to collect and when to collect them. More information is in the data source section and more detailed information is in the technical specifications.

Confidence intervals

In this section, measures of statistical uncertainty pertaining to estimates (95% confidence intervals) are shown in all data tables and represented in data visualisations by black bars. If the intervals for comparison groups do not overlap – that is, they do not include the same values in the range – the difference between groups can be generally inferred to be statistically significant.

More information about confidence intervals Expand

This section reports confidence intervals in the data tables and visualisations. A confidence interval is a range of values that quantifies the statistical uncertainty in estimates that result from natural or random variation. For example, in the number of services provided and the number of persons using services over time. There are also non-random sources of uncertainty, such as incomplete reporting, that are not captured by confidence intervals.

Generally, confidence intervals describe how different an estimate could have been if the underlying conditions stayed the same but random fluctuations had led to a different set of data. Accordingly, it is recommended that confidence intervals are reported alongside a number estimate.

Confidence intervals are calculated with a stated probability (commonly 95%); this means we can be 95% confident that the confidence interval includes the true value if the assumptions made in the construction of the confidence interval hold. Larger numbers of observations yield more precise estimates with narrower confidence intervals. Confidence intervals can be used to perform tests of statistical significance. If the 95% confidence intervals do not overlap – that is, they do not include the same values in the range – the difference can be said to be statistically significant (note that differences can be significant in a subset of cases where the ranges do overlap).

In this section, 95% confidence intervals are shown in most figures and all tables.

Further information about confidence intervals, including calculation methods, statistical assumptions behind the calculation and sources of variability can be found in the data source section.

Consumers

All people who receive clinical care in public sector specialised mental health services – including psychiatric inpatient, residential and ambulatory (non-admitted) services – should be included in the NOCC.

Nationally during 2019–20, NOCC measures were collected for 202,234 people, which is 43.3% of the 467,062 people who received clinical care from public sector specialised mental health services (Table NOCC.1).

The proportions of people who received clinical care in public sector specialised mental health services who were included in the NOCC were higher among people aged 75–84 (47.1%) and 45–54 (45.9%) and lower among people aged 0–17 (40.9%) and 18–24 (41.6%) (Table NOCC.1).

Demographics

During 2019–20, half of the consumers included in the NOCC were male and half were female (49.5% and 50.4% respectively). There were 71,768 people aged between 25 and 44 years, accounting for 35.5%. There were 19,563 Aboriginal and Torres Strait Islander peoples, accounting for 9.9%.

People living in Major cities made up the majority of consumers included in the NOCC (65.2%) and people living in Remote and very remote areas made up the smallest proportion (3.3%). For socio-economic status, people living in areas of most disadvantage made up the largest proportion (25.0%), while people living in areas of least disadvantage made up the smallest (14.6%) (Figure NOCC.1).

Figure NOCC.1: Demographic characteristics of consumers included in the NOCC, 2019–20

There were 7,744 children aged 10 years and under, accounting for 3.8% of all consumers in the NOCC in 2019–20 (Table NOCC.2). As this is a relatively small number of consumers, the ability to undertake comprehensive reporting and disaggregations is limited. As such, data relating to children aged 10 years and under are not further reported in this section. Reports can be generated via other NOCC reporting products (Web Decision Support Tool and Reports Portal).

Service setting

Under the NOCC specifications, clinical and casemix measures may be completed at collection occasions. The 3 collection occasion types are Admission, Review, and Discharge. A person may have multiple collection occasions.

During 2019–20 across all age bands, the majority of collection occasions were in ambulatory
service settings – accounting for 57,239 (88.2%) collection occasions for young people aged 11–17 years, 274,816 (71.6%) for people aged 18–64, and 48,671 (83.1%) for people aged 65 years and older.

Inpatient settings accounted for 7,556 (11.6%) collection occasions for young people aged 11–17 years, and 100,208 (26.1%) for people aged 18–64. Within the inpatient setting, the majority of service programs provided acute care, accounting for 97.5% of inpatient collection occasions for 11–17 year olds, 95.1% for people aged 18–64, and 94.5% for people aged 65 years and older (Tables NOCC.3 and NOCC.4).

There were nearly 8,800 collection occasions in residential services accounting for no more than 2.3% in any age band (Table NOCC.3).

The remainder of this section reports data for ambulatory and acute inpatient service settings.

Mental health legal status

Mental health legal status indicates whether the person was treated on an involuntary basis under the relevant state or territory mental health legislation during care.

Overall during 2019–20, Involuntary status was recorded for higher proportions of discharge collection occasions in acute inpatient care than in ambulatory care – up to 40.9% for inpatient and up to 12.9% for ambulatory, depending on age (Figure NOCC.2).

Figure NOCC.2: Involuntary mental health legal status recorded at discharge, by age band and setting, 2019–20

Involuntary mental health legal status by age band Expand

Data differences for consumer- and clinician-rated measures Expand

The suite of clinician-rated Health of the Nation Outcome Scales (HoNOS) provides information about the mental health-related problems experienced by consumers at admission to a mental health service that are rated by a clinician to have a clinically significant impact on the consumer.

During 2019–20, many consumers in all age bands were facing more than one clinically significant problem. Overall, Emotional problems (adolescents) and Depressed mood (adults) were common along with Other mental health and behavioural problems for adults, indicating the presence of co-morbid problems (Figure NOCC.3).

Across all age bands, Hallucinations more frequently affected consumers in acute inpatient care than ambulatory care.

Figure NOCC.3: Clinically significant problems for consumers at admission, by age band and setting, 2019–20

Children and adolescents (11–17 years) Expand

Adults (18–64 years) Expand

Older persons (65 years and older) Expand

Principal diagnoses for consumers

The principal diagnosis recorded at discharge provides an indication of the treated prevalence of specific mental illnesses in specialised mental health care services. Principal diagnosis is recorded as a code from the International Classification of Diseases and Related Health Problems (ICD-10-AM, 11^th Edition).

Mental disorder not otherwise specified was among the most frequent principal diagnoses recorded at discharge during 2019–20 for all age bands and settings (Figure NOCC.4). This suggests heterogeneity in the clinical presentation of mental health diagnoses for consumers.

Of specified mental health-related diagnoses, Depressive episode was among the five most frequent principal diagnoses for all age bands and settings.

Figure NOCC.4: Five most commonly recorded mental health-related principal diagnoses for consumers at discharge, by age band and setting, 2019–20

Children and adolescents (11–17) Expand

Adults (18–64 years) Expand

Older person (65 years and older) Expand

Clinical outcomes of care

Information gathered at collection occasions can be organised into consumer groups, which pairs the episode type (completed, ongoing, closed) with the setting in which treatment is provided (acute, inpatient, ambulatory).

Clinical outcomes can only be calculated on a subset of episodes in the NOCC dataset – those with a matched pair of collection occasions where the same clinical measure has been completed twice for a particular consumer during an episode of care.

The number of episodes with matched pairs are low for consumer-rated measures and high for clinician-rated measures. During 2019–20, consumer-rated clinical outcomes could be calculated for 7.2% to 30.4% of episodes, depending on consumer group and age band. Clinician-rated clinical outcomes could be calculated for 73.3% to 93.8% of episodes (Figure NOCC.5).

Caution should be applied in comparing outcomes using clinician-rated with consumer-rated measures because it cannot be assumed they are the same consumers.

Figure NOCC.5: Episodes for which consumer outcomes can be calculated, clinician- and consumer-rated measures, by age band and consumer group, 2019–20

More about consumer groups Expand

Consumer episodes can be classified into clinical outcomes of Improvement, No change, or Deterioration. These classifications rely on significance testing to identify change.

Clinical outcomes during 2019–20 vary depending on consumer group and age band. There are some overall patterns in the data that apply across all age bands (Figure NOCC.6).

The highest proportions of Improvement compared to other clinical outcomes were for consumers who accessed completed acute inpatient care. This held for both clinician- and consumer-rated measures using available data. People in this consumer group showed improvement for between 57.9% and 73.4% of episodes.
People were more likely to show No change compared to other clinical outcomes if they were in ongoing ambulatory care, for both clinician- and consumer-rated measures. People in this consumer group showed no change for between 46.7% and 68.6% of episodes.
Deterioration was the least common outcome compared with other clinical outcomes across all consumer groups. Consumers showed deterioration for between 1.3% and 17.5% of episodes.

Figure NOCC.6: Clinician- and consumer-rated outcomes for consumers, by age band and consumer group, 2019–20

Children and adolescents (11–17) Expand

Adults (18–64) Expand

Older persons (65 years and over) Expand

Data source Expand

The National Outcomes and Casemix Collection (NOCC) is a nationally agreed data collection for the routine collection and reporting of consumer outcomes using clinical measures. Under the National Mental Health Policy 1992, Australian governments committed to national monitoring of the effectiveness of public mental health services. The primary objective of the NOCC was to establish the routine use of outcome measures in all publicly funded or managed mental health services, where such measures contribute both to improved practice and service management (Burgess et al. 2015). The NOCC has been progressively implemented in state and territory public sector specialised mental health services from 2001 with all jurisdictions reporting by June 2005.

The NOCC captures information about consumers’ health and wellbeing during their mental health care using standardised clinical measures, which is used to report on outcomes.

The NOCC also gathers ‘casemix’ information, which is information about the mix of people who are receiving mental health services according to their clinical status and the nature of the care they are receiving. The casemix information collected in the NOCC supports the introduction of the first version of the Australian Mental Health Care Classification (AMHCC) (Independent Hospital Pricing Authority 2018).

Public reporting of the NOCC

National indicator set: Change in consumers’ clinical outcomes

The Key Performance Indicators for Australian Public Mental Health Services (KPIs) were developed for the purpose of improving public mental health services. KPI 01 Change in mental health consumers’ clinical outcomes draws on data from the NOCC for reporting. These indicators contribute to measuring the performance and progress of mental health services in Australia.

The original KPI set was released in 2005, with the aim to measure and improve the performance of public mental health services. The indicators have been revised over time through the former National Mental Health Performance Subcommittee (NMHPSC) of the former Mental Health Information Strategy Standing Committee (MHISSC), to drive and incorporate data improvements, jurisdictional implementation and results of nation-wide projects (NMHPSC 2013).

Change in mental health consumers’ clinical outcomes was added to the national KPI set in 2011 (NMHPSC 2013).

The Key Performance Indicators are published on Mental health services in Australia (MHSA). Due to differences in methodology there is variation in the NOCC data that are reported for the KPIs, this section, and other reporting tools.

Other reporting tools

Other NOCC public reporting products focus on the clinical utility of the collection, through the publication of ‘normative’ reference data for the clinical measures that assist clinicians and other users to better understand the outcomes and variability in the population under care.

A Web Decision Support Tool allows users to compare an individual consumer’s scores at a single point in time, or change in scores over time, against normative data from ‘like’ consumers around Australia. In addition, scores on clinician- and consumer-rated measures can be displayed side-by-side which facilitates engagement with the consumer/family around different perspectives on mental health status.

A Reports Portal allows users to create tailored reports that provide different statistical summaries of the NOCC data, for example the change in scores on various measures across the course of given episodes. More granular reports can be created by selecting from a range of variables, for example age, measure (including item level), service setting, collection occasion, collection reason, jurisdiction, diagnosis, sex, legal status.

In the Web Decision Support Tool and the Reports Portal, NOCC data are reported at national and state/territory levels.

NOCC protocol

The collection of the standard clinical measures is guided by an underlying conceptual model and national protocol. The clinical measures included in the NOCC protocol are at Box NOCC.1. The measures are specific to the consumers’ age group (Child or adolescent aged less than 18 years; Adult aged 18–64 years; and Older person aged 65 years and over) and may be used for the purpose of measuring outcomes or describing casemix.

Box NOCC.1: Clinical measures in the National Outcomes and Casemix Collection (NOCC)
Clinical measures	Age group: Children and adolescents	Age group: Adults	Age group: Older people	Purpose: Outcomes	Purpose: Casemix
Clinician-rated measures:
Health of the Nation Outcome Scales (HoNOS)		●		●	●
Health of the Nation Outcome Scales for Older People (HoNOS 65+)			●	●	●
Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA)	●			●	●
Life Skills Profile (LSP-16)		●	●	●	●
Resource Utilisation Groups - Activities of Daily Living Scale (RUG·ADL)			●		●
Children's Global Assessment Scale (CGAS)	●				●
Factors Influencing Health Status (FIHS)	●			○	●
Mental Health Legal Status	●	●	●	○	●
Principal and Additional diagnosis	●	●	●	○	●
Phase of care	●	●	●	○	●
Consumer and carer-rated measures:
Kessler Psychological Distress Scale - Plus (K10+), Behavior and Symptom Identification Scales (BASIS-32), or Mental Health Inventory (MHI-38)^a		●	●	●
Strengths and Difficulties Questionnaire (SDQ)^b	●			●

● Measure is used for the specified purpose of measuring outcomes or describing casemix.

○ Not an outcomes measure but is important for the interpretation of outcome data.

a) These measures are completed by the consumer. The specific measure used varies across states and territories – K10+ (New South Wales, Northern Territory, South Australia and Western Australia), BASIS-32 (Australian Capital Territory, Tasmania and Victoria), and MHI-38 (Queensland).

b) The NOCC includes three versions of the SDQ: SDQ-PC (parent report measure for children aged 4–10 years); SDQ-PY (parent report measure for youth aged 11–17 years); and SDQ-YR (self-report measure for youth aged 11–17 years).

Clinical measures are completed at key Collection occasions during the consumer’s episode of mental health care (at admission, review and discharge). The measures are specific to service setting (inpatient, residential and ambulatory) and the consumer’s age group. Outcome measures that are collected on at least two occasions allow assessment of change in health status.

Casemix measures are used to describe the mix of people who are receiving mental health services, grouped according to their clinical status and the pattern of services they are receiving. Casemix measures need only be collected at the single most appropriate point for describing and classifying each episode.

Other data elements in the NOCC provide context for interpreting the information gathered using the clinical measures. These include defining attributes of collection occasions, for example mental health provider entity identifier, person identifier, age group, mental health service setting, reason for collection, collection occasion date and person-level socio-demographic characteristics.

More information can be found on the Australian Mental Health Outcomes and Classification Network (AMHOCN) website:

Scope and coverage

NOCC data are reported annually, based on financial year. An individual consumer’s measures are not linked across years. The NOCC does enable an individual consumer’s clinical status and functioning to be described at different points of treatment within a single year. However, many consumers, due to the nature of their mental illness, receive care for longer periods and often across multiple settings and organisations. The approach used to report outcomes from the NOCC separates consumers’ care into segments – for example, inpatient versus ambulatory care – within a single year, rather than tracking outcomes across treatment settings and time.

NOCC coverage estimates are reported as an indicator of the extent to which the NOCC protocol has been successfully implemented. They are derived by comparing the number of consumers with clinical ratings in the NOCC in a reporting period (the numerator) to the overall number of consumers reported as receiving clinical care from state and territory public mental health services in the same reporting period (the denominator).

The numerator is derived from the NOCC. For the purposes of coverage estimates, counts of consumers included in the NOCC are unique at the jurisdictional level within the reporting period. The denominator is sourced from aggregated data supplied by jurisdictions to the AIHW for the purposes of calculating MHS Key Performance Indicator 9 (KPI 9) New client index. General and specific caveats affecting the quality of these data are provided in the Key Performance Indicators for Australian Public Mental Health Services tables (200kB XLSX).

The consumer’s age at each collection occasion may not align with the NOCC protocol in terms of the age group specific services received and the measures completed. This may occur, for example, when consumers aged less than 18 years receive ‘adult services’, or consumers aged 18–64 years receive ‘older persons’ services, etc. Over the 6-year period covered by this section, approximately 2.1% of collection occasions did not align. For the purposes of this section, this small percentage of collection occasions has been excluded, resulting in a small underestimate of coverage.

Statistical units

The statistical counting unit used to describe coverage of the NOCC is the consumer, a unique individual within a jurisdiction.
The statistical counting unit used to describe NOCC volume is a collection occasion, a meaningful point during a period of contact between a consumer and a mental health service organisation within the reporting period.
The statistical counting unit used to describe outcomes is an episode of mental health care, the period of contact between a consumer in a single setting within a mental health service organisation bounded by the ‘first’ and ‘last’ collection occasions within the reporting period.

More information about the statistical counting units and the conceptual basis for the reporting of the NOCC can be found in the Reporting Framework for the National Outcomes and Casemix Collection.

Data quality information

Data validation

Data are supplied annually by all states and territories and are validated to ensure the data conform to the NOCC protocol under the NOCC Technical Specifications ‘business rules’. Jurisdictional representatives respond to any issues before the data are accepted as the most reliable current data collection. This process may highlight issues with historical data. In such cases, historical data may be resupplied to ensure data are consistent. Only data that form valid sequences of collection occasions within non-overlapping episodes of mental health care are used for public reporting. Further information about the NOCC data set business rules can be found on the AMHOCN website.

Data quality over time

Data should be consistent across most jurisdictions and across years within most jurisdictions, with the following exceptions.

The Australian Capital Territory transitioned to a new information system in 2016–17. This has impacted the integrity of the unique counts of consumers that were supplied for the purposes of calculating NOCC coverage.

The Australian Capital Territory does not reliably report principal type of admitted patient care program provided by specialised inpatient mental health services. In this section, all Australian Capital Territory inpatient services are considered ‘acute care’.

New South Wales transitioned to new information systems in the 2015–16 and 2016–17 periods. This occurred along different timelines region by region. The change had an impact on the ability of staff to record data as they were trained and adjusted to the new systems.

The NOCC Technical Specifications were updated with effect from 2015–16 to include new data elements for country of birth, Aboriginal and Torres Strait Islander status, and consumers’ area of usual residence. The latter, reported as Statistical Area Level (SA2) from the Australian Statistical Geography Standard, is used to derive socio-economic status and remoteness measures. Partial reporting of these measures (4 jurisdictions) commenced in 2015–16; full reporting commenced in 2016–17. These measures are reported only for years in which there was full reporting.

The concepts episode type and consumer group are based on information about service setting, collection occasion and the reason for collection. Reason for collection is a NOCC data element comprising domains that describe key events triggering admission (e.g., new referral), review (e.g., 3-month review) or discharge (e.g., no further care) collection occasions. The Australian Capital Territory did not reliably report the reason for collection data element with respect to ambulatory discharges before 2018–19. This means that estimates for ‘Completed’ ambulatory episode types and the ‘Completed ambulatory’ consumer group are not available prior to 2018–19.

Clinical measures

The collection of routine outcome measures in everyday clinical practice may be challenging and gaps in collection can subsequently occur. It is important to understand these challenges as they can impact on the volume of data that is available for reporting and introduce systematic biases. An important challenge in the design of the NOCC protocol has been to minimise the burden of collection.

There are limited exceptions to the protocol for defined instances when the collection of measures is not required – for example, if it is not appropriate to offer the measure where the consumer’s current clinical state is of sufficient severity that it could not be reliably completed and/or would cause additional distress; or if an episode of mental health care is too brief to allow meaningful opportunity to show change at the time of discharge; or when the care of the consumer is transferred from the ambulatory service to an inpatient or residential service of the same Mental Health Service Organisation. In these instances, the collection occasion is not considered to be in-scope for reporting.

Other challenges reflect the reality of everyday clinical practice – for example, when the consumer is not available to be offered the measure at the intended collection occasion, such as at discharge. In these instances, the collection occasion is considered to be in-scope for reporting.

Clinician-rated measures

HoNOSCA (Health of the Nation Outcome Scales for Children and Adolescents):

The HoNOSCA is modelled on the HoNOS and designed specifically for use in the assessment of child and adolescent consumer outcomes in mental health services. It comprises 15 scales assessing specific aspects of the youth’s mental health (13 items), and environmental aspects related to lack of information or access to services (2 items). Each scale is rated on one of five levels of severity (0 = no problem, 1-4 = minor problem to very severe problem). A rating of 2 or more on each scale indicates a clinically significant problem (Burgess et al. ). A total score is obtained by summing the scores on the first 13 scales (range 0-52) 2009 (Gowers et al. 1999)

HoNOS/HoNOS 65+ (Health of the Nation Outcome Scales for working age adults and older adults):

The focus of the HoNOS is on health status and severity of symptoms. It consists of 12 scales rated on one of five levels of severity (0 = no problem, 1–4 = minor problem to very severe problem) that cover problems that may be experienced by people with a significant mental illness. A rating of 2 or more on each scale indicates a clinically significant problem (Burgess et al.2009). A total score is obtained by summing the ratings on each individual scale (range0–48). The HoNOS 65+ version consists of the same set of 12 scales and is scored in the same way. However, the accompanying glossary has been modified to better reflect the problems and symptoms encountered when assessing older persons (Burns et al. 1999; Wing et al. 1994; Wing et al. 1998).

Consumer-rated measures

SDQ-YR (The Strengths and Difficulties Questionnaire Youth Report):

The SDQ is a brief behavioural screening measure. The NOCC includes self-report (the SDQ-YR for youth) for consumers aged 11–17 years. Each version includes 25 items on psychological attributes; additional items vary across versions. The reference period for the psychological attributes items is the last 6 months. These items are rated on 0-2 scale; some items are reverse scored so that a high score indicates greater difficulty. A Total Difficulties score is obtained by first calculating scores for four scales that each contain 5 of the 25 psychological attribute items (Emotional Symptoms Scale, the Conduct Scale, the Hyperactivity Scale, and the Peer Problem Scale), then summing those scale scores (range 0-40) (Goodman 1997).

Adults and Older Persons

Kessler Psychological Distress Scale (K10)/K10 Plus (K10+):

The K10 is a self–report measure intended to yield a global measure of ‘non-specific psychosocial distress’ based on ten questions about the level of nervousness, agitation, psychological fatigue and depression in the relevant rating period. A total score for the 10 questions is generated by the sum of individual responses (1=None of the time, 2=A little of the time, 3=Some of the time, 4=Most of the time and 5=All of the time) (Kessler et al. 2002). The K10+ contains additional questions to assess functioning and related factors; there is no summary score for these items. The NOCC includes the K10LM (the label ‘LM’ stands for Last Month) which uses the rating period of the previous four weeks, and the K10L3D (the label ‘L3D’ stands for Last 3 Days) which is designed for use in inpatient settings.

Behaviour and Symptom Identification Scale (BASIS-32):

BASIS-32 comprises 32 items that cover the major symptoms and functioning difficulties often experienced by people as a result of a mental illness, across five domains (relation to self and others, daily living and role functioning, depression and anxiety, impulsive and addictive behaviour, psychosis). All items are rated on a 5-point scale (from 0 for least difficulty to 4 for greatest difficulty). A total score is obtained by calculating the average ratings on 30 of the individual items (only one of items 2, 3, 4 is included in this calculation, range 0-4) (Eisen et al. 2000; Eisen et al. 1994 )

Mental Health Inventory (MHI-38):

The MHI-38 was designed to measure general psychological distress and well-being in the general population, therefore includes positive aspects of well-being (such as cheerfulness, interest in and enjoyment of life) as well as negative aspects of mental health (e.g., anxiety and depression). The respondent rates on a scale the degree (frequency or intensity) to which they have experienced a particular symptom or state of mind in the past month using either a six-point scale (1-6) or a five-point scale (1-5). A Mental Health Index score calculated by summing the ratings on the 38 individual items (range 38-226) such that higher scores indicate greater wellbeing and less psychological distress (Veit and Ware 1983).

Outcomes methodology

In order to measure outcomes:

the same measure must be collected, and
on two collection occasions that form a logical sequence, and
within an episode of mental health care, and
within a single setting.

When these conditions are met, these are called matched pairs.

The NOCC protocol defines specific instances when the clinical measures are not in-scope for collection; these instances are a complex mix of factors that vary across age bands, settings, types of collection occasions, and the specific measure. This can impact the number of collection occasions that are available for describing the clinical characteristics of consumers at different points in their engagement with services, and for describing the outcomes of clinical care.

Not all collection occasions are eligible to form matched pairs. Specifically, discharge ratings on the clinician- and consumer/carer-rated measures are not required for brief episodes of ambulatory care (14 days or less) or brief acute inpatient care (3 days or less) because this brief period does not provide a meaningful opportunity to measure change. In addition, discharge ratings on the clinician- and consumer/carer-rated measures are not required when the consumer is transferred to an inpatient or residential setting within the same organisation, because the measures will be collected upon admission to the new setting.

Episodes of mental health care for an individual consumer are derived from the sequence of collection occasions recorded within the annual reporting period (refer to the Reporting Framework for the National Outcomes and Casemix Collection for further information). Episodes can only be derived if there is a valid sequence of collection occasions (e.g., an admission collection occasion followed by a discharge collection occasion). Episodes cannot be derived if there is an invalid sequence of collection occasions (e.g., a review collection occasion followed by an admission collection occasion) or where there is only a single collection occasion reported for a consumer in the reporting period.

Not all episodes are in-scope for reporting on outcomes based on the consumer-rated measure, specifically because some consumers are not offered the measure.

Outcomes classification

Public reporting of the outcomes from the NOCC is based on an effect size methodology. Specifically, mental health outcomes – that is, the difference or change between scores at the start and end of an episode of mental health care – were classified using an effect size metric as significant improvement, no significant change or significant deterioration. The advantage of this method is that change values derived from the different consumer-rated measures are converted into standardised units so that they can be combined for national reporting. For each measure, a medium effect size threshold of half a standard deviation (Cohen 1988) of the score was calculated from all admission collection occasions, separately for acute inpatient and ambulatory settings and for each outcome measure.

The effect size thresholds for each measure reported in this section are shown in Box NOCC.2. For example, for episodes in which consumer outcomes were based on the HoNOS family of measures, this corresponded to an absolute threshold of change score of 3 in both acute inpatient and ambulatory settings. Outcomes were then classified as significant improvement if the change score was 4 or more, no significant change if the change score was between -3 and 3, and significant deterioration if the change score was -4 or less.

Box NOCC.2. Medium effect size thresholds by measure
Measure	n^a	SD	Absolute threshold of change score^a	Interval of change scores for ‘significant improvement’		Interval of change scores for ‘no significant change’		Interval of change scores for ‘significant deterioration’
Measure	n^a	SD	Absolute threshold of change score^a	Max.	Min.	Max.	Min.	Min.	Max.
Acute inpatient settings
HoNOSCA	19,009	7.6	3	52	4	3	-3	-4	-52
HoNOS	247,221	7.4	3	48	4	3	-3	-4	-48
HoNOS 65+	21,446	7.5	3	48	4	3	-3	-4	-48
BASIS-32	167	0.9	0.4	4	0.5	0.4	-0.4	-0.5	4
K10	46,946	11.0	5	40	6	5	-5	-6	-40
MHI-38^b	439	37.3	18	-188	-19	-18	18	19	188
SDQ-YR	12,087	6.0	3	40	4	3	-3	-4	-40
Ambulatory settings
HoNOSCA	93,105	6.8	3	52	4	3	-3	-4	-52
HoNOS	376,144	6.4	3	48	4	3	-3	-4	-48
HoNOS 65+	70,634	6.3	3	44	4	3	-3	-4	-44
BASIS-32	18,240	1.0	0.5	4	0.6	0.5	-0.5	-0.6	-4
K10	74,197	10.6	5	40	6	5	-5	-6	-40
MHI-38^b	9,947	38.7	19	-188	-20	-19	19	20	188
SDQ-YR	50,184	6.1	3	40	4	3	-3	-4	-40

SD = standard deviation; Max.=maximum; Min.=minimum.

a) Represents all admission collection occasion scores in these settings.

b) The intervals of change for the MHI-38 have opposite signs to the other measures because, unlike the other measures, higher scores indicate better mental health. Therefore, a negative change score indicates improvement and a positive change score indicates deterioration.

Use of confidence intervals

This section makes use of confidence intervals to reflect some of the statistical variability (‘uncertainty’) in estimates derived from the NOCC. It is acknowledged that there are different views on the appropriateness of using inferential statistics, such as confidence intervals, for population parameters (Redelings et al. 2012), noting that the NOCC is intended to comprise the complete population receiving care from public sector specialised mental health services. We adopted the approach used by Public Health England, which recommends that a confidence interval should be presented alongside a point estimate whenever an inference is being made from a set of observations to the underlying process or ‘risk’ that generated them (Eayres 2008; Redelings et al. 2012). In this section, confidence intervals are shown in most figures and are included in the NOCC tables.

A confidence interval is a range of values that is used to quantify the random variability or fluctuations that can occur naturally, for example in the numbers of services used and of persons using services over time. Generally, confidence intervals describe how different an estimate could have been if the underlying conditions stayed the same but random variability had led to a different set of data (Eayres 2008).

A confidence interval does not quantify all variability inherent in a statistic. In the NOCC, a key source of variability is incomplete reporting. This can occur when a clinician does not collect a measure on a particular collection occasion as prescribed by the NOCC protocol, or when a consumer is not available to be offered a consumer-rated measure on a given collection occasion, for example at discharge where the consumer is ‘lost to follow-up’ and ‘administratively’ discharged. Other sources of non-random variability include systematic differences between jurisdictions in their implementation of the national protocol. For example, although the national protocol was updated in 2017–18 to capture the data element Phase of care, Victoria had not commenced its collection by 2019–20. Systems are in place to encourage standardised data collection, and to check for patterns of non-random variability (refer to Data validation, above), however, some non-random variability is likely to remain (Kreisfeld and Harrison 2020).

The width of the confidence interval is determined by 3 factors. The first factor relates to the extent of variability in the phenomenon being measured. In this section, almost all estimates derived from the NOCC are proportions, calculated by dividing the numerator by the denominator. The underlying distribution of a proportion is assumed to follow a binomial distribution, and the corresponding variability is taken into account in the calculation of the confidence intervals. Following the approach recommended by Public Health England (Public Health England 2018), the Wilson Score method was used to calculate the confidence intervals (Newcombe & Altman 2000; Wilson 1927). This method has the advantage of generating an interval when the numerator, and therefore the proportion, is zero. Because the binomial distribution is non-normal, the resulting confidence intervals are asymmetrical. That is, the size of the margin of error between the lower 95% value and the estimate will not necessarily be equal to the size of the margin of error between the upper 95% value and the estimate.

The second factor is the ‘level of confidence’, the desired probability that the interval includes the true value. In reporting of public health measures, a 95% level of probability is commonly used, and means that we can be 95% confident that the true value lies within the interval. Confidence intervals can be used to test for statistical differences between estimates. If the 95% confidence intervals for two reported estimates do not overlap, then there is 95% confidence that the difference between them is statistically significant. This is considered a conservative method; it is not always the case that overlapping confidence intervals do not indicate a statistically significant difference (Public Health England 2018). More exact methods are available but have not been used in this section.

The third factor is the population size from which the estimate is derived. Larger population sizes yield more precise estimates with narrower confidence intervals. In this section, estimates are provided for groups that vary widely in clinical population size. For example, there is at least a 30-fold variation in the number of people receiving clinical care from specialised mental health services in the Northern Territory compared to New South Wales. Similarly, there is wide variation in the size of some population subgroups, for example between the number of Indigenous Australians receiving care compared to non-Indigenous Australians, between the number of people living in very remote locations compared to major cities, and between the number of people receiving care in residential services compared to ambulatory services. In the absence of information about the precision of the estimate, small differences between groups or small fluctuations for a group over time could be incorrectly interpreted as meaningful (AIHW: Kreisfeld and Harrison 2020; Redelings et al. 2012)

Key concepts

Key concept	Description
Age band	A more detailed classification of age than age group. For consumers aged less than 18 years, age bands (less than 4 years, 4–10 years and 11–17 years) correspond to the groups specified by the NOCC protocol to be offered different versions of the consumer-rated and carer-rated measures.
Age group	The age group to which the patient or client has been assigned for the purposes of the NOCC protocol. Generally, Adult is defined as persons between the age of 18 and 64 years inclusive, an Older person is defined as persons aged 65 years and over and a Child or adolescent is defined as persons aged less than 18 years of age. In some circumstances a person may be legitimately assigned to a different age group to that in which they would be assigned on the basis of their actual age. For example, a person aged 60 years who was being cared for in an inpatient psychogeriatric unit may be assigned to the Older person age group.
Clinician-rated measure	Clinical measures are particular surveys or forms that are used to gather information about a person's clinical mental health status and functioning. Clinician-rated measures are completed by the clinician (mental health provider) about the consumer’s mental health. The NOCC includes the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) for children and adolescents, the Health of the Nation Outcome Scales (HoNOS) for adults, and the Health of the Nation Outcome Scales 65+ (HoNOS 65+) for adults aged 65 years and older.
Collection occasion	An occasion during an episode of mental health care when the required dataset is to be collected in accordance with a standard protocol. Three collection occasion types within an episode of mental health care are identified: Admission, Review, and Discharge.
Comorbid problems	The following are comorbid problems that clinicians consider when rating the Other mental and behavioural problems scale of the HoNOS (for adults) and HoNOS 65+ (for older persons): A Phobias – including fear of leaving home, crowds, public places, travelling, social phobias and specific phobias. B Anxiety and panics. C Obsessional and compulsive problems. D Reactions to severely stressful events and traumas. E Dissociative (‘conversion’) problems. F Somatisation – persisting physical complaints in spite of full investigation and reassurance that no disease is present. G Problems with appetite, over- or under-eating. H Sleep problems. I Sexual problems. J Problems not specified elsewhere including expansive or elated mood.
Confidence interval	A statistical term describing a range (interval) of values used to describe the uncertainty around an estimate. Generally speaking, confidence intervals describe how different the estimate could have been if the underlying conditions stayed the same but variability in sampling (i.e. selecting a different sample from the population) had led to a different set of data. Confidence intervals are calculated with a stated probability – usually 95% level of confidence – that, if the assumptions inherent in the calculation of the interval hold, the true value lies within the interval.
Consumer-rated measure	Clinical measures are particular surveys or forms that are used to gather information about a person's clinical mental health status and functioning. Consumer-rated measures are completed by the consumer about their own mental health. The NOCC uses the Strengths and Difficulties Questionnaire Youth Report (SDQ-YR) for children and adolescents, and the Behaviour and Symptom Identification Scale (BASIS-32), Kessler Psychological Distress Scale (K10+), or Mental Health Inventory – 38 (MHI-38) for adults, depending on the state or territory in which the consumer receives mental health care.
Consumer group	Consumer group refers to a classification of episodes types according to the setting in which treatment occurred. Three main episode types reported in this section are: Completed acute inpatient; Completed ambulatory; and Ongoing ambulatory.
Duration	The period of contact in an episode of mental health care. Duration is calculated as the number of days between collection occasions that form the start and end of the episode, including the episode start date.
Episode of mental health care	For the purposes of the NOCC, a period of more or less continuous contact between the consumer and a mental health service organisation within a single setting and for which there is both a ‘Start’ and an ‘End’ clinical rating within the reporting period. Two business rules apply to episodes: a) one episode at a time; and b) change of setting implies a change of episode.
Episode types	A classification of episodes of mental health care defined on the basis of the type of collection occasion, and reason for collection, at the ‘Start’ and ‘End’ of the episode, within the annual reporting period. The three categories are: Completed, Ongoing, and Closed. Completed episodes are those that started and ended within in the reporting period (e.g., Admission to Discharge). Ongoing episodes were still open at the conclusion of the reporting period (e.g., Admission to Review, or Review to Review). Closed episodes were already open at the commencement of the reporting period and closed within the reporting period (e.g., Review to Discharge).
In-scope	‘In-scope’ refers to the collection of information as specified in the NOCC protocol. In-scope collection occasions are collection occasions for which a given measure type is expected to be completed or offered as specified in the NOCC protocol. In-scope episodes are episodes for which a matched pair of ratings was expected to be completed according to the NOCC protocol.
Matched pair	A pair of collection occasions that form a valid sequence within an episode of mental health care, and for which the same measure was able to be rated on both collection occasions. A valid sequence is when collection occasions are logically ordered, for example an Admission collection occasion followed by a Discharge collection occasion. Conversely, an example of an invalid sequence is a Review collection occasion followed by an Admission collection occasion. In this section, NOCC ratings for an episode are categorised according to their completion status as follows: Matched pair and No matched pair.
Mental health legal status	Whether a person was provided care on an involuntary basis under the relevant state or territory mental health legislation, at some point during the period of care preceding the collection occasion.
NOCC coverage	The extent to which consumers included in the NOCC protocol are representative of the population receiving clinical care from public sector specialised mental health services. Coverage is derived by comparing the number of persons with at least one valid NOCC measure to the overall number of persons reported as receiving clinical care from public sector specialised mental health services.
NOCC protocol	The minimum requirement for the collection of the NOCC measures. Together, the three concepts of collection occasion (Admission, Review, Discharge), service setting (Inpatient, Residential, Ambulatory) and the consumers’ age group (Children and adolescents, Adults, Older persons) determine what measures to collect and when to collect them.
Outcome	A change in health status that can be attributed to specific health care investments or interventions (CIHI 2021).
Outcome classification	A classification of the extent of change between the clinical ratings at the ‘Start’ and ‘End’ of an episode of mental health care. Classification is based on statistical testing using an effect size metric. The categories are Significant improvement, No significant change, and Significant deterioration. A ‘medium’ effect size of 0.5 (Cohen, 1988)is used to assign change scores to one of the 3 outcome categories. A medium effect size is equivalent to an individual change score of at least one half (0.5) of a standard deviation. Individual episodes are classified as: ‘significant improvement’ if the effect size index is greater than or equal to positive 0.5; ‘significant deterioration’ if the effect size index is less than or equal to negative 0.5; or ‘no significant change’ if the index is greater than negative 0.5 and less than positive 0.5.
Period of care	The period bound by one collection occasion and another, and immediately preceding the current collection occasion.
Principal diagnosis	The diagnosis established after study to be chiefly responsible for occasioning the patient or client’s care during the period of care preceding the collection occasion. The principal diagnosis must be a valid code from the International Statistical Classification of Diseases and Related Health Problems, 10th Revision, Australian Modification (ICD-10-AM) (11th Edition).
Public sector specialised mental health services	Publicly funded or managed services with a primary function to provide treatment, rehabilitation or community health support targeted towards people with a mental disorder or psychiatric disability. These activities are delivered from a service or facility that is readily identifiable as both specialised and serving a mental health care function.
Service setting	The setting in which the episode of mental health care takes place. The categories are as follows. Inpatient: overnight care provided in public psychiatric hospitals and designated psychiatric units in public acute hospitals Residential: overnight care provided in residential units staffed on a 24-hour basis by health professionals with specialist mental health qualifications or training and established in a community setting which provides specialised treatment, rehabilitation or care for people affected by a mental illness or psychiatric disability Ambulatory: non-admitted, non-residential services provided by health professionals with specialist mental health qualifications or training.

References Expand

AIHW: Kreisfeld R and Harrison J 2020. Indigenous injury deaths: 2011–12 to 2015–16. Injury research and statistics series no. 130. Cat. no. INJCAT 210. Canberra: AIHW.

Burgess P, Pirkis J and Coombs T 2015. Routine outcome measurement in Australia. International Review of Psychiatry 27(4):264–75.

Burgess P, Trauer T, Coombs T, McKay R and Pirkis J 2009. What does 'clinical significance' mean in the context of the Health of the Nation Outcome Scales? Australasian Psychiatry 17(2):141–8.

Burns A, Beevor A, Lelliott P, Wing J, Blakey A, Orrell M, Mulinga J and Hadden S. 1999. Health of the Nation Outcome Scales for elderly people (HoNOS 65+). British Journal of Psychiatry 174:424–7.

CIHI (Canadian Institute for Health Information) 2021. Outcomes. Viewed 18 May 2021.

Cohen J 1988. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, New Jersey: Lawrence Erlbaum Associates Inc.

Eayres D 2008. Commonly Used Public Health Statistics and their Confidence Intervals. Viewed 10 February 2021.

Eisen SV, Dickey B and Sederer LI 2000. A self-report symptom and problem rating scale to increase inpatients' involvement in treatment. Psychiatric Services 51(3):349–53.

Eisen SV, Dill DL and Grob MC 1994. Reliability and validity of a brief patient-report instrument for psychiatric outcome evaluation. Hospital and Community Psychiatry 45(3):242–7.

Goodman R 1997. The Strengths and Difficulties Questionnaire: a research note. Journal of Child Psychology and Psychiatry 38(5):581–6.

Gowers SG, Harrington RC, Whitton A, Lelliott P, Beevor A, Wing JK, Jezzard R 1999. Brief scale for measuring the outcomes of emotional and behavioural disorders in children: Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA). British Journal of Psychiatry 174:413-6

Independent Hospital Pricing Authority 2018. Australian Mental Health Care Classification v1.0 User Manual. Sydney: IHPA.

Kessler R, Andrews G, Colpe L, Hiripi E, Mroczek D, Normand S, Walters E and Zaslavsky A 2002. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine 32(6):959–76.

NMHPSC (National Mental Health Performance Subcommittee) (2013) Key Performance Indicators for Australian Public Mental Health Services, 3^rd edn, NMHPSC, Australian Health Ministers Advisory Council’s Mental Health Drug and Alcohol Principal Committee (MHDAPC).

Newcombe RG and Altman DG 2000. Proportions and their differences. In: Altman DG, Machin D, Bryant T, Gardner M (edn) Statistics with confidence. 2nd edition. London: BMJ Books, 46–8.

Public Health England 2018. Technical Guide: Confidence Intervals. Version: 25 May 2018. Viewed 10 February 2021.

Redelings M, Sorvillo F, Smith L and Greenland S 2012. Why confidence intervals should be used in reporting studies of complete populations. The Open Public Health Journal 5:52–4.

Veit C and Ware JJ 1983. The structure of psychological distress and well-being in general populations. Journal of Consulting and Clinical Psychology 51(5):730–42.

Wilson EB 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158):209–12.

Wing J, Curtis R and Beevor A 1994. ‘Health of the Nation’: measuring mental health outcomes. Psychiatric Bulletin 18:690-1.

Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S and Burns A 1998. Health of the Nation Outcome Scales (HoNOS). Research and development. British Journal of Psychiatry 172:11-8.

Consumer outcomes in mental health care Print this page