Australia’s Disability Strategy 2021–2031 Outcomes Framework: Second annual report

Methodology for reporting on the progress of measures over time

To report on the progress over time of each measure in the Outcomes Framework, the most recent data point is compared to the baseline data (that is, the latest data available before December 2021, the starting point of the Strategy).

The change observed between the most recent data and the baseline data is used to assign one of the following 4 categories:

• Progress: measures with 2 or more data points, where the latest observed change since the baseline is in the direction that the Strategy wants to achieve.
• No change: measures with 2 or more data points, where the latest data remain similar to the baseline.
• Regress: measures with 2 or more data points where the latest observed change since the baseline is in the opposite direction to what the Strategy wants to achieve. 
• Not enough data: measures where there is only a single data point available (usually the baseline).

Measures with no available data (future measures) are listed at the end of each domain section.

Measuring progress

Commenting on whether a measure has improved or not, considers the following:

• Confidence: Is there confidence in the change (that is, that the change is real and not a product of chance or some underlying uncertainty in the data)?
• Importance: Is the change important (that is, the size of the change)?

Confidence

Determining confidence in the change is based on a technical evaluation. Where appropriate, statistical methods are used to assist with determining confidence where there are 2 or more data points.

For sampled data, sampling error is considered for determining whether the change is statistically significant, based on overlapping confidence intervals and z-tests.

For administrative data, the following criteria are considered:

• the source of the data
• whether the data are complete for the full population or only covers a sub-set
• the timeliness of the data
• the reliability of any disability indicator including:
  • the likely accuracy of the data collected
  • the level of missing data and whether that level is changing over time
• whether there are any changes to, or inconsistencies in, the metadata between sub‑populations (for example, jurisdictions), or over time
• the magnitude of any observed difference
• whether there is any seasonality observed in the data
• where historical data are available for the measure:
  • whether the observed difference from the benchmark is consistent with past observations and trends (based on linear regression analysis)
  • the size of any change compared to the variability historically observed in the data.

Importance

Importance refers to whether an observed change is worthwhile in the context of a set of subject-specific considerations. For example, a change may be ‘real’ in statistical terms, but not important if the change is small.

Information is provided in relation to the size of the change, so that readers can determine whether the change is an important one.

To avoid the misinterpretation of very small changes, those that are <0.5 of a percentage point in either direction have been classified as ‘no change’. 

Regression analysis

The linear regression analysis used for reporting looks only at changes over time for the summary measure. It does not control for multiple characteristics, as would be done, for example, with multiple regression modelling.

The Australian Institute of Health and Welfare undertook all regression analysis.

Rounding

Percentages in the report are generally rounded to whole numbers except for those less than 10% which are rounded to 1 decimal place. Exceptions include for the reporting on latest results, where rounding to whole numbers would obscure differences between the baseline and the latest results. 

Impact of COVID-19 on baseline results

The Strategy’s start date, and the baseline data point for many indicators, fall within the period that restrictions were still in place in Australia to reduce the spread of COVID-19 in the community. This should be taken into account when reviewing changes over time against the baseline, particularly in relation to measures in the Employment and financial security outcome area.