4.10 Evaluability issues

The key evaluability issues that apply to this Report fall into two main groups:

• Internal data limitations
• Broader challenges in evaluating suicide prevention programs

4.10.1 Internal data limitations

As outlined in Section 4.3, the pre-existing project documentation/data had a number of shortcomings. At the core of these is the fact that this pre-existing documentation/data was specifically designed to address DoHA's standard reporting requirements and was not compiled with this Evaluation in mind.

Among the key shortcomings identified were:

• Inconsistency in data quality and availability across projects
• Data collection categories that were not aligned with other key external data sources
• Inadequate data collection for key variables that were critical to the Report and the overall Evaluation
• Missing or incomplete data
• Short timeframes for the Evaluation.

While the project survey addressed many of these gaps, a more refined analysis of specific activities based on target groups and setting was not possible prior to the implementation of the MDS. A number of MDS specific limitations also exist. These include:

• Missing data as a result of non-submission of monthly MDS data by the projects
• Inconsistencies in data quality as a result of data entry errors or omissions by projects
• The six-month time frame for MDS implementation meant that the full spectrum of activities was not captured for all projects (see Chapter 6 for further details).

4.10.2 Broader challenges in evaluating suicide prevention programs

Although internal data limitations have been outlined (Section 4.3), it is also important to acknowledge the significant challenges related to the evaluation of suicide prevention programs, which are well recognised in the sector. These include:

• The actual number of people who take their own lives is a statistically rare event. This makes it difficult to achieve the statistical power that is necessary to identify patterns and causation, or to draw conclusions about reductions in the suicide rate.
• There is limited suicide data on specific target groups, data on protective and risk factors, pathways to suicide and mental health statistics. This creates difficulties in understanding the impact of programs on target groups.
• Issues of attribution: suicide prevention programs do not operate in isolation. They are provided in an environment where other contextual factors are present. For example, the presence of other programs or improvements in economic or social circumstances can also have an effect on the suicide rate. It is therefore difficult to separate out these effects from the program itself.
• Barriers exist to establishing longitudinal effects of programs on reductions in the suicide rate. Small program size and short program duration can diminish the statistical power of studies and thus limit the ability to establish causation and assess the effects of the program.
• While the quality of Australian deaths information is high by world standards, the ABS acknowledge that '[t]here remain considerable challenges in improving the quality of suicide data, particularly in relation to timeliness, consistency of process across jurisdictions and improving the identification of Aboriginal and Torres Strait Islander peoples at the time of death.'³² Some have argued that ABS figures underestimate the total figures.³³
• The issue of evidence. Much has been written on suicide prevention generally, yet the issue of what constitutes evidence of success remains contested. This is because:
  • While randomised control trials or quasi-experiments are often considered the gold standard in terms of evidence, the conduct of such trials is often unfeasible/inappropriate in the suicide prevention context because of the complexity of causality described above and for ethical and/or funding reasons.
  • Reliance on peer-reviewed publications as an evidence source is itself problematic because of publication biases and the lag that exists between innovation, established practice, research and publication.
  • Established practice is often considered to afford the best opportunities to collect evidence and as a consequence, this can make it hard to achieve a balance between innovation and established practice in the published studies.
  • Reliance on peer-reviewed publications as the primary source of information may fail to acknowledge valuable local insights that smaller projects have to offer. ^34,35

³² DePoy & Gitlin, Introduction to research
³³ RFG Williams, DP Doessel, S Jerneja & D de Leo, 'Accuracy of Official Suicide Mortality Data in Queensland', Australian & New Zealand Journal of Psychiatry, DOI: 10.3109/00048674.2010.483222, vol 44, no 9, 2010, pp.815-822.
³⁴ DoHA, LIFE: Research and Evidence in Suicide Prevention.
³⁵ M Nordentoft, 'Crucial Elements in Suicide Prevention Strategies', Progress in Neuro-Psychopharmacology & Biological Psychiatry, vol 35, no 4, 2010, pp.848-53.