Evaluation of suicide prevention activities

4.3 Documentation/data quality: implications

Page last updated: January 2014

An initial review of the documentation/data provided by DoHA identified some challenges. These challenges not only had implications for the extent of analysis which could be conducted but also highlighted limitations with existing data collection to support the implementation of the Evaluation Framework in subsequent stages of the Evaluation. The following issues were highlighted:

  • Data quality and availability was inconsistent across projects
  • Due to the small number of EDRs received, access to disaggregated activity level data was limited
  • Information regarding target groups and geographical coverage was limited
  • Data collection categories that were in use showed a lack of alignment with key external data sources such as the Australian Institute of Health and Welfare (AIHW), Australian Bureau of Statistics (ABS) or the National Health Data Dictionary (Version 15)
  • Activities under LIFE Action Areas were not clearly articulated
  • Important contextual information was absent for many projects, for example, issues/challenges encountered during the start-up phase, externalities such as non-NSPP funding, volunteer staff inputs and local factors were not always in these reports
  • Evaluation reports were not available for all projects as not all projects were required to conduct evaluations under their funding agreements. The quality of the evaluation reports received was variable.
These findings highlighted the need for more consistent and comprehensive data collection in the overall Evaluation. A Minimum Data Set (MDS) was developed by AHA for this purpose (Appendix C), informed by the initial documentation/data review and by the needs of the Evaluation Framework. The first draft of the MDS was presented to project representatives at workshop consultation forums held throughout Australia in August 2012 at which all 49 projects were represented.

These consultations, in turn, highlighted a further range of issues that needed to be considered in both the Mid-term Assessment and the broader Evaluation. These issues included:

  • Individual projects engage in multiple LIFE Action Areas and use multiple approaches and are therefore not easily clustered for analysis purposes
  • Not all projects were required to complete EDRs. Narrative-style reports were the norm for many projects, thus restricting the level of statistical analysis possible
  • Project staff expressed concern that reliance on pre-existing data for the Mid-term Assessment might not provide a comprehensive picture of their activities, achievements, and the overall journey of their projects. In particular, concerns were raised regarding:
    • Whether AHA was in receipt of a complete set of reports/documentation. Until that point, all communication with the projects had been through a third party (either DoHA Central Office or State/Territory Office). Project representatives were concerned that reports/documentation may have been overlooked.
    • Reliance on existing reports/documentation could mean that important contextual information relating to the development and evolution of projects could be lost.
In response to the issues raised during these forums, an online survey (Appendix D) was developed by AHA to address the gaps in the pre-existing data and to provide projects with the opportunity to provide direct input to the Mid-term Assessment. Further refinement of the MDS was also undertaken following feedback from project representatives at the workshops (see Section 4.8 for details of the final MDS).

Additionally, based on the feedback obtained at the workshops and discussions with DoHA, the list of high-risk groups identified in the LIFE Framework (Section 3.3.1) was expanded in both the survey and MDS to include:

  • Children
  • Youth
  • People with a mental illness
  • People who engage in self-harm
  • People from Culturally and Linguistically Diverse backgrounds (CALD)
  • Refugees
  • Older people
  • People with Alcohol and other Drug (AOD) problems
  • Workforce settings
  • Communities that experience redundancies
  • Communities that experience natural disasters.