An initial review of the documentation/data provided by DoHA identified some challenges. These challenges not only had implications for the extent of analysis which could be conducted but also highlighted limitations with existing data collection to support the implementation of the Evaluation Framework in subsequent stages of the Evaluation. The following issues were highlighted:
- Data quality and availability was inconsistent across projects
- Due to the small number of EDRs received, access to disaggregated activity level data was limited
- Information regarding target groups and geographical coverage was limited
- Data collection categories that were in use showed a lack of alignment with key external data sources such as the Australian Institute of Health and Welfare (AIHW), Australian Bureau of Statistics (ABS) or the National Health Data Dictionary (Version 15)
- Activities under LIFE Action Areas were not clearly articulated
- Important contextual information was absent for many projects, for example, issues/challenges encountered during the start-up phase, externalities such as non-NSPP funding, volunteer staff inputs and local factors were not always in these reports
- Evaluation reports were not available for all projects as not all projects were required to conduct evaluations under their funding agreements. The quality of the evaluation reports received was variable.
These consultations, in turn, highlighted a further range of issues that needed to be considered in both the Mid-term Assessment and the broader Evaluation. These issues included:
- Individual projects engage in multiple LIFE Action Areas and use multiple approaches and are therefore not easily clustered for analysis purposes
- Not all projects were required to complete EDRs. Narrative-style reports were the norm for many projects, thus restricting the level of statistical analysis possible
- Project staff expressed concern that reliance on pre-existing data for the Mid-term Assessment might not provide a comprehensive picture of their activities, achievements, and the overall journey of their projects. In particular, concerns were raised regarding:
- Whether AHA was in receipt of a complete set of reports/documentation. Until that point, all communication with the projects had been through a third party (either DoHA Central Office or State/Territory Office). Project representatives were concerned that reports/documentation may have been overlooked.
- Reliance on existing reports/documentation could mean that important contextual information relating to the development and evolution of projects could be lost.
Additionally, based on the feedback obtained at the workshops and discussions with DoHA, the list of high-risk groups identified in the LIFE Framework (Section 3.3.1) was expanded in both the survey and MDS to include:
- Children
- Youth
- People with a mental illness
- People who engage in self-harm
- People from Culturally and Linguistically Diverse backgrounds (CALD)
- Refugees
- Older people
- People with Alcohol and other Drug (AOD) problems
- Workforce settings
- Communities that experience redundancies
- Communities that experience natural disasters.