Outcome measurement represents a significant change and challenge for many projects and for the Department, as it represents a new and expanded mode of reporting. It is important, therefore, to highlight the key issues that may arise so that change can be appropriately managed. These issues were identified through consultation with the projects and the Department throughout this evaluation.
As indicated in each of the causal mechanism models (Figure 12-2 to Figure 12-5), outcome measurement in the NSPP context presupposes data collection at two levels:
12.4.1 Project level
One of the key lessons learned from administering the MDS is that there is a general willingness among NSPP-funded projects to participate in data collection. This is evidenced by the fact that all 47 eligible projects submitted data during the snapshot period (October 2012 to March 2013). All required data was provided by 44 of the 47 projects (94%) and partial data by the remaining three projects. Furthermore, positive feedback was provided either in writing or verbally from many of the projects. Examples of written feedback include:Overall, the portal is brilliant. It makes entering the data quick and simple.
The reports are really useful for us. We’ll be able to use them for our business purposes here.
What you have designed is very comprehensive.
Discussions with project representatives at the workshops held in August 2012 also indicated a willingness to collect outcome data.
Despite this general willingness, a number of barriers were evident that could have implications for the introduction of outcome measurement at project level:
- Engagement with projects highlighted that the majority are focussed on the business of service delivery.
- For some, outcome measurement is perceived as research and therefore outside their core business of service delivery.
- Many lack training in outcome measurement, ie, selection, identification, administration and analysis of appropriate tools.
- Project effects have generally been reported using narrative accounts or invalidated tools in the past.
- Projects were generally unaware of the range of outcome measurement tools currently available. They also indicated that a repository of NSPP-appropriate outcome measurement tools and guidance in the use of these tools would be welcomed.
- Where resistance occurs, it is likely to be on the grounds of inadequate time and limited staff availability. In some cases this may be justified, particularly among those projects with large volumes of clients and few staff.
- For some projects, outcome measurement may be impractical, eg, those operating crisis lines where they do not have the capacity to follow-up.
- Loss to follow-up is likely to be raised as an issue by projects for establishing long-term outcomes.
- Non-compliance may be an issue. As noted above, not all projects completed the MDS. This is also likely to be the case with outcome measurement.
Additional considerations for the Department include:
- Cessation of funding will mean that project staff will be unable to follow up with service users in later years to establish the long-term outcomes of project activities. Funding duration at project level may need to be reviewed or alternative follow-up arrangement considered.
- Assessing NSPP-funding applicants to determine their capacity to undertake the required level of service delivery and data collection (including outcome measurement).
- Investment in capacity building at project level to ensure compliance with service delivery and data collection and outcome measurement requirements. This could involve expanding the role of existing NSPP-funded projects. For example, under its funding agreement, the National Centre of Excellence in Suicide Prevention is tasked with 'conducting educational workshops on a range of topics agreed by the department to provide NSPP projects with additional support, for example, evaluation and data collection practices'.
- Facilitating collaborations and partnerships between projects and the research sector to bridge the evidence-practice gap. Again, this could involve expanding the role of existing NSPP-funded projects. For example, in its funding agreement, the Suicide Prevention Australia (SPA) Strategic Partnership is tasked with 'building and consolidating alliances with researchers and centres of excellence'.
- Introduction of accountability measures for non-compliant projects.
- Outcome evaluation at state/territory and national levels is a matter for public health specialists. Top of page
12.4.2 National level
While the introduction of outcome measurement at project level will do much to address the lack of information on project-specific outcomes, it is important to note that no matter how well outcome measurement is undertaken at project level, it is not possible to establish a direct correlation between individual project-level activities and reductions in the suicide rate nationally. It is for this reason that 'governments look at suicide rates as the main outcome measure to determine the efficacy of their policies' in the absence of other more proximal measures being available.114Much has been written on the problematic nature of suicide data in Australia (see Section 4.10.2). If changes in national suicide statistics are to serve as a surrogate measure of long-term project outcomes, reliable national suicide statistics are as crucial to outcome measurement in the NSPP.
Consultations with key stakeholders in the preparation of this report highlighted the need to include suicide attempts, not just completed suicides as key indicators of change. This would require consideration of implementing a suicide attempt register and the follow-up of people who used services as a result of attempted suicide or self-harm, to establish what happens in the 12 months after contact with the service or emergency department.
Data linkages are crucial if an analysis at this level is to be achieved.
114 Williams et al, 'Accuracy of Official Suicide Mortality Data in Queensland', p819.