Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Executive summary

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

This evaluation
Key findings—the Child Health Check Initiative
Key findings—the Expanding Health Service Delivery Initiative


This report provides findings and recommendations from an evaluation of the Child Health Check Initiative (CHCI) and the Expanding Health Service Delivery Initiative (EHSDI). The evaluation, commissioned by the Australian Government Department of Health and Ageing (DoHA), has been undertaken independently and the interpretation, conclusions and recommendations in this report are those of the authors.

The two programs (the CHCI and the EHSDI) represented contrasting approaches to addressing the health needs of remote Aboriginal communities in the Northern Territory (NT). The programs were implemented as part of the Northern Territory Emergency Response (NTER).

The CHCI was launched as a centrally-driven program, without following a normal policy development process or engaging in consultation with stakeholders outside central government. Policy decisions focused on the clinical and logistical aspects of the child health checks, and engagement with the NT health sector was largely on specific implementation issues. The lack of suitable policy development meant there was insufficient regard for the diversity of the people, systems and processes already operating in the NT, and insufficient focus on existing bottlenecks within the NT health system. The program achieved some credible successes in areas such as dental, hearing and ear, nose and throat (ENT) health; however, its overall impact was dulled by a lack of precision in the way it interacted with communities and the existing health care system.

The EHSDI, on the other hand, is an ongoing program being built on a rich history of innovation and health system development in the NT. The Australian Government, NT Government and Aboriginal Community Controlled Health Organisations (ACCHOs) are playing an active part. The program is building on commendable models of Aboriginal controlled primary health care (PHC) services as well as governance, quality and accountability approaches that have been developed in response to the specific context of providing PHC in NT remote Aboriginal communities. While the program faces ongoing challenges, the EHSDI’s achievements to date are a cause for celebration.

A considerable gap remains between the health outcomes of the Indigenous people of the NT and non-Indigenous Australians in the NT and across Australia. The developing PHC service models in the NT have the potential to contribute to closing this gap provided they are sustained and adequately resourced.

This evaluation

The approach taken to evaluating the two programs differed. The CHCI was largely completed at the time of this evaluation, therefore the approach taken was summative. The EHSDI, on the other hand, was at an early stage of implementation. Given the more emergent nature of the program, we took a formative approach to its evaluation. The choice of evaluation approaches has affected not only the way we designed the evaluation but also the way we have reported on it. We are able to report on the CHCI program, its effectiveness and impacts. For the EHSDI, we are reporting to the program to support continuous improvement in its ongoing implementation.Top of page

The evaluation incorporated three overarching objectives of effectiveness, efficiency and appropriateness, as well as four specific objectives for the CHCI and six for the EHSDI.

The CHCI evaluation objectives

  1. Assess the extent to which the child health checks reached the target population.
  2. Identify the prevalence and, if possible, the severity of the health conditions found through the child health checks and validate these findings with data from other sources.
  3. Assess the extent to which requested PHC, allied health and specialist follow-up services have been received, gaps in existing health service delivery and barriers to completing follow-up treatment.
  4. Explore the possibility of undertaking more complex evaluative analyses which could include questions about:
    1. whether or not the CHCI has led to improvements in health service delivery for Aboriginal and Torres Strait Islander children
    2. the health status of children in relation to the social determinants of health and access to comprehensive primary health care
    3. treatment outcomes.

The EHSDI evaluation objectives

  1. Assess the impact and sustainability of the EHSDI on PHC service delivery and on the equitable distribution of resources.
  2. Assess the extent to which Aboriginal and Torres Strait Islander people were engaged and empowered to contribute to health service planning, governance and responsiveness of services.
  3. Assess the impact and sustainability of the Remote Area Health Corps (RAHC) on health workforce availability and flexibility in the NT.
  4. Assess the efficiency of the EHSDI in terms of how well it has maximised health service delivery with the available funds.
  5. Assess the effectiveness of the EHSDI in achieving change in health status.
  6. Assess the impact of the regional reform process on:
    1. efficient and effective operation of health services
    2. clinical governance, including quality of health services delivery
    3. information systems and planning capacity.

The CHCI evaluation objectives have largely been met, including analysis of the reach of the program, provision of follow-up services and impacts of the CHCI on the health system and health status. Where the CHCI evaluation objectives had already been significantly addressed by the Australian Institute of Health and Welfare (for example objective 2), we have provided additional commentary and discussion.

Addressing the EHSDI evaluation objectives was more complex, as the evaluation was formative and conducted in a changing and developing context. The discrete elements of the EHSDI are intended to work together as part of a wider process of reform in the NT PHC system. We have addressed the evaluation objectives of the impact and sustainability of the EHSDI on health service delivery, distribution of resources, the PHC workforce and engaging Aboriginal and Torres Strait Islander people in health service planning and governance of the wider reform process. It was premature to address the effectiveness of the EHSDI in achieving change in health status (objective 5). We have, however, considered the likelihood of the initiative contributing to improved health status in the future. Although regional reform had not yet progressed sufficiently to assess its impact on the areas outlined in objective 6, we have included discussion on the more immediate impacts of the regionalisation process in the report.

The evaluation was conducted from June 2009 to March 2011 and involved four key phases of work:
  • Phase 1: Design—this involved developing an evaluation design to address each of the evaluation objectives.
  • Phase 2: Implementation—the evaluation objectives were addressed through a mixed method design involving: key informant interviews; case studies; workshops on EHSDI-related issues with the program partner organisations; an analysis of existing population health data sets with regard to the CHCI; review and analysis of program and program-related data; and review of program documents and research literature. The case studies were based on five remote Aboriginal communities within four regions and aimed to collect information on people’s experiences and views of the CHCI and the EHSDI.
  • Phases 3: Consultation, communication and dissemination; and Phase 4: Analysis and reporting— these phases overlapped and involved feedback and consultation with a range of audiences including regional health staff and staff of the three agencies leading the reform and development of PHC services in the NT—the Aboriginal Medical Services Alliance of the NT (AMSANT), the NT Department of Health and Families (DHF)1, and DoHA.

Strengths and limitations

The evaluation accounted for context by collecting information through case studies and by considering what was happening in child health and the NT PHC system before and outside the CHCI and EHSDI. The formative evaluation of the EHSDI was responsive to the emerging nature of the program and engaged with participants during the evaluation process to facilitate ongoing improvement in program implementation. The evaluation reported on the views of 175 participants from all levels of the NT PHC system.

The evaluation approach was limited by several factors including difficulty in generalising from context-rich data and in attributing changes to either the CHCI or EHSDI program when there are many other initiatives targeting remote Aboriginal communities. Some of the quantitative data analysis undertaken for the evaluation of the CHCI was limited by the complexities inherent in identifying and comparing the group of children who received a child health check and the group of children who did not. The results from the analyses that could be done should be treated with caution.Top of page

Key findings—the Child Health Check Initiative

The design of the CHCI did not follow normal policy protocols

The CHCI arose from an intense national political process that precluded the development of any specific policy documents. In this environment the Medicare Benefits Schedule (MBS) Item No. 708 (an existing health check program for Aboriginal and Torres Strait Islander children aged less than 15 years) became the de facto policy for the child health checks; however, it was not designed for the specific conditions in remote NT communities and is usually carried out as part of an ongoing relationship between a child and a health clinic or health practitioner. This was not the case in the delivery of the CHCI. The lack of adequate policy processes meant that the child health checks did not follow international best practice for screening programs. There is no evidence of officials considering recognised best practice guidelines for designing screening programs such as those issued by the World Health Organization (Wilson and Jungner 1968).

There was a ‘bottleneck’ in referral systems

Approximately 70 per cent of all children who had a child health check received at least one referral to followup care. The NT health system, which was not adequately coping with the existing number of referrals from PHC, did not have the capacity, processes, infrastructure or workforce to provide an increased volume of follow-up health services in a short time frame. This aggravated a ‘bottleneck’ in the pathway to referred services. The proportion of children who had been given a referral following a child health check but who had not been seen by the follow-up service was 19.6 per cent for PHC services, 34.2 per cent for ENT specialist services, 39.8 per cent for dental services, 42.0 per cent for paediatric services, 45.4 per cent for tympanometry and audiometry services, and 57.4 per cent for other specialist services.

The CHCI brought considerable resources into the NT health system

The NT remote PHC sector had long been under-funded (McDermott et al 1997; Rosewarne & Boffa 2004; Mooney and Henry 2004) and the extra financial resources that accompanied the CHCI were welcome and needed. From 1 July 2007 to 30 June 2010 a total of $54.469 million was spent on the initiative ($17.935 million on the child health checks and $36.535 million on follow-up services). This represents an average cost per child of $1,691 for the child health checks, and an average cost per follow-up service of $1,181 for dental services and $1,842 for hearing/ENT services. By way of comparison, the benefit paid for the MBS Item No. 708 child health check is around $200 per child and research suggests that the average cost of a 60-minute face-to-face medical consultation in remote Aboriginal communities in the NT in 2003–04 was around $504. While these costs are not directly comparable, it does suggest that the CHCI, as a new health screening program, involved a substantial financial outlay (including administrative costs) and that the benefits that occurred in areas such as follow-up dental and hearing services could have been achieved more directly.

There was a lack of engagement with, and disruption to, existing systems

The centrally-driven approach to the CHCI meant that there was insufficient consideration of the needs of the people, systems and processes already operating in the NT. Teams of visiting professionals often conducted the checks in a ‘fly-in/fly-out’ model. Many of these teams appeared to do their best to engage with local health service staff and the community. Nevertheless, for many health services the checks disrupted normal clinic business and other services were sometimes suspended while the checks were carried out. This represented a significant opportunity cost as staff attention was diverted to conducting the checks, supporting visiting teams or working to overcome community scepticism and fear about the checks.

Those who received a child health check were more likely to have been previously hospitalised and were more likely to participate in existing child health screening programs

The CHCI achieved an overall coverage rate of between 57 and 65 per cent of the total eligible population. The program achieved greatest coverage in small communities and among children aged 2–9 years and lowest coverage in large communities and children aged 14–15 years. While not directly comparable, the coverage rate for younger children, estimated at between 56.4–69.4 per cent, compares with a 69.0 per cent coverage rate for the routine but less comprehensive NT GAA2 program.

Comparison of the health characteristics of the populations who did and did not receive a health check suggests that there was little difference in the health status of the two groups. Those who received a child health check were more likely to have been previously hospitalised and more likely to have attended existing child health screening programs available in the NT, compared to the population who did not receive a check. Top of page

New service delivery models for hearing/ENT and dental were program successes

Funding provided through the CHCI enabled the development of new service delivery models for hearing/ENT and dental services. Improved case management practices in hearing/ENT services have resulted in more precise tracking of children through the system, reducing the risk of children falling through the gaps. A new dental service delivery model helped overcome workforce shortages during the roll out of follow-up services. These new models have the potential to provide more efficient service delivery in the future, providing these services continue to be funded and are developed within the context of a comprehensive PHC approach.

Data generated was used effectively to focus attention on the NT

The CHCI added a significant amount of data to the PHC system. There is evidence that the CHCI data has been used to focus Australian Government attention on the needs of Aboriginal children in the NT. DoHA has regularly been asked at Senate Estimates hearings about the progress of the checks and follow-up referrals, as well as the prevalence of the specific conditions found. The CHCI dataset provided the basis for DoHA responses to the committee. The CHCI data provided decision makers with quantifiable, tangible and timely evidence of the health needs of Aboriginal children and of the ability of health services to meet these needs, in a way that was not previously possible.

There is evidence that larger NT health provider organisations have made use of the CHCI data to improve programs and to inform reviews on the treatment of some conditions; however, smaller organisations and providers with fewer resources do not have the capacity to make use of the data. There is a widespread view among NT health professionals that the data did little to increase local understanding of the population health needs of Aboriginal children as this information was already well known to health providers.Top of page

Key findings—the Expanding Health Service Delivery Initiative

Engagement with existing processes

An important feature of the EHSDI is its engagement with existing reform processes in the NT. The EHSDI was preceded by a 15-year period of development in the NT health system under programs such as the Coordinated Care Trials, Primary Health Care Access Program, the establishment of the NT Aboriginal Health Forum (AHF) and the Pathways to Community Control framework to support the transition to community control. The EHSDI was built on the successes and lessons learned from these initiatives.

The EHSDI significantly increased funding to the NT remote health system

The EHSDI also significantly increased the financial resources available to the NT health system. The initiative added around 17 per cent to remote PHC funding in the NT in 2008–09 and around 29 per cent in 2009–10. The health workforce increased by 251 full-time equivalent positions over a two-year period. EHSDI funding enabled increased investment in remote health infrastructure such as staff accommodation and clinic buildings.

The RAHC added to workforce capacity in the NT remote health sector

The RAHC model focuses on one workforce issue—recruiting health professionals for short-term placements—and has been successful in deploying 439 health professionals to more than three quarters of all NT remote health services. Other longstanding workforce issues remain including the difficulty in recruiting and retaining permanent staff. The current arrangements for recruiting, training and supporting Aboriginal Health Workers are also a major concern. The future of the RAHC needs to be considered within the context of effectively and sustainably addressing these wider workforce issues.

Regionalisation was under-scoped and under-resourced

The NT PHC sector is engaged in a complex change process which is attempting to balance the health system’s need for efficient health service provision based on economies of scale, with the desire for greater community control of health services. Regionalisation processes have been under-scoped and under-resourced, with a lack of clarity on the tasks involved, actions required and who is appropriate to undertake them.

Communication within and across the NT AHF partners concerning increasing Aboriginal community control and regionalisation has not been consistent and there appear to be varying degrees of acceptance of the concept. The process will remain problematic while mixed messages from the leading organisations persist.

There is insufficient policy capacity in the NT

The EHSDI reform process has not been accompanied by sufficient increased policy and administrative capacity in the major partner organisations in the NT. The bulk of the implementation policy work has been undertaken by the Primary Health Reform Group (PHRG); however, the existing policy capacity is insufficient to sustain the pace and effectiveness of the reforms. The organisations involved in implementing the PHC reforms need to re-assess where the current policy capacity is being applied and ensure that there is sufficient policy capacity in the right place to effect the intended changes.

Existing governance frameworks in the NT have served the developments in the sector well

The NT AHF has an important role in the strategic governance of the EHSDI and the PHRG has been formed to oversee the implementation of the program. The extensive development of structures and relationships before the EHSDI meant that the sudden increase in resourcing could be managed within existing frameworks.The commitment to partnership between the Australian Government, NT Government and Aboriginal community controlled health sector, and to NT-based decision making, is commendable.

The system is moving towards more equitable distribution of financial resources

Before the EHSDI, health services received funding from a number of sources including DoHA program funding, DoHA regional health service funding and the NT Government. The amounts received by each service varied considerably with some providers receiving much higher levels of funding per head of population than others. The EHSDI funding for service expansion was distributed using a benchmark methodology to determine which proposed Health Service Delivery Areas (HSDAs) were priorities for
additional funding. This contributed to a more equitable distribution of resources at a regional level.

The benchmark will need further development before it can be routinely used as a costing model for delivering services in remote areas. There is still a considerable way to go before the health services for remote populations have the funding required to provide a full range of PHC services.

The NT Aboriginal Health Key Performance Indicators (NT AHKPIs) are a significant achievement of the current reform process

The NT AHKPIs are a unique feature of the NT remote health system, providing a system-wide tool to gather data on health service activities and outputs. With further development, including expanding the scope of the indicators and supporting health service staff to use the results to inform health service planning, the NT AHKPIs have the potential to both support continuous quality improvement activities and strengthen accountability processes.

2 - The GAA program is the NT Government’s ongoing program aimed at improving the growth and nutritional status of children 0–4 years of age who live in remote areas. It involves routinely collecting measurements of children’s weight, height/length and haemoglobin. In March 2009 the DHF began piloting a new program called Healthy Under 5 Kids. This program complements the GAA.

Top of page