Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

2.3 Information sources and methods

Evaluation of the Child Health Check Initiative and the Expanding Health Service Delivery Initiative - Final Report

Page last updated: 17 April 2012

The evaluation involved a mixed-method design, drawing on multiple sources of information and involving four phases of work. Details on the activities undertaken at each phase are included in Appendix C.

The methods used in the evaluation included:

  • key informant interviews
  • case studies, involving interviews, observation, administered questionnaires and document review
  • workshops on EHSDI-related issues with program partner organisations
  • analysis of existing population health data sets for CHCI
  • review and analysis of program and program-related data
  • review of program documents and research literature.
A description of each method follows. Appendix C includes two matrices that demonstrate how the mixedmethod design was used to address each of the evaluation objectives.

2.3.1 Key informant interviews

We interviewed informants who had played a key role in the NTER, the CHCI, the EHSDI or wider NT PHC system reform. The majority of informants were current or past staff of Australian Government departments such as DoHA and the Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA); the DHF; and AMSANT, including members of the NT AHF and the PHRG. Other informants were representatives of professional bodies such as the Australian Medical Association (AMA) and the Australian Indigenous Doctors’ Association (AIDA); staff of the RAHC; and a co-chair of the Board of Inquiry into the Protection of Aboriginal Children from Sexual Abuse. The interviews collected largely qualitative information on peoples’ experiences and views on the CHCI, the EHSDI and the NT health service and system more generally.

2.3.2 Case studies—the CHCI and EHSDI

Five communities within four regions (Barkly, Central Australia, East Arnhem and Katherine East) were selected by the project team and confirmed by the evaluation partners as suitable case studies for collecting more detailed information on people’s experiences and views of the CHCI and the EHSDI. This could then be added to the key informant interviews at a national and NT level. Community participants’ views were seen as crucial for understanding the impact of the programs.

The case study sites were selected to reflect a range of regions and communities in population size, location (regions in the Top End and Central Australia), the type of health service (government or community controlled), progress toward regional reform and funding (the per capita gap in funding to the EHSDI benchmark).

Case study methodology

We invited people in each community to participate in interviews with a community member and/or health service staff acting as an intermediary between the evaluation team and community members. Interviews were mainly face to face, although some were completed by telephone. A group of practitioners who had completed placements in the NT under the RAHC were also interviewed by telephone. The interviews were based on a set of 12 core case study questions (see Appendix C). Each interview took from 20–90 minutes.

At the regional level the participants included members of regional steering groups, members of area health services or boards, DHF Area Service Managers, and regional administrative, clinical and support staff. We got a strong sense that people were willing to participate in the evaluation and share their experiences and stories, in spite of the generally high level of consultation fatigue and perceptions of interference in remote Aboriginal communities from evaluators and researchers. A number of participants clearly felt that talking to us and reflecting on their experiences with the CHCI in particular, would help them to move on from these experiences.

We were able to interview the majority of staff in the five community health clinics, including some staff who had been at the clinics at the time of the CHCI but had since left. We also interviewed enough parents and guardians to give us confidence that we were capturing a variety of experiences. The one exception to this was the relatively low number of male parents/guardians who participated in interviews. This is probably a reflection of the interview process, with women being more likely to be identified as parents/guardians, rather than men not wanting to talk about the CHCI given its association with child sexual abuse and the past portrayal of Aboriginal men as the main perpetrators of such abuse. No males refused to be interviewed and those that were interviewed spoke freely about their experiences. Top of page

We interviewed 154 people as part of the case studies, including 87 people in the five case study communities.

2.3.3 Workshops—the EHSDI

We held two workshops as part of the formative evaluation of the EHSDI. The workshops involved participants from the evaluation partner organisations including members of the MoU Management Committee, the PHRG and the IAG. The workshops were designed to identify and discuss issues and ideas relating to ‘hot topics’. These were identified and agreed with participants at the time and then fed back to those working on the EHSDI programs to help improve the ongoing implementation of the EHSDI. The evaluation team played the role of a critical friend in the workshops, offering an outsider’s view on the topics and asking challenging questions about the program in a constructive and collaborative way.

The first workshop was held in October 2009. It involved 17 participants and focused on three issues— partnership, capacity and communication. The second workshop was held in May 2010. It involved 21 participants and focused on the issue of sustainability. The issues and ideas identified in the workshops were reported back to the evaluation partners. The information has also contributed to the analysis in this final report.

2.3.4 Comparing data from the CHCI to other health datasets

The evaluation draws on existing child health check and follow-up datasets held by AIHW, and additional analyses of these collections specific to our case study sites. The ability to compare data from the CHCI with other sources was limited.

We worked with the DHF and the AIHW to determine whether the population of children who received a child health check was similar to the population eligible for a child health check but who did not have one. This involved matching the child health check dataset with a number of datasets held by the DHF, including the NT Hospital Morbidity dataset, the NT Midwives dataset, and the Growth Assessment and Action (GAA) dataset. The results are presented in Section 4.2.

We identified a number of potential sources of data for comparative analyses but met with limited success.

To inform current and future child health check programs, we proposed to work with DHF and AIHW to undertake a similar analysis for the GAA program, which is the NT Government’s ongoing program aimed at improving the growth and nutritional status of children 0–4 years of age living in remote areas.6 The GAA was selected because it has comprehensive coverage (around 79 per cent of resident children aged 0–4 years) and because it is included in the core PHC services in the NT (DHF 2009a). We were unable to negotiate suitable access to GAA data to complete this analysis within the time frame of this evaluation.

We also considered doing a similar analysis for the Healthy School-Age Kids (HSAK) program, which is the NT Government’s ongoing program aimed at improving the health, wellbeing and learning outcomes of schoolage children living in remote communities in the NT. This program is more comprehensive than the GAA as it involves checks for a wide range of conditions including trachoma, skin conditions, hearing, and dental health; however, the program’s coverage is not as comprehensive and for this reason we did not proceed with the comparative analysis.

The AIHW had already completed several comparative analyses covered by evaluation objective 2 (validating the prevalence and severity of conditions found through the child health checks with other data sources), the findings of which are summarised in Table 3. The AIHW commented on the comparability of these data sources with the CHCI dataset, noting differences in the methods of data collection and the way the participants were recruited. Differences included: children being recruited into a research study through a sampling strategy as opposed to voluntarily attending a health check; the definition of the conditions considered; the age of children being considered; the geographical area covered by the collections; and the timing of collection (AIHW and DoHA 2009).

For a number of conditions the findings from the CHCI data were similar to those from the other data sources. The most notable exception is for middle-ear condition, although the data sources for this condition are not directly comparable for the reasons given above and due to the differences in diagnostic criteria, the type of equipment available to those making the assessments, and the skill and experience of the assessors themselves.Top of page

Table 3: Summary of findings from comparisons of CHCI data to other data sources
ConditionCHCI data (prevalence)Other data source (prevalence)
Ear disease33% (aged 4–5 years)31% (aged 4–5 years)(a)
Middle-ear condition30% (aged 0–5 years)91% (aged 6–30 months)(b)
Skin sores10% (aged 4–15 years)
9% (aged 0–14 years)
16% (aged 4–15 years)(a)
6% (aged 0–14 years)(c)
Scabies8% (aged 0–14 years)13% (aged 0–14 years)(c)
Ring worm6% (aged 4–15 years)
6% (aged 0–14 years)
8% (aged 4–15 years)(a)
15% (aged 0–14 years)(c)
History of rheumatic heart disease1.1% (aged 0–14 years)0.5% (aged 0–14 years)(d)
History of asthma6% (aged 0–14 years)7% (aged 0–14 years)(e)
Anaemia11% (aged 4–15 years)
27% (aged 0–4 years)
11% (aged 4–15 years)(a)
25% (aged 6 months–4 years)(f)
Stunting6% (aged 0–4 years)11% (aged 0–4 years)(f)
Underweight11% (aged 0–4 years)14% (aged 0–4 years)(f)
Wasting10% (aged 0–4 years)10% (aged 0–4 years)(f)
Smoker in household77%82%(e)

(a) - HSAK screening program.
(b) - Morris et al 2001 study of middle-ear infection.
(c) - East Arnhem Regional Healthy Skin Project.
(d) - Top End and Central Australian rheumatic heart disease registers.
(e) - National Aboriginal and Torres Strait Islander Health Survey 2004–05.
(f) - GAA program.
Source: AIHW and DoHA (2009).

We identified two recent datasets that had not been compared to the CHCI data—the Audit and Best Practice in Chronic Disease (ABCD) program dataset and a recent study by Peter Morris (Morris and Leach 2009), as well as earlier studies (Morris 1998; Morris et al 1999; Morris et al 2005). We wanted to see if it was possible to compare the prevalence of ear disease among Indigenous children in the NT with the data collected through the CHCI and to determine if any differences were significant, and the reasons behind the differences.

Comparisons were limited by several factors, not the least of which was the purpose for which this data was collected. The CHCI datasets were not developed with specific research purposes in mind, and were not intended to be used for comparative analyses. The methodology used to diagnose and classify conditions varied widely. The teams involved in Morris’ study, for example, were highly trained in recognising and diagnosing ear disease in children; whereas this was not always the case with the CHCI teams. The CHCI looked at point prevalence of ear disease whereas ABCD looked for a history of chronic ear disease.

Conditions like anaemia which are more readily comparable (that is, it is either present or not), presented difficulties because of the different age bands being checked and variations in geographic locations covered in the reporting. The numbers involved in the ABCD programs were relatively small. Coupled with the issues of accessing data at a community level, the small numbers would limit the reliability of comparisons at the community level where data from the CHCI and ABCD could be matched.

Similar prevalence and incidence of conditions was found by the CHCI as by other screening programs, except where there is a specific disease or condition focus for the screening program in which higher rates were generally found.

2.3.5 Review and analysis of program and program-related data

In addition to the health datasets, the evaluation examined financial data on allocation and expenditure on the CHCI and the EHSDI; deployment data collected on the RAHC; workforce data relating to the EHSDI; and health workforce and population data more generally.

We used this data for a variety of purposes:
  • to directly address evaluation questions on the RAHC and the distribution of resourcing
  • to validate data collected through other sources (in particular the qualitative information collected through interviews)
  • to help develop the context for the evaluation
  • as a means of understanding the environment in which the results of the evaluation will be used.

2.3.6 Review of program documents and research literature

Throughout the evaluation we reviewed a number of documents relating to the programs, including:
  • Ampe Akelyernemane Meke Mekarle—Little Children are Sacred from the Northern Territory Board of Inquiry into the Protection of Aboriginal Children from Sexual Abuse (Anderson and Wild 2007)
  • submissions on the 2009 review of the NTER and the Australian Government and NT Government response to the review (FaHCSIA 2009b)
  • program documents including area health service plans and program status reports
  • policy papers on the systems-wide reform elements of the EHSDI (such as on NT AHKPIs, CQI, core services and hubs)
  • research literature on PHC in remote communities and Aboriginal health.
The evaluation team sourced documents from the evaluation partners, the PHRG website, published peer review journals, and through the internet.
Our literature search included a number of media sources covering information leading up to the NTER. These sources were particularly important given there was no definitive policy document on the CHCI; however, we do acknowledge that media sources are not a proxy for policy documentation and are not treated in that way in this evaluation.

References are cited in this report and a full reference list is included.

2.3.7 Profile of evaluation participants

Table 4 shows the profile of the participants in the case studies, key informant interviews and workshops. There was a total of 175 participants.

Table 4: Profile of evaluation participants
Participant groupNumberPer cent
Parents/guardians of children2917
Health service staff4123
Other community members106
Members of regional steering groups, area health services or boards63
Regional administrative, clinical and support staff1810
RAHC practitioners106
NT Government sector2011
NT non-government sector15(a)9
Australian Government sector1810
Australian non-government sector (including individuals)85

(a) - Eight of whom were AMSANT officials.

6 - The GAA program involves routinely collecting measurements of children’s weight, height/length and haemoglobin. In March 2009 the DHF began piloting a new program called Health Under 5 Kids. This program complements the GAA.

Top of page