A print friendly PDF version is available from this Communicable Diseases Intelligence issue's table of contents.

Observational methods are important in the measurement of vaccine effectiveness (VE) as experimental designs cannot be used for measurement of vaccines already on the vaccination schedule. Furthermore, efficacy measured in clinical trials under ideal conditions may differ to effectiveness in the field under non-ideal conditions and in different populations. In addition to post-licensure surveillance, observational VE studies are particularly important when disease incidence does not predictably decrease with increased vaccine coverage, when high proportions of vaccine failure among reported cases suggest a problem with the vaccine or when issues arise that were not predicted in pre-licensure evaluations. Commonly used study types for evaluating VE include cohort studies, household contact studies, case-control studies, the screening method and case-cohort studies. There are many potential biases in all observational VE studies which should be considered in the study design and analysis stage. Of the five observational study types reviewed, cohort studies undertaken during an outbreak investigation offer the simplest means of VE estimation and is the preferred study design where the situation permits. Where this is not possible the screening method is the most economical and rapid method. It is essential that the effectiveness of all vaccination programs be evaluated. As new vaccines are introduced to the schedule, booster doses are added and the timing of doses changed, the role of observational methods in the evaluation of VE will become even more important. To date, few observational VE studies have been undertaken in Australia, suggesting the under-utilisation of these methods. Commun Dis Intell 2002;26:451-457.

Top of page

Introduction

Vaccine efficacy is the percentage reduction of disease incidence in a vaccinated group compared with an unvaccinated group, under ideal conditions. Vaccine efficacy studies are typically undertaken pre-licensure using double-blind randomised controlled trials, with all participants initially susceptible to the disease.¹ Once a vaccine has been shown to be efficacious and is licensed, the use of a placebo is unethical. Therefore an experimental design cannot be used for vaccines on the vaccination schedule and so observational methods must be employed. Furthermore, efficacy measured in clinical trials under ideal conditions may differ to effectiveness in the field under non-ideal conditions and in different populations.²

Vaccine effectiveness depends upon vaccine efficacy but is also affected by other factors such as transportation and storage at appropriate temperatures ('cold chain') and proper administration and timing of doses. The terms 'vaccine effectiveness' and 'vaccine efficacy' are often used interchangeably and the abbreviation VE is used for both vaccine efficacy and effectiveness.³ In this review VE is used as an abbreviation for vaccine effectiveness.

The Australian vaccination schedule is constantly changing as new vaccines are introduced, booster doses are added and the timing of doses changed.⁴ To maintain public and provider confidence in vaccination programs, it is essential that the effectiveness of new vaccines and changes to the schedule of existing vaccines be evaluated. The evaluation of current vaccines/schedules should also be monitored to enable detection of variations in effectiveness over time which may result from changes in the target population or in the epidemiology of the disease. In the case of new vaccines these effectiveness studies may be incorporated as a component of post-licensure surveillance. In addition to post-licensure surveillance, observational vaccine effectiveness studies are particularly important when disease incidence does not predictably decrease with increased vaccine coverage, when high proportions of vaccine failure among reported cases suggest a problem with the vaccine or when issues arise that were not predicted in pre-licensure evaluations.² However, it should be noted that when most of the population are vaccinated, most cases will be vaccine failures, so a high proportion of vaccine failures is not necessarily indicative of a declining vaccine effectiveness or efficacy.

A number of observational methodologies can be used in the assessment of vaccine effectiveness, some of which may be incorporated into routine surveillance of vaccine preventable diseases. This paper discusses the potential biases and limitations of observational VE studies, outlines five commonly used study types and provides examples from the literature of where these study types have been used.

Calculating VE

All VE studies involve comparison of the relati
ve risks of disease in the vaccinated group(s) with the unvaccinated group(s), hence any study type from which relative risk can be estimated can be used to calculate VE.² The standard equation for calculating VE as a percentage is:²



where ARU is the attack rate in the unvaccinated group and ARV is the attack rate in the vaccinated group. Rearranging the formula gives the following:²



where is equivalent to the relative risk. In case-control studies the relative risk is approximated by the odds ratio.

Protection against what? Defining the study question

Immunisation may produce more than one effect, both at the individual and population level.³ Individual effects include the production of an immunologic response, protection against infection or in some cases only against disease or severe disease, a reduction in the degree or duration of infectivity, or even behavioural effects such as changes in the rate of contact with potentially infectious sources.³ Population effects include a reduction in transmission of disease and/or infection. When designing a study to estimate VE, it is important to clearly define the question of interest, in particular whether individual and/or population effects are of interest, as this determines the appropriate choice of unit of observation, comparison group, parameter of effect, and level of information required.³

The question of interest is dependent upon the objective of the control program. If the objective of a control program is to reduce morbidity then high coverage with a vaccine protecting only against disease, or even severe disease, may be satisfactory.⁵ In contrast if herd immunity or eradication is the goal, then the vaccine must clearly protect against infection.⁵

Top of page

Potential biases in all study types

Any factor which differentially raises or lowers the apparent attack rate in either the vaccinated or unvaccinated group will bias the VE estimate. There are many potential biases in observational VE studies that need to be minimised in either the design or analysis phase of a study. In addition, the results should be presented in such a way that enables the reader to judge the extent to which potential biases have operated and to estimate their impact on the estimation of VE.²

Case definition

Ideally, case definitions should be sensitive and specific. Whilst a high sensitivity gives a more precise estimate of VE, the point estimate is not unduly affected by a low sensitivity as long as the case definition has equal sensitivity in the vaccinated and unvaccinated groups.² This is a problem with pertussis, as vaccinated persons often experience a milder form of the disease which is less likely to fit a clinical case definition. In these situations, it is the effectiveness of the vaccine against more serious disease, rather than against all disease or infection, that is being estimated.

For VE estimation the specificity of a case definition is generally more crucial than its sensitivity, as the misclassification of other illness as cases would equalise the attack rates in the two groups resulting in a falsely low VE estimate.² The rarer the disease, and the greater the incidence of misclassified illness, then the greater the bias toward low VE.² This bias is even greater when the case definition has low sensitivity.²

Point estimates of pertussis VE increase with increasing specificity of clinical case definitions² or when based on clinically severe or bacteriologically positive cases.⁵ The same phenomenon has been observed in pertussis vaccine trials, where the greater the clinical severity of cases accepted as pertussis, the higher the VE estimates.⁶ However, although laboratory confirmation increases specificity, it may lead to other biases as a result of problems with case ascertainment.²

Case ascertainment

In a pre-licensure trial, bias in case ascertainment is minimised by randomisation and by blinding the observer to vaccination status, neither of which is generally possible in an observational study. In observational studies, vaccinated and unvaccinated persons are self selected groups who may not have equal access to health care services, hence equal case detection cannot be assured.²

Studies using passively notified cases are particularly prone to bias in case ascertainment, as individuals with disease may not all have an equal probability of being notified. If notifications correlate with good public health practice and easy access to medical services and hence are associated with high vaccine uptake, then vaccinated individuals may be preferentially notified, resulting in an underestimate of VE.⁵ Or if, as is the case with pertussis,^6,7 the vaccine gives greater protection against more severe disease and there is a correlation between clinical severity and the probability of a physician recognising and then notifying a case, VE will be overestimated.⁵ A more serious problem occurs if, independently of disease severity, unvaccinated cases are more (or less) likely to be recognised and/or notified than vaccinated cases. For example a physician's knowledge that a child is fully vaccinated against pertussis could reduce the index of suspicion that an illness is in fact pertussis, resulting in an overestimate of VE.⁵ The extent of this bias is difficult to estimate.

A pertussis outbreak investigation in the United Kingdom in 1987 found that only 31 of 90 children with bouts of coughing lasting for two or more weeks followed by whooping, vomiting or choking/turning blue (probable cases), were notified.⁸ Using notified cases only the VE estimate was 88 per cent. This fell to 75 per cent when probable cases were included and 68 per cent when the case definition included all children with bouts of coughing lasting at least 2 weeks.⁸ The author found that notified children were younger and less likely to be vaccinated, suggesting that children were less likely to be diagnosed and notified as pertussis if they were known to have been vaccinated.

Top of page

Ascertainment of vaccination status

Classification errors in vaccination status reduce VE estimates unless there is a bias towards misrepresenting vaccinated cases as unvaccinated.⁵ Studies relying on parental recall of a child's vaccination status tend to overestimate vaccination coverage, whereas studies which require verification with written records may underestimate vaccination coverage.⁹

VE estimation for diseases against which more than one dose of vaccine is necessary for full protection require information on the number of doses of vaccine given. If partial vaccination affords some protection against disease, then the way partially vaccinated cases are handled in the analysis can affect the VE estimate. If partially vaccinated cases are classified as unvaccinated but still receive some protection, the attack rate in the unvaccinated is lowered, whereas classifying partially vaccinated cases as fully vaccinated will raise the attack rate in the vaccinated. If the effectiveness of the full course of vaccination is being measured, then cases who are partially vaccinated should be excluded from the analysis wherever possible.²

Comparability of vaccinated and unvaccinated groups - potential confounding

In randomised controlled trials potentially confounding variables are randomly distributed among the experimental and control groups. In observational studies the groups may differ in many ways, only some of which may be recognised by the investigator.² Unrecognised or unmeasurable differences between the experimental and control groups such as increased susceptibility due to poor nutrition in unvaccinated marginalised groups may pose serious threats to validity. However, the most important potential confounder in VE studies is exposure to disease. VE calculations generally assume equal exposure to infection in vaccinated and unvaccinated individuals or groups. Exposure to infection may in turn be associated with variables such as age and place of residence.

For a variable to be considered as a confounder it must be independently related to both the risk of disease and to vaccination status. Not all variables which differ in frequency between vaccinated and unvaccinated groups fulfil this requirement. Orenstein et al² give the example of a case-control study whereby cases, by definition, will have been more exposed to disease than controls but this difference in exposure does not bias the VE estimate unless the probability of exposure is also related to the probability of being vaccinated. If groups who have a greater risk of exposure (e.g. children who attend day care) are more likely to be vaccinated, then VE will be underestimated.

The indirect effects of vaccination can affect the probability of exposure in both vaccinated and unvaccinated groups, but not necessarily to an equal extent. If VE is estimated at a population level, where the study group is comprised of exposed and unexposed individuals, then whether or not the vaccine has been administered randomly will have a significant effect on the estimate.¹⁰ If groups with high vaccine coverage are at low risk of exposure to infection, for example due to herd immunity, and VE is viewed as the degree of protection afforded to an individual who has been exposed to the disease, then clustering of vaccine status in the population may produce falsely high VE estimates.^5,10 Similarly, groups with low vaccine coverage may have greater exposure to infection resulting in falsely low VE estimates.² However, if the overall effectiveness of the vaccination program is being studied, then it is appropriate to include the indirect protection of vaccination in the calculation.

Age

Age may be associated with both the probability of vaccination and the probability of having had prior exposure to the disease. Where immunity from disease and/or vaccination is acquired at a young age and diminishes with time, age may be a proxy measure for time since vaccination. Data should be analysed separately for narrow age groups or otherwise standardised for age.⁵

Prior disease

If prior disease is not associated with vaccination status then VE estimates will be unbiased.2 However, vaccinated and unvaccinated groups may differ with respect to prior disease, in which case ignoring previous histories may bias the VE estimate.² However, the effect of this bias must be weighed against the problem of obtaining valid histories, which for some diseases may not be feasible.

Top of page

Study types

A variety of observational methods may be used to estimate VE including the well established cohort and case-control design. Each methodology has its advantages and disadvantages and methodological problems have been identified for all study types.

Cohort studies

A cohort design is most appropriate when a discrete population at risk can be defined.¹¹ Most cohort studies of VE have been retrospective and have generally been undertaken as part of an outbreak investigation. The cohort, often based in a school, child care centre or geographically defined area, is defined and the vaccination status of all members of the cohort is ascertained. The relative risk of disease in the vaccinated compared with the unvaccinated group is then calculated thus enabling the calculation of VE. Examples of disease outbreaks in Australia where VE has been measured using a cohort study design include a pertussis outbreak in an Australian Capital Territory school¹² and measles outbreaks in Western Australia,¹³ the Australian Capital Territory,¹⁴ Central Australia¹⁵ and Queensland.¹⁶

Orenstein et al¹ list five criteria which minimise bias in cohort studies which are part of an outbreak investigation:

• absence of substantial prior disease activity in the studied age group;
• both vaccinated and unvaccinated individuals are included in the study population;
• adequate numbers in the population in the age group to be studied;
• high overall attack rate; and
• good vaccination records available.

Household contact studies

Household contact studies are used to measure the secondary attack rate of disease in household contacts of index cases. VE is calculated by combining the total population of the households under study, excluding the primary and co-primary cases, to form vaccinated and unvaccinated cohorts.² The methodology corrects for potential differences in exposure between vaccinees and non-vaccinees, thus reducing the bias that may result from differential exposure.¹ Orenstein et al¹ comment that next to outbreak investigations, this technique has been evaluated most and is an acceptable alternative to outbreak investigations. However, Fine and Clarkson⁵ point out that the relative simplicity of the household secondary attack method should not be taken as license for its uncritical application and interpretation. No Australian household contact studies were identified in the literature.

Pertussis VE estimates derived from household secondary attack rates are generally lower than those obtained by other methods, regardless of diagnostic criteria used for case ascertainment.¹⁷ One possible bias in these study types, which relates to pertussis vaccines and level of exposure, is the assumption that these vaccines are less effective under conditions of heavy exposure such as that within households.⁵ The study of family contacts of ascertained cases, which are highly selected populations, may introduce bias. If vaccine uptake is non-random, then most or all of the vaccinated individuals in the study will be included because of a prior vaccine failure in the household (i.e. the index case). Possible risk factors for vaccine failure are likely to be shared by members of the household thus introducing a bias against the vaccine.

Again assuming non-random vaccine uptake and the likelihood that household contacts share the same vaccination status of the primary case, studies of situations in which pertussis is introduced to the household by a vaccinated case may be biased in favour of the vaccine.⁵ This arises from the reduced severity of disease in vaccinated persons which may result in close contacts being exposed to fewer bacilli than the contacts of unvaccinated individuals, thus reducing the risk of infection preferentially among vaccinated contacts and raising VE estimates.

Households with larger rather than smaller numbers of cases are more likely to be identified and included in a study.5 This ascertainment bias is likely to lower vaccine effectiveness as households in which the vaccine is working best would be selectively excluded from the study.⁵

Top of page

Case-control

In a case-control study, cases are selected on the basis of having the disease of interest, and controls on the basis of being comparable to cases but without having the disease so that the odds ratio of vaccination can be calculated. The traditional VE equation cannot be used in case-control studies¹ as cases represent one sampling fraction of all cases and the controls represent a different sampling fraction of the population that is not ill.¹⁸ As the sampling fraction is unknown, the total populations of vaccinated and unvaccinated people cannot be calculated and therefore neither can attack rates.¹ However, for rare diseases, the odds ratio approximates relative risk and so can be used to estimate VE. Although the VE estimate will be erroneously high when the attack rate in vaccinated persons is greater than 10 per cent,¹ in non-outbreak situations this is usually not the case and therefore the error will not be of an important magnitude.¹ In Australia case-control studies have been used to estimate VE for measles in Western Sydney¹⁹ and for Haemophilus influenzae type b infection in Aboriginal children in Western Australia.²⁰

Screening method

Using the screening method, VE is estimated by comparing the proportion of cases who are vaccinated (PCV) with the proportion of a comparable group in the population who are vaccinated (PPV). The standard VE equation can be rearranged to give the screening method equation, which is:²¹



The screening method is a simple and rapid way of estimating VE which has been used to estimate pertussis VE in the Netherlands,²² the United States,²³ Nova Scotia,²⁴ the United Kingdom,²⁵ and New Zealand.²⁶ In Australia it has been used to estimate the VE of Haemophilus influenzae type b²⁷ and, more recently, pertussis.²⁸ The screening method is particularly useful for routine monitoring of VE or in circumstances where data on the vaccination status of cases only are available. Provided that any biases remain reasonably constant, the screening method may be used for monitoring changes in VE over time. It should not be relied upon for precise VE estimates.¹ An overestimate in PPV will result in an overestimate of VE and this error is particularly noticeable when vaccine coverage is greater than 80 per cent.¹

Care must be taken to stratify the data by possible confounding variables such as age and location. If different population groups have different coverage figures then the groups should be analysed individually. Farrington²¹ illustrates the effect of pooling population coverage figures in an example of two cohorts, A and B, of equal size. In cohort A there are 100 cases, 50 of whom are vaccinated and the PPV is 0.9. In cohort B there are 10 cases, one of whom is vaccinated and PPV is 0.5. The screening method VE estimate is 89 per cent in each cohort. However, if the cohorts are combined, then there are 110 cases, 51 of whom are vaccinated, while the combined value of PPV is 0.7 which produces a VE estimate of only 63 per cent.²¹

Case-cohort

This study type is also known as case-base and is similar to the screening method except that vaccination status is sampled in population controls rather than using an assumed true value of PPV.²¹

Top of page

Discussion

In summary, there are a variety of observational methods which can be used to estimate VE, none of which is perfect. The screening method is the most economical and rapid means of determining whether there is a major problem with the vaccine, as all that is required is a reliable estimate of the proportion of cases who are vaccinated and an estimate of the vaccine coverage in the population at risk.¹ If the screening method results suggest that VE is lower than expected, this should be confirmed by more rigorous methods. Of the more accurate observational methods available, cohort studies undertaken during an outbreak investigation offer the simplest means of VE estimation and is the preferred study design where the situation permits.¹ The most appropriate study design will depend upon the specifics of the particular situation such as availability of resources, access to records, the number and distribution of cases and the availability of population coverage data.

Whilst results obtained using observational methods may be distorted due to unavoidable bias, it may still be possible to calculate a sufficiently good estimate of VE for operational purposes.²⁹ Potential biases should be considered in the design phase of a VE study and steps taken to minimise them if possible. All reports of VE studies should include a discussion of the biases which may have been operating and their possible effects on VE estimates. Provided that these steps are taken, observational methods provide valuable tools for the evaluation of vaccination programs. To date, few observational VE studies have been undertaken in Australia, suggesting the under-utilisation of these methods.

Top of page

Acknowledgements

We are grateful to Ross Andrews, Heather Gidding and Stephen Lambert for their valuable comments.

Top of page

References

1. Orenstein WA, Bernier RH, Dondero TJ, Hinman AR, Marks JS, et al. Field evaluation of vaccine efficacy. Bull World Health Organ 1985;63:1055-1068.

2. Orenstein WA, Bernier RH, Hinman AR. Assessing vaccine efficacy in the field. Further observations. Epidemiol Rev 1988;10:212-241.

3. Halloran ME, Struchiner CJ, Longini IM. Study designs for evaluating different efficacy and effectiveness aspects of vaccines. Am J Epidemiol 1997;146:789-803.

4. McIntyre P, Amin J, Gidding H, Hull B, Torvaldsen S, et al. Vaccine preventable diseases and vaccination coverage in Australia, 1993-1998. Commun Dis Intell 2000;24 Suppl:S1-S83.

5. Fine PE, Clarkson JA. Reflections on the efficacy of pertussis vaccines. Rev Infect Dis 1987;9:866-883 (Review).

6. Fine PE. Implications of different study designs for the evaluation of acellular pertussis vaccines. Dev Biol Stand 1997;89:123-133 (Review).

7. Cherry JD, Olin P. The science and fiction of pertussis vaccines (commentary). Pediatrics 1999;104:1381-1384.

8. Palmer SR. Vaccine efficacy and control measures in pertussis. Arch Dis Child 1991;66:854-857.

9. Lister S, McIntyre P, Burgess M, O'Brien ED. Immunisation coverage in Australian children: a systematic review. Commun Dis Intell 1999;23:145-170.

10. Mühlemann K, Weiss NS. Can herd immunity influence the assessment of vaccine efficacy in nonrandomized studies? (letter). Am J Public Health 1997;87:113.

11. Chen RT, Orenstein WA. Epidemiologic methods in immunization programs. Epidemiol Rev 1996;18:99-117.

12. Herceg A. Bordetella pertussis in an ACT school: outbreak investigation and vaccine efficacy study. Commun Dis Intell 1993;17:284-286.

13. Jeremijenko AM, Kelly H, Patel M. The high morbidity associated with a measles outbreak in a West Australian town. J Paediatr Child Health 1996;32:382-385.

14. Cheah D, Lane JM, Passaris I. Measles vaccine efficacy study in a Canberra high school: a study following a measles outbreak. J Paediatr Child Health 1993;29:455-458.

15. Patel M, Lush D. Measles vaccine effectiveness in Central Australian Aboriginal children vaccinated at or after eight months of age. Aust N Z J Public Health 1998;22:729-730.

16. Gidding HF, Hills S, Selvey L, Roberts LA, Johnston S. An outbreak of measles in a rural Queensland town in 1997; an opportunity to assess vaccine effectiveness. Commun Dis Intell 1999;23:240-245.

17. Fine PEM, Clarkson JA, Miller E. The efficacy of pertussis vaccines under conditions of household exposure. Int J Epidemiol 1988;17:635-642.

18. Schlesselman JJ. Case-control Studies. Design, Conduct, Analysis. New York: Oxford University Press, 1982.

19. McDonnell LF, Jorm L, Patel MS. Measles outbreak in western Sydney. Vaccine failure or failure to vaccinate? Med J Aust 1995;162:471-475.

20. Bower C, Condon R, Payne J, Burton P, Watson C, et al. Measuring the impact of conjugate vaccines on invasive Haemophilus influenzae type b infection in Western Australia. Aust N Z J Public Health 1998;22:67-72.

21. Farrington CP. Estimation of vaccine effectiveness using the screening method. Int J Epidemiol 1993;22:742-746.

22. de Melker HE, Schellekens JFP, Neppeelenbroek SE, Mooi FR, Rümke HC, et al. Reemergence of pertussis in the highly vaccinated population of the Netherlands: observations on surveillance data. Emerg Infect Dis 2000;6:348-357.

23. Guris D, Strebel PM, Tachdjian R, Bardenheier B, Wharton M, et al. Effectiveness of the pertussis vaccination program as determined by use of the screening method: United States, 1992-1994. J   Infect Dis 1997;176:456-463.

24. Halperin SA, Bortolussi R, MacLean D, Chisholm N. Persistence of pertussis in an immunized population: results of the Nova Scotia enhanced pertussis surveillance program. J   Pediatr 1989;115:686-693.

25. Ramsay M, Farrington C, Miller E. Age-specific efficacy of pertussis vaccine during epidemic and non-epidemic periods. Epidemiol Infect 1993;111:41-48.

26. Blakely T, Mansoor O, Baker M. The 1996 pertussis epidemic in New Zealand: vaccine effectiveness. N Z Med J 1999;112:118-120.

27. Herceg A. The decline of Haemophilus influenzae type b disease in Australia. Commun Dis Intell 1997;21:173-176.

28. Torvaldsen S, Simpson JM, McIntyre PB. Effectiveness of pertussis vaccination in New South Wales, Australia, 1996 to 1998. Eur J Epidemiol; In press.

29. Smith PG, Rodigues LC, Fine PEM. Assessing the protective efficacy of vaccines against common diseases using case-control and cohort studies. Int J Epidemiol 1984;13:87-93


This article was published in Communicable Diseases Intelligence Volume 26, No 3, September 2002