The GAVI Alliance welcomes the Institute for Health Metrics and Evaluation (IHME) report

The GAVI Alliance welcomes the Institute for Health Metrics and Evaluation (IHME) report and is now working with the independent GAVI data task team, established earlier this year, to take forward and unravel its implications. The head of IHME, Dr. Chris Murray, has been invited to join the task team.

Experts agree that there exists no single gold standard for measuring the effectiveness of aid programme results and performance. Aid agencies rely on administrative or self-reported data, household surveys and ad hoc audits.

Household survey data, as used by Chris and his team, tends to generate systematically lower coverage data than that produced by WHO and UNICEF, perhaps because it depends on mothers remembering if, when and how often their children were vaccinated.

The decision by the GAVI Board in 2000 to use administrative data for the ISS initiative was a conscious one. Timely data is required for performance management. In addition, the Board felt strongly about strengthening the capacity of governmental data systems and not creating parallel processes.

An independent evaluation commissioned in 2007 confirmed that ISS has helped improve immunisation rates in countries. However, it also recognised that some countries may under- or over-report their immunisation rates. As a result, GAVI commissioned a further study by the Swiss Tropical Institute to look into immunisation data collection and quality in the ISS programme.

Earlier this year, GAVI established a task team of international experts to assess these studies and how data is used by the Alliance. Members include the World Health Organization, UNICEF, the World Bank, the IHME, UNAIDS, the Norwegian government, health ministries from developing countries, the Harvard School of Public Health and The Lancet journal.

Pending the task team's analysis of the two studies' methods and findings, including proposals for a revision to improve the ISS programme, GAVI has suspended further ISS payments; the Alliance is also reviewing countries where there is significant variance and reporting when using other methodologies.

As the IHME study notes, there is no finding of misuse or misappropriation of ISS funds. In the past, when GAVI has identified potential misuse in any of its programmes, support was immediately suspended; investigations were initiated in accordance with national legal/judicial requirements and funds recovery efforts were launched.

GAVI has been a pathfinder for performance-based funding that has resulted in more children being immunised. Performance based funding could have great potential in improving access to health for poor people. Two challenges remain:

  • Further research to understand why different survey models and administrative reporting deliver differing results;
  • Better design, including checks and balances on performance funding in order to meet management goals, particularly as immunisation coverage has significantly increased in recent years.

As David Bishai notes in a commentary published in The Lancet, the real impact of the IHME study should be to encourage researchers to understand why survey reports do not agree with administrative reports.

The WHO/UNICEF position is as follows: 

WHO and UNICEF stand by their vaccination coverage estimates based on various sources of information (country reported data, surveys from countries and those reported in literature, information on vaccine supply, changes in immunization policy, and consultation with regional and national immunization experts.) 

Trends in immunization coverage continue to be positive and due to the increase more than 2.5 million deaths are prevented every year. Global vaccination coverage has risen from 20% in 1980 to 81% in 2007. 

WHO recognizes the challenge of generating accurate vaccination coverage data, particularly in the developing world, where reporting systems need strengthening. The new methodology proposed by Lim et al. will assist some countries to improve their data quality by pointing out inconsistencies. 

However, some points of concern in the proposed methodology need to be addressed first, such as: 

  1. the use of survey data from one country to fill gaps in vaccination coverage estimate models for other countries with limited or no coverage data; 
  2. the heavy emphasis on surveys which, in practice, have differing levels of quality and rely on the mother's/caregiver's recall of when and if their children were vaccinated. 

Subscribe to our newsletter