Back to
During the course of 2008, the GAVI Alliance undertook an assessment of ISS, which culminated in the establishment of an independent task team to review the quality of data and the resulting estimates of coverage that this performance-based programme relies upon.
Team members included academics and technical experts from a range of institutions, including the University of Aberdeen and the Harvard School of Public Health, the World Bank, and UNAIDS, among others.
From December to March, the team assessed the metrics that are currently used to measure ISS delivery and performance, reviewed relevant data collection sources and systems and compared these with alternative processes.
Data Task Team Recommendations
The experts recognised the value of GAVI's performance-based funding system and recommended that the Alliance continue to implement the ISS programme. The team was informed that ISS has helped increase basic DPT3 coverage in 62 countries by 15%, from 65% to 78%.
The background material provided to the experts also highlighted that, although ISS represents just 11% of GAVI's business, it has been a pathfinder for country-driven development ownership and has helped establish a solid foundation in poor countries for the introduction of basic and new vaccines.
The experts agreed that there exists no single "gold standard" for measuring the effectiveness of aid programme results and performance. The team recommended that GAVI continue to use the annual estimates compiled by WHO and UNICEF that relies on multiple data sources to monitor progress of immunisation programmes and calculate reward payments.
The experts also advised GAVI to support the strengthening of the WHO/UNICEF estimation process to ensure systematic and transparent use of all data sources and made a number of specific recommendations to strengthen that process.
In order to provide an incentive to countries to increase immunisation coverage, as opposed to just the number of children immunised, the experts also recommended that future reward calculations be based in part on the number of children immunised and in part on the proportion of children immunised.
The group also recommended that GAVI:
- Continue to support the strengthening of country administrative data systems;
- Monitor trends in disparities of immunisation coverage among different groups;
- Support ongoing initiatives to improve the availability of health outcomes data;
- Consider introducing rewards for specific countries to reduce inequities in DTP3 coverage;
- Endorse and apply principles from IHP+ and the Paris Declaration to ensure that GAVI support is fully aligned with best practices in aid and development;
- Support efforts to strengthen methods for measuring immunisation coverage and quality of service delivery; and,
- Implement a systematic evaluation of the Data Quality Audit, the monitoring and evaluation tool used to ensure the accuracy of data collection processes.
ISS resumed
The task team also recommended that GAVI consider the reinstatement of ISS payments as soon as possible, whilst continuing to work with countries GAVI has identified as having data discrepancies.
At a meeting in early March, GAVI agreed to lift the suspension of ISS payments to all countries except for seven, which require further analysis. (GAVI originally identified 12 countries with discrepancies but removed five from the list after assessing their data collection processes and recognising that significant upgrades and improvements have been made which should prevent major discrepancies in the future.)
The GAVI Alliance, in particular through the Health Metrics Network, WHO and UNICEF, is working with these remaining countries to address the data issues identified. The GAVI Board recognised that all payments to these countries were made according to the design of the ISS programme. Secondly, it was again recognised that no misuse or misappropriation of ISS funds has been identified in these countries.
ISS design evaluation
Data quality issues are an ongoing concern for everyone working in development and some level of difference between data sources will always exist. The key is to have the right systems in place to provide checks and balances and make appropriate management decisions.
It is clear that the ISS has been a very successful program that has used innovative means to extend immunisation services to an unprecedented number of children. And GAVI is committed to implementing the task team's recommendations to strengthen data quality and the checks and balances in the ISS programme. At the same time, as a learning organisation, GAVI will seek to share additional findings on ISS to provide input for the creation of the next generation of performance-based programmes for health and development.