This evaluation was commissioned to ensure the successes, challenges, and lessons learned from the COVAX Facility and COVAX AMC are independently evaluated and documented – both from a learning and an accountability perspective.
More specifically, the Formative Review and Baseline Study assesses what has worked well and less well to date in the design, implementation, and results of the COVAX Facility and AMC, albeit less focus on the later given the implementation time period, from when COVAX was conceptualized in 2020 through to the end of 2021, although it recognizes subsequent relevant shifts over Q1-Q2 2022. The evaluation was conducted over the period March 2022 – March 2023.
The Formative Review and Baseline Study builds on the Evaluability Assessment and Evaluation Design (EA/ED) phase conducted by Itad over the period August 2021 - March 2022. The EA/ED phase aimed to:
- assess the readiness for an evaluation, including the coherence and completeness of the COVAX Facility and COVAX AMC design, the availability of data to answer the evaluation questions (EQs), and the usefulness of doing so; and
- set out an appropriate and robust multi-stage evaluation design, that can be utilized over the life course of the COVAX Facility and COVAX AMC.
It is to be noted this evaluation focuses on Gavi’s role in administering the COVAX Facility, but considers the links to and ways of working with other agencies in meeting the COVAX Facility and AMC’s objectives.
The evaluation used a mixed-methods and complexity-aware design underpinned by a theory-based approach. Four evaluation modules provided a framework to organize the Evaluation Questions (EQs) and employ different methods:
- Module 1: COVAX Facility and AMC design – A political economy analysis was used, and the development and in-depth analysis of an overall Theory of Change ( ToC) and nested ToCs for five programmatic sub-areas. These analyses were mainly based on information collected using in-depth desk reviews of relevant articles, reports and studies, as well as from Key Informant Interviews (KIIs) and six country case studies (Brazil, Colombia, Democratic Republic of the Congo, India, Senegal, Vietnam) selected for deep dive data collection.
- Module 2: COVAX Facility and AMC implementation – Programme implementation process tracing, benchmarking and contribution analysis methods were used as was root cause analyses based on in-depth information collected through desk reviews of relevant COVAX reports and other documents, individual and small group KIIs; consultations with global experts as well as experts based in countries with experience implementing COVAX, including through the six country case studies.
- Module 3: COVAX initial results – Secondary data analysis was conducted on key indicator data reported by COVAX and collected from other relevant data sources, as well as in-depth desk review of critical reports to determine impressions of the COVAX Facility and AMC contribution to the overall initial results regarding allocation, supply, distribution and vaccine coverage. Contribution analysis supported an understanding of COVAX Facility and AMC contribution relative to ToC components. A rapid literature review was also used to discuss the COVAX Facility and AMC contribution to reduction in morbidity and mortality.
- Module 4: Lessons learned – An in-depth systematic review of the findings across all three modules was conducted and several consensus building meetings were facilitated with the evaluation team to identify the top lessons learned that would be relevant for course corrections and well as planning for future pandemics. A priority list of lessons was developed further through sense-making workshops in October and November with key COVAX stakeholders to obtain their inputs and pressure test the lessons generated by the evaluation team.