EVALUATION OBJECTIVES

This evaluation was commissioned to ensure the successes, challenges, and lessons learned from the COVAX Facility and COVAX AMC are independently evaluated and documented – both from a learning and an accountability perspective.

More specifically, the Formative Review and Baseline Study assesses what has worked well and less well to date in the design, implementation, and results of the COVAX Facility and AMC, albeit less focus on the later given the implementation time period, from when COVAX was conceptualized in 2020 through to the end of 2021, although it recognizes subsequent relevant shifts over Q1-Q2 2022. The evaluation was conducted over the period March 2022 – March 2023.

The Formative Review and Baseline Study builds on the Evaluability Assessment and Evaluation Design (EA/ED) phase conducted by Itad over the period August 2021 - March 2022. The EA/ED phase aimed to:

  • assess the readiness for an evaluation, including the coherence and completeness of the COVAX Facility and COVAX AMC design, the availability of data to answer the evaluation questions (EQs), and the usefulness of doing so; and
  • set out an appropriate and robust multi-stage evaluation design, that can be utilized over the life course of the COVAX Facility and COVAX AMC.

It is to be noted this evaluation focuses on Gavi’s role in administering the COVAX Facility, but considers the links to and ways of working with other agencies in meeting the COVAX Facility and AMC’s objectives.

METHODS

The evaluation used a mixed-methods and complexity-aware design underpinned by a theory-based approach. Four evaluation modules provided a framework to organize the Evaluation Questions (EQs) and employ different methods:

  • Module 1: COVAX Facility and AMC design – A political economy analysis was used, and the development and in-depth analysis of an overall Theory of Change ( ToC) and nested ToCs for five programmatic sub-areas. These analyses were mainly based on information collected using in-depth desk reviews of relevant articles, reports and studies, as well as from Key Informant Interviews (KIIs) and six country case studies (Brazil, Colombia, Democratic Republic of the Congo, India, Senegal, Vietnam) selected for deep dive data collection.
  • Module 2: COVAX Facility and AMC implementation – Programme implementation process tracing, benchmarking and contribution analysis methods were used as was root cause analyses based on in-depth information collected through desk reviews of relevant COVAX reports and other documents, individual and small group KIIs; consultations with global experts as well as experts based in countries with experience implementing COVAX, including through the six country case studies.
  • Module 3: COVAX initial results – Secondary data analysis was conducted on key indicator data reported by COVAX and collected from other relevant data sources, as well as in-depth desk review of critical reports to determine impressions of the COVAX Facility and AMC contribution to the overall initial results regarding allocation, supply, distribution and vaccine coverage. Contribution analysis supported an understanding of COVAX Facility and AMC contribution relative to ToC components. A rapid literature review was also used to discuss the COVAX Facility and AMC contribution to reduction in morbidity and mortality.
  • Module 4: Lessons learned – An in-depth systematic review of the findings across all three modules was conducted and several consensus building meetings were facilitated with the evaluation team to identify the top lessons learned that would be relevant for course corrections and well as planning for future pandemics. A priority list of lessons was developed further through sense-making workshops in October and November with key COVAX stakeholders to obtain their inputs and pressure test the lessons generated by the evaluation team.

COVAX Facility and COVAX AMC Formative Review and Baseline Study

Panel discussion on the evaluation

Related content

Formative Review and Baseline Study

FINDINGS AND CONCLUSIONS

Design principles and processes

  • COVAX was a bold and ambitious proposal to avoid unequitable allocation of vaccines. Many elements were innovative and untested, and there was limited knowledge or prior experience. Many were considered design risks worth taking, but some were questioned or unclear.
  • With hindsight, some assumptions underpinning the design were perhaps too optimistic – e.g. the behavior of HICs in terms of scale and aggressiveness of bilateral procurement or its impact on vaccine markets.
  • The design process was driven by small subset of stakeholders without meaningful engagement of countries/civil society – done in the interest of rapidly finalizing the approach to an emergency. Greater inclusion happened in time, but design largely reflected the priorities of those engaged early.

Governance and management

  • The COVAX Facility and AMC’s establishment was a huge achievement within the operational context. The scope of innovation and pace of implementation created a significant burden for Gavi and had implications for management capacity, efficiency and effectiveness.
  • Gavi was well placed to facilitate good governance of a multi-stakeholder effort to rapidly scale up global vaccination, but the scope and scale of the COVAX Facility and AMC placed a heavy burden on Gavi’s existing governance arrangements and partner working relationships. 
  • While a strong management team was created, it was under-resourced for the scale of its responsibilities, with staff stretched across multiple roles, overworked, and in some cases burned out. However, the very strong mission-driven culture of the Office of the COVAX Facility enabled it to rapidly implement a hugely ambitious agenda.

Risk and communication

  • The initial design was agreed in mid-2020 without a full understanding or analysis of risks and their implications. This led to some design decisions being taken which were considered in retrospect to be overly risk averse, limiting the COVAX Facility and AMC’s programmatic progress. Nonetheless, strong risk management systems and processes were established over time, notably drawing on the Gavi AFC engagement to supplement the Secretariat’s capacity.
  • External communications supported several strategic objectives. However, the Office of the COVAX Facility was intentionally restrained in calling out stakeholder behavior where it was inconsistent with the objective of equitable access, which had implications for perceptions of COVAX. There was a notable shift in approach in late 2021, partly to address this and to respond to public criticism.

Resource mobilisation, market shaping and supply

  • A need-based, opportunistic and ambitious fundraising strategy was implemented, drawing on Gavi’s pre-existing capacity and donor relationships. While it was not possible to raise cash resources immediately in 2020, resource mobilization was highly successful, generating +$10 billion in pledges by end 2021.
  • The COVAX Facility ultimately lacked the market power to meet its market-shaping objectives in the early phase of the COVID-19 pandemic. The COVAX Facility and AMC did, however, achieve the lowest-in-market prices for lower-middle income countries (LMICs) and low-income countries (LICs), and its early deal with the Serum Institute of India (SII) did help to expand vaccine supply.
  • COVAX fell short of its 2021 delivery target by more than 1bn doses, primarily as a result of difficulty securing vaccine supplies. This shortfall had several causes: early and aggressive buying by high-income countries, the halt to India’s vaccine exports, and possibly lower priority given to COVAX by some manufacturers. This gap was partially filled by donations, although supply from this source was slowed by the need to put in place complex arrangements.

Allocation and vaccine delivery support

  • No two allocation rounds were conducted in the same way in 2021, and each round involved different processes. The approach evolved as a response to a challenging operating environment, and until Round 7 (September 2021) allocation was broadly in line with the WHO Allocation Framework and the principle of proportional allocation. Up to this point, allocation did not factor in other non-COVAX sources of vaccine supply, and as a result did not optimize global equality or equity as much as it could have.
  • Gavi did not take on a substantial role in vaccine delivery support through to mid-2021, but did provide $150 million in critical support for cold chain equipment and technical assistance. Amid concern about the lack of vaccine delivery support, Gavi mobilized and approved $775 million to support vaccine delivery in June 2021. However only a small amount of this funding had been made available to countries by the end of 2021, and many stakeholders noted that country needs were not met in a timely way.

Results

  • The COVAX Facility and AMC made a substantial contribution to supply of vaccines in LICs (79% of all vaccines delivered to LICs), moderate contribution in LMICs and small contribution in upper-middle-income countries (UMICs) and HICs.
  • Almost 1 billion doses to 144 countries were delivered by the end of 2021 and the COVAX Facility and AMC played an important role in ensuring doses could be delivered in-country. 833 million (87%) of those doses went to AMC participants – close to the AMC target of 950 million doses.
  • Deliveries to AMC countries were small and sporadic for the first six months of 2021, rising slowly but steadily in Q3 and picking up significantly in Q4. Given their reliance on COVAX, LICs received vaccines much later and in lower volumes than HICs throughout 2021.
  • While global vaccine coverage was highly inequitable (comparing coverage in LICs with other countries), COVAX vaccine supplies played a major role in scaling up coverage in LICs, as compared to a more modest role in LMICs and a small role in UMICs and HICs.
  • Limited vaccine supplies constrained coverage in LICs, as well as inadequate country readiness and roll-out capacity, as well as dose expiry issues. Political commitment to vaccine roll-out and lack of demand were also issues in some countries. These challenges were also noted in LMICs but were less prominent.
  • Respondents from AMC participating countries widely acknowledged the value of Gavi and Alliance partner support in strengthening country readiness for vaccine roll-out, particularly for cold chain capacity.
  • Despite unequal access to vaccine supplies and coverage, distribution of vaccines within countries was broadly equitable: groups at highest risk were prioritized and there were no significant differences between vaccination rates for men and women in most countries.
  • Unintended consequences include the significant contribution of the COVAX Facility to vaccine supplies in India in 2021, outside its AMC participant agreement; shifts in resources away from routine immunization and associated declines in coverage, and increased interest in regional procurement.

For the evaluation recommendations and Gavi Alliance management response please see the full report and Evaluation Management Response.

Evaluability Assessment and Evaluation Design Study (2022)

FINDINGS AND CONCLUSIONS

  • Overall, the assessment concluded the COVAX Facility and COVAX AMC design is coherent and complete enough for a robust evaluation and recommended a 10-year horizon evaluation (2022-2030), in line with the initially envisaged life span for the COVAX Facility and COVAX AMC.

For the evaluation recommendations and Gavi Alliance management response please see the full report and Evaluation Management Response.

Last updated: 8 May 2023

Subscribe to our newsletter