Evaluation objectives
The evaluation covered two initiatives under Gavi’s initial (immediate and interim) response to COVID-19, namely:
- Respond and Protect (R&P): which aimed to provide immediate funding to health systems to enable countries to protect health care workers, perform vital surveillance and training, and purchase diagnostic tests; and
- Maintain, Restore, and strengthen (M&R&S): which aimed to support countries adapt and restart immunisation services, rebuild community trust, catch-up those who have been missed both before and during the pandemic, and investing in strengthening immunisation systems.
The main objectives of the evaluation were fourfold: 1) assess the design, implementation process, efficiency, and effectiveness of R&P; 2) assess the design, implementation process, efficiency, and effectiveness of Gavi’s Maintain, Restore, and Strengthen (M&R&S) in terms of immunisation services at the country level; 3) describe the main successes, challenges, and lessons learned from Respond and Protect and M&R&S; and 4) assess how effectively countries executed the flexibilities funds and how Gavi and the Vaccine Alliance mitigated risk (to the extent possible).
The evaluation was commissioned to generate evidenced-based data and learning to inform ongoing programmatic initiatives to M&R&S and inform the future direction of Gavi’s response to epidemic/pandemic situations.
Methods
The evaluation adopted a mixed-methods approach involving structured document review, Key Informant Interviews (KIIs) at the global, regional, and country level, country case studies, and learning-focused reviews of other funders’ experiences organized around four interrelated workstrams focusing on the following: (1) right design, (2) right ways, (3) right results, and (4) cross-cutting lessons learnt. The evaluation used a Theory of Action (ToA) - for both R&P and M&R&S – grounded in conribution analysis.
Findings and Conclusions
- There was a clear and compelling rationale for Gavi’s initial COVID-19 response: in terms of enabling countries flexible use of existing Gavi funds to support a timely pandemic response. Whilst this entailed going beyond it’s core business (albeit with intended purpose to protect frontline vaccinators and therefore RI) it is hard to imagine a scenario where Gavi did nothing to respond, given the potential impact on its strategic goals.
- We can conclude, in terms of the primary objective of enabling countries to make quick decisions on reprogramming existing funds, that the R&P flexibilities were a qualified success. It is too early to conclude whether M&R&S will achieve its goals, given a) limited data availability; b) that its implementation is ongoing as it continues to provide a useful framing for Gavi’s efforts to refocus on routine immunisation after the initial pandemic response; and c) that evaluation was not tasked to provide a summative judgement on M&R&S.
- Adapting existing Gavi systems was insufficient to ensure uptake of M&R&S and protect RI. Due to a range of factors, including limited incentives to apply, uptake of M&R&S was low and RI coverage was subsequently seen to have dropped. Available evidence does not allow us to comment on causality i.e., that increasing M&R&S uptake would have mitigated impacts on RI (although that was its goal). However, experience suggests that better incentives to apply, better communication and roll-out of M&R&S and strengthening EPI team capacity could have increased uptake. We also recognize that M&R&S was one part of Gavi’s overall COVID-19 response alongside e.g., COVAX, advocacy efforts.
- Gavi Secretariat staff felt that Gavi did not go further in developing more innovative measures to protect RI because its prevailing culture (in terms of attitude to risk and focus on protecting previous gains) and systems (in terms of decision making and prioritisation, partnership, staff resources) presented obstacles that could not easily be overcome within available time and resources. As noted in conclusion 7, it was also not clear how significant the risk was to RI.
- Whilst some Gavi stakeholders felt that Gavi could have given stronger priority to its core mandate (RI) instead of diverting to focus on the immediate COVID-19 response, this was not always practically feasible given country-level constraints to respond to COVID-19 and RI in parallel.
- Gavi had limited availability of data on uptake and performance of R&P and M&R&S as a result of its justifiable decisions to reduce transaction costs for countries to allow them to focus on the emergency response. Lack of data in turn prevented Gavi from both systematically reflecting on the appropriateness of its offer and from learning lessons about what worked. Gavi also suffered from lack of timely access to strategic data (in terms of external systems to track effectiveness) which could have helped to respond sooner to the double dip of RI coverage.
- Notwithstanding these data challenges, uptake appears to have been low, especially for M&R&S. Initially this was considered acceptable given understanding of COVID-19 impact on RI coverage, but with the publication of WUENIC data in 2022 (which saw the biggest falls in RI coverage for 30 years) the low uptake of M&R&S could be interpreted as a missed opportunity.
- Low uptake appears to have been linked more to lack of operating capacity in Gavi and country EPI teams than to concerns about the relevance of the flexibilities offered under R&P and M&R&S; and, whilst comparable organisations experienced similar challenges, the need for surge capacity (both within the Secretariat and at country-level) is highlighted as a key lesson.
For the evaluation recommendations and Gavi Alliance management response please see the full report and Evaluation Management Response.
