Ethics approval
Table of Contents
We received Institutional Review Board (IRB) approval from the Sierra Leone Ethics and Scientific Review Committee (SLERC 20220210), Yale University (2000031541) and Wageningen University (WUR 20220222). The research protocol was pre-registered at the ISRCTN Registry (study identifier ISRCTN17878735). All study participants completed informed consent.
The study was implemented in close collaboration between the researchers, the Government of Sierra Leone’s Expanded Programme on Immunization (EPI) at the MoHS, their National COVID-19 Emergency Response Centre and Concern Worldwide (an international NGO who partners with MoHS on health projects). This collaboration came together because all partners had the joint goal of addressing barriers to vaccine adoption in rural Sierra Leone. Although all partners are responsible for the research design, only the Ministry of Health team was responsible for actually distributing and administering vaccines. We had a memorandum of understanding in place to govern this collaboration.
Village study sample
To determine the sample size, we ran a power calculation assuming a 5% significance level with 80% power. We assumed an intra-cluster correlation of 0.15 as decisions to take a vaccine are probably highly correlated within a village. Average village populations are 2,480 people. We assumed an eligible population of 50% and a baseline vaccination rate of 2.5%. Based on the treatment effects reported in the literature for similar studies, we took a conservative approach and set our expected minimum detectable effect at 0.05. We oversampled slightly and the final design included 150 communities across the three treatment groups in a 1:1:1 ratio.
We chose study sites in collaboration with the MoHS. We started with the 2015 Sierra Leone census, which contains data on 20,659 communities in 166 chiefdoms across 16 districts. We selected 7 largely rural districts (Koinadugu, Falaba, Karene, Kambia, Tonkolili, Bombali and Port Loko), limiting the sample to 8,784 communities in 54 chiefdoms. We then restricted our sampling frame to communities that, according to the 2015 census, had no health clinic within 5 miles (about 8 km) of the community centre, the standard PHU catchment area (Extended Data Table 10), resulting in 1,849 communities. From this list, we excluded very small communities that contained fewer than 19 structures and communities for which latitude and longitude were missing. The final sampling frame consisted of 420 communities located in 49 chiefdoms and 7 districts. Within each district, we then matched communities on the following strata: (1) the share of the population that was immunized; (2) the age of the population; (3) literacy levels; and (4) the distance from the closest clinic. This allowed us to identify communities that had the most similar characteristics within a district and used this to assign the most similar communities to one of the treatment groups and establish comparable ‘triplets’. This resulted in 106 triplets in total. We then randomly selected 50 triplets using district as a blocking variable. The final list included 9 triplets each for Koinadugu and Falaba districts, 8 triplets for Karene district and 6 triplets each for Port Loko, Tonkolili, Kambia and Bombali districts.
Randomization
Randomization to vaccine access treatments
Within each of the 50 triplets, we randomized villages into control, door-to-door and small-group treatment groups. This resulted in 50 villages assigned to control, 50 to door-to-door and 50 to small group (Supplementary Fig. 1). The sample was well balanced on observable characteristics (the F-statistic at the bottom of Extended Data Table 2 is small and not significant).
Household-level random assignment to door-to-door treatment
Within the villages randomly assigned to the door-to-door treatment group, we randomly selected up to 20 residential structures from the community census list to receive a visit from the social mobilization team.
Data collection
Community census listing and baseline survey
Before any intervention activities took place, the research team implemented a community census to enumerate all households in all 150 villages. The research team went door to door to each residential structure and asked how many households resided in the structure. They then interviewed each household head to create a roster of those who ‘eat from the same pot; and reside under the same roof for at least the past 9 months (aside from newborn babies).’ For each household member, enumerators asked about the sex, age and vaccination status. The total census includes N = 29,588 people. Migrant household members who were temporarily away on the day of the visit would have been missed from this listing.
Next, the research team randomly selected a sample of 20 households per village from the households listed in the census to conduct a short (baseline) survey with the household head to record household characteristics (age, sex and education), access to land and food security. The total baseline sample included N = 2,240 respondents.
Exit and endline surveys
After the interventions were implemented, the research team conducted an exit survey of those who received a vaccine at each mobile vaccination clinic. The survey recorded the vaccination status verified using visual inspection of the vaccination card, as well as age and sex.
During the exit survey, enumerators also recorded where people came from and their district and village name (if different from the implementation site). To assess between-village spillovers, we then matched the names of reported villages back to our list of control villages. Using a hard match on district names and then a Levenshtein distance metric to match village names, allowing for a string distance of 2, we found only 8 matches. Using a more conservative cut-off of 1, no overlap was found. Our within-sample spillovers were small or non-existent owing to the large physical distance between pairs of sample villages. The minimum straight-line distance between project treatment and control villages was 8.5 miles (13.7 km), which would take at least 2–3 h to traverse by foot. Any spillover benefits largely accrued to others who were not part of the experimental pool.
For a subsample, the research team conducted a follow-up survey to capture knowledge of COVID-19 and COVID-19 vaccines as well as trust in various sources of information. We used data from 878 respondents in 45 villages for which we observed triplets (that is, where we had information on all treatment groups and a 1:1:1 ratio). We collected data from a total of 105 villages (50 control, 30 door-to-door and 25 small-group treatment group villages); however, only for 45 villages did we observe all three treatment groups and therefore provide a clean comparison. Respondents in this subsample of villages were highly similar to those in the overall sample. An overall F-test did not reject the equality of means: P = 0.668 (Extended Data Table 10).
In treatment villages, these questions were part of the exit survey and implemented 1 day after intervention activities were completed. In control villages, households were visited only once. From a design perspective, we would have ideally captured outcomes at both baseline and endline in each village. It was, however, highly unlikely that these remote places would have been visited by other health personnel from the MoHS or NGOs in the 5-day period between baseline and endline, or that a large number of people would have incurred the cost of visiting the community health clinic for receiving a COVID-19 vaccine. In addition, the costs of revisiting communities in these remote locations are high (the largest line item on the budget relates to transportation costs; Extended Data Table 9). We verified that there was no vaccination drive conducted during this period. Furthermore, we use the fact that our baseline survey was conducted over a few weeks across communities to inspect the temporal trends in the data. A simple regression of baseline vaccination rates on the date of the baseline survey did not reveal any trend. This reduces the concern that our choice to not revisit control villages affects the conclusions we draw.
Research assistants were blinded with respect to treatment groups and study hypothesis.
Intervention details
Timeline of activities
The research team collaborated closely with the Ministry of Health vaccination team. Both the team of vaccinators and social mobilizers from the MoHS and enumerators in charge of the survey received extensive training on implementation protocols. Only those individuals who were considered proficient after examination were retained for implementation or data collection. Within each village the teams followed several steps outlined below (see Supplementary Fig. 1 for further details). On day 1–2, the research team implemented census listing and baseline surveys described above. On days 3–5, the social mobilizer team engaged in small-group and door-to-door mobilization, the MoHS performed a vaccination drive and the research team conducted exit surveys in treatment villages. On day 6, the research team implemented endline surveys.
Social mobilization
The MoHS trained community mobilizers on COVID-19 vaccine safety and efficacy, vaccine types and availability. All mobilizers were trained on how to respond to questions and to counter any misinformation about COVID-19. They were also trained on WHO-recommended safe practices relating to COVID-19 and were instructed to maintain social-distancing protocols and to wear masks when social distancing could not be guaranteed. Additional masks were made available for free for community members.
Community social mobilizers arrived at the village before the mobile vaccination teams. The community mobilizer engaged with local community leaders, including the town chief, section chief, paramount chief, mammy queen, town elders, youth leaders, community health officers, imams, and any other relevant authorities, to seek permission to organize a village information session. The information session took place at a central location, often the community centre or any other convenient location amenable to safe COVID-19 practices.
At the information session, the mobilizer informed community members about COVID-19, available vaccines and evidence about the safety and efficacy of vaccines in preventing transmission and severe illness. People were also informed about the mobile vaccination team and operating procedures during the vaccination drive. They encouraged participants to spread this message to other members of the community not present during the meeting.
In two treatment villages, the MoHS vaccination team did not receive permission from village authorities to conduct the vaccination drive.
Door-to-door campaign
In 50 of the 100 villages randomly selected for treatment, community mobilizers approached up to 20 structures randomly selected from the census list, after the group information session was completed. The proportion of each community assigned to treatment therefore varied with the population of the community. In four small communities, all structures were assigned. Owing to logistical complexities and costs, in some communities, mobilizers did not include highly remote village structures (more than 15 min walk from the village centre). This excluded a total of 10 structures (including 40 people aged ≥12 years). Social mobilizers met in private with residents and delivered the same information as was presented at the community meeting. In addition, they addressed people’s concerns in private. If the individuals were immediately convinced to get vaccinated, the social mobilizer would guide them to the vaccination site before moving on to the next household. Neighbours not assigned to receive a home visit were present during the information session in a few cases. In 75% of the communities, these ‘compliance issues’ were limited to representatives of three or fewer control households, and the majority of communities had no non-compliance of this kind.
Small-group mobilization
In the other 50 treatment villages, after the group information session, social mobilizers searched for small groups of people around the village to converse with. Such groups included women washing clothes around the river, individuals gathered at the ataya (tea) shops, residents playing a game of draughts, groups of people around the mosque or church or farm, or residents gathered near the town chief’s house. Social mobilizers repeated the same information presented during the community information session. If people inside the small group had already taken the vaccine before this second session, they were invited to talk about their experience. After the session, if residents wanted to take the vaccine, the social mobilizer would guide them to the vaccination site before moving on.
Mobile vaccination drive
Vaccines were transported in approved cool boxes or vaccine carriers appropriate for transportation to remote locations. In each treatment village, the MoHS mobile vaccination teams worked with community leaders to select a suitable venue for the vaccination drive. The venue was chosen with the following requirements in mind: it needed to accommodate a waiting area (with some shelter); an arrival and check-in area where patient information can be gathered, maintaining confidentiality; a space for clinical assessment and vaccine administration, including vaccine preparation, maintaining patient confidentiality, privacy and social distancing; an area and system for post-administration observation of patients.
Individuals below 12 years of age were excluded from vaccinations. MoHS teams determined on-site whether a person deemed ‘at risk’ (for example, pregnant or suffering from severe disease) would also be excluded. After the vaccine was administered, recipients were asked to remain in close proximity to the vaccination team for a minimum of 15 min in the event that they experienced any unexpected side effect.
Vaccine teams were compliant with MoHS requirements for the storage, preparation, administration and disposal of the vaccine and associated materials. They followed infection prevention and controls and checked the eligibility of people to be vaccinated using the patient checklist.
Mobile teams adhered to MoHS guidelines on informed consent to receive COVID-19 vaccination, ensuring it was taken only by people with the mental capacity to consent to the administration of the vaccines, and taken freely, voluntarily and without coercion. Participants were allowed to withdraw consent at any time.
All vaccine teams received training on vaccinations, including the management of adverse events following immunization. All such events had to be reported using national reporting systems to the MoHS.
Statistical analysis
To estimate the impact of the intervention on the adult vaccination rate (Extended Data Table 3), we estimated intent-to-treat effects using OLS on individual-level data as follows:
$${Y}_{i,j}={\alpha }_{k}+{\beta }_{1,j}{T}_{{\rm{pooled}}}+{{\epsilon }}_{i,j}$$
(1)
where Yi,j is the vaccination status of individual i, in village j, Tpooled is the village assignment to either door-to-door or small-group treatment groups, αk is a vector of randomization block fixed effects (that is, triplet) and ϵi,j are heteroscedasticity-robust standard errors clustered at the village level. We estimated effects using a linear estimator (OLS) that accounts for high dimensional fixed effects49. In additional analyses, we added to the right-hand side of this equation Yi,j,bl, the baseline vaccination status, and Xj, the vector of covariates that were unbalanced at baseline. We also estimated equation (1) at the village level and for each group, by estimating both β1,jTdoor to door and β2,jTsmall group for the door-to-door and small-group treatment groups, respectively (Extended Data Table 1).
To estimate the vaccination count (Extended Data Table 4), we estimated a village–level intent-to-treat effect using OLS on village-level data as follows:
$${Y}_{j}={\alpha }_{k}+{\beta }_{1,j}{T}_{{\rm{pooled}}}+{{\epsilon }}_{j}$$
(2)
where Yj is the number of people vaccinated in village j, Tpooled is the village assignment to either door-to-door and small-group treatment groups, αk is a vector of randomization block fixed effects (that is, triplets) and ϵj is the heteroscedasticity-robust standard error. We estimated equation (2) for several types of respondents. That is, those who were part of the village census, migrants, returnees and those not present during census, and those from other villages, and added Xj, a vector of covariates that were unbalanced at baseline.
To assess the individual-level effect of the door-to-door campaign, we restricted our sample to the 50 villages assigned to the door-to-door campaign (that is, Tdoor to door = 1), and estimated intent-to-treat effects using OLS as follows:
$${Y}_{i,s}={\alpha }_{j}+{\delta }_{i}{T}_{{\rm{door\; to\; door}}}+{\mu }_{i,s}$$
(3)
where Yi,s is the vaccination status of individual i in structure s (hut or house), Tdoor to door is the individual-level assignment to receive a visit by the social mobilization team to a structure, αj is a vector of randomization block fixed effect (that is, the village) and μi,s is the heteroscedasticity-robust standard error clustered at the structure level.
For the survey-based outcomes on COVID-19 vaccine knowledge and trust, we estimated equation (1), replacing the dependent variable with the survey responses described above, using the subsample of 45 villages where this dataset was collected and we had data on the full randomization blocks.
For our analysis of the treatment effects by subgroup, we estimated equation (1) separately for men, women, various age groups (18–24 years, 25–54 years and >55 years), and sample splits based on whether the household head had any schooling, owns any land or reduced portions of food. To test for differences across subgroups, we estimate equation (1) and interact the subgroup variable with treatment.
In the results presented in Extended Data Tables 6 and 7, we also adjusted for the fact that we conducted multiple tests on the same dataset by implementing FDR corrections and report the FDR q values50. We also report the bootstrapped P value51 to account for regressions with a small number of clusters.
Literature review of vaccination uptake RCT studies
We conducted a literature review of articles in PubMed published between 1 January 2000 and 7 January 2023 using the search terms ‘(vaccin*[Title/Abstract] OR immun*[Title/Abstract]) AND additional search term[Title/Abstract]) AND (Randomized Controlled Trial[Publication Type])’, with the following additional search terms: ‘access’; ‘community-based’; ‘cost effect*’; ‘demand’; ‘hesitant’; ‘incentive*’; ‘intervention*’; ‘mobile’; ‘nudge*’; ‘rural’; and ‘supply’. These searches returned 3,615 unique articles. We screened out articles that were not related to vaccine uptake or that did not use a RCT, which reduced the sample to 141 articles. We appended a further 20 relevant studies that were identified by snowballing and rejected 17 papers that did not have a control group, did not report the percentage-point change in vaccine uptake or did not include a test statistic. The final list of 144 articles comprises 234 distinct interventions for which we can report a percentage-point change relative to a control group (Supplementary Table 2). Of these, 33 interventions (14%) reported information about the cost of the intervention per vaccine administered. This cost specifically refers to the cost of implementing the intervention and does not include the cost of the vaccine itself. Studies that did not unequivocally state the cost of the intervention per vaccinated person were not included in our cost-effectiveness comparisons. Two studies reported the cost in currencies other than US dollars52,53, and these costs were converted to the US dollar equivalent for the year the study was published, using exchange rate data from the respective countries’ national statistics agencies. We did not analyse publication bias.
Deviations from pre-registered hypotheses
We pre-registered our research protocol and hypotheses at the ISRCTN registry (study ISRCTN17878735).
We report on our main hypothesis in Fig. 2 and Extended Data Table 3. In addition to reporting on our main pre-registered outcome (adult vaccination rate), we also report on the total immunizations given per vaccination site because many more people showed up to our temporary clinics from neighbouring villages or were not present during the pre-intervention census, and we had not anticipated this. Figure 3 and Extended Data Table 4 therefore report on the count of all individuals (aged 12 years and above) who visited our clinics to receive a vaccination. This metric is necessary to correctly compute the cost-effectiveness.
The heterogeneity analysis reported in Fig. 5, in which we analysed whether vaccination rates differ by age, sex, schooling and wealth variables, was not pre-specified and followed heterogeneity tests that are common in the vaccine literature54.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.