Does Experience of Vaccination Improve Vaccine Confidence and Trust? Policy Feedback Effects of Mass COVID-19 Vaccination in the United States Free

Abstract

Context: Policy feedback research has demonstrated that a highly tangible policy that shapes public attitudes through direct and day-to-day experiences often reshapes public opinion, with the effect of generating supportive or skeptical constituencies that determine the sustainability of future programs. This article examines the impact of mass vaccination efforts on attitudes toward vaccines in a context of high vaccine hesitancy in the United States.

Methods: The authors analyzed 73,092 survey responses from 9,229 participants in the longitudinal data from the Understanding America Study project, covering December 2020 to July 2023. Using two-way fixed-effects ordinary least squares regression and ordinal logistic regression, the authors estimated the changes in attitudes toward vaccines, including trust in vaccine manufacturing and approval processes, following COVID-19 vaccinations.

Findings: COVID-19 vaccination was associated with improved perceptions of vaccine effectiveness and social benefits and reduced mistrust in vaccine-related processes. However, it did not significantly alleviate concerns regarding vaccine side effects and severe illness stemming from vaccination. The strongest effects were observed among initially hesitant respondents who eventually received vaccinations.

Conclusions: The experience of COVID-19 vaccination generally improved attitudes and confidence in COVID-19 vaccines among the US public, particularly among vaccine-hesitant people. These effects could have positive impacts on future immunization programs by mitigating vaccine hesitancy.

Mass immunization requires a large majority of people with varying perspectives on vaccines to get vaccinated. Vaccination therefore poses a collective action problem: those opposed to vaccination have the incentive to “free ride” on those who are more willing to bear the concerns of vaccine side effects, however small (Bradley and Navin 2021; Raymond, Kelly, and Hennes 2021). COVID-19 has been no exception: from the onset of the pandemic, governments at various levels in the United States made significant efforts, both voluntary and more coercive, to boost the public's vaccine uptake. These included financial support from federal and state governments, logistical support for vaccine distribution, requirements for vaccine proof in public spaces and for air travel, and nationwide vaccine promotion campaigns targeting the general public.

The US government expedited the development and distribution of COVID-19 vaccines by coordinating efforts between federal agencies and pharmaceutical companies through Operation Warp Speed and the rapid authorization of the Pfizer/BioNTech vaccine by December 2020 (Slaoui and Hepburn 2020). While this accelerated rollout initially sparked public concern about the vaccine's effectiveness and safety, and despite considerable hesitancy among a broad portion of the public, more than 80% of the population eventually received the vaccine, either by choice or because of vaccine requirements and mandates (Mathieu et al. 2021). This study asks how the mass distribution of COVID-19 vaccines—both through more voluntary measures and more coercive ones—affected the public's confidence in vaccines, and it discusses the implications of such an influence for future immunization programs. In examining this question, we draw on policy feedback theory: the idea that citizens' experiences with policy can shape the politics of an issue and the public's perception of government more broadly.

The literature on policy feedback effects has observed that in addition to policies remedying a specific problem, policies create politics by altering the incentive structure around the issue (Béland 2010; Daugbjerg and Kay 2020; Mettler and Sorelle 2018; Pierson 1993; Schattschneider 1964). Moreover, people's experiences with a policy can also improve or undermine their perceptions of that policy and government more broadly, determining the future pathway of policy decisions and implementation of policies (Campbell 2003; Mettler 2005; Moynihan and Soss 2014; Soss 1999). Efforts to promote mass vaccination have the potential to produce either positive spillover effects if the public learns from experience that vaccines are in fact safe and effective, or negative spillover effects if the vaccination experience is viewed as overly coercive and associated with detrimental outcomes. Vaccination is a good candidate for the study of policy feedback effects because the public is likely to view vaccines as a product of government action and the vaccination experience as an interaction with the state.

How did the experience of COVID-19 vaccination affect attitudes toward vaccination among the US public? We seek to answer these questions with data from the Understanding America Study (UAS) project, which tracked changes in people's attitudes toward COVID-19 vaccines and vaccine-related processes during the pandemic. With data from 73,092 responses (9,229 respondents), the study aims to examine how mass vaccination efforts affected public attitudes toward COVID-19 vaccines and mistrust in vaccine-related processes. We do so by comparing differences in the effects of vaccination along with respondents’ vaccine hesitancy before vaccination.

COVID-19 Vaccination as Policy Experience

Policy feedback theory suggests that existing policies reshape the political landscape of future policy making and implementation (Béland 2010; Pierson 1993). This perspective understands public policies as part of evolving institutions rather than the result of discrete choices, and it sees them as redistributing resources, altering power relations among groups, and creating new interest groups and organizations (Béland 2010; Campbell 2012; Daugbjerg and Kay 2020). Through this cyclical process, policies can reinforce or undermine existing political arrangements and power dynamics, potentially leading to significant transformations in political circumstances over time. Attitudinal policy feedback studies stress how policy decisions and implementation influence public attitudes and preferences (Fernandez and Jaime-Castillo 2013; Svallfors 2010). For instance, policy makers may aver that even initially unpopular policies, once implemented, may become more popular over time if their implementation demonstrates tangible benefits to society. Conversely, poorly implemented popular policies might become less popular as their shortcomings become evident with experience.

To date, most research on policy feedback has examined the impact of distributive and redistributive policies on the building up of political coalitions (Campbell 2003; Mettler 2005), attitudinal change (Campbell 2012), and perceptions of government (Soss 1999). Fewer policy feedback studies have focused on the effects of what Pacheco (2013) refers to as “highly tangible policies.” Highly tangible policies include visible policies that the public directly experiences on a daily basis, such as sin taxes, bans on trans fats, bicycle helmet and seat belt laws, recreational marijuana legalization, and menu calorie labeling, which often have the explicit intent of changing both attitudes and behaviors. These policies provide easily understandable benefits or sanctions; therefore, they shape people's attitudes toward and support for similar policies.

Vaccination, like the examples given above, is a highly tangible policy with which citizens have direct and recurrent experiences. In routine immunization, people directly engage with vaccination, from scheduling appointments to receiving vaccine doses and potentially experiencing physical reactions. During the COVID-19 pandemic, most people in the United States and in many other countries had such experiences with COVID-19 vaccines during the widespread rollout of the vaccines. Early on, the federal government launched the Operation Warp Speed program in May 2020 to enable joint efforts between US federal agencies—such as the Department of Health and Human Services, the Centers for Disease Control and Prevention, and the Food and Drug Administration (FDA)—and pharmaceutical companies to allow funding and streamline regulatory processes for vaccine development (Slaoui and Hepburn 2020). As a result of this full support from the government, the first COVID-19 vaccine by Pfizer/BioNTech became available at a remarkable pace in December 2020 by receiving emergency use authorization from the FDA. A large portion of the public initially expressed great skepticism toward these vaccines during the development process and their early introduction, fearing they had been developed too hastily. Nevertheless, nearly 80% of the US population received at least one dose of COVID-19 vaccine (Mathieu et al. 2021). Many who were initially hesitant eventually got vaccinated, either voluntarily or in response to some form of requirement. These experiences with COVID-19 vaccines could shape people's perceptions of the benefits and risks of the vaccines, and potentially their attitudes toward immunization programs in general.

Recognizing the role that a positive policy experience plays in fostering favorable attitudes toward future programs, policy makers are advised to carefully take into account the broader ramifications of their decisions rather than focusing solely on addressing immediate needs (Salamon and Elliott 2002). The present widespread hesitancy toward vaccines is a prime example of policy feedback effects, as historical negative experiences with vaccines have left a lasting impact that contributes to ongoing skepticism toward mass vaccination campaigns. The “swine flu affair” of 1976—when swine flu failed to materialize, but approximately 450 cases of Guillain-Barré syndrome were found among those who had received the flu shot (approximately 1 per 100,000)—is believed to have left a lasting negative impact on people's perceptions of mass vaccination (Eschner 2017).

Although compulsory vaccinations, such as vaccine requirements for travel, might be justified in certain situations because of the overriding need to protect the general public's health and to overcome the free-rider problem (Canning et al. 2021; Gostin, Cohen, and Shaw 2021; Malone and Hinman 2007; Navin and Attwell 2019; Sween, Ekeoduru, and Mann 2022), such measures have the potential to intensify preexisting vaccine resistance and erode trust in government, threatening future vaccination campaigns (Bardosh et al. 2022; Gostin, Salmon, and Larson 2021). Recently, Buttenheim and colleagues (2020) conducted an online experiment in which they determined that mandating vaccination for school entry slightly increased hesitancy among parents, even when they did not have actual vaccination experience. Consequently, public health actors have generally preferred less coercive, voluntary measures to promote vaccination when possible (Bardosh et al. 2022; Dubov and Phung 2015).

However, there are good reasons to think that the experience of COVID-19 vaccination—whether voluntary or coerced—could result in positive feedback loops. Despite widespread public concern about vaccine side effects, the incidence rate of severe adverse events following COVID-19 vaccination was exceedingly rare. Abara and colleagues (2022) estimated that the expected rates of potential adverse events within 42 days following COVID-19 vaccination were approximately 0.000225% for Guillain-Barré syndrome, and 0.000143% for myopericarditis, illustrating the rarity of these events. Furthermore, as COVID-19 vaccines were distributed among the population, social distancing and mask regulations were relaxed, and the economy gradually returned to normalcy. Thus, we can expect the overwhelming majority of people to have had positive experiences with COVID-19 vaccines, which may improve their overall perceptions of vaccines and their social benefits and may contribute to reduced skepticism toward their development.

Several studies have explored the potential for a positive feedback loop resulting from COVID-19 vaccination, although they did not examine causality using nationally representative longitudinal data. One study conducted in Bangladesh found that people who had a seamless vaccine process and did not experience adverse events were significantly more satisfied with their vaccine experience and more likely to recommend COVID-19 vaccines to others (Islam et al. 2021). Similarly, a qualitative study involving 28 primary health care workers in Hong Kong showed that some interviewees expressed that they became less nervous about vaccination and more confident in encouraging others to receive vaccines after their own vaccination (Ng, Chu, and Lau 2022).

The feedback effects of COVID-19 vaccination may be stronger among people with vaccine hesitancy. For example, when people are hesitant about COVID-19 vaccines, the experience of vaccination may give them a chance to address their concerns about the vaccines. Early in the vaccine development process in December 2020, large swaths of the US public were hesitant to get the vaccine: 39% reported that they wanted to wait and see, 9% said would only get vaccinated if they were required to, and 15% stated they would definitely would not get it under any circumstance (Hamel et al. 2020). However, as time has gone on, the proportion of reluctant individuals that have gotten the vaccines has increased, particularly among those with undecided positions who said they would wait and see (Hamel et al. 2020).

It is also worth noting that vaccine hesitancy, reluctance to vaccinate, and antivaccine sentiment are not evenly distributed across the population in the US context. Rather, vaccination has been one of the most politically polarizing and partisan policies in recent history (Choi and Fox 2022; Hornsey et al. 2020; Howard 2024; Jones and McDermott 2022; Oberlander 2024). Additionally, racial disparities in hesitancy and willingness to vaccinate were identified early in the vaccine development process (Callaghan et al. 2021; Fox et al. 2022; Jacobi and Vaidyanathan 2021). It is therefore possible that government efforts to encourage vaccination could potentially backfire and feed into existing narratives about government overreach or medical racism, especially among groups who perceive these initiatives as overly coercive, regardless of whether the experience of vaccination was positive or not. Thus, we also consider the possibility that the experience of vaccination could make groups that were initially hesitant less confident in vaccines and their social benefit.

Research to date has rarely tested the potential attitudinal changes resulting from the mass distribution of vaccination empirically, and there is still very little knowledge of how the experience of COVID-19 vaccination affects attitudes toward these vaccines and immunization programs. Thus, this article asks the following questions: First, did the experience of COVID-19 vaccination improve people's attitudes toward COVID-19 vaccines and reduce mistrust in the vaccine manufacturing and approval processes in the United States? Second, do the effects of the vaccination experience differ by the level of vaccine hesitancy? This study examines these questions by analyzing nationally representative longitudinal survey data.

Methods

The empirical analysis used the two-way fixed-effects (TWFE) ordinary least squares (OLS) regression and ordinal logistic regression to conduct a statistical comparison of the outcomes—that is, four vaccine confidence measures and four trust measures—before and after vaccination. We examined both models for the robustness check. To check and minimize the influence of preexisting trends, we conducted visual inspections and applied a narrow bandwidth in statistical testing.

Data

The analysis used the COVID-19 panel from the UAS project provided by the Center for Economic and Social Research from the University of Southern California. This panel provides online longitudinal survey data that include more than 9,000 participants who agreed to receive email invitations. The sample was representative of the US population and was recruited based on address-based sampling and through postcard invitations initially (Kapteyn et al. 2020). The UAS project launched the COVID-19 panel survey in March 2020 to monitor the impact of the COVID-19 pandemic on US society. As of this writing in April 2024, the UAS project currently has 25 national sample batches. The rate of active panel members has ranged from approximately 50% to 99% of those who have ever participated in the survey. The average response rate in the study period was 78% (appendix 1).

The initial dataset included 144,158 responses from 9,491 respondents in 31 survey waves of the national batch—waves 2–6, 8, and 10–34. The main analysis used 15 waves from the 20th to the 34th that include questions on the outcome variables. Earlier waves were used only for measuring vaccine hesitancy before COVID-19 vaccination, using the willingness to receive COVID-19 vaccines. Waves 1, 7, and 9 were excluded because they did not include both the willingness and outcome questions. The 31 waves covered the period from April 1, 2020, to July 9, 2023 (appendix 1). The 20th wave was fielded starting December 9, 2020, and the first vaccination for COVID-19 occurred during the 21st wave. The data collection period per wave was approximately three to four weeks until wave 24, five weeks in waves 25 and 30, and two months in wave 31.

The initial dataset from waves 20 to 34 included 73,092 responses from 9,229 respondents. There were 2,015 responses with missing values (2.8%) in the variables examined in the analysis (see appendix 2 for the number of missing values by variable and survey wave). Panelists who did not take part in a survey wave were contacted again for participation in subsequent waves. Therefore, we first imputed 13 responses with missing values in education and income (1 value in education and 13 values in income), which are relatively more stable than perceptual variables, by using the preceding and the subsequent waves. We then excluded 2,002 responses with missing values for willingness to vaccinate, vaccination status (dropping “unsure” as well), COVID-19 infection, age, education, and income. We did not conduct multiple imputations for these values because we considered that the potential bias from a relatively small portion of missing values constituting less than 5% of the sample may exceed the benefits (Jakobsen et al. 2017). Then, we dropped two types of inconsistent respondents: 42 respondents (425 responses) who showed inconsistent vaccination statuses (i.e., initially indicating that they received a vaccine but later reporting that they did not), and 43 respondents (498 responses) who provided inconsistent race and gender, which are not likely to change in the short term, across waves. Last, we dropped 2,895 respondents (6,524 responses) who did not complete the survey before or after vaccination, to ensure that all respondents included have observations to compare before and after vaccination. The majority of exclusions at this stage were either those who started participating in the survey in waves 30–34 when the interval of waves became wider, or those who answered the survey in the first two waves and did not return after vaccination. After these procedures, 63,643 responses from 6,098 respondents were included in the analysis (see appendix 3 for additional details of the data cleaning process).

Variables

We examined two versions of outcome variables: original measures using the full four-point Likert scale and dichotomized measures. The analysis used six attitudinal outcome variables: four vaccine confidence measures and two measures of mistrust in the process. The first two variables measured positive attitudes toward COVID-19 vaccines’ effectiveness (i.e., “COVID-19 vaccines are useful and effective”) and social benefits (i.e., “COVID-19 vaccines provide benefits to society”). The next two variables measured concerns regarding adverse events, that is, harmful side effects (i.e., “COVID-19 vaccines have many known harmful side effects”) and serious illness (i.e., “COVID-19 vaccines lead to serious illness and death”). These confidence variables were coded to assign higher values to indicate higher levels of agreement with the statement (i.e., 0 for strongly disagree to 3 for strongly agree). For binary versions of these, we coded them as 1 if respondents agreed or strongly agreed with the statement and 0 if they disagreed or strongly disagreed. Last, two mistrust outcomes were measured by using questions asking how much respondents trusted the vaccine manufacturing and approval process: “How much do you trust the process in general (not just for COVID-19) to develop safe vaccines for the public?” and “How much do you trust the governmental approval process to ensure the COVID-19 vaccine is safe for the public?” These variables were coded to assign higher values to higher levels of mistrust (i.e., 0 for “fully trust,” 1 for “mostly trust,” 2 for “somewhat trust,” and 3 for “do not trust”). The binary versions of these were coded as 1 only for “do not trust” by considering that the other choices indicate some levels of trust.

The key independent variable measures people's COVID-19 vaccination status in wave t. The vaccination status was measured by using the following question: “Have you gotten vaccinated for the coronavirus?” Respondents were considered vaccinated if they answered “Yes” (coded as 1) and not vaccinated if they answered “No” (coded as 0).

Considering that the effects of vaccination can be more salient among vaccine-hesitant people, we also examined the interaction between the vaccination variable and respondents’ vaccine hesitancy before COVID-19 vaccination. To create a time-invariant category of vaccine hesitancy, we measured the hesitancy variable by dividing the individual average of answers to a 5-point Likert scale question asking about the respondent's willingness to vaccinate for COVID-19 across all available survey waves before vaccination (Choi and Fox 2022). The question was: “How likely are you to get vaccinated for coronavirus once a vaccine is available to the public?” The original choices were: very unlikely, somewhat unlikely, somewhat likely, very likely, and unsure, in sequence. Before calculating the average, we adjusted this arrangement by assigning higher values to indicate higher levels of hesitancy. We also positioned “unsure” in the middle, that is, 0 if very likely, 1 if somewhat likely, 2 if unsure, 3 if somewhat unlikely, and 4 if very unlikely. We then divided the individual average of this 5-point Likert scale measure into four categories: very low hesitancy for values less than 1 (58% of respondents), low for values between 1 and less than 2 (22%), high for values between 2 and less than 3 (12%), and very high for values of 3 or higher (7%). The selection of these cut points was data driven and was based on visual inspection rather than theoretical considerations. This was to ensure that each group had a sufficient number of observations across all waves and to mitigate noticeable fluctuations, particularly surrounding the event of COVID-19 vaccination. The number of survey waves before vaccination ranged from 1 to 29, and the mean was 19. Four percent of the respondents completed the survey fewer than three times.

There was one time-variant control variable included in the analysis: testing positive for COVID-19 in wave t. This was measured as 1 if respondents answered that they had been diagnosed with having the coronavirus. We also reported respondents’ race, sex, age, educational attainment, and household income in appendix 4 but did not include these variables in statistical analysis as they are largely time invariant.

Identification Strategy

The analysis used two specifications: TWFE OLS and ordinal logistic regression. These models have both pros and cons. A TWFE OLS model helps calculate an interpretable average effect of COVID-19 vaccination across vaccine hesitancy groups and uses the full sample, but it loses some information by dichotomizing the ordered outcome variable. Although a TWFE ordinal logistic model can mitigate this limitation by using the full information in the outcome variable, this drops observations without variations in the outcome variable. Its marginal effect is also less intuitive than the marginal effect of a TWFE OLS model. Therefore, we tested with both specifications to check whether the results of both models converged.

where i indexes individuals and t indexes survey waves, three waves before and after vaccination. $Yit$ denotes the six outcome variables for individual i in wave t. Importantly, in statistical analyses, we excluded $β1$⁠, which captures the difference associated with the first dose of vaccination for COVID-19. Coefficients in $β2$ explain the difference in the effect size compared to the base category, the very-low-hesitancy group. $𝜗it$ indicates the COVID-19 infection. $σ$ and $τ$ set unobserved time-invariant individual variations and wave effects fixed. In the equation, the marginal effect is calculated as the sum of $γ$ estimates weighted by the size of each vaccine hesitancy group.

Importantly, we excluded responses following the second dose of vaccination in the analysis, as these could potentially have additional influences on attitudes. For those who received both the first and second doses within a single survey wave period (53% of respondents), we categorized them as having received the first dose only. We included them in the analysis because excluding such a large portion could undermine the representativeness of the sample.

In addition to testing the overall effects of COVID-19 vaccination on outcomes, we also ran models stratified by party identification and race, with results reported in appendices 9 and 10. Party identification and race have been some of the strongest predictors of vaccine hesitancy and uptake in the US context. We might expect the experience of vaccination, especially vaccination that might feel compelled, to have a different impact on subsequent attitudes among those who may be particularly hesitant as a result of legacies of medical mistrust or political ideology. As party identification and race variables are time invariant, they would drop out of our fixed-effects models if we enter them as controls. Instead, we stratify our models by party identification and race to examine heterogenous effects.

Standard errors were clustered at the individual level for both models. Analyses were conducted by using Stata version 18 (StataCorp LP, College Station, TX). The statistical significance of the results was determined based on the 95% confidence level and two-tailed tests.

Results

Table 1 presents the sample characteristics by the level of vaccine hesitancy. More than 70% of respondents in the study sample received at least one dose of COVID-19 vaccines by the 29th survey wave, fielded between June 9 and July 21, 2021. As of the 31st survey wave, there were substantial differences across the four groups in the vaccination rate, confidence in vaccine safety, and mistrust. The percentages of those who thought COVID-19 vaccines cause serious side effects and illness and mistrust in the vaccine manufacturing and approval processes were the lowest in the group with the lowest level of hesitancy, and these percentages increased along with the level of vaccine hesitancy. The percentages of belief in vaccine effectiveness and social benefits were the lowest in the group with the highest vaccine hesitancy; however, they were relatively comparable across the four groups. Other characteristics were comparable across the groups.

There were differences in the sample characteristics between respondents with and without COVID-19 vaccination (appendix 5). Compared to respondents who had been vaccinated for COVID-19, nonvaccinators tended to agree with the effectiveness and social benefits of COVID-19 vaccines while thinking that COVID-19 vaccines would cause serious side effects and illness and mistrusting the vaccine manufacturing and approval processes. They were more politically conservative or independent/other, younger, low education, and low income (appendix 5).

See appendix 6 for the trend of the COVID-19 vaccination rate in the sample divided by the level of vaccine hesitancy before the vaccination. The vaccination rate increased over time in all groups, although the groups showed different paces. The vaccination rate in the very-low-hesitancy group reached 90% in wave 29 and decreased to 74% in the last survey wave, as the latter waves with longer data collection periods included more nonvaccinators. The vaccination rates in the other groups were 78% (low), 64% (high), and 33% (very high) in the last survey wave. While there were still many people who received COVID-19 vaccines in latter waves in the low- and high-hesitancy groups, the percentage of nonvaccinators in the very-high-hesitancy group was still very high (61%) until the last wave. By considering the possible oversampling of nonvaccinators in the latter waves, we provided a robustness check result without survey waves 32–34 in appendix 8.

Figure 1 compares the trends of the six outcome variables among the four hesitancy groups, centering on the vaccination event. In the figure, the darker markers indicate more hesitant respondents. This figure provides two notable observations. First, vaccine-hesitant respondents’ positive attitudes toward COVID-19 vaccines tended to increase after vaccination, and negative attitudes decreased. In particular, there were noticeable discrete changes in the average negative attitudinal outcomes in most groups around the vaccination event (fig. 1, graphs C–F). Such jumps were not found when plotting survey waves on the x axis, indicating that they were not likely to be driven by certain external events (see appendix 6 for the entire sample and appendix 7 for the vaccinated sample).

In addition to the jumps observed around vaccination, there were preexisting improvements in vaccine confidence, especially in perceived effectiveness and social benefits of COVID-19 vaccines in the very-high-hesitancy group (fig. 1, graphs A–B). This group's average perceived effectiveness and social benefits of COVID-19 vaccines increased by 0.14 point in the three survey waves before vaccination.

In most groups, we also observed gradual changes leading up to the fourth wave preceding vaccination. Therefore, to minimize the influence of these preexisting trends, we limited the subsequent statistical analysis to include only up to three survey waves before and after vaccination (the dotted vertical lines in the figure).

Figure 2 confirms the first hypothesis that COVID-19 vaccination improved vaccine confidence while addressing mistrust in the vaccine process, on average, for four of the outcome variables, including effectiveness, social benefits, and mistrust outcomes. The figure compares the marginal effects of COVID-19 vaccination on each of the outcomes, based on the TWFE OLS models. After getting the first dose, respondents were 5 percentage points more likely to answer that COVID-19 vaccines are effective [95% CI = 0.02, 0.07; p < 0.001] and 5 percentage points more likely to answer that COVID-19 vaccines provide social benefits [95% CI = 0.02, 0.07; p < 0.001]. Respondents were 9 percentage points less likely to mistrust the manufacturing process [95% CI = –0.11, −0.06; p < 0.001] and 10 percentage points less likely to mistrust the approval process [95% CI = –0.13, −0.07; p < 0.001]. However, we did not find significant differences for concerns about harmful side effects and serious illness from vaccination.

Tables 2 and 3 present the estimated results from the two specifications of OLS and ordinal logistic models. These results similarly confirm the second hypothesis about whether the effects of COVID-19 vaccines would be stronger among vaccine-hesitant groups, although there are variations across the six outcomes. In sum, the evidence suggests that the effect of COVID-19 vaccination was relatively stronger for perceived effectiveness and social benefits and mistrust in the process; however, it was weak for addressing concerns about harmful side effects and serious illness.

Table 2 presents the results from the TWFE OLS models explaining the association between COVID-19 vaccination and outcome variables. The results are based on the bandwidth including three survey waves before and after vaccination and excluding responses following the second dose. Given the interactions included, the coefficients of the vaccination variable in the first row explain the probability difference in the very-low-hesitancy group. The interaction terms indicate the difference in the effect size compared to the very-low-hesitancy group. Overall, COVID-19 vaccination significantly increased positive attitudes toward COVID-19 vaccines and decreased mistrust in the vaccine manufacturing and approval processes, but its effects were relatively limited in addressing concerns about vaccine side effects and illness. Moreover, the effect size tended to increase along with the degree of vaccine hesitancy. Although nonvaccinators were oversampled in the very-low-hesitancy group in survey waves 32–34, the results were largely replicable when excluding these latter survey waves (appendix 8).

The effects were most salient for mistrust outcomes. In the very-low-hesitancy group, the probabilities of mistrust in the manufacturing and approval processes significantly decreased by 5 percentage points [95% CI = –0.08, −0.02; p < 0.001] and 7 percentage points [95% CI = –0.10, −0.03; p < 0.001], respectively. In other words, after experiencing COVID-19 vaccination, people were 5 percentage points less likely to mistrust the vaccine manufacturing process and 7 percentage points less likely to mistrust the approval process. The effect size for mistrust in the manufacturing process was 9 percentage points higher in the low-hesitancy group [95% CI = –0.13, −0.04; p < 0.001] and 7 percentage points higher in the low-hesitancy group [95% CI = –0.14, −0.00; p = 0.035]. For mistrust in the approval process, the effect size was 10 percentage points higher in the low-hesitancy group [95% CI = –0.15, −0.04; p < 0.001] and 11 percentage points higher in the high-hesitancy group [95% CI = –0.17, −0.04; p < 0.001]. It was comparable between the very-high-hesitancy group and the very-low-hesitancy group.

The effect of COVID-19 vaccination on perceived effectiveness and social benefits of COVID-19 vaccines was significantly higher in the high-hesitancy and very high-hesitancy groups, and it did not make significant differences in the very-low-hesitancy group. For the effectiveness outcome, the difference in the probability of agreement was 8 percentage points higher in the high-hesitancy group [95% CI = 0.03, .014; p = 0.004] and 18 percentage points higher in the very-high-hesitancy group [95% CI = 0.09, 0.26; p < 0.001]. For the social benefits outcome, it was 7 percentage points higher in the high-hesitancy group [95% CI = 0.01, 0.12; p = 0.015] and 13 percentage points higher in the very-high-hesitancy group [95% CI = 0.05, 0.21; p = 0.003]. The effect in the very-low-hesitancy group was insignificant and was comparable to the low-hesitancy group.

We also found that people in the low-hesitancy and very-high-hesitancy groups were less likely to think that COVID-19 vaccines cause serious illness after getting the vaccine. While the vaccination did not make a significant difference in the very-low-hesitancy group, the effect size was 5 percentage points higher in the low-hesitancy group [95% CI = –0.10, −0.00; P = 0.038] and 19 percentage points higher in the very-high-hesitancy group [95% CI = –0.27, −0.10; p < 0.001].

Interestingly, people tended to think that COVID-19 vaccines were effective when they were diagnosed with COVID-19. Among those who had previously been infected with COVID-19, the probability of thinking vaccines were effective increased by 12 percentage points [95% CI = 0.02, 0.21; p = 0.014].

Table 3 presents the results from the TWFE ordinal logistic models. As shown in the table, the estimation process excluded observations where the outcome remained unchanged, with counts ranging from 5,355 to 7,858 across the six models. Overall, the results were largely consistent with the results from TWFE OLS models, although there were differences in the levels of significance in some groups. The effect of COVID-19 vaccination was most salient for the mistrust outcomes and tended to be stronger in the more vaccine-hesitant groups.

For mistrust outcomes, COVID-19 vaccination was associated with a 33% decrease in the odds ratio of moving to a higher category of trust in the vaccine manufacturing process [95% CI = 0.55, 0.82; p < 0.001] and a 36% decrease for trust in the approval process [95% CI = 0.52, 0.78; p < 0.001]. In the other groups, the odds ratio for mistrust in the manufacturing process further decreased by 32% in the low-hesitancy group, indicating a higher impact of vaccination [95% CI = 0.51, 0.91; p = 0.009]. The difference in the high-hesitancy group was insignificant, while it was significant in the OLS model. For mistrust in the approval process, the odds ratio was 29% lower in the low-hesitancy group [95% CI = 0.53, 0.95; p = 0.021] and 39% lower in the high-hesitancy group [95% CI = 0.42, 0.88; p = 0.009]. It was comparable between the very-high-hesitancy group and the very-low-hesitancy group.

The effect of COVID-19 vaccination on perceived effectiveness was not significant in the very-low-hesitancy group; however, it was stronger in vaccine-hesitant groups. The odds ratio was 76% higher in the high-hesitancy group [95% CI = 1.14, 2.70; p = 0.010] and 128% higher in the very-high-hesitancy group [95% CI = 1.34, 3.89; p = 0.002].

For the social benefits outcome, COVID-19 vaccination was associated with a 34% increase in the odds ratio in the very-low-hesitancy group [95% CI = 1.05, 1.72; p = 0.018], while it was insignificant in the TWFE OLS model. Among the vaccine-hesitant groups, only the very-high-hesitancy group showed a significant difference in the odds ratio, which was 71% higher [95% CI = 1.01, 2.90; p = 0.044].

Similar to the results from the TWFE OLS models, we did not find consistent evidence that COVID-19 vaccination addressed concerns about serious side effects and illness. All estimates were insignificant for the side effects outcome. For the illness outcome, it was insignificant in the very-low-hesitancy group; however, the odds ratio was 69% lower [95% CI = 0.18, 0.54; p < 0.001] when compared to the very-low-hesitancy group.

Getting COVID-19 was associated with a higher odds ratio in the effectiveness outcome, consistent with the TWFE OLS model. When people were diagnosed with COVID-19, the odds ratio increased by 78% [95% CI = 1.12, 2.82; p = 0.015].

We might expect the effects of vaccination to differ by respondents’ party affiliation and race, considering that COVID-19 vaccination has become a highly salient political topic across US society, sharply dividing the public's attitudes along partisan lines. However, we did not find differing effects when testing in the subsamples of Democrat and Republican respondents (appendix 9) and white, Black, and other racial groups (appendix 10). Following vaccination, confidence in COVID-19 vaccines tended to improve, while mistrust in the vaccine manufacturing and approval processes decreased in all subsamples.

Limitations

Our findings need to be contextualized within the specific COVID-19 moment and may not be generalizable to other circumstances or even national settings. While we found that COVID-19 vaccination provided an opportunity to generate more positive attitudes toward these vaccines and their manufacturing and approval processes, unpleasant experiences with vaccination could potentially increase doubt in the safety and efficacy of vaccines and mistrust in government, both among those who were previously hesitant and among those with more initial confidence—an issue we have not explicitly examined here. We did not examine heterogeneous effects by vaccine side effects experience. This question can be further explored in future research. Media reporting of rare adverse events can also amplify concerns and anxieties around vaccines that personal experience cannot fully counter. While this article has implications for vaccine mandates, we do not explicitly test the effects of mandates, given the complexity of modeling this would entail and the fact that mandates were never universally implemented in the United States. The results of this study should not be interpreted as evidence supporting mandatory vaccination. However, previous research has shown that even the perception that a mandate may be imposed can affect vaccination behavior and attitudes (Schmelz and Bowles 2022). We also did not measure effects on broader attitudes toward government (only trust in public health institutions) and confined our analysis to more short-term effects, given the recency of COVID-19 vaccination and data availability.

Discussion

By decomposing the effects of vaccination experience with differing levels of vaccine hesitancy, the analysis demonstrated that people's experience with COVID-19 vaccination increased their confidence in COVID-19 vaccines and decreased mistrust in the vaccine manufacturing and approval processes, on average. Strikingly, we observed generally higher impact of vaccination in more hesitant groups, with larger effect sizes in these groups. This implies that COVID-19 vaccination not only increased vaccine confidence but also mitigated underlying hesitancy. However, this was not consistently significant in addressing concerns about vaccine-related side effects and illness, suggesting that improved confidence in vaccines can coexist with lingering reservations.

The overall study findings suggest a fresh perspective: the experience of vaccination, as a highly tangible policy, can serve as a tool in improving people's attitudes toward vaccines and vaccine-related processes that are critical to the success of immunization programs. Most earlier studies on vaccination and vaccine confidence primarily focused on identifying the factors of vaccine hesitancy and uptake. By shifting this focus to viewing vaccination itself as a policy intervention, this study underscores the potential use of mass vaccine rollouts in addressing vaccine hesitancy and enhancing vaccine confidence. The government can also consider facilitating positive vaccination experiences as part of its vaccination efforts, to promote trust and confidence in immunization programs. Providing safe vaccines with a smooth process can be a means to recover public confidence in vaccines in countries and communities where vaccine confidence plummeted, particularly in the aftermath of the COVID-19 pandemic.

The finding that the effects of vaccination was consistent across different population segments suggests that the impact of vaccination on vaccine confidence was widespread and not limited to specific groups. This highlights the universal potential of positive vaccination experiences to rebuild trust in vaccines, particularly in groups where skepticism would be more salient. In the United States, researchers have made several attempts to develop interventions to address the partisan and racial gaps in vaccine hesitancy, such as financial incentives (Algara and Simmons 2023) and messaging (Dhanani and Franz 2022). Our study adds that fostering positive vaccination experiences could serve as another effective intervention.

However, it is worth noting that the effects of COVID-19 vaccination were relatively limited in addressing concerns about vaccine side effects and vaccine-induced illness. At minimum, given how rare immediate side effects are, we might expect people's concerns about side effects to decrease after an experience with vaccination. This indicates that addressing concerns about potential side effects is a more difficult challenge compared to influencing beliefs about the benefits of COVID-19 vaccination, such as mitigating virus infection and disease severity.

Our findings are limited in their ability to answer the question of whether vaccine mandates are helpful or harmful to vaccine confidence, because we do not know whether this eventual vaccination was voluntary or was undertaken in response to a mandate or the impression that there could be future compulsion. Nevertheless, we observe that hesitant populations that ultimately received vaccinations generally experienced improved attitudes toward vaccines. Two interpretations of this are possible. It is possible that if these experiences were undertaken voluntarily, vaccine confidence and trust in vaccine-related processes were improved primarily because of the impression that vaccination was uncoerced. On the other hand, if people's attitudes improved following mandated vaccination, we could also conclude that mandates improved, and did not erode, confidence in vaccines and vaccine-related processes. Future studies that can better causally identify whether vaccination was voluntary or mandated can help us to tease these questions out.

References