There’s No Perfect Way To Measure COVID-19 Vaccination Equity, But Here’s Where To Begin

Two years into the COVID-19 pandemic, and more than a year into the nationwide vaccination campaign, more than 80 percent of vaccine-eligible Americans have rolled up their sleeve for a shot. About 70 percent are considered “fully vaccinated,” even as that term seems increasingly antiquated in the era of booster doses and Centers for Disease Control and Prevention (CDC) guidance around “staying up to date” with your vaccines.

CDC data and a growing body of research show that historically marginalized populations, including Black, Hispanic, and Indigenous communities, are at higher risk of mortality from COVID-19. Recognizing this, the White House rolled out a set of policies in early 2021 designed to increase vaccination rates in the hardest-hit and most vulnerable communities. Achieving vaccination equity across racial and ethnic groups remains a focus for the CDC, with the agency describing equity as “when everyone has fair and just access to COVID-19 vaccination.” Numerous states also launched initiatives in early 2021 focused on racial equity in vaccine distribution. One year later, how can we assess their success at achieving equitable vaccination?

We have been keeping close tabs on this issue of vaccination equity as part of our ongoing work in support of the Rockefeller Foundation’s Equity-First Vaccination Initiative (EVI), an initiative with national-, local-, and community-based organizations in five cities to implement hyper-local strategies to improve vaccine access and increase vaccine confidence. The EVI seeks not only to promote equitable access to COVID-19 vaccines but to narrow racial and ethnic disparities in actual vaccination rates. Building on research specific to COVID-19 and related to measuring health equity overall, we have used several approaches to quantify vaccination equity and monitor progress toward this goal.

Below, using data from the United States overall, Illinois, and Chicago, we describe the strengths and limitations of three approaches for measuring and visualizing progress toward equity in vaccination uptake. In addition, we highlight the limitations common to all three approaches that make it difficult to measure whether vaccination is equitable across racial and ethnic groups. We display the equity measures at three levels of geography, both to allow for comparisons and to describe how data limitations can vary by geography.

We chose to focus on Chicago (one of the EVI cities) because the city of Chicago’s Data Portal provides publicly available, high-quality, and timely data by race and ethnicity over time. These analyses could be replicated for other cities that provide timely data disaggregated by race and ethnicity, although the availability of such data remains far from universal. We focus on the “fully vaccinated” population (as opposed to those who only received their first dose, or booster recipients) because getting people two shots was the initial goal of the vaccination campaign and because of the incomplete availability of booster data by race/ethnicity at all geographic levels. We also note that the reduction in risk of severe COVID-19 outcomes is greater when going from unvaccinated to “fully vaccinated” than from “fully vaccinated” to boosted.

Approach 1: An Equity Goal Of 100 Percent Vaccination For All Groups

An intuitive measure of whether vaccinations are being delivered equitably is to determine whether the percentage of individuals fully vaccinated is the same across all racial or ethnic groups. This approach allows for straightforward comparisons of vaccination rates across racial/ethnic groups and geographies. This approach also lends itself well to tracking progress over time toward a unified, aspirational goal: having 100 percent of eligible individuals fully vaccinated. Exhibit 1 illustrates that vaccination rates are generally higher in Chicago than in Illinois and nationally, although the Black or African American rate is the lowest in every geography.

Exhibit 1: Percentage of vaccine-eligible population that is fully vaccinated, by level of geography and race/ethnicity

Sources: The Chicago Department of Public Health, the Illinois Department of Public Health, and the Centers for Disease Control and Prevention, as of May 2, 2022. Population denominators for the vaccine-eligible population are estimated from the American Community Survey, 2019 1-Year Estimates.

In this exhibit, we define “fully vaccinated” as receipt of two doses of the Pfizer-BioNTech or Moderna vaccines, or one dose of the J&J/Janssen vaccine. Hispanic or Latino individuals may be of any race, while the other groups shown reflect data for non-Hispanic individuals in the racial group. Non-Hispanic individuals of other races are not displayed because smaller population sizes make data less reliable. Those with missing race/ethnicity information in the vaccination data are not displayed because there is no available population denominator for this group (that is, the American Community Survey does not include an unknown race/ethnicity category).

A limitation of this approach is that it is especially vulnerable to missing race/ethnicity data. In exhibit 1, it appears that vaccination rates are much lower nationally than in Chicago or Illinois. However, lower national vaccination rates are partially an artifact of missing data: Race or ethnicity is missing for nearly one-quarter of the fully vaccinated population nationally, versus just 2–3 percent in Chicago and Illinois. The large amount of missing data makes it difficult to understand the true size of racial/ethnic disparities. We know that the overall fully vaccinated rate is about 70 percent nationally, and that rates for all races and ethnicities are higher than shown in exhibit 1. However, we do not know how much higher the rate for each group is because we do not know the share of vaccinated individuals of unknown race/ethnicity in each group.

Furthermore, the approach of showing the progress toward 100 percent vaccination for each race and ethnicity group (as in exhibit 1) masks the size of each population group and whether the inequities displayed affect a large share of the overall population. This approach to measuring equity also fails to consider disparities in the COVID-19 burden across these populations and where inequities in vaccination rates may compound the already inequitable burden the pandemic has placed on different groups. Therefore, approach 1 should be considered more a measure of the equality of vaccination receipt than a true measure of equity. Other approaches to measuring equity described below can better account for these factors but are more complex.

Approach 2: An Equity Goal Where Share Of Vaccinations Matches Share Of Population

Another approach is to look at whether the racial/ethnic distribution of the fully vaccinated population matches the racial/ethnic distribution of the vaccine-eligible population. This relates to the first approach in that these distributions would align if the vaccination rate for each race/ethnicity group were the same, but this approach to visualizing equity offers different benefits and drawbacks. Exhibit 2 illustrates how racial and ethnic distributions vary across city, state, and national levels, and in turn, what portion of the overall population is affected by vaccination inequities. This visualization lends itself well to presenting information on the racial/ethnic distribution of the fully vaccinated population, both with and without individuals for whom racial and ethnic information is unknown.

Exhibit 2: Percentage of fully vaccinated and vaccine-eligible population distribution, by level of geography and race/ethnicity

Sources: The Chicago Department of Public Health, the Illinois Department of Public Health, and the Centers for Disease Control and Prevention, as of May 2, 2022. The leftmost bars for each level of geography reflect the distribution of the vaccine-eligible population based on 2019 American Community Survey 1-Year estimates. The center bars show the distribution of the fully vaccinated population including all fully vaccinated individuals in the denominator, including those of unknown race/ethnicity. The rightmost bars present the distribution by race/ethnicity for individuals for whom race/ethnicity is known. The “Other” category includes non-Hispanic individuals who are American Indian/Alaska Native, Native Hawaiian/Other Pacific Islander, multiracial, or another race not separately listed.

This exhibit highlights the difference in the race/ethnicity distribution of the population in the city of Chicago when compared to the statewide and national distributions, with Black or African American and Hispanic or Latino individuals accounting for a much larger share of the population in the city. Despite the differences in the distributions, we see that in Chicago, Illinois, and nationally, Black or African American individuals are a smaller share of the fully vaccinated than of the total vaccine-eligible population, even when restricting to the vaccinated population with known race/ethnicity information. At all three levels of geography, Hispanic or Latino individuals account for about the same share of the fully vaccinated as their share of the vaccine-eligible population. The White population accounts for about the same share of the fully vaccinated in Chicago and Illinois, but nationally this group appears underrepresented among the fully vaccinated, even when considering only those with known race/ethnicity data.

Across all levels of geography, the Asian population and individuals in the “other” race group appear to be vaccinated at higher rates relative to their population shares than other groups However, as exhibit 2 shows, both populations are comparatively small. Moreover, high rates among the “other” race category may stem in part from data limitations, with 2019 census data likely undercounting these groups relative to how people self-identify their race/ethnicity when getting vaccinated (we discuss this limitation in more detail further below).

Additional limitations of this approach include the difficulty of easily comparing vaccination rates across geographies and race or ethnicity groups, since vaccination rates are not directly shown and each group accounts for a different share of the vaccine-eligible population in each geography. Finally, this approach does not consider differential burdens across groups and whether vaccinations have successfully targeted populations most impacted by COVID-19. As with the first approach, it should be considered more a measure of equality than of equity.

Approach 3: An Equity Goal That Accounts For Differential Deaths Due To COVID-19

Drawing on HIV-prevention efforts and early research on strategies for prioritization of the COVID-19 vaccine, we constructed a measure of how well the vaccination effort is reaching the racial/ethnic groups most in need. Specifically, in exhibit 3, we calculated an “equity index” with a numerator of the proportion of fully vaccinated who are in the racial/ethnic group and a denominator of the proportion of COVID-19 deaths that are in the racial/ethnic group.  

A value of 1 for this index reflects one interpretation of equity—that the share of fully vaccinated individuals who are in a particular race/ethnicity group matches that group’s share of deaths due to COVID-19. For example, a group would have an index of 1 if it accounts for both 25 percent of individuals fully vaccinated and 25 percent of deaths due to COVID-19. Values below 1 signify inequity, meaning that a group has experienced a larger share of deaths than its share of the fully vaccinated population. While this information could be presented in a similar way to exhibit 2 (that is, using multiple stacked bars), constructing indices for each racial/ethnic group can facilitate examining trends over time (not shown here).

Exhibit 3: Percentage of fully vaccinated as a proportion of percentage of COVID-19 deaths, by level of geography and race/ethnicity

Sources: The Chicago Department of Public Health, the Illinois Department of Public Health, and the Centers for Disease Control and Prevention, as of May 2, 2022. This index is calculated with a numerator of fully vaccinated in the racial/ethnic group divided by the total number of fully vaccinated individuals and a denominator of deaths in racial/ethnic group divided by all deaths. Non-Hispanic individuals of other races and those with missing race/ethnicity information are included in the totals when calculating the numerator and denominator proportions but we do not display index values for these groups.

Exhibit 3 shows that in Chicago, Illinois, and nationally, the Black or African American community accounts for a smaller share of the fully vaccinated population than would be equitable based on the proportion of deaths experienced due to COVID-19 (index value <1). This finding is unchanged even when excluding those with unknown race/ethnicity (for example, when assuming that 10 percent rather than 8 percent of fully vaccinated individuals nationally are Black or African American). Conversely, the Asian population accounts for a smaller share of deaths than the fully vaccinated (index value >1) at all levels of geography. The findings are more mixed for the Hispanic or Latino and White populations. For example, the White population has an index value of about 1.5 in Chicago but less than 1 in Illinois and nationally. Equity index values by race/ethnicity will vary across cities and states, depending on the distributions of the vaccinated population and the deaths due to COVID-19 in each area.

A strength of this approach is that it considers the differential burden of COVID-19 across racial and ethnic groups, a critical element of assessing equity. It illustrates the severity of the inequities faced by the Black or African American population by accounting both for this group’s overrepresentation among deaths and underrepresentation among the vaccinated. For example, while exhibit 1 shows that the vaccination rate for the White population in Chicago is about 25 percent higher (14 percentage points) than the rate for the Black or African American population, the equity index value is nearly 175 percent higher (exhibit 3). This equity index is especially useful for understanding inequities in the early stages of the vaccine rollout, when an appropriate goal (considering vaccine supply constraints) was to target the vaccine to the groups most severely impacted by the pandemic—as with the prioritization of vaccination among older adults.

However, it can be particularly difficult to explain and interpret this index once a large proportion of every group is vaccinated. If everyone were vaccinated, resulting in each group’s share of the vaccinated matching its share of the population, achieving equity indices of one would require the distribution of deaths to match the population distribution as well. For groups with index values above one, this could only happen by experiencing a disproportionate share of deaths moving forward. Thus, this index may be most instructive in the early phases of vaccination campaigns by demonstrating whether vaccinations are being successfully targeted to the groups bearing disproportionate death burdens. Over the longer term, its greatest utility may be for highlighting the magnitude and persistence of inequities when accounting for both the impacts of deaths and differential vaccination rates.

Last, we note a limitation common to all approaches: the inability of administrative data to capture the rapidly evolving race/ethnicity distribution of the US population and changes in how people self-identify within the sociopolitical construct of race. Our analyses and those of others (including the CDC) use 2019 population data to calculate what proportion of each race or ethnicity group is vaccinated. These data are three years old in a country that saw its Hispanic or Latino population grow by 23 percent from 2010 to 2020 while the White, non-Hispanic population declined. For this reason, we anticipate that Hispanic or Latino vaccination rates are a bit lower than calculations using 2019 population data suggest and that White, non-Hispanic vaccination rates are a bit higher.

Meanwhile, more people than ever identify as multiracial or of a race other than the socially defined racial groupings commonly tabulated in census data. While this can allow for a more granular understanding of the US population, it also compounds challenges that arise from inconsistencies in how race/ethnicity groups are described in federal, state, and local data collections. In some cases, race/ethnicity vaccination rates can appear to exceed 100 percent. Because their small sizes make data less reliable, we are not able to show American Indian or Alaska Native, Native Hawaiian or Other Pacific Islander, multiracial, and other races in exhibits 1 and 3, and we group them when presenting the population and fully vaccinated distributions in exhibit 2. Ideally, efforts to encourage more consistent, disaggregated data collection (for example, for the American Indian or Alaska Native populations) will yield higher-quality data that enhances our understanding of the experiences of less populous groups in the future.

Measuring And Visualizing Equity Moving Forward

As the pandemic evolves, public health practitioners, policy makers, and researchers should continue to measure progress toward equitable vaccination and keep in mind the strengths and limitations of different approaches for measuring and visualizing equity (exhibit 4). While many limitations are common to all approaches, each approach has unique strengths. For example:

• If the goal is to present easy-to-understand information to a broad audience or to show trends over time toward a clear, common goal, reporting the percentage of the population that is fully vaccinated may be the preferred approach (exhibit 1).
• If providing context about relative population size is important when monitoring vaccination progress—for example, showing that the Black or African American and Hispanic or Latino populations are a much larger share of the Chicago population than statewide or nationally—reporting the percentage of those fully vaccinated along with the distribution of the vaccine-eligible population may be the preferred approach (exhibit 2).
• If the goal is to consider disease burden (to more fully assess equity as opposed to equality) or to understand if vaccinations are reaching the groups most severely affected in the early days of a vaccination campaign, reporting the percentage of the fully vaccinated as a proportion of the percentage of deaths due to COVID-19 may be the preferred approach (exhibit 3).

While we have presented three approaches to assessing vaccination equity throughout the vaccination effort (and have therefore used cumulative numbers) we also note that, to focus efforts based on more recent trends, it might be preferable to only examine recent data (for example, limit the analysis to the vaccinations in the previous month). However, this introduces additional interpretation challenges when considering the evolution over time in the race/ethnicity distribution of the remaining unvaccinated population. Future work might also consider exploring similar questions using data on booster doses if data by race/ethnicity become more widely available. It may also move beyond the raw comparisons made here to account for differences by race/ethnicity in factors that affect the risk of severe outcomes from COVID-19, such as age distribution, comorbidities, or occupation.

Overall, selecting the best approach(es) to examine equitable vaccination requires tradeoffs and considerations of the objective, audience, and data availability. While the approaches we have described are focused on COVID-19 vaccination across racial and ethnic groups, others have examined equitable vaccination using community-level measures of social vulnerability and described similar considerations for measuring health disparities more broadly. A common theme is that promoting equity requires an understanding of existing inequities, which in turn demands high-quality, disaggregated, geographically granular data, and clear methods to present those data.

Exhibit 4: Summary of the strengths and limitations of three approaches for measuring and visualizing equitable vaccination by race and ethnicity

Source: Authors’ notes.

Authors’ Note

The authors received financial support from the Rockefeller Foundation for the research and analysis presented in this article. The findings and conclusions contained in the article are those of the authors and do not necessarily reflect positions or policies of the Rockefeller Foundation.