Death is always a difficult topic to discuss, and death has been in the news a lot during this tragic coronavirus pandemic. Many news stories focus on states, counties, or cities that have the most cases or the most deaths. A related statistic is the case fatality rate, which is the number of deaths due to COVID-19 divided by the number of confirmed cases. (For more information about case fatality rates, see "Understanding COVID-19 data: Case fatality rate vs. mortality rate vs. risk of dying.") For this statistic, it is not always appropriate to focus on the highest values. When a county or state has only a few confirmed cases, the estimate of the case fatality rate can vary widely from day to day. The examples and graphs in this article can help you to understand the variability in a rate statistic.

This article discusses visualizing the case fatality rate for COVID-related deaths in North Carolina counties. A subsequent article discusses all US counties.

Fatalities are a grim topic. Each marker on each graph represents people who have died. A goal of these graphs is to identify counties that have extreme case-fatality rates. Decision-makers can then direct resources to the higher-than-expected communities and learn from the communities that have lower-than-expected rates.

Data visualization of the case fatality rate

Data visualization is an essential tool for researchers, decision-makers, and the general public. But you need to be careful when you visualize rates. If you use a simple bar chart, the viewer's eye is drawn towards the highest rates. This might not be appropriate when the sample size is small.

An alternative visualization is called a funnel plot (Spiegelhalter, 2005). The funnel plot is a scatter plot that can help to identify communities where the case fatality rate is higher than expected or lower than expected. This article emphasizes two main statistical ideas:

1. An estimate of a rate is highly variable when the sample size is small.
2. A funnel plot can help you visualize the rates relative to a reference rate (which is often the overall rate). A funnel plot can help you decide whether a sample rate is much higher or lower than the reference rate by taking the size of the sample into account.

Variability in estimates of proportion

Let's get specific. The case fatality rate is a proportion: the number of deaths due to COVID-19 (the numerator) divided by the number of confirmed cases (the denominator): Rate = Deaths / (Confirmed Cases). When the denominator is small, a small change in the numerator causes a large change in the estimate of the rate. For example, consider a hypothetical county in which initially only one person has tested positive for the coronavirus and no one has died. The county might experience the following progression over time:

• First week: 0 deaths and 1 confirmed case. The case fatality rate is 0.0.
• Second week: 1 death and 2 confirmed cases. The case fatality rate is 1/2 = 0.5 or 50%.
• Third week: 2 death and 5 confirmed cases. The case fatality rate is 2/5 = 0.40 or 40%.
• Fourth week: 2 death and 10 confirmed cases. The case fatality rate is 2/10 = 0.2 or 20%.

When a public-health official looks at the case fatality rate for this county, should she be unduly concerned when she sees that the rate jumped from 0% to 50% in one week? No. That spike is due to one additional death and occurred because the denominator was so small. For counties with 20 or fewer cases, one or two new deaths can cause the case fatality rate to spike. Looking only at the raw case fatality rates can be misleading. Therefore, you should not use a table or bar chart to visualize the counties that have the highest rates.

A funnel plot for case fatality rates

Instead, you can visualize each county as a point in a funnel plot. A sample funnel plot for counties in North Carolina is shown below. In a funnel plot, the horizontal axis represents the number of confirmed cases (the denominator) and the vertical axis represents the estimate of the case fatality rate. For comparison, you can add a reference line that represents an overall case fatality rate. You can also add curves that indicate the usual range of variability for an estimate as a function of the sample size.

The blue-gray funnel-shaped area between these curves is the "range of variability." Estimates that are within the range of variability are not much different from the reference rate. Estimates that are outside the range are higher than expected or lower than expected. The last section of this article provides references to the theoretical justification of a funnel plot, as well as links that show how to create a funnel plot in SAS.

In the funnel plot, each marker represents one of the 78 counties in North Carolina that have five or more confirmed cases of coronavirus. The graph shows the names of the 10 counties that have 100 or more confirmed cases. Not surprisingly, these are among the most populous counties in NC, including the counties that are near the big cities of Charlotte, Raleigh, Durham, and Greensboro.

The points along the left side of the funnel plot represent counties that have few confirmed cases. As you can see, the empirical estimates for these counties vary between 0.0 (no reported deaths) to 0.2 (one death among five cases). The other high rates correspond to the fractions 2/15, 1/8, and 2/18. Although the raw rates are high, these are inside or near the boundary of the funnel-shaped region. There is not sufficient evidence to conclude that these rates are significantly different than the overall case fatality rate in North Carolina, which is 1.9%.

The two counties that stand out in this funnel plot are Johnston County and Guilford County. On 15APR2020, both had case fatality rates that were much higher than you would expect if their rate was the state average of 1.9%. Johnston County (just south of Raleigh) had an outbreak at a nursing home, which resulted in four of the county's nine deaths.

There are also two counties that have fewer-than-expected rates: Durham County and Wake County. Durham County (which contains the city of Durham) has had 330 cases and two deaths. Wake County (which contains Raleigh and SAS World Headquarters) has had 151 confirmed cases and one death.

In this graph, the reference rate is the overall case fatality rate in NC (1.9%), but you can create a similar graph by using some other rate, such as a national rate or a target rate.

It is important to remember that the denominator of the estimate is the number of confirmed cases of infection. Therefore, the estimate depends on the availability of tests, the number of people tested, and other factors that might be beyond the control of county health officials.

Summary

It is a statistical fact that estimates that are based on small samples are highly variable. In other words, it is not unusual to observe an extreme rate for a small sample. A funnel plot is a good way to visualize many estimates that are based on samples of different sizes. The funnel plot incorporates the size of the sample into the visualization. The graph includes curves that indicate an acceptable range of variability for each sample size. If a sample rate is far outside the region, the sample can be examined more closely to understand why the rate is extreme.

A funnel plot is a statistical tool that applies to any rate, but this article used the case fatality rate for the COVID-19 pandemic. A funnel plot can help decision-makers identify unusual rates that merit additional attention. If a rate is abnormally high, the community might need additional support or assistance. If a rate is abnormally low, the community might be doing something well that other communities can learn from.

Further reading

• Spiegelhalter, D. J. (2005), "Funnel plots for comparing institutional performance," Statistics in Medicine, 24(8), 1185-1202.
• Wicklin, R. (2011), "Funnel plots for proportions," The DO Loop blog.
• Wicklin, R. (2018), "A funnel plot for immunization rates," The DO Loop blog.
• You can download the data and the SAS program that creates the funnel plot in this article.