Funnel plots for proportions

I have previously written about how to create funnel plots in SAS software. A funnel plot is a way to compare the aggregated performance of many groups without ranking them. The groups can be states, counties, schools, hospitals, doctors, airlines, and so forth. A funnel plot graphs a performance metric versus a measure of uncertainty (often the size of each sample) and overlays "control limits" that enable you to assess whether extreme values are "unusual" or whether they fall within the realm of statistical variation.

When the performance metric is normally distributed, the funnel chart is similar to the analysis of means plot. However, you can also create funnel charts for quantities such as rates, proportions, and ratios of proportions. All of these variations are discussed in Spiegelhalter's 2005 paper on funnel plots.

This article describes how to construct a funnel plot for proportions in SAS. Spiegelhalter lays out four steps for creating the funnel plot of the proportion of cases that satisfy some condition for multiple groups (see Appendix A.1.1):

  1. Compute the proportion for each category.
  2. Compute the overall proportion.
  3. Compute control limits by using quantiles of the binomial distribution. This step is tricky. The binomial distribution is a discrete distribution, which means that you have to use an interpolation scheme in order to produce smooth control limits. (If your sample sizes are large, you might prefer to use a normal approximation to the binomial limits.)
  4. Plot the proportion for each category against the sample size and overlay the control limits.

Step 3 is very interesting. However, this post does not discuss the details of Step 3.

The proportion of children placed into adoption within 12 months

A recent post on Spiegelhalter's blog gives data and produces a funnel plot for the proportion of children that are placed with adoptive parents within 12 months. The data is aggregated for various regions in England, which are called "local authorities." (You can download the data in a CSV file.) To give you some idea of the data, one authority (Kent) placed 162 out of 235 children (69%) with adoptive parents within 12 months. Another (Chesire East) handled only 15 cases and placed 10 of the children (67%).

A funnel plot (shown below) is a way to compare the proportion of children that are placed within 12 months for each of 143 regions. The horizontal line shows the nationwide proportion of children that are placed within 12 months. The funnel-shaped curves are the 95% and 99.8% quantile curves under the assumption that proportions are sampled from a binomial distribution for which the probability of "success" (placing a child) equals the national average. Because a statistic computed on a small sample has larger variability that the same statistic computed on a large sample, the two funnel-shaped curves for these data are wide to the left and narrow to the right.

Only those local authorities that are above the funnel-shaped curves can be considered "above average" at placing children into adoption. Similarly, only the handful of local authorities that are below the curves can be considered "below average." The authorities that are between the innermost curves are all "average," and you should avoid saying that one is "better" that another, even though one has placed 85% of their children whereas another has placed only 65%.

You can download the SAS program used in to create the funnel plot. The remainder of this post describes each of the four main steps that are used to construct the plot.

Compute the proportion for each category

The Adoptions data set contains 143 observations and three variables. Each observation is for a local authority (the Authority variable). The Trials variable contains the number of children that are available for adoption. The Events variable contains the number of children placed into adoptive homes within 12 months.

The following statements read the data into SAS/IML vectors and compute the proportion of children that were placed:

proc iml;
use Adoptions;
read all var {Events Trials};
close Adoptions;
 
Proportion = Events/ Trials;

Compute the overall proportion

The following statements compute the proportion of children (theta) nationwide that were placed within 12 months. This overall proportion is used to compute the expected number of children that would be placed if each local authority had a success rate of theta in placing children. The observed and expected values are written to a SAS data set, and the overall proportion is stored in a macro variable for later use:

/* write theta to macro to use as reference line */
call symputx("AvgProp", theta);
 
/* write results to data set */
results = Proportion || Expected;
labels = {"Proportion" "Expected"};
create Stats from results[colname=labels];
append from results;
close;

Compute control limits

The following statements compute the control limits by using the formulas in Appendix A.1.1. Notice that the control limits do not depend on the data, only on the national proportion. In this example, the control limits are plotted at equally spaced values between the minimum and maximum sample sizes. The 95% and 99.8% control limits are computed and saved to a SAS data set for graphing.

/* plot limits at equally spaced points between min & max */
minN = min(Trials); maxN = max(Trials);
n = T( do(minN, maxN, (maxN-minN)/20) );
n = round(n); /* binomial parameter must be integer */
p = {0.001 0.025 0.975 0.999}; /* lower/upper limits */
/* compute matrix with four columns, one for each CL */
limits = j(nrow(n), ncol(p));
do i = 1 to ncol(p);
   r = quantile("Binomial", p[i], theta, n);
   /* adjust quantiles according to method in Appendix A.1.1
      Spiegelhalter (2005) p. 1197 */
   numer = cdf("Binomial", r  , theta, n) - p[i];
   denom = cdf("Binomial", r  , theta, n) - cdf("Binomial", r-1, theta, n);
   alpha = numer / denom;
   limits[,i] = (r-alpha)/n;
end;
 
/* write these control limits to data set */
results = n || limits;
labels = {"N" "L3sd" "L2sd" "U2sd" "U3sd"};
create Limits from results[colname=labels];
append from results;
close;
quit;

If you want to compute the control limits for each local authority (for example, to flag those that are outside of the 99.8 control limits), you can set n = Trials instead of using evenly spaced values of n.

Create the funnel plot

There are now three data sets: the original data, the observed and expected proportions, and the control limits. The first and second data sets can be merged, but the third has a different number of observations. The following DATA steps combine the data into a single data set. The SGPLOT procedure then creates the funnel plot.

/* merge data with proportions */
data Adoptions; merge Adoptions Stats; run;
/* append control limits */
data Funnel; set Adoptions Limits; run;
 
/* Plot proportions versus sample size. Overlay control limits */
proc sgplot data=Funnel;
band x=N lower=L3sd upper=U3sd / nofill lineattrs=(color=lipk)
     legendlabel="99.8% limits" name="band99";
band x=N lower=L2sd upper=U2sd / nofill lineattrs=(color=gray)  
     legendlabel="95% limits" name="band95";
refline &AvgProp / axis=y;
scatter x=Trials y=Proportion;
keylegend "band95" "band99" / location=inside
     position=bottomright;
yaxis grid values=(0.3 to 1 by 0.1); xaxis grid;
run;

There are five local authorities who are "below average" in placing children into adoptive homes within a 12-month time frame. The graph indicates that Hackney (44%; 24 out of 55 cases), Brent (51%; 23 out of 45 cases), Nottinghamshire (55%, 52 out of 95 cases), Derby (57%; 54 out of 95 cases), and Nottingham (59%; 59 out of 100 cases) might need additional support and resources to better handle their caseloads in a timely fashion.

If desired, you can use the technique that I described in a previous post to label the observations that are beyond the control limits.

tags: Statistical Graphics, Statistical Programming

12 Comments

  1. Charles
    Posted February 1, 2012 at 3:48 pm | Permalink

    I am not able to use the sgplot with SAS 9.1 although the proc IML appears to work. How can I et around the problem of not having sgplot?

    Many thanks,
    charles

  2. John
    Posted February 23, 2012 at 10:51 am | Permalink

    Hi Rick,

    Thanks for this very useful post! I have a large cohort of patients. How do I change the code for the control limits to use the approx normal distribution as you suggest?

    I have tried modifying the section below without success.
    r = quantile("Binomial", p[i], theta, n);
    /* adjust quantiles according to method in Appendix A.1.1
    Spiegelhalter (2005) p. 1197 */
    numer = cdf("Binomial", r , theta, n) - p[i];
    denom = cdf("Binomial", r , theta, n) - cdf("Binomial", r-1, theta, n);

    thanks for your help,

    John

  3. John
    Posted May 21, 2012 at 12:09 pm | Permalink

    Hi Rick,

    Why aren't the control limits perfectly symmetrical?

    Thanks,
    John

  4. Kevin Williams
    Posted November 17, 2012 at 5:17 pm | Permalink

    Based on article Funnel plots for proportions - the overall proportion is calculated but what is the calculation for the overall variance ?

    • Posted November 19, 2012 at 8:54 am | Permalink

      It's not needed for this computation. However, the variance of a Bernoulli variable is p*(1-p), where p is the probability of success. For this data, the variable Proportion is an estimate of p.

  5. RK
    Posted October 10, 2013 at 12:15 am | Permalink

    I notice that the funnel plot code does not quite work for very small sample sizes (<5) ? The upper CI especially instead of gradually decreasing in values show a small spike and then decreases in values as expected.
    1) Can we correct for this ?
    2) Should we worry about even doing a funnel plot for SS < 5 ?

    • Posted October 10, 2013 at 9:50 am | Permalink

      Good questions. Spiegelhalter already corrects for small sample sizes by using a more complicated formula to interpolate the binomial quantiles. See my article on "The Normal Approximation to the Binomial Distribution."

      I suspect that most people try not to use subjects (doctors, hospitals, schools,...) that have fewer than 5 observations. The variance of the proportion is too large in those cases. Suppose you are investigating the mortality rate of patients treated by doctors. If a doctor treats three patients and two die, is that incompetance or bad luck? Better to get more data. For example, collect data over several years instead of using the past month.

      For a different approach, See Rodriguez and Lewellen (2004), Example 10, which uses logistic regression to adjust and estimate proportions. Note, however, that even in that paper they exclude doctors with fewer than 10 patients.

      • RK
        Posted October 10, 2013 at 12:50 pm | Permalink

        How would a ratio plot compare? x-axis would be exposure, the Y-axis would be the observed divided by the expected exposure adjusted rates, i.e. Ratio. Any insights on this ? I have done some reading and it appears that we would use poisson exact control limits. Is there SAS code out there that does this?

2 Trackbacks

  1. [...] Funnel plots for proportions [...]

  2. [...] always returns a discrete value. A consequence of this fact was featured in my article on "Funnel plots for proportions." Step 3 of creating a funnel plot is complicated because it computes a continuous approximation to [...]

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <p> <pre lang="" line="" escaped=""> <q cite=""> <strike> <strong>