Students with missing test scores are often highly mobile students and are more likely to be low-achieving students. It is important to include these students in any growth/value-added model to avoid selection bias, which could provide misleading growth estimates to districts, schools and teachers that serve higher populations of these students. While more simplistic growth/value-added estimates may require that students have the same set of predictors (prior test data points) or that students have all required predictors, this often has the result of excluding certain kind of students. This would disproportionately affect educators serving those types of students.
Some value-added models do not require that students have the same set of predictors or all required predictors, and this approach includes more students in the growth measures. When estimating students’ entering achievement, value-added models should consider the quantity and quality of information available to each student, as well as student mobility among schools from year to year.
To accomplish this without imputing student test scores, EVAAS uses a sophisticated modeling approach that provides more reliable estimates of growth.1
As a simple example, consider the following scenario. Ten students are given a test in two different years. The goal is to measure academic growth (gain) from one year to the next. The left side of the figure shows all students taking both tests. The right side of the figure shows the same students, some of whom now have missing scores. Two simple approaches when data are missing are to calculate the mean of the differences, or to calculate the differences of the means.
When there are no missing data, these two simple methods provide the same answer: 5.8 in the left table.
However, when there are missing data, each method provides a different result: 9.6 vs. 4.0 on the right.
As illustrated above, a more sophisticated model is needed to address this problem. The approach used by EVAAS estimates the means in each of these cells using relationships between students’ test scores as if there were no missing test scores. The models provide more reliable and less biased growth measures without imputing any data. Furthermore, EVAAS uses much more student data (across all tested grades and subjects) to obtain these relationships in the growth estimates for districts, schools and teachers.
The problem of missing data is very common, especially in school systems with high mobility, and must be taken into consideration. From a philosophical perspective, it is important that as many students as possible be included in the district and school growth measures so that highly-mobile student populations receive the same level of attention as non-mobile ones. At the teacher level, it is critical so that teachers are not advantaged or disadvantaged by the student populations they serve.
1 Wright, S. P. (2004). “Advantages of a Multivariate Longitudinal Approach to Educational Value-Added Assessment Without Imputation.” Paper presented at National Evaluation Institute, online at http://www.createconference.org/documents/archive/2004/Wright-NEI04.pdf.
