Value-added myth busting, Part 4: Value-added models cannot measure growth of students who have missing data or are highly mobile

Nadja Young
Nadja Young|May 17, 2013

Students with missing test scores are often highly mobile students and are more likely to be low-achieving students. It is important to include these students in any growth/value-added model to avoid selection bias, which could provide misleading growth estimates to districts, schools and teachers that serve higher populations of these students. While more simplistic growth/value-added estimates may require that students have the same set of predictors (prior test data points) or that students have all required predictors, this often has the result of excluding certain kind of students. This would disproportionately affect educators serving those types of students.

Some value-added models do not require that students have the same set of predictors or all required predictors, and this approach includes more students in the growth measures. When estimating students’ entering achievement, value-added models should consider the quantity and quality of information available to each student, as well as student mobility among schools from year to year.

To accomplish this without imputing student test scores, EVAAS uses a sophisticated modeling approach that provides more reliable estimates of growth.1

As a simple example, consider the following scenario. Ten students are given a test in two different years. The goal is to measure academic growth (gain) from one year to the next. The left side of the figure shows all students taking both tests. The right side of the figure shows the same students, some of whom now have missing scores. Two simple approaches when data are missing are to calculate the mean of the differences, or to calculate the differences of the means.

When there are no missing data, these two simple methods provide the same answer: 5.8 in the left table.

However, when there are missing data, each method provides a different result: 9.6 vs. 4.0 on the right.

As illustrated above, a more sophisticated model is needed to address this problem. The approach used by EVAAS estimates the means in each of these cells using relationships between students’ test scores as if there were no missing test scores. The models provide more reliable and less biased growth measures without imputing any data. Furthermore, EVAAS uses much more student data (across all tested grades and subjects) to obtain these relationships in the growth estimates for districts, schools and teachers.

The problem of missing data is very common, especially in school systems with high mobility, and must be taken into consideration. From a philosophical perspective, it is important that as many students as possible be included in the district and school growth measures so that highly-mobile student populations receive the same level of attention as non-mobile ones. At the teacher level, it is critical so that teachers are not advantaged or disadvantaged by the student populations they serve.

1 Wright, S. P. (2004). “Advantages of a Multivariate Longitudinal Approach to Educational Value-Added Assessment Without Imputation.” Paper presented at National Evaluation Institute, online at http://www.createconference.org/documents/archive/2004/Wright-NEI04.pdf.

Post a Comment

Data visualization can help state transportation leaders drive infrastructure improvements

Melissa Savage
Melissa Savage|May 2, 2013

In my last blog, I discussed the growing commitment among governors to infrastructure investment, and to coming up with innovative ways to find dollars to fund the needed improvements to the transportation network.

I heard it over and over again during the Transportation Research Board (TRB) Annual Meeting in January and again during the American Association of State and Highway Transportation Officials (AASHTO) Washington Briefing in February.  State DOTs are drowning in data.  They have data on traffic flow and volume, crashes, weather, road use, equipment and assets, freight, commercial vehicles, driver’s licensing and motor vehicle registrations, and now many include economic development, community and land use data, as well.

The question on many state DOT leaders minds is how to use the data in a way that not only helps them make day-to-day and long term operational decisions but also, as recommended by several panelists at the AASHTO meeting, to communicate with elected officials at the federal, state and local level as well as transportation advocacy groups and road users.

Data visualization is a term that keeps cropping up in the transportation world.  And at both the TRB and AASHTO meeting, panelists urged state DOT executives to find compelling ways to share their data visually to tell their story to core audiences including members of congress, US Department of Transportation staff and their customers.

And while simply showing their data is important, the real power resides in having the ability to:

  • interactively explore the data in minutes or even seconds
  • apply predictive and descriptive analytics to data of any size to spot previously unknown patterns identify key relationships
  • uncover insights that would otherwise stay hidden

Crash hotspots analysis via SAS Visual Analytics

State DOT IT department are busy places—frequently facing backlogs of requests for ad hoc reports that can take days to produce.  Data visualization can offer state DOT executives fast access to their data on mobile devices while they are meeting with their congressional delegation.  Users can explore data on their own and easily create new ways of looking at data and dynamically filtering and grouping variables which can help reveal insights that might spark further analysis.

Gone are the days of static PDF annual reports.  With data visualization, DOT leaders can provide governors and other state leaders with up-to-date, more accurate information that they need to follow through on their commitments to infrastructure improvement.

Post a Comment

Identity theft? Next step, government fraud

Carl Hammersburg
Carl Hammersburg|April 23, 2013

Sometimes, it is good to start with a confession.  I filed my taxes at the last minute. It was past time to get some money back from the IRS before they could waste $60,000 on another Star Trek spoof video.  Normally I'm one of those people that files in February, as soon as I have everything in hand.  Filing late gave me some concern, as it allowed a lot more time for someone else to "file" for me.  As in, steal my identity, file a bogus return for a huge refund, and mess up my life royally.

Why was I concerned this could happen?  Because according to a recent Javelin report, 12.6 million consumers had their identities stolen last year.  A shocking 1 in 4 notified of a data breach ended up being victims of identity theft.  Guess what, I was notified of a data breach by one of my credit cards late last year.  They had immediately canceled the card and issued one with another number, but that doesn't solve the cascade of effects.

A great ABC News article that ran last month highlighted a recent report from the Federal Trade Commission about the rapidly growing tide of complaints of identity theft.  In that FTC report, identity theft complaints made up 18% of all complaints, nearly double that of the next category, debt collection complaints.  Of the identity theft complaints, 43% of consumers reported that their identity was used for, guess what, TAX theft!  Either using the identity to gain a false tax refund, or working under your identity to stick you with the tax bill at the end of the year for wages you never earned.  Within Florida, the leading state for the identity theft crisis, 75% of the stolen identities were used to file false tax refunds or for government benefits, like welfare or unemployment.  Maybe I'm lucky I live in Washington State, about as far away from Florida as possible while still staying in the U.S.

What can be done about all of this?  Well, for consumers, start protecting your data and your identity as soon as possible.  If you have a breach, watch closely for months afterwards, as well as the next tax filing cycle.  For government agencies, it is past time to start doing something about this.  First step, break down data silos and start sharing data between agencies.  Match up birth and death records, other public records, SSN matches and filings between programs.  See if people really are who and where they say they are.  Then, add analytics.  Proper predictive models, anomaly detection and looking for networks of individuals will quickly show the rings of identity theft and false filings.  That will protect not only the government agencies and funds, but also protect the consumers and constituents living in your states.  We've already seen states like Kentucky, Michigan and Louisiana and others start down this path, utilizing tools like the SAS Fraud Framework with an enterprise view to protect themselves and their citizens.  While some of those are nascent projects, it is the right step.  What about the rest of the states and the Federal government, not to mention large cities and counties?  It's not too late to catch up.  Don't be last, or you may end up knocking Florida from its dubious perch.

 

Post a Comment

Busting myths of education value-added analysis, Part 3: Simple growth measures provide better information to educators.

Nadja Young
Nadja Young|April 15, 2013

Welcome to Part 3 of the value-added Myth Busters blog series. I have heard a variation of this many times.

“Why shouldn’t educators just use a simple gains approach or a pre- and post-test? They can trust simpler methodologies because they can replicate and understand them more easily.”

Simple growth measures might be sufficient if we were working with perfect data. However, student assessment data is far from perfect:

  • In a perfect world…
    • All students would start the year on grade level.
    • All students would progress at the same rate.
    • Student would never miss a standardized test.
    • Students would perform at peak levels on test day.
    • All large-scale achievement tests would be perfect measures of student attainment and would account for student progress.
  • But in the real world…
    • Not all students begin the year on grade level.
    • Not all students progress at the same pace.
    • Some students miss their standardized test and have missing data.
    • Student and teacher mobility exists within the school year.
    • Shared instructional practices exist, such as team teaching, push-in, pull-out, etc.
    • Tests are on differing scales, are not all vertically aligned, and change over time.
    • All tests contain measurement error and are just an estimate of what a student knows on that given day. Some may underperform on test day.

There is clearly some statistical rigor necessary to provide precise and reliable growth measures given the above analytical problems.  This is even more critically important in any reporting used for educator evaluations.

What is the downside to using more simplistic methodologies?

Growth estimates based on simple calculations are often correlated with the type of students served by the educators, rather than the educator’s effectiveness with those students. In other words, high-achieving students tend to show higher growth. Conversely, low-achieving students tend to show lower growth. This turns the growth model into more of a status model, which we already have by looking at achievement data alone. Such models often unfairly disadvantage educators serving low-achieving students and unfairly advantage educators serving high-achieving students. Empirical evidence from any growth model should be examined to see how strong a relationship exists.

The bottom line:

If we want growth and value-added models to level the playing field for all educators regardless of the students they serve, they must be rigorous enough to adequately account for students’ entering achievement levels and the various challenges associated with assessment data listed above.

Post a Comment

State leaders must take the wheel of transportation infrastructure reform

Melissa Savage
Melissa Savage|April 3, 2013

A few weeks ago I found myself in a room full of fellow transportation geeks (a term I use with great respect) at the annual American Association of State and Highway Transportation Officials (AASHTO) Washington Briefing.  One panel in particular really got the room buzzing with talk about the transportation fiscal cliff we are getting ready to drive over.

As car manufacturers continue to improve fuel efficiency (which is a good thing for the environment and my wallet, but a bad thing for the highway trust fund which derives revenue from gas taxes) and infrastructure funding remains a hostage to the Federal morass, transportation geeks are left wondering how to find the dollars needed to simply maintain what we have?  Furthermore, how can we tell the transportation story to elected officials and road users (tax payers) in a way that interests them and helps communicate the dire situation we face?

Financially strapped state departments of transportation can only put the brakes on the slow decline of the transportation system so much.  The money isn’t there to make major improvements and you can forget about building new roads.  All they can do is fill potholes, which is kind of like putting a Band-Aid on a severed limb.

The good news is that transportation funding issues were highlighted in the majority of speeches governors made to their state legislative bodies in early 2013.  The signals that state governments will most likely be the entities that drive the solution to the transportation funding challenge.

One example is in Virginia, where Governor Bob McDonnell proposed an aggressive legislative package designed to piece together funding sources to make needed infrastructure investments.  That Northern Virginia is one of the regions most crippled by a crumbling road system that cannot support growing demand was certainly a factor in Gov. McDonnell’s decision to move so aggressively.  The other reason is that the old way of doing business simply isn’t going to cut it.

State DOTs have recognized this for quite some time.  They have sought out innovative approaches to conducting day-to-day operations and long-term planning.  And as they face the future, they are open to looking at new ways of doing business.

As Governors make the move and commitment to make infrastructure improvements, they recognize that effective communication is a key to the success of the overall plan.  Any increases in taxes and fees, and how that new money will be spent, will need to be effectively communicated to the average road user, freight companies, and policymakers.  Transparency and accountability will be important.

One panelist urged state DOT leaders to use the tons of transportation data to tell their story and make their case. That will be the topic of my next post.

Post a Comment

Busting myths of education value-added analysis, Part 2: It is harder to show growth with high-achieving students

Nadja Young
Nadja Young|March 27, 2013

Welcome to Part 2 of the value-added Myth Busters blog series…have you heard this one before?

Educators serving high-achieving students are often concerned that their students’ entering achievement level makes it more difficult for them to show growth. “How can my students show growth if they are already earning high A’s on all of their assessments?”

Myth Busted:

With EVAAS, educators are neither advantaged nor disadvantaged by the type of students that they serve. The modeling reflects the philosophy that all students deserve to make appropriate academic progress each year. Students are compared against themselves and the amount of progress they have previously made across all tested grades and subjects.

Additionally, standardized assessment scales typically have enough “stretch” to assess measurable differences between students who score close to the top score. That is one of the base criteria SAS looks for before running any value-added analyses. If a test does not have enough “stretch,” SAS would not provide value-added estimates on the assessment. As such, EVAAS provides reliable and valid measures of growth for students, regardless of their achievement level.

As with Myth #1, actual data may be the most readily apparent evidence. The graph in Figure 1 plots the average entering achievement for each school in Tennessee against its growth index (the value-added estimate divided by its standard error) for the state math assessment in grades 4-8 in 2012. These data come from Tennessee’s public website.

FIGURE 1: GROWTH INDEX V. AVERAGE ACHIEVEMENT BY SCHOOL

Regardless of the school’s achievement level, there is essentially no correlation to the growth index. In other words, the dots representing each school do not trend up or down as achievement increases; the cluster of dots is fairly even across the achievement spectrum. Schools serving both very high-achieving (on right), or low-achieving (on left), populations of students are demonstrating great gains with their students. In fact, many of them are above the zero line, meaning they made more than the expected amount of growth with their students in 2012.

Post a Comment

Are Your Wages Killing You? Study Opens The Door For Workers Comp Fraud

Carl Hammersburg
Carl Hammersburg|March 20, 2013

Personally, I love studies.  They help put things into context, and when done well, provide an independent and hopefully unbiased view of the forces that shape our lives.  They are also a great way to see government funds used in strange ways.  For example, the new NIOSH (National Institute for Occupational Safety and Health) study linking hypertension (aka high blood pressure) and wage rates.  The study, published in the European Journal of Medicine, covered 5,651 employed persons in the U.S. between 1999 and 2005.   Based on the results, it estimates that just a five percent increase in inflation-adjusted wages for 110 million employed workers between 25 and 65 would result in 66,000 fewer cases of hypertension a year.  Doubling wages brings a 25-35 percent drop in hypertension in younger workers, and for women, a 25-30 percent decrease.

The upshot of all of this?  If you are diagnosed with hypertension, which can lead to more serious heart disease and death in some cases, there is now a perfect link to file a workers' compensation claim, blaming it all on your job.  As a job-related condition, you are eligible for medical coverage, time loss, and potentially a pension.  Separating the work impacts from diet, other stress in life, smoking and weight is possible, but difficult, and the laws of states vary on how much the work contribution needs to be.  What an amazing open door for fraud!  All you need to do is drink a bunch of coffee before heading into the doctor, show that high number on the blood pressure, and ta dah, there's your claim!  For some people, just going to the doctor spikes their blood pressure, which the medical industry calls "white coat hypertension".  Even easier to fake.

As more and more conditions that are really just side-effects of life become diagnosed as diseases, and better yet, have links to employment, the door for fraud for private health insurerers, Medicaid and Medicare, and workers' comp insurers and providers is flung wide open. They are exposed to skyroketing costs and higher thresholds to provide fraud, waste or abuse.  The best defense is to put strong systems in place like the SAS Fraud Framework that take a comprehensive view of individuals from many perspectives, including anomalies about their behavior and condition, predictive models based on past cases, and connections through social link analysis to others.  That will help identify connections to past fraud or abuse they have committed, connections to other bad claims and claimants, as well as suspect medical providers.  Knowing the risks doesn't solve everything, but it helps target investigative and audit resources, and gives ammunition to convince would-be fraudsters to shut down claims early.

Come to think of it, I've been diagnosed with hypertension for years now.  Maybe it was 21 years of low government wages.  Time to file a workers' comp claim?  No, maybe I'll just ask my boss for a raise.

Post a Comment

Busting myths of education value-added analysis, Part 1: You must control for demographics

Nadja Young
Nadja Young|March 13, 2013

In the past five years, value-added models have been increasingly adopted by states to support various teaching effectiveness policies. As educators make the paradigm shift from looking at only achievement data to incorporating growth data, many misconceptions have developed. Compounding this issue is the fact that not all value-added and growth models are alike. What may be true for a simplistic model, may be a myth for a more statistically robust methodology. Therefore, I'm launching a series of blog posts that will serve as “myth busters” for common value-added misconceptions, based on SAS’ 20+ years of experience working with education value-added analyses.

Tell me what additional myths you are hearing and I’ll try to address them in future posts!

Myth #1:

Student growth is correlated with certain demographic variables, so value-added models should control for demographics.

It is widely known that students with certain socioeconomic or demographic (SES/DEM) characteristics tend to score lower, on average, than students with other SES/DEM characteristics. There is concern that educators serving those students could be systematically disadvantaged in the modeling.

Myth busted:

It is understandable that this myth has persisted because most educators know the impacts that poverty can have on student achievement, as seen in Figure 1, which contains publicly available data from Tennessee’s TVAAS website.

FIGURE 1: ENTERING ACHIEVEMENT V. PERCENT TESTED ECONOMICALLY DISADVANTAGED BY SCHOOL

To combat this issue, some value-added models take the “kitchen sink” approach and try to control for every student characteristic possible in order to level the playing field. However, these adjustments are not statistically necessary for the most sophisticated value-added models, such as those used by SAS EVAAS.  Because the models used by SAS incorporate more student assessment data across all grades and subjects, each student serves as his or her own control. While a picture is worth a thousand words, actual data may be the most readily apparent evidence.

The graph in Figure 2 plots the percentage of tested students who are considered economically disadvantaged at each school in the State of Tennessee against the school’s growth index (the value-added estimate divided by its standard error) for the state math assessment in grades four through eight.  Figure 3 provides similar information for the percentage of minority students.  Regardless of the school’s student characteristics, there is essentially no correlation to the growth index. In other words, the dots representing each school do not trend up or down as the percentage increases; the cluster of dots is fairly even across the spectrum.

FIGURE 2: GROWTH INDEX V. PERCENT TESTED ECONOMICALLY DISADVANTAGED BY SCHOOL

 FIGURE 3: GROWTH INDEX V. PERCENT TESTED MINORITY BY SCHOOL

Adjustments for student characteristics may be important with any model where you do not see such a level playing field, or from a communications standpoint. However, with EVAAS’ value-added models, no teacher is advantaged or disadvantaged by the types of students they serve because all students (rich, poor, Caucasian, or minority) can demonstrate growth.

 

Post a Comment

Michigan’s holistic view of fraud is what’s required to combat organized criminal networks

Greg Henderson
Greg Henderson|March 4, 2013

SAS announced yesterday that Michigan will use the SAS Fraud Framework for Government to, initially, combat fraud, waste and abuse in the state’s unemployment insurance and food stamp programs. Those two programs are good focus areas and I’m confident they will lead to the state recovering funds, avoiding losses and saving tax payer money. Even better, the enterprise system they are building allows data from other agencies to be added over time to tackle fraud in other areas. This data sharing will lead to visibility across programs. Organized fraudsters are coordinated and adaptable, and take a more sophisticated, enterprise approach to defrauding government programs than many states do protecting them.

I wrote in a recent Government Security News article:

“Fraud rings, organized criminal networks and collusion are now more pervasive in government, and take a holistic view across programs. Professional fraud rings are not just exploiting one government program, or even one at a time. They are very creative in exploiting the gaps across the spectrum.

Thwarting these criminals requires an enterprise approach, where departments and agencies share a common technology layer with common investigative capabilities. So, if I’m investigating a public assistance program provider for both claims fraud and tax evasion, the investigation can be coordinated between the two departments. For example, one customer I talked to recently manually cross-matched tax fraud investigation information with Medicaid investigations and found an 85 percent overlap. The fact is bad people do bad things across the spectrum.”

Michigan is one of several states embracing a holistic, statewide approach to fighting fraud, waste and abuse. I hope to be able to share more examples soon. In the meantime, I am always on the lookout for examples of fraud fighting across programs and agencies. Please share examples in the comments section.

Post a Comment

If you build it, they will get stuck in traffic

Melissa Savage
Melissa Savage|February 26, 2013

I’m a huge movie buff.  I love all kinds of movies but sports films are at the top of my list. Field of Dreams is one of my favorites not only because it’s a great story but also because, in my opinion, it has one of the best movie lines.

Spoiler alert! Just kidding, you’ve had 24 years to see the movie and it’s one of the most famous film quotes of all time.

“If you build it, he will come.”  And so begins Ray Kinsella’s journey of plowing his corn to build a baseball field so that, ultimately, he can have an otherworldly reunion with his long-deceased father, a former baseball player.

One of the other reasons I like that line so much is that it relates to my work with transportation analytics.  That saying holds true in the transportation world, with one small change, “If you build it, they will come.”  In droves. This fact stirs concerns in the court of public opinion before and during any major development project.  It is particularly relevant to me as the county where I live plans to down a forest to build a baseball park.

"It's great to see you too, son, but the parking here is terrible."

The quote has been getting quite a bit of play on my local news. As the developer moves forward with plans for mixed use housing, a movie theater, the baseball field and an outdoor mall, nearby residents of the area surrounding the new minor league ballpark believe that, while the development will surely bring economic development and other benefits, it will also bring an increase in traffic that the current infrastructure simply can’t support. Residents are worried that the capacity just isn’t there and once the development opens, people living in and around the development will be stuck in traffic morning, noon, and night. (You might say they’re concerned too many people will “go the distance” to visit. Yeah, that’s another Field of Dreams reference.)

That’s a real fear in Northern Virginia where traffic is already bad.  Planners and developers believe they have done their due diligence, but residents in this area have heard those promises before.  And while no one has a crystal ball, they do have data about the area including demographics, road sensors, employment and taxes.  By using transportation analytics and models, developers and planners could make forecasts and decisions with the best possible information, removing much of the guess work.

Analytics can be a powerful tool whether you’re trying to decide if you should spend millions of dollars to build a new large scale development, or you’re considering whether or not to plow under your corn to have a catch with your father’s ghost…or members of the 1919 Black Sox.

Post a Comment