Value-added data informing and improving TN Schools of Education

Nadja Young
Nadja Young|May 8, 2012

With the rapid changes in our education systems regarding new standards, assessments, accountability and evaluation, teachers are rightfully feeling the pressure of being underprepared. The majority of teachers were not trained or certified with these rigorous systems in place. Recognizing that higher education institutions need to play an active role in the continuous improvement process, President Obama released a reauthorization plan for the Higher Education Act with a specific focus on reforming teacher preparation programs. This plan highlighted Tennessee and Louisiana as national leaders on this front, with North Carolina, New York and a handful of other states following suit.

I recently spent some time talking to the TN Higher Education Commission (THEC) to learn exactly what they are doing to lead the pack. THEC collects K-12 value-added data to evaluate teacher preparation program effectiveness. While TN’s Report Card on the Effectiveness of Teacher Training Programs is a model for other states to follow, they don’t stop there. Tennessee’s higher ed institutions also educate pre-service teachers and administrators on value-added data literacy to improve student and school outcomes. Pre-service teachers are teachers in undergraduate, training, or alternative certification programs who will soon join the ranks of teachers in the classroom. So TN takes a two-pronged approach with accountability and support.

Accountability: Emily Carter, THEC’s Higher Education Program Coordinator, uses teacher effect scores to evaluate how well different programs are preparing teachers, and identify areas for improvement. After collecting all of the information (academic background, placement and retention, etc.) on completers for a certain cohort year, SAS then provides a value-added score for each teacher preparation program. Who were TN’s top performers in 2011?  Three teacher prep programs were actually able to produce teachers with higher student achievement gains than veteran teachers – Teach for America Memphis, Teach for America Nashville, and Lipscomb University. Read more in the 2011 Report Card.

Carter also discussed planned future improvements to provide more detailed information that is disaggregated for each program (math vs. English/language arts vs. science, etc.). Additionally, TN will develop new growth measures for teachers in traditionally non-tested grades and subjects to soon be incorporated into the report card. Given more data, there are additional research opportunities using these results. For example, is a pre-service teacher’s SAT/ACT score a reliable predictor of their future effectiveness? Does their college GPA or choice of major impact their teaching effectiveness?

Support: If a goal of the Report Card is to improve the services offered to future educators, then additional support is critical. Katrina Miller directs THEC’s First to the Top office managing several programs to improve the teacher pipeline in higher education institutions. Katrina worked with SAS to develop modular data literacy curriculum that is integrated into the pre-service curriculum. This eight-hour package of modules teaches future educators and leaders how to use value-added data to differentiate instruction in order to meet the needs of all students.

Dr. Deborah Boyd, a Professor of Education and Associate Dean for Lipscomb University’s College of Education, currently uses this curriculum and expands upon it. The college's coursework in research discusses the different types of data that teachers will receive about their students, instruction, and their own practice. The college also incorporates value-added reports into case study assessments for graduate students. Perhaps this is a contributing factor to Lipscomb’s superior preparation of teachers, as reflected in the Report Card.

What is your state doing to measure the effectiveness of teacher prep programs and to support them in producing 21st Century educators?

Post a Comment

Four E's of crash analytics, part 2: Enforcement

Melissa Savage
Melissa Savage|April 30, 2012

Last week I introduced this series. Today I begin to dive deeper into each of the four E's. First up? Enforcement.

In 2009, there were 5.5 million police-reported traffic crashes.  Law enforcement officers work diligently to prevent crashes by enforcing traffic safety laws pertaining to, among other things, seat belt use, child passenger protection, speeding, driving while impaired and distracted driving. Studies have shown that increased enforcement and educational campaigns can yield big results in terms of changing driver behavior.  One example, “Click it Or Ticket”, has helped achieve a nationwide seatbelt use of 85 percent—saving an estimated 72,000 lives between 2005 and 2009 alone.

National Highway Traffic Safety Administration adheres to five essential components of traffic safety law enforcement, which break down roughly like this. There must be traffic laws and guidelines for enforcing them, law enforcement officers must respond to violations and issue necessary citations, which must be adjudicated and appropriate sanctions handed out.

Law enforcement officers are usually the first to arrive at a crash scene of a crash, and are responsible for capturing important data, including:

  • Weather and pavement conditions at time of crash
  • When the crash happened
  • Violations committed
  • Fatality and injury information
  • Complete information about all vehicles involved in the crash
  • Driver information, including driver’s license information, license status and conviction history
  • Other crash scene information, including whether it happened at an intersection and what the traffic volume was like at the time of the crash.
  • Information about any commercial vehicles involved, the driver and their load

The data is typically housed in the state crash database and used to report state-specific crash information to the federal government, help make resource allocation decisions and prioritize traffic safety programs. By applying analytics to that data, state DOTs, public safety agencies and traffic safety offices can make proactive resource and funding decisions about education and enforcement campaigns that will yield the greatest return on investment.

Additionally, the data can help traffic safety agencies make decisions on how to best use each of the four E’s. For example, if an analysis of the data shows an increase in alcohol-related crashes at a particular intersection, police can optimize resources by stepping up DUI enforcement in that area.  Another challenge facing law enforcement is identifying repeat offenders and unlicensed drivers. Applying analytics to traffic citations, driver history and motor vehicle title and registration data would provide law enforcement officers with the ability to identify high risk drivers and target their enforcement efforts.

Post a Comment

The four E's of crash analytics: Part 1

Melissa Savage
Melissa Savage|April 19, 2012

The leading cause of death for Americans between 5 and 34 isn’t what most would expect.  It’s not disease or acute illness. It’s traffic crashes. The good news is that the number of highway deaths has steadily decreased the last few years, yet crashes still cause more than 30,000 fatalities and send two million people to the emergency room each year.

Not only are the deaths and injuries resulting from these crashes devastating to those involved, the societal and economic costs can be expensive. AAA estimates that crashes cost society $299.5 billion annually and believes that an investment in data collection and analytics is imperative to ensure that resources are spent in a way that yields the greatest possible results. A study conducted by the University of Michigan Transportation Research Institute shows that in Michigan alone the total societal costs associated with motor vehicle crashes reached $9.1 billion in 2009—outpacing societal costs for all other types of crime combined.

In an effort to combat these high costs, deaths and injuries, state departments of transportation (DOT) work closely with law enforcement agencies, state traffic safety offices and the National Highway Traffic Safety Administration (NHTSA) to plan and implement policies that can help reduce the number of motor vehicle crashes. One approach is through the Four E’s of traffic safety—Enforcement, Engineering, Education and Emergency Medical Services.

Over the past few years, the Four E’s approach has contributed to a steady decline in both fatality and injury rates. While this decline in traffic crashes and the resulting deaths and injuries is good news, NHTSA, law enforcement, state DOTs and state highway safety offices are not resting on their laurels—they continue to work toward keeping the roads as safe as possible.

The ultimate goal these groups are striving toward is ambitious, but they believe achievable. Toward Zero Deaths (TZD) is a data-driven highway safety strategy that focuses on changing the driving culture, with the goal of no deaths on the nation’s highways. Statistically, that means to get the fatality rate per vehicle miles traveled (VMT) to zero. Today, the rate is 1.14 fatalities per 1 million VMT. The TZD initiative relies on data from crashes and police stops, in concert with the four E’s, to determine priority areas and make changes to policies and programs.

The Four E’s play an important part in road safety. Each component is essential and when taken together as a unified approach, has had great success helping to achieve the lowest crash rate in decades. By addressing the four components in a holistic way, NHTSA, state DOTs, law enforcement and state traffic safety offices hope that they can prevent crashes from happening in the first place. Those groups are exploring how technology can improve and transform the way traffic safety advocates, traffic safety engineers and other key stakeholders use the Four E’s.

Data from the Four E’s can include:

  • Vehicle speed
  • Traffic volume at the time of the crash,
  • Law enforcement crash investigation information
  • Emergency medical response information
  • Road sensor and design data
  • Effectiveness of public education campaigns

This can be analyzed holistically to help decision makers create strategies for comprehensive traffic safety improvement plans. Unfortunately, the data often resides in different agencies, in various databases, formats, and types of hardware. Integrating the data and creating a holistic view of traffic safety would allow for a coordinated approach to preventing crashes. Furthermore, analytics gives road designers, law enforcement officers, emergency medical responders and those designing public education campaigns the ability to identify trends and develop highways safety plans and interventions that will have the best return on investment in terms of reducing the crash rate.

I will dive deeper into each of the four E's in future posts. Stay tuned!

Post a Comment

More than “teaching to the test”: Value-added ROI persists throughout a student’s life

Nadja Young
Nadja Young|March 30, 2012

A 23-year Harvard and Columbia University study was recently published shedding new light on the long-term impacts of teachers with both high and low value-added estimates. Researchers Chetty, Friedman, and Rockoff tracked math and reading assessment data on over 2.5 million students from 1989-2009. They then incorporated 90% of these students’ tax records from 1996-2010 to analyze the long-term outcomes highlighted below. For anyone interested in the return on investment for a high value-added (VA) teacher, you now have it expressed in student outcomes and life opportunities:

College Attendance
“Being assigned to a higher VA teacher in a single grade raises a student’s probability of attending college significantly,” through age 25. At age 20, it is relative to a mean of 37.8%. (pg. 36-37 of the report) Additionally, “there is a highly significant relationship between the quality of colleges students attend and the quality of teachers they had in grades 4-8.”
Earnings
“Being assigned to a higher value-added teacher has a clear, statistically significant impact on earnings.” (Pg. 39) Replacing a poor teacher with an average one would raise a single classroom’s lifetime earnings by about $267,000, the economists estimate. (pg. 48) Additionally, “increases in teacher VA raise earnings even in subgroups that are relatively unlikely to attend college, suggesting that better teaching has direct returns in the labor market independent of its effects on college attendance.” (pg. 40)
Neighborhoods
Having a 1 SD increase in teacher VA raises neighborhood quality by 0.063 percentage points (0.5% of the mean), measured by the percent of college graduates living in that neighborhood. “The impact on neighborhood quality more than doubles at age 28, consistent with the growing earning impacts documented above.” (pg. 42)
Parenthood
Having a 1 SD higher VA teacher in a single year from grades 4-8 reduces the probability of a teen birth by roughly 1.25%. (pg. 41)
Retirement Savings
Mixed results were consistent with the college attendance and earnings findings: “In schools with low college attendance rates, students who have high VA teachers, find better jobs by age 25 and are more likely to start saving in 401(k)’s. In schools with high college attendance rates, students with high VA teachers are more likely to be in college at age 25 and thus may not obtain a job in which they begin saving for retirement until they are older.” (pg. 42)

Critics of value-added analysis say that it places too much focus on standardized testing, since assessment data is used as an input for the models. A concern is that this will motivate “teaching to the test” and lead to a narrowing of the curriculum.

First, we need to define what “teaching to the test” really means. If it means teaching to all of the standards and objectives in the curriculum, which reflect what we want students to learn in preparation for a final exam that measures their proficiency and readiness to move on to the next academic level, then in my book that is simply called “teaching.” If it means a majority of instructional time is spent on “drill and kill” review sessions for the final exam, then that is cause for concern due to the reduction of engagement, inquiry-based, or project-based learning.

Second, this argument highlights a misconception of the way value-added analysis works. In reality, while teaching to the test might have a positive impact on student achievement, it might actually have a negative impact on value-added estimates, which measure progress. (See my October blog to better understand the difference between measuring achievement and progress.)

In value-added analysis, all students count, regardless of their achievement level.  By teaching a narrowed curriculum, very low achieving as well as very high achieving students will have limited opportunities to make appropriate academic growth.  In essence, the teacher or school is less likely to be highly effective from a value-added perspective.

The above mentioned research study gives me the feeling that the high value-added teachers did much more than “teach to the test” to yield the magnitude of positive outcomes described. You don’t reduce teenage pregnancy while increasing college attendance, earnings, and retirement savings with your students by merely “drilling and killing” with test review. Those teachers most likely engaged their students to create inquisitive lifelong learners, and incorporated college and career exploration into their discussions and assignments.

I think that a great follow-up study would be to interview those high VA teachers to find common instructional practices that contributed to their success. In the meantime, at least we have some empirical evidence showing the importance of measuring teaching effectiveness with value-added analysis. It can help all teachers improve and contribute to similar outcomes and opportunities for their students.

 

Post a Comment

Promises, (over)promises

Vincent Talucci
Vincent Talucci|March 23, 2012

As public safety officials leaf through their favorite criminal justice periodical they are greeted with pages and pages of analytics advertisements. These ads are laden with promises of robust and scalable solutions, improved efficiencies and, yes, the promise of prediction.

While reading the advertisements, the mental conversation may go something like this:

  • “Wait, they can help my agency forecast crime and crash occurrence? That’s terribly exciting!” (And accurate.)
  • “Let me get this straight – based on forecasts, we can implement responses to improve future outcomes? That’s even more exciting!” (And this too is accurate).
  • “So if we combine forecasting along with building appropriate responses – that’s a powerful combination! And they can base the predictions on something other than existing or historical data? That is unbelievable!” (Indeed it is.)

These bullet points (though wrought with faux excitement and blatant overuse of exclamation points) parallel actual conversations. While the first two points have legitimate merit, the third bullet is the overpromise that should cause wariness. However, wariness should not impede a public safety leader’s strive for innovation – it should prepare you well to embrace or debunk the value, practicality and reality of the next great thing.

Ask the questions -- does the next great thing fit into my agency’s:

  • Strategic plan?
  • Budget?
  • Acceptance by the community?
  • Needed return on investment?
  • Existing IT infrastructure?
  • (Fill in the blank)?

I believe analytics is the next great thing for public safety – but it is certainly not an overnight panacea for all things ailing your agency or community. Implementing advanced or predictive analytics will reap benefits but will take work, time, understanding and above all else, data.

So with the ever changing landscape of shiny boxes that promise criminal justice utopia, like forecasting despite a dearth of data, I say “Bravo!” Oh, and that is immediately followed by a hearty “Prove it.”

When looking to assess or acquire the next great thing, talk to trusted, practical sources. Be skeptical and ask the tough questions that make your prospective solution provider think deeply, and maybe get a bit uncomfortable. Such an approach will help you separate the promise from overpromise.

Post a Comment

Fallen interstate sign shows importance of asset management

Melissa Savage
Melissa Savage|March 15, 2012

A few weeks ago, in Northern Virginia, a 30 foot highway sign fell onto I-66 and landed on a passing pickup truck.  Fortunately, no one was hurt, but it drew media attention and caused motorists in the area to wonder about the safety of other signs and the transportation network in general. 

Wrong kind of sign from above? (Courtesy of VDOT)

According to Virginia Department of Transportation officials, there are more than 260 similar signs hanging over roads throughout Northern Virginia.  VDOT mobilized crews immediately following the incident but 10 days after the sign fell, only about 25 of the signs had been inspected and three (with cracked bolts) had been removed.  According to VDOT, signs like the one that fell are inspected every five years—the one that fell was last inspected in 2007.  

State DOT inspectors across the country have a big job, with serious safety implications if they miss something.  Not only do they have to worry about the signs hanging from overpasses and along highways, but also about the overall safety and reliability of bridges and roads. 

To say this workload is overwhelming is an understatement—just consider the 260 signs in Northern Virginia that need to be inspected and those are just the signs similar to the one that fell. It took ten days to inspect 25 signs—VDOT hopes to speed up the process by adding workers to the crew but at the rate they are going it could take months to complete the work.

What if state DOTs were able to take a more proactive approach to these inspections?  Instead of being placed in a reactionary mode and working as fast as possible to find the faulty signs, what if the inspectors knew which signs needed immediate attention and could direct work crews to conduct those repairs first?

Having the ability to take a proactive approach with maintenance and asset management through predictive analytics, state DOT inspectors would know which bridges, signs and roads needed repair first and could prioritize the dispatch of work crews accordingly.  This would allow them to prioritize not only daily work, but also long range transportation planning. By collecting data on all aspects of each asset like height, weight, type of materials, condition of joints, bolts and reflective material for signs and volume for roads and bridges, state DOTs could apply data analytics to determine and target maintenance efforts more effectively and efficiently.

As my colleague, Bill Coleman blogged about a few months ago, when government agencies are able to manage their assets (roads, bridges, signs, and equipment) by using predictive analytics, they can keep public confidence high and more importantly, in the case of the transportation system, keep it moving safely and reliably.

Post a Comment

SAS Curriculum Pathways and the path to Common Core success

Nadja Young
Nadja Young|February 20, 2012

Teachers have more than enough to juggle each day, lacking the time to search for, and find, high-quality curricular resources online. When I would search for lesson plan supplements, I would often get lost in Google's abyss of results, spending far too much precious time sifting through mediocre materials. Until one day, I saw the light. Mrs. Angie Stephenson, a neighboring English teacher, brought her freshmen into my classroom to use my computer lab for SAS Curriculum Pathways. She had her class upload their essays into the Writing Reviser tool allowing SAS' artificial intelligence to identify possible errors. Students then self-edited their work by making their own determination of how to improve their run-on sentences, fragments, dangling modifiers, verb tense issues, relative and dependent clauses, and more.

SAS Curriculum Pathways Writing Reviser

In the era of increased class sizes and the need to have students practice writing constantly to improve their proficiency, SAS Curriculum Pathways is the answer. What’s even better? This $75 million philanthropic effort of SAS is offered at no cost to educators!

I discovered this resource in my seventh year of teaching, and I truly hope that more educators explore it now in preparation for the Common Core State Standards (CCSS).  SAS Curriculum Pathways provides more than 200 Interactive Tools, 200 “read, research and respond” inquiries, 600 Web Lessons and 70 Audio Tutorials in the areas of Math, ELA, Social Studies, Science, and Spanish. These resources are mapped to each state’s standards; English language arts and math resources also map to the CCSS. Curriculum Pathways has won numerous recent awards for these advancements, including ones from eSchool News, District Administration and Technology & Learning magazines.

SAS announced today record registrations in 2011. Over 18,000 schools nationwide now take advantage of this powerful free resource. However, some schools combine Curriculum Pathways with other SAS educational tools, such as EVAAS, to fully leverage student data and individualize learning opportunities for all. See how one high school in Granville County, NC, has increased graduation rates over 20% in two years, while decreasing absences and discipline incidents. Teachers first used EVAAS’ individual student projections to identify which students needed to be challenged, or to be retaught concepts in new ways. They then used Curriculum Pathways to differentiate instructional activities based on specific student needs. NC’s Mooresville Graded School District even caught the attention of the NY Times with their successful 1:1 initiative. By combining EVAAS and Curriculum Pathways, they were able to maximize the use of their laptops with this technology-rich content. The engaging lessons capture students’ attention to more deeply explore challenging material. Spread the word and share this free resource with your education colleagues!

Post a Comment

Analytics helping transportation officials get the job done in tight financial times

Melissa Savage
Melissa Savage|February 9, 2012

The American Association of State Highway Transportation Officials recently released a top 10 list of transportation issues that will be “talked, written or tweeted and legislated about” in 2012.  

As expected, funding constraints and Congressional action on reauthorization appear on the list but the group also notes that natural disaster preparation, senior mobility, increasing intercity rail, traffic safety and meeting environmental regulations will be hot topics in 2012.

Number two on the list is getting the job done in a tight financial environment.  With reauthorization lagging at the federal level and many state budgets in bad shape, state Departments of Transportation have had to look for ways to conduct day-to-day operations and long-term planning efficiently.

In North Carolina, by using data analytics, the state DOT has been able to shave weeks—about 20 percent—off the overall planning timeline, translating into an estimated cost savings of $500,000. Working in partnership with the North Carolina Department of Water Quality, the NC DOT used LIDAR (a process using laser to chart topography) data to narrow the list of acceptable choices for new road construction from hundreds of options down to a handful.  This huge reduction allowed NC DOT to quickly determine the best possible options and focus planning efforts on those areas.  

Analysis showing predicted streams and wetlands within proposed road corridors

The pilot project earned an award from the Federal Highway Administration and NC DOT has been fielding requests for more information from the US Army Corp of Engineers, the US Environmental Protection Agency and state and local transportation agencies. As state DOTs continue to look for ways to do more with less, data analytics can yield significant time and money savings, keeping projects on time and under budget.

Post a Comment

Predictive policing is passé

Vincent Talucci
Vincent Talucci|February 2, 2012

My colleague, Steve Serrao, just published a blog post on the proliferation of varied law enforcement approaches and their related phraseologies. After reading, I concur – hence, this blog’s title. I am not a huge fan of the term “predictive policing”. While others may love it and live it, and I absolutely respect that, I have tried to remove it from my policing lexicon. (Wait – I used passé and lexicon in the same blog – that’s terrible).

One reason I dislike the term?  Given I work with SAS – a leader in the use of analytics – on law enforcement topics, anytime anyone with whom I may be remotely affiliated with stumbles upon an article, blog, or Facebook post about predictive policing, it is matter of time before I am inundated with the forwarded article, blog, or Facebook post. While I appreciate information sharing, I get the concept!

And, no, I have never seen “Minority Report.”

So with the above as context, I do get the concept – I have been doing policing stuff for a while and have daily discussions with law enforcement thought leaders who really get policing stuff. We discuss policing at the policy level and operational level; we look at its strategic value and its tactical value; and, more importantly, we look at how policies, operations, strategies and tactics can best serve their respective constituents and communities- (and as an aside- think of the possibilities when all this is synchronized!). While we talk about a lot of important policing stuff, one thing we really never focus on – policing naming conventions.

This is not inconsistent with (forgetting my day job for a second) how I think and feel as a citizen. For example, as a citizen I want to know that my kids are as safe as possible walking to school; I want to know that speeding on the main thoroughfare is being addressed; and, I want to know my police department is doing everything possible to solve the spate of burglaries in my neighborhood. Never once as a citizen have I thought, “Wow, I really wish our police employed an intelligence-led (or predictive or community or neighborhood or information-led) policing approach.”

While as a professional, I understand and embrace the nuances, differences and complementary aspects of varying policing styles; however, as a citizen, those terms matter very little to me – and my guess is that they matter very little to most citizens. What is front of mind for citizens? My kids are safe, speeding is down, the burglar has been arrested, and if I call for help, help will arrive.

While I absolutely get the predictive policing concept, I also believe the term in and of itself sets an unreasonable (and sometimes unreachable) goal for many agencies – predicting where that next crime will occur. While I understand that is not the actual message, not fully understanding or articulating the concept and its limitations could lead to confusion and put the police and community at odds when a crime does occur.

While I have difficulty with the term, I believe that predictive analytics can help law enforcement in measurable and meaningful ways by improving service delivery and maximizing their abilities to do an already difficult job. Regardless of an agency’s policing strategy, or what it is called, predictive analytics has much to offer in helping police better protect and serve.

 

Post a Comment

Local government impact of rising group medical and insurance costs, and what can be done

Bill Coleman
Bill Coleman|January 24, 2012

The single most costly employee benefit for any organization is health insurance, and the price is going up. From 2003-2009, the costs per hour worked for employee health insurance increased from $1.03 to $2.00. These costs continue to increase from 5%-7% per year. The reality is that employee health insurance costs will continue to be a drag on operating  budgets that are already stretched too thin.

Cities and counties are service provider organizations whose budgets are heavily dependent on head count. The greater these benefit costs, the fewer the personnel. This could result in lower service levels in important areas like public safety, environmental protection, safety inspections, and infrastructure maintenance, to name just a few.

Many human resource organizations have made an effort to control these costs. They've implemented programs to improve the health of their employees and/or identify life style choices that reduce the individual’s risk of developing or worsening already existing problems.  Examples include wellness, health screening and consumer driven health programs, cash incentives and others. These programs require separate funding which further adds to the cost of this important benefit. The question is,  how does the organization determine if these programs are having the desired effect on health benefit costs? Many organizations use group health insurance premium costs as the sole measure for program effectiveness.

One must go several steps further to truly measure cause and effect of these programs on medical and insurance costs. HR organizations need to track things like:

  • Employee and dependent participation in the program
  • Employee demographics
  • Changes in behavior/life style
  • Medical costs over time
  • Number and types of workers compensation claims
  • Safety classes attended
  • Group health insurance claims, submitted by type and amount
  • Cost of prescription drug claims and the relation to claimant history
  • Type of work done by the employee/dependents
  • Health insurance premium costs and other relevant data.

This information needs to be tracked and analyzed over time, and related to past claims history of all employees/dependents covered by any plan. It is important that this same analysis be done for those not participating in the various incentive programs as a control group.

The key to analyzing the immense amount of relevant data to answer performance questions is having state of the art data integration, data quality, business intelligence and predictive analytics tools. With those tools, HR can easily tell which programs or combination of programs are helping control or reduce medical and insurance costs, including workers compensation costs, and which are not. It can predict the likely future costs given the employee history and demographics. Programs can then be developed to control costs in advance. The strategies for cost control are limitless if one can understand the relevant data and the interrelationships among the various data elements.

Post a Comment