Sample blog post: Briggs and Domingue review of teacher effectiveness study

Los Angeles Times teacher effectiveness study deeply flawed

A widely reported analysis done of teachers in the Los Angeles Unified School District is a deeply flawed measure of teacher effectiveness, according to a review released today.

On Aug. 14, 2010, the Los Angeles Times published a controversial article that highlighted the results of a study conducted by Richard Buddin, senior economist at the RAND Corporation. Buddin's analysis, "How Effective Are Los Angeles Elementary Teachers and Schools?" erroneously asserted that teacher quality can be validly measured using student performance on the math and reading portions of the California Standardized Test. The L.A. Times also published a database of LAUSD teachers by name and their individual effectiveness ratings, which were based on Buddin's flawed analysis.

A review of Buddin's analysis was performed by Derek C. Briggs, chair of the Research and Evaluation Methodology Program at the University of Colorado at Boulder, and Colorado-Boulder doctoral student Ben Domingue. Key among their findings was that Buddin's approach, known as "value-added modeling," failed to consider several critical factors that impact student achievement, such as long-term achievement patterns and influence from their peers.

Buddin's model leads to overly broad and simplistic generalizations about teacher effectiveness, often resulting in negative consequences for many potentially effective teachers and their students, Briggs and Domingue found. Buddin's value-added approach can lead to a number of false positives and negatives - many teachers may have been falsely categorized as either effective or ineffective.

Briggs and Domingue developed a stronger alternative model that included students' long-term achievement patterns and peer influence. The researchers then compared the results of their model and Buddin's, and found that only 60.8 percent of teachers would retain the same effectiveness rating in both models for math outcomes, while only 46.4 percent of teachers would retain the same effectiveness rating for reading outcomes.

Notably, Briggs and Domingue found that Buddin's model could not be independently replicated, casting serious doubts about the validity of the entire analysis. Ensuring research can be fully replicated, which Buddin's could not be, is one of the very foundations of the scientific research model.

The New York City Board of Education is planning to follow Los Angeles' lead and publish ratings of 12,000 New York City teachers, based on an approach similar to Buddin's.

It would be highly irresponsible for the New York City Board of Education to publish flawed teacher rankings based on faulty research.

It's important to find valid, complete ways of measuring teacher quality that are based on more than just standardized test results. Test scores are a mere snapshot of student achievement; they should not be the primary measure of teacher effectiveness, and they certainly shouldn't be used to rank teachers in such a public manner.