Skip to main content

Welcome, the Hub connects all projects

Voices From The Field

MSPnet Blog: “What value is added? VAM under review”

See All Blog Posts

posted April 23, 2015 – by Brian Drayton

It’s no longer news that actual expertise doesn’t count for much in policy debates, except under special circumstances. The “climate change debate” in this country is much on my mind these days — when people who cheerfully proclaim their lack of credentials have no qualms about rejecting the advice of people who are deeply knowledgeable about the subject. Evolution is another one.
But education policy provides a lot of great (read: upsetting/alarming) examples, too. Many of them have to do with evaluating and judging schools, teachers, or students. The enthusiasm for “growth models,” or “value-added” models as the basis for judging teacher quality, brings together a cornucopia of unfounded social-engineering fads. The Educational Researcher for March 2015 is a special issue dedicated to reviewing the state of knowledge about VAMs, and I recommend it as a thought-provoking crash course.

The evidence is that VAMs at present are best seen as possibly-helpful adjuncts to more reliable and insightful ways of answering questions about teacher and school quality.  There are so many unsupported assumptions built into even the best of these systems that it is astonishing that anyone wants to use them to make important decisions.  Many serious people, however, do want to use them, and some of them are in a position to make it happen, regardless of the counter-evidence.  Wonder why?

As I worked through this set of articles, I found myself noticing some turns of phrase that help provide some insight into the thinking of people for whom VAMs (and some other policy tools) remain attractive despite the crippling weaknesses.  Add them to your Menagerie of Untamed Assumptions:

1.  The “last one standing is the winner” logical fallacy.  More than one of the authors in this collection, after doing a workmanlike job of analyzing some issues with VAMs, end by saying something like “But even though the VAM-based system has serious flaws, we should still take it seriously because the older systems aren’t satisfactory.”   This fallacy turns up in the rhetoric of Intelligent Design proponents, too.  The basic structure is, “There is Your Theory, and My Theory.  If I can show that Your Theory is wrong, then My Theory must be better (even if the evidence is to the contrary).”   Doesn’t follow at all.  Just because the old system is bad doesn’t mean that any new system is better.

2. The “tertium non datur” fallacy.  Related to #1:  The only choice before is Your Theory or My Theory– there is no third alternative.  Linda Darling-Hammond’s article provides a “modest proposal” for alternative approaches, and Golding et al.’s confusingly titled article “Make room value added” presents some interesting ideas — and data – on ways that some districts around the country are increasing the rigor and value of classroom observations as a way to examine the quality of teachers’ practice, a strategy that teachers generally feel is more “grounded” and more just than more quantitative methods are.

3. The “Looking under the lamp-post” fallacy.  You know the joke about the man going home from the bar, who sees an inebriate crawling around on the ground in the light cast by a lamp-post.  He goes to the crawler and asks if anything’s the matter.  “Oh, I lost my keys, and I’m looking for them.”   Did you lose them just here?  “Oh, no, I’m looking here because the light’s better.”    Just because you can measure something doesn’t mean it’s the thing you should be measuring.  It’s a fallacy even more to be avoided, if you can’t actually measure it very well.  Maybe there’s another way to answer the question?

4. The increasing prominence of ‘economese’ in ed talk.   By this I mean the way that so much discussion of educational policy is cast in economic terms which can divert the gaze from essential questions. It’s not only that the most persistent emphasis in policy documents  these days is the “fit” between school and workplace (where “workplace” overlooks the majority of jobs people will end up taking).  I am just noting that it’s now commonplace to talk about schools’ “productivity” in terms proper to manufacturing;  or to talk about the education “industry,” and students or parents as “consumers”, teachers as “entrepreneurs,” — or as “human capital.”

This last phrase turns up a lot in these articles, and the more I read about “human capital” theory the more questions I have.  Does the term “human capital” refer to the teachers themselves (as in some authors), or the skills and knowledge that a teacher may have or acquire?  Is the term used to abstract away from personality, so that instead of thinking about your personnel, consisting of people with faces, histories, experience, etc., you can talk about units of capital.  Consider the interesting sentence “Many new teacher human capital systems are designed so that school systems can explicitly use value-added information to decide…who is allowed to stay in the labor market.”  (pg. 89 of the ER issue).  Somehow,  I do not easily relate the language of learning and teaching (human growth) to language in which the CEO of a School Administrative Unit is implementing a new Teacher Human Capital System.  It feels as though there are two very different enterprises going on, with different values and imperatives.

Blog comments have been archived, commenting is no longer available.
This blog post has 2 comments, showing all.

Norm-referenced accountability

posted by: Michael Culbertson on 4/24/2015 11:43 am

Thanks for your interesting post--a clear and succinct presentation of a number of common fallacies in accountability thinking. To your list, I think I would add the "norm-referenced accountability" pitfall.

Statistical challenges aside, one big difficulty with VAM-based accountability systems, in my view, is that the value-added scores are population-relative (or norm-referenced) measurements, not absolute measures of "educational value." (The same also applies to many accountability systems based on student growth percentiles, for that matter.) This means that any given number on the VAM scale may or may not reflect progress toward our social goals with respect to student achievement: Just because a teacher is above average doesn't mean the teacher is doing well enough to reach our educational goals if the population as a whole is in need of improvement. Just because a teacher is below average doesn't mean the teacher isn't doing well enough to reach our educational goals if the population as a whole is stellar.

If your goal is to hire the best available teacher for an open position, a norm-referenced score may help you. But, if you start firing the least well-performing teachers irrespective of how well or poorly they are actually perform, you will always continue expending (wasting) energy on firing and re-filling positions, because there will always be some teacher on the bottom of the norm-referenced stack, even if your entire staff is composed of Hollywood blockbuster-worthy teachers-of-the-year. This may sound extreme, but it reflects the underlying logic of an accountability system that views performance only as relative to the population.

Norm-referenced teacher measures could be useful for giving kudos to outstanding teachers and for directing limited professional-development resources and supports to teachers who need them most, but it makes little sense to hold teachers accountable for where they stand in the teacher line-up. Accountability is about whether you meet your performance expectations, and it's nonsensical to expect all teachers to perform above average. We made the switch from norm-referenced to criterion-referenced student measures--relying on norm-referenced measures for teachers feels like a regression in educational policy.

post updated by the author 4/24/2015

What is it we're counting?

posted by: Brian Drayton on 4/27/2015 7:57 am

Michael, thanks for this - I hadn't thought much about the point you make. The "norm referenced" approach has a lot of appeal for two reasons that are troubling, it seems to me.

1. First is that it fits with the preference for competition as the way to identify value. This is so endemic to American culture that it often goes unquestioned. It is also so naturally tied to management practices about the apportionment of blame or praise, or a share of limited resources. Then the drive to rank-order is also subject to the law* of efficiency the drive for a metric that is easy to understand, easy to apply, and easy to interpret.
(*I almost wrote "iron law," but in policy no law is set in iron-- you make it stubborn and rigorous for some people or circumstances, but not for others. As in demands for "research-based" policies, not to mention other exx. you might think of. Laws are more like "guidelines". )

2. Second, however, is just as troubling. Ordering by rank can be done on the basis of a sorting rule that can be fairly content-free. If we move to a criterion-referenced approach, then we have to develop criteria, and that means deciding what (in this case) good teaching looks like.
Interestingly, this is what we have some more experience and fairly good research on. It;s also the approach that teachers and other informed observers are more inclined to trust.
As the ER articles I cited in my post showed, it's possible to take this approach, and improve its rigor and even its efficiency but it takes time and sustained attention: what Dewey might call inquiry. Not something that policy makers are comfortable with, nor is our system well-suited to that kind of ambition. When will we accept that economic values are not always the right ones for the job?