You can’t compare apples and oranges. That’s one of the most common assumptions in examining differences between options. Comedian George Carlin solved it easily enough though: “They keep saying you can’t compare apples and oranges. I can. An apple is red and distinctly non-spherical; an orange is orange and nearly spherical. So, what’s the big problem?”
Data-minded baseball analysts have also solved the conundrum with the statistical concept of Wins Above Replacement (WAR), a method for analyzing a player’s overall value. This statistic quantifies and synthesizes all of a player’s contributions to a team, and then assigns an overall numerical value to the player expressed as how many more wins a team could expect from using that player instead of a different player. The formula is complicated, but it’s generally accepted by baseball executives, writers, and fans as a valuable tool for comparing players.
For example, the player most valuable to his team in 2011 was Los Angeles Dodgers outfielder Matt Kemp whose 2011 WAR was 10. This means that the Dodgers could expect to win 10 more games if Matt Kemp played every game than if someone else was playing in his place.
The all-time worst WAR belongs to Kansas City Athletics shortstop Joe “Froggy” DeMaestri. His 1957 WAR was a negative 4.9. In other words, the 1957 Athletics could expect to lose almost five games more with DeMaestri on the field than if any other player were there instead. (I mean no disrespect to Mr. DeMaestri. He had a solid major league career before 1957.)
So what does WAR have to do with schools? Well, we see a lot of changes implemented in schools these days, but we don’t see much examination of how the changes compare to what else we could have been doing. How many more “wins” could we expect if we did something other than a specific implemented change?
For example, schools in our district have ten Late Start days each year when students arrive to school 100 minutes later than usual. That’s 1,000 minutes less schooling over the span of a year. During the Late Start time, teachers gather in our district’s version of Professional Learning Communities and conduct a variety of conversations and collaborations. These are then declared as successful by our decision-makers, and the next year we do it again. But no one has ever asked the question “Do our students benefit more from the Late Start activities than they would if they were in school for those 1,000 minutes per year?” In other words we made a change, declared it successful, and repeated it, but never compared it to any kind of control situation or examined other possible options. We simply accept the continued recommendations of those who recommended the change in the first place.
Consider schools that move from grouping students by level to non-leveled instruction, or vice versa. Presumably the decision-makers have good reasons for making the change, but do they compare the results of their decision to other possible options, or do they simply find a way to declare success as validation for their original decision?
What about schools that implement widespread standardized test preparation while displacing time that would have been devoted to curriculum? If those test prep efforts seem to make test scores go up, it’s easy to declare success. But would the students have experienced at least as much success or maybe even more on those tests if the displaced curricular material was made available to them? (Of course, the even larger question is, Which is more advantageous to a student in the long run—standardized test preparation or the displaced curricular material?)
All school-related changes can be considered as positive developments if we accept two statements as facts: (1.) Policy-makers and decision-makers are always right, and (2.) A change that works in one context will also work in another context. If those two statements are true, then what schools need to do immediately snaps into sharp focus. Of course, only a lunatic would accept either of those statements as true. Evidence, experience, and intuition all strongly suggest that neither of those statements is worthy of acceptance as fact.
No change happens in a vacuum. A new policy or practice not only replaces an old policy or practice, but it also takes the place of other options that were not implemented for whatever reason. How can we compare the successfulness of the policy we chose to implement as compared to the potential for success of the options that were replaced or not chosen? Those metrics exist but they are way over my head. But if baseball can do it, why can’t educators? At a minimum, we need to ask comparative questions, and question the assumptions behind school-based changes.
Let’s at least ask the question, “Why is X better than Y?” Just because X produces some positive results, does that automatically mean it’s a better option than Y or any other option? Of course not. Maybe Y would have produced even more positive results. To simply accept X as “good enough” is a disservice to our students and school communities.
I see a lot of education policy-makers and decision-makers implementing big changes and then declaring them successful without any meaningful comparison to unchosen options. Then other policy-makers and decision-makers like the sound of that “success” and implement the same change in a different context. Then those folks declare it a success without examining other possible options, and the next think we know, cottage industries of consultants, publishers, and material creators have sprung up to support the change. Then a mentality emerges that says, “It must be a good idea because we’re spending so much money on it.”
But no one has stopped to ask, “Is this really the best idea? How do we know?” (The corollary to these unasked questions seems to be another unexamined assumption: We must be correct because so many others are copying us.)
The WAR rating did not exist in 1957 when Froggy DeMaestri played in almost every game for the Kansas City Athletics as a statistical liability that is only visible with hindsight. The team ended up 7th out of 8 teams in the American League standings that year. He went on to play full-time for two more seasons, and saw his 1957 error total increase from 13 to 22 in 1959. DeMaestri’s 1957 batting average of .245 dipped to .219 in 1958 before rising to .244 in 1959, his final full season. In other words, he didn’t improve before he faded from the major leagues.
No analogy is perfect, and baseball and schools are very different universes. But this much is true: The “W” in Wins Above Replacement, when translated to schooling, relates to our students’ learning. When we make a change, learning is likely to be affected. Let’s be sure we get it right, or at least examine the options and assumptions behind any changes we make.