The Failures of Quantification and Statistics

Why is it that no matter how many new tests we come up with - no matter how many new teaching methods we devise - the kids seem to do worse and worse?

Albert Einstein's school performance problems have probably been exaggerated to make his life appear more dramatic, but he did clash with the authorities of the the Luitpold Gymnasium and he did fail the entrance exam to the Eidgenössische Polytechnische Schule in Zürich. There was clearly a problem with the system's ability to test his intelligence.

Then, as now, scientists, statisticians and the general public were very optimistic about the measurability of seemingly-elusive aspects of life: character, intellect, disposition. Darwin had published The Origin of Species relatively recently, and that had spawned fields as diverse as eugenics, phrenology and psychometrics - which in turn gave us the IQ test. Indeed, if success in society seems to be as heritable as a particular beak shape, then there must be some way to objectively measure it, even at an early age.

Many of these techniques are now discredited as junk science, pseudoscience or outright fraudulence. However, today - armed with an impressive new array of techniques and equations, as well as the vast processing and information-gathering powers of computers - we again are attempting to employ objective testing to understand our world. This time, however, it is our children and their teacher's performance which is under the most intense scrutiny, as mandated by the No Child Left Behind Act.

I could, here, launch into a polemic about how NCLB is yet another attempt by the Radical Right to destroy public education, as a strong public education system tends to produce unruly liberals who want rights, whereas without public education the masses become a source of cheaper labor and are less likely to rebel against the corporate powers-that-be. But, nefarious as I think their ultimate goals are, the success of this coup ultimately rests on the readiness of the public to believe that intelligence is objectively testable. For years before that, one of the most important factors in college acceptance was the SAT. And while most colleges still demand a personal statement, there is a general sense that a dazzling personal statement might make the difference in a borderline case, but it would certainly not allow someone with poor grades and test scores entrance into an elite university.

And, indeed, why shouldn't people believe that we can test something as tiny and elemental as a single human being? After all, this is the era of targeted internet advertising, predictive cell phone text, first-person-shooter bots, high-frequency trading, climate change, high-stakes election manipulation, and mortgage-backed securities so complex that no one short of an autistic savant can understand them.

Because of these incredibly powerful tools, we have become a more information-centric culture, rather than a knowledge culture. Writers and journalists have lamented that, with all of the myriad, punchy distractions of modern life, they rarely have time to sit down and absorb an in-depth article or paper. As information becomes more punchy and more graspable to machines - indeed, in no small part because gaining PageRank is almost equivalent to gaining advertising dollars - so too that information becomes shallow.

Also, terse information is enticing because it seems easy. For example, with the click of a button, I can pull up statistics on unemployment in any county in the nation. In fact, I can probably get unemployment statistics for every county in the nation as far back as we've been collecting unemployment statistics. But the ease of acquiring this information belies the difficulty of interpreting it.

First, many people simply take the information at face value, as if it simply leaped into being from the all-knowing Mind of God. In reality, there is serious debate over whether the official unemployment rate hides a larger "real" unemployment rate. If you simply count the can only find part-time work and those who have given up, the unemployment rate goes up to 17%.

Secondly, the total unemployment rate includes people of all ages, races and genders. Some are doing better than others. If you look at demographics such as young men, the unemployment rate jumps to 20% officially, and probably 50% unofficially. The statistics for young black men are even more dire. How does it bode for our future as a nation if we cannot provide work for millions of young men? I am not trying to be chauvinistic here. Rather, I am merely addressing the undeniable fact that many of these men - especially working-class men - have been raised to regard men as breadwinners, and the fact that men are intractably more aggressive, assertive and prone to random acts of violence than women. These men will grow up disillusioned, poor and angry, and create instability and social unrest as they mature.

I could go on dissembling our official unemployment numbers, but it should be clear at this point that the official number hides far more complexity than it reveals. Indeed, the most important piece of knowledge in unemployment is how it is actually affecting people on the ground, and while quants could no doubt come up with some arcane way of measuring unhappiness, I skeptically assert that the only way to acquire real knowledge of this sort is to meet the people and ask them what they are doing to cope with their situations. Any statistic attempting to derive the same information would, by necessity, be to simple to be useful or too opaque to be understood.

Similarly, we can understand the financial crisis as a crisis of bad information. We entered a period of incredible real estate price inflation. This inflation was caused in no small part by giving loans to people who couldn't afford them, but for a time, these people were able to get off the hook by selling their property to yet another sucker and making a nice profit. But eventually, the ability of Americans to buy homes was exhausted, and prices started leveling off. Suddenly, the massage therapists and housemaids who owned five investment properties apiece were left with no way to pay off their debts, and all of those "Safe as Houses" securities started to explode.

What happened here was that math, rather than being used to increase our understanding of the world, was used to distort it. The big banks were allowed to invest in these highly volatile securities because the risk-assessment algorithms did not account for a "black swan." The securities were passed off as better than they were because of corrupt bond rating agencies and a byzantine system of security construction. Meanwhile, AIG was allowed to sell unregulated insurance contracts it couldn't fulfill - then, when they came due, Goldman-Sachs alumni/operatives in the government swept in to use taxpayer dollars to pay off Goldman-Sachs and the other counterparties.

The reality is that the theoretical truth which investors and quants saw was wholly divorced from the brute facts underlying it.

Tests fall short for precisely the same reason as bizarre mortgage-backed securities. They attempt to objectively assess a student's competency in a variety of areas. Increasingly, however, the schools are simply teaching the students how to perform well on the tests, rather than actually teaching them how to read, write, and creatively solve mathematical and scientific problems. Just as traders maximize their own bonuses by fooling the risk-assessing algorithm, so too schools maximize test scores by teaching to the test at the expense of real education. In other words, there is no system that cannot be gamed.

In a sense, the teachers are far more noble than bond traders. A bond trader has a choice between making more money in a year than a middle-class person will make in a lifetime and making more than ten or twenty middle-class people. A teacher has a choice between having a middle-class job as a teacher and having a working-poor job at Starbuck's. But in terms of outcomes, they have more in common than not. They are both performing societally disadvantageous activities as a result of a system of perverse incentives over which they have no control.

What is the alternative?

The fundamental dysfunction gripping public schools in this nation is a lack of local responsibility, local funding and local control. On a local level, the teachers can subjectively evaluate the students, the administrators can subjectively evaluate the teachers, and the community/PTA can subjectively evaluate everyone. The statistical system of assessment is really an abdication of the individual responsibility of people and of true competency. Tax revolts, such as Proposition 13, may not have reduced the total tax burden of the communities, but they did shift power away from local governments and towards federal and state governments. These federal and state entities have no option other than statistically assessing schools, as the cost of administering a bureaucracy large enough and competent enough to handle America's school system would be prohibitively large. In order for America's schools to work, communities have to control their schools, including the all-important power of the purse, and those schools must cease to be beholden to some far-away political body which they will never see, and which will never truly see them.