The Signal and the Noise (a review)

Nate Silver's famously accurate political forecasts on his blog FiveThirtyEight gave me a little emotional stability going into the 2012 presidential elections. He hits almost every election right, and I liked how his predictions didn't vacillate with every new event.  And my favorite candidate had a 91% chance, the morning of.  So I read his book.  

It turns out his method is simple.  Silver, who now works for the New York Times, averages many political polls after weighting each on the basis of the polsters' past successes, the sample size, and when the poll was taken. He delivers the result in terms of probabilities, which allow for things unknown and imperfections in the model. He's smart, and startlingly accurate -- so when I heard he had written a book on probability I ordered it immediately. I ordered myself three copies by accident, such was my enthusiasm.

Bayesian Probability – the heart of the book -- is a simple statistical technique that corrects for shortcuts in our thinking. It was one of the lessons in Kahneman's extraordinary Thinking Fast and Slow. Bayes factors in the prior expectations in addition to several estimated probabilities for events occurring and not occurring. The method requires constant revision as new data come forward. I like it so much I've installed a Bayes app for my smart phone.

Silver is a serious gambler, we learn, and gamblers must be especially honest in their forecasts. They put their money where their mouth is.  Silver’s own games have been mostly poker and baseball and he devotes a full chapter to each of these. You wouldn’t know it from the cute titles (such as “How to Drown in Three Feet of Water”), but each chapter is sharply focused on a particular subject: 1. mortgage collapse, 2. politics, 3. baseball, 4. weather, 5. earthquakes, 6. GDP, 7. epidemics, 8., Bayesian probability itself, 9. chess, 10. poker, 11. stocks, 12. climate change, and 13. terrorism. Each of these, it turns out, is quite an interesting challenge as they vary in their complexity, the amount of data which are available, forces for bias, and the quality of the models used in forecasting.

I've drawn several sweeping conclusions. Good forecasting requires a lot of reliable data, a clear understanding of the baseline, and while a causal model is not essential it will help sort the signal from the noise. The forecast will benefit from multiple viewpoints (though not weighted equally), and from both quantitative and qualitative methods. It must be constantly reassessed as new data come forward. Computers are excellent for processing information, but if you put enough data into them they will detect patterns and relationships that don't really exist. Usually human judgment should figure in -- in part, and interestingly, because people have good vision.

People are irrational in predictable ways. Recent, local, and highly publicized events appear more important than they actually are. We tend to select information to support prior biases –especially when there is a lot of information available (and so this is more of a problem, in the informaton age). We easily attribute actual mistakes to bad luck. We are overconfident of our opinions and wedded to them. We can easily discount the possibility of extreme events.  And we are inclined to detect meaningful patterns where there are none. The author speculates, as would I, that there are evolutionary reasons for many of these.

There are some pretty impressive successes in forecasting, though. Chess is now a computer's game because all the usefull skills and combination of profitable moves can be programmed. Poker is fairly easily forecasted, because it’s relatively easy to calculate probabilities, and the human element is fairly small. Silver played himself for many years, ending only when on-line gaming rules drove out the poorer players -- the "fish," he calls them; one day he realized he was one.

Baseball also has a fixed rulebook and scads of data, but with a more complex human element which is more difficult to predict.  In these cases, Silver says, a wider net for information is better – interviews, profiles .. in other words, an old-fashioned talent scout.  More impressive still are the weather forecasts, now quite accurate eight days in advance. These also incorporate a great deal of data, and they benefit from the immutable laws of physics, and from elaborate models which have been refined and corrected by a steady stream of feedback data. Interestingly, commercial services offer 15 day forecasts even though anything beyond eight is less accurate than the baseline (climatological data alone, which is no forecast at all).   Apparently people sometimes prefer to feel informed, even with misinformation. The commercial forecasts, which are based on the government's models, are also jiggered.  The Weather Channel rarely predicts a 50% chance of rain, because their viewers are more comfortable with 60% or 40% -- and the chance of precipitation is routinely overstated (people enjoy unexpected sunshine, and they really hate bad surprises!).   Newscasters and political pundits are terrible forecasters -- they are often biased (towards sensationalism, or a particular party) and there is no cost to them, of being wrong.

If weather has become easy to forecast, climate is not. There is far less corrective feedback data, as climate changes so slowly, and so the models are inferior.  They're also more complicated as they must incorporate the greenhouse effect, and quite a lot more chemistry.  Add to this the strong political bias, because the causes, effects, and costs fall differently around the world. Many players are biased by self interest, others by pure contrariness, and because the science is incomplete some simply get confused and deny climate change altogether. However, there is wide agreement that C02 and other greenhouse gasses increase warming, human activities are increasing these gasses, and the world is slowly warming (~1.5dC/century), at least partly as a result of human activity. These details are debated: how fast it will happen, how temperatures and precipitation will change in specific regions, economic impacts, and how effective we might be slowing or reversing it.  But if you every hear someone claim that there is no evidence that the global climate is changing, that scientists are merely promoting a political agenda, or that we don't know enough about climate to predict anything with confidence ... that is not signal. That's noise.

There are things even more difficult to predict than climate: Earthquakes, largely because the most relevant data are deeply buried and hard to gather. And terrorism – for much the same reasons. Thankfully (I suppose) both of these seem to have a power law distribution such that when plotting frequency and magnitude on two logarithmic axes, events form a straight line. This is useful predicting even the frequency of events larger than any before.   So we know how likely, but we don't know when.  Or where.

That’s enough review – I've skipped a lot of my highlights and I highly recommend reading the book through. At times Silver gets drawn into irrelevant details such as building up a story which only ends in being summarily denied an interview.   He likes baseball a whole lot more than I do.  But mostly the book is rock-hard practical.

Other quibbles -- the word data is plural; a datum is, but the data are. And there were a few typos (though when I find one in a book of this quality it's like uncovering a nice little fossil).   It's not short – 454 pages, followed by 80 pages of notes and references – but I wanted more.  There should be chapters on the job market, on real estate, marriage, religious predictions, the efficacy of legislation,such as gun control, marijuana legalization, etc.  These are even more important than baseball, or poker.  I do think we need another book.

Mr. Silver?

Human Evolution: A Geographer's Perspective

Not long ago, I received an email asking me to speak to a church group about the evolution of humans from a geographer’s perspective.  It was a member of the Ethical Humanist Society in Skokie Illinois; he'd seen my blog. I knew enough about the organization to quickly agree, but I asked for several months, to go over my books and notes on hominids and humans and their migration out of Africa. At first I had in mind a sort of a Jared Diamond approach; I wanted to reread his books  (Guns, Germs, and Steel .. and The Third Chimpanzee -- I tried, but could never get through Collapse), Darwin’s The Descent of Man, and an Anthro text called Darwin’s Legacy.  But the more weekends I tinkered with my thoughts, the more the scope of my talk expanded, and I chose in the end to weave together themes from many of my existing blogs, culminating with a new set of ideas which has intrigued me. This blog is a summary of my talk which I gave to full house Dec. 2. 　

As I put down my thoughts in narrative here, I will embed links to my various essays and will mention specific books so anyone interested in going deeper can do so easily.  As my date approached, whenever　I timed myself honestly I could not nearly roll through my address in 45 minutes, so I invoked what most people call natural selection, evolution, or survival of the fittest; it is simply the process of elimination.  I jettisoned whole chunks, distilled others, and then I cut even more to allow myself some digression -- to switch it up a bit with something spontaneous if the mood struck. After all, innovation is an essential part of evolution too.

I must admit I was a bit outside my comfort zone as this was the first time I was to speak on evolution in a formal setting.  I worried, as is normal for me, that something might go terribly, crashingly wrong.  I teach cartography and GIS, and while I’ve been designing a class on biogeography I haven’t nearly rolled it out yet.  But I have plenty of raw enthusiasm, quite a few thoughtful perspectives, formal coursework, and my heavily annotated science books which are listed to the right. This was quite a welcome opportunity too.  By now I’ve exhausted my family members on the topic. To be fair, they are still receptive to my ruminations -- but only in small doses.

A little background about Ethical Humanists.  A few years go when I took an online survey to determine which religion best matched my views, I hadn't known of ethical humanists but it turned out I was one.  I’m no authority, but Humanists seem a bit like Unitarians I’ve known and many liberal Quakers I grew up with -- keenly interested in knowledge, goodness, and community -- and pointedly light on the supernatural talk. Like the Unitarian church I briefly attended in Southern Illinois, these Ethical Humanists bring in speakers.  I recently saw Dan Barker, author of Godless, address this very group; he is a former evangelical preacher and is now an articulate, lighthearted and very effective spokesman for those who have shed their religious beliefs.

And as I was milling about before the talk I picked up a little card summarizing the mission statement of the parent organization: American Humanists.  Excerpts from their manifesto include
　

"Humanism is a progressive philosophy of life that, without supernaturalism, affirms our ability and responsibility to lead ethical lives of personal fulfillment that aspire to the greater good of humanity."
　
"Knowledge of the world is derived by observation, experimentation, and rational analysis." …
　
"Humans are an integral part of nature, the result of unguided evolutionary change."　　

I found that refreshing; clearly I could expect a kind and educated audience and the last bit was particularly helpful with my last-minute jitters. Here is their web site -- they have events all through the week, and for children, too.
　

My title was "Human Evolution: A Geographer’s Perspective" and I started off with my standard clarification of the scope of geography -- a discipline which is oddly inconspicuous in America but has become much easier to explain with the advent of Geographic Information Systems.  GIS uses a variety of digital maps for the same location.  One might be of winds, another of population, and a third map of hospitals.  And each one carries an inventory; imagine a spreadsheet wherein each row represents one little mark on the map: a weather station, census tract, or particular hospital.  Respective columns contain speed and direction for various times of the day; everything from the U.S. census; and all information about hospital facilities.  Stack these maps, and GIS will let you count the number of young children 5 miles downwind from a chemical plant, and determines whether nearby pediatric wards could handle an explosion.  The databases are relational, with location being the relational link.  This is how geographers think -- location is our hook, and we can hook into all things for which location matters. 　
　

I also quickly ran through some of the more common misunderstandings about evolution -- common, that is,　among the 45% or so of Americans who believe in evolution at all (I know, WTF!).  Evolution is not a drive toward perfection, humans didn't come from chimps, very few traits result from a single gene, biology actually does interact with environment, and so on.  You only need three things for evolution to occur, according to Richard Dawkins in the seminal The Selfish Gene. 1. something must duplicate, 2. it must do it imperfectly, and 3. it must matter in the real world.  That's it.  Then it evolves.  And I defined species, using the biological definition: groups which don't naturally have fertile offspring.  Species, I'll add, are usually the result of geographical barriers separating populations which are then groomed by selective pressures or drift apart randomly.  The exception to this is called sympatric speciation, where different groups fall into different niches within the same locale.  There is much about this last one which is still a mystery to me; I understand why they would go into different niches but as populations drift apart, but why would the hybrid (like a mule), vigorous in all other ways, be sterile?  I've read expert opinions, but  I still don’t know.

I showed the well known map of human migration out of Africa about 50,000 years ago, shortly after the Toba super-volcano pressed human populations into the thousands -- to near extinction -- for thousands of years.  That genetic bottleneck is thought to explain why there is very little variation in the human genome today.  There is said to be more variation in a single social group of chimps as there is in the entire human race. The routes humans took from there is to the left. 

Then I rolled out the story of how I traced my own DNA, using National Geographic’s Genographic Project and 23andme.　
　
As I was preparing my talk I very much wanted to comment on 3.5 billion years of life, not just the last 200,000 years in which humans have existed.  Here's how I did it: first I described ring species, using the example of the circumpolar Laurus Gull, all the way around at 80 degrees north. Laurus characteristics vary only slightly everywhere; they satisfy the biological definition of a single species by producing fertile offspring, with neighbors.  However, where the ends of this ring meet the gulls won’t have anything to do with one another. The Herring Gull and the lesser Black-Backed Gull are therefore different species if you face east, but the same species if you're looking west!  And here is where it becomes relevant to humans. There is a similar and utter lack of species breaks when you trace humans all the way to the first spark of life; it’s incremental variation, all the way down.  Never would one generation be unable to produce fertile offspring with the next generation and this is true not only going down but back up the trunk, branches, and twigs, to all living things today.  You can travel from the woman to the ladybug on her shoulder if you crawl from one leaf (her), back through time, all the way to the junction, then back up another branch and twig to the bug.  You'll never step from one species to another.　

So, to understand humans it only makes sense to start 3.5 billion years ago when all life began.  And we know life only started once, because all genetic code from protozoa to great blue whale is written in the same language with the same four letters: nucleotides A, C, G, and T.
　
When I was practicing this next bit, which is really the only new thing I had to offer, I always ran long so I created a graphic to help me skip through it.  My thesis is that plants and animals were driven by four essential needs (gathering food, finding mates, evading predators, and protecting young), and in response evolved patently spatial strategies: improved perception and control of the environment, and improved navigation through it.

I've ruminated before about first life.  From there forward, the strand which would become human usually opted for complexity, given the option.  There is nothing shabby about simplicity -- bacteria took that route and ... how does that saying go:  "He who laughs last, last best."  We tooled up, while they stripped down.  They hunker. We lumber.  You might summarize that our ancestors were both imprudent and lucky.  Good fortune and has blissed us with brains and consciousness. 

Digression: It is simply awesome to be self-reflective; it's spectacular, really. Dumbfounding.  I must add that I also feel like a spectator;; I don't presume to think I am independently charged.  On the topic of free will I’m with Sam Harris -- we only feel that we have free will because we can’t predict what we are going to decide next!  But it makes sense that we feel so self-directed.  As Robert Wright clearly explains in The Moral Animal, self delusion has some clear evolutionary advantages.　
Now we come to the meat of this story.  Several brilliant authors have elucidated on live from its origins to modern day.  Richard Fortey's Life, Matt Ridley's Genome, Nick Lane in Life Ascending, Bill Bryson's Short History of Nearly Everything, and of course Richard Dawkins' The Ancesters Tale... they all take us on a long long journey to the past.
　

I won't touch on every invention, but if you read the chart carefully I think you’ll appreciate the spatial dimension of each juncture.  It's commonly thought that we first duplicated in porous rock by the deep sea vents' stable gradient of heat and pH.  You might say we were born in bondage.  When we finally developed our own cell wall we were free to blithely drift and duplicate for more than a billion years, without a care (so to speak) until the moment a mutation caused one bit to eat another.  Nutritious! It was rewarded by survival, and procreation.  So predation 2.4bya, placed a premium for both parties on movement so when one of the cellular creatures grew a tail, actually a flittering hair, a "fagellum," it too was rewarded in offspring.  Soon we (our ancestors) all had little wigglers.  Two billion years later, when a flotation bladder accidentally became a rudimentary lung, we crawled onto land.  But there, cold nights were particularly important – there had never been such temperature fluctuations underwater. Warm bloodedness emerged, soon as it could, simply because of the survival advantage of being first up for breakfast.  Our eyes had developed around this time, so we could scan our environment for predators and (conversely) for food.  In the broad view this was nothing special.  Eyes have happened about 40 times.  Judged against the other eyes, human eyes are just mediocre.

The amniotic egg about 200 million years ago allowed us to have young in even where there were no ponds.  This is a spatial explosion!  Next came the secondary palette and we could run with food in our mouths, still breathing.  Larger meals, greater territory still. 　
　
There is a short period in our history 145-65 mya, where we reversed our general trend toward complexity and territorial expansion.  There were dinosaurs about; they better at eating than we were.  So we hunkered down for nearly 100 million years, burrowing, developing timid habits, in a spatial sense constricting ... until that wonderful astroid landed east of theYucatan Peninsula, causing rains of buring sulfur and effectively removing our opressors.  It was suddenly safe not only to emerge from underground, but to take to trees, and we did, thereby improving our stereo vision and building up bony eyesockets. The process of elimination (by fatal falls) certainly helped with these new innovations.　

We were never destined to greatness, remember.  It's not fate, but luck. 　
　
Eventually homo habilis (the handyman) started making tools: first, a stone knife, which was simply an improved, replacable tooth and nail. This cleverness paid off in food, as man attacked beast, and no doubt in mates -- as man attacked man. Lots of animals make tools, but this was a particularly good one.  Not as good as the spear, though, and I’ve recently read that the spear may have spurred a social revolution. Prior to the spear and bow the behavior of hanging back, whenever a group tried to take down a large animal, paid off in survival tokens; so everyone tended to be a little tentative. Then spears made cooperative hunting possible by limited the personal risk. it built community; it's the prisoner's dilemma, pure and simple. 　
　
But then came the big leap -- language -- 100,000 years ago. We had been getting smarter, no doubt -- in the previous 2 million years our brains had tripled in size but with language we could suddenly tap into the brains of others. My 16 year old son recently observed that language is a lot like telepathy, and it's true.  When we developed written language we could access the thoughts of people far away (and those of some dead people too). Language was not the first wireless network, however; it was probably second to mirror neurons. It is commonly believed (among scientists) that mirror neurons allow one to actually experience the experience of others. I watch you grasping a coffee mug, my brain feel grasping -- my grasping neurons fire. Some have speculated that this is how fish school, birds flock, and why yawns are infectious.  Mirror neurons may very well have been the dawn of empathy.

Then just 600 years ago movable type printing press brought an immense knowledge base to the literate commoner, and it allowed the literate commoner to share his intelligence too. Do you see how this leaps and stretches across space?

Each bar in the diagram represents a much shorter time frame, reflecting the acceleration of change. The bar on the right, spanning just 5,000 years, might as well have focused on innovations in transportation -- wheel through boat, horse, horseshoe, steamboat, bicycle, locomotive, car, airplane, helicopter, jet, and rocket ... but I’ve chosen instead to emphasize communication.

After telegraph (1843), telephone (1876) radio (1901), television (1925), photocopier (1958), satellite (1963), worldwide web (1989), with the advent of the computer, internet, and the inexpensive mobile phone we are witnessing a revolution in information as important as what happened with language 100,000 years ago – but with 100,000 times the spatial bang. The two maps show internet connectivity (node to node, local connections ignored) and facebook friends. But think of google searches, netflicks, wikipedia, facebook, skype, MOOCs, Youtube, and all maknner of mobile apps. Isn't it interesting to speculate that while no other animal is even aware of it, homo sapien sapien brains around the world are in the process of coalescing in a very real sense, into one brain.  The new brain still has a slippery grip, but it's a global grip.  An interplanetary reach even, as we recently have sent an extension of our own eyes to Mars.
　
I ended my talk with a little fun, speculating about ideas, which are virus-like replicators and therefore also subject to evolution. Much like genes, ideas duplicate, imperfectly, and they are filtered by the external world. A good joke, bad news, and useful information is repeated, while other noise fizzles out. Like genes, concepts can also team up to duplicate more reliably. 　

For example, think of this. Combine empiricism, conjecture, hypothesis testing, evidence based reasoning, statistics, peer review, transparency, and a few more concepts and you have the scientific method, which has spread.  To carry it further (in my way of thinking) science often comes up against religion, which have at their core just three little ideas: 1. there is an invisible, mysterious entity, 2. it can and will hurt you if you don’t follow its rules.  And 3. the first rule is believe in it.  This circular core could not withstand scrutiny by itself, so is protected by its own cell: faith, of the sort that means belief without evidence.  Now comes the hat-trick: this faith is presented as a virtue.  And then distractions are attached, such as ritual, song, cathedrals, art, diet, dress, scripture, etc.  hence the mad,  circular, core concepts are sufficiently insulated that the whole package replicates, most asily in the lesser educated minds, and those of children -- imprinted, like a duckling for a lifetime.  And that's all you need but if becomes lucrative there will be those to promote and defend it for other reasons as well.
　
This idea of ideas having their way with us -- whether it’s science, religion, or neighborhood gossip -- well I find it ... er .. infectious.
　
Looking back on my hour, there were two big surprises. The first was that while I was arguing that human evolution can be seen as a series of advances in spatial awareness and control, I nearly stepped off the stage.  The humor in that was not lost on the crowd, I’m happy to report.  Secondly, when someone asked me if I’d written my thoughts down, and I said I’d probably do it in my blog, there was actually applause! This, recall, is the same blog my vigilant statistics tracker regularly reminds me that almost no one reads. That encouragement probably pushed me over the edge and I actually took the time to do it.  Everyone, but particularly if you are an Ethical Humanist, and if you've read this far ... bliss you.