Do You Know What You Don't Know?

Tonight, I've been running a bunch of simulations for a project for my agent-based modeling class. While the models were running, I listened to this intriguing podcast with Russ Roberts interviewing Nassim Taleb, author of The Black Swan.

Taleb discussed the differences between events that follow Gaussian distributions (normal distributions) and those that follow power law distributions. Most of conventional statistics is based on normal distributions while many events in life actually follow power law distributions, making many of the tools used for data analysis and risk-management very flawed. This affect fields as diverse as medical research (according to Taleb, 80% of epidemiological studies are not replicable), finance, and economics.

To get at the distinction between a normal distribution and a power law distribution, think about comparing the height or weight of people to their wealth. Height and weight of adult males can certainly range from short to tall, but you'll never meet a 6-inch-tall man or a 50-foot tall giant. The constraints on this range make using a normal distribution an appropriate representation for the variation in this population. On the other hand, you might encounter someone from Botswana who makes $300 per year while Bill Gates is worth billions. The distribution of wealth follows a power law, with a high number of low-wealth people and a small number of high-wealth people.

A problem arises when researchers and analysts inappropriately use statistical tools designed for normal distributions for things that follow a power law. Stock markets, earthquakes, book sales, etc. all are examples of things that are sometimes prone to unforeseeable rare events (both positive and negative). The troubling thing about power laws is that they are unpredictable. One of the key insights of Taleb is that it is important to know what we don't know and once we do, it is possible to plan for the unexpected. We often get into our biggest trouble when we assume we know much more than we do.

Listen to this fascinating podcast and be sure to check out Taleb's book. I ordered a copy on Wednesday and it should get here later this afternoon.

See my previous post on The Black Swan and go to EconTalk's website for more links related to this talk.