Take Sample Size into Account
To assess past events properly, consider the relationship between where the activity is on the luck-skill continuum and the size of the sample you are measuring. One common mistake is to read more into an outcome than is justified. Howard Wainer, a distinguished research scientist for the National Board of Medical Examiners and an adjunct professor of statistics at the University of Pennsylvania, makes this point by identifying what he calls, “the most dangerous equation.” Derived by Abraham de Moivre, a renowned French mathematician, the equation states that the variation of the mean (average) is inversely proportional to the size of the sample. This says that small samples display much larger variation (measured by standard deviation) than large samples in activities that involve a large dose of luck.24 You can visualize the mean and standard deviation with the bell curve, the shape that traces the distribution. The largest number of observations is close to the top of the bell, near the mean, or average. From the top of the bell, the curve slopes down the sides symmetrically with an equal number of observations on each side. Standard deviation is a measure of how far the sides of the bell curve are from the average. A skinny bell curve has a small standard deviation, and a fat bell curve has a large standard deviation.
A small number of results tell you very little about what's going on when luck dominates, because the bell curve will look fatter for the small sample than it will for the overall population. Wainer deems this the most dangerous equation because ignorance of its lessons has misled people in a wide range of fields for a long time and has had serious consequences.
Wainer offers an example to illustrate the point: the rate at which people contract cancer of the kidney in the United States. He provides a map showing that the counties in the United States with the lowest rates tend to be rural, small, and in the Midwest, South, and West. He then shows a map of the counties with the highest rates. They tend to be rural, small, and in the Midwest, South, and West. This is simply de Moivre's equation at work: if you're closer to the luck side of the luck-skill continuum, small sample sizes will exhibit large variations and will lead to unreliable conclusions. Wainer then shows the rate at which people contract cancer of the kidney as a function of the population of any given county, and it is visually clear that small counties have the highest and lowest rates of incidence of cancer while large counties have rates that are closely clustered. A small population equals a small sample and therefore a wide variation.25
Failing to understand de Moivre's equation has led to some significant blunders in making policy. One example is the effort to improve the education of children. Seeking reform, policy makers proceeded in a seemingly sensible way by asking what kinds of schools had children who scored well on tests. The next step was to restructure other schools to look like the ones producing the outstanding students. As you would guess by now, small schools are substantially overrepresented among the schools that scored the highest. This led to a movement toward reducing the size of schools. In fact, the private and public sectors spent billions of dollars to implement a policy aimed at reducing the size of schools.
A closer look at the data shows that small schools were not only overrepresented among the schools that scored the highest, they were also overrepresented among the schools that scored the lowest. Further, Wainer offers evidence that, toward the end of their secondary education, students at larger schools actually score better on average than those at small schools, because larger schools have the resources to offer a richer curriculum, with teachers who can specialize in a subject.26
Here's the main point: if you have an activity where the results are nearly all skill, you don't need a large sample to draw reasonable conclusions. A world-class sprinter will beat an amateur every time, and it doesn't take a long time to figure that out. But as you move left on the continuum between skill and luck, you need an ever-larger sample to understand the contributions of skill (the causal factors) and luck.27 In a game of poker, a lucky amateur may beat a pro in a few hands but the pro's edge would become clear as they played more hands. If finding skill is like finding gold, the skill side of the continuum is like walking into Fort Knox: the gold is right there for you to see. The luck side of the continuum is similar to the tedious work of panning for gold in the American River in California; you have to do a lot of sifting if you want to find the nuggets of gold.
Most business executives try to improve the performance of their companies. One way to do that is to observe successful companies and do what they do. So it comes as no surprise that there are a large number of books based on studies of success. Each work has a similar formula: find companies that have been successful, identify what they did to achieve that success, and share those attributes with other companies seeking similar success. The approach is intuitively appealing, which explains why the authors of these studies have sold millions of books.
Unfortunately, this approach comes with an inherent problem. Some of the companies were lucky, which means that there are no reliable lessons to learn from their successes. Michael Raynor and Mumtaz Ahmed at Deloitte Consulting teamed up with Andrew Henderson at the University of Texas to sort out how skill and luck contribute to the way that companies perform. First, the researchers studied over twenty thousand companies from 1965–2005 to understand the patterns of performance, including what you would expect to see as the result of luck. They concluded that there were more companies that sustained superior performance than luck alone could explain.
Next, they examined the 288 companies that were featured in thirteen popular books on high performance and tested them to see how many were truly great. Of the companies they were able to categorize, they found that fewer than 25 percent could confidently be called superior performers. Raynor, Ahmed, and Henderson write, “Our results show that it is easy to be fooled by randomness, and we suspect that a number of the firms that are identified as sustained superior performers based on 5-year or 10-year windows may be random walkers rather than the possessors of exceptional resources.”28
The authors of those how-to studies found success and interpreted it to create lessons that they could peddle to a credulous audience. Yet only a small percentage of the companies they identified were truly excellent. Most were simply the beneficiaries of luck. At the end of the day, the advice for management is based on little more than patterns stitched together out of chance occurrences. You have to untangle skill and luck to know what lessons you can take from history. Where skill is the dominant force, history is a useful teacher. For example, by well-established methods, you can train yourself to play music, speak a language, or compete in athletic games such as tennis and golf. Where luck is the dominant force, however, history is a poor teacher.
At the heart of making this distinction lies the issue of feedback. On the skill side of the continuum, feedback is clear and accurate, because there is a close relationship between cause and effect. Feedback on the luck side is often misleading because cause and effect are poorly correlated in the short run. Good decisions can lead to failure, and bad decisions can lead to success. Further, many of the activities that involve lots of luck have changing characteristics. The stock market is a great example. What worked in the past may not work in the future.
An understanding of where an activity is on the luck-skill continuum also allows you to estimate the likely rate of reversion to the mean. Any activity that combines skill and luck will eventually revert to the mean. This means that you should expect a result that is above or below average to be followed by one that is closer to the average. Recall Charlie, the student who knew eighty out of one hundred facts but was tested on only twenty of them. If he scored a 90 on the first test because the teacher happened to select mostly questions he could answer, you would expect the score on the second test to be closer to 80, as his good luck would be unlikely to last.29
The important point is that the expected rate of reversion to the mean is a function of the relative