Survivorship bias

Survivorship Bias (Don’t Believe the Hype)

I remember the first personal development book I ever purchased. It was called “The Magic of Thinking Big” by David Schwartz. The promise on the book jacket jumped off the bookshelf: “Millions of readers have acquired the secrets of success through The Magic of Thinking Big. Achieve everything you always wanted: financial security, power and influence, the ideal job, satisfying relationships, and a rewarding, happy life.”

Sold! Who wouldn’t want to achieve everything they have always wanted? Up until then, it never occurred to me there was a “formula for success”. What a revelation. All I had to do was learn the magic formula and apply it. It’s a compelling story that we all want to believe.

That book was my gateway to consuming a flood of books, videos, podcasts, and training courses to help me achieve my goals.

I was a competitive bodybuilder in my twenties (I won the Mr. New Zealand bodybuilding title in 1993). As I was coming up through the ranks I devoured all the magazines and books by the “bodybuilding gurus”; Arnold Schwarzenegger, Joe Weider, etc.

Then I studied the “personal development gurus”; Brian Tracy, Zig Ziglar, Jim Rohn, Tony Robbins, etc, and books on how those “Millionaires next door” came to be that way.

Later on, I studied the “business gurus”; Jim Collins, Tom Peters, Peter Drucker, Seth Godin, etc, the biographies of successful entrepreneurs, business coaching frameworks (Scaling up, EOS, etc), and business research from the likes of McKinsey, Bain, Stanford, and Harvard.

I tried to devour pretty much all of the best-selling non-fiction books you see in the airport bookstores. I set a goal of reading 1 business book or personal development book per week. 50 books per year was my target and I achieved this standard for more than 10 years in a row. I spent thousands of dollars and thousands of hours reading books.

It quickly dawned on me that all these gurus and books follow a similar approach. Observe what successful people do and try to reverse engineer it. Package it up and sell it as a guaranteed “success formula” that will transform your life.

The success formulas are pretty similar too, whether it is for business, sports, or just life in general: Visualize the future you want. Set lofty goals. Copy the strategies used by the winners. Model the behaviors used by the winners. Be “bold”. Work hard. Put in 10,000 hours. Track your progress. Never quit. Stay focused. Keep believing. Eventually, you will win!

The general principles seem plausible. Of course, learning the core principles and working hard is important in any endeavor. But after a while, it dawned on me. The authors were glossing over a factor that plays a critical role in success.

Life is not fair!

As a novice bodybuilder, I studied the training routines and diets of the champions. I trained harder than most, even to the point of fainting after workouts. I became stronger than most. I ate cleaner foods than most.

But here’s the big lesson, so did everyone else in my peer group. Some of my friends at the gym trained harder than me. Some were stronger. Some ate cleaner diets. Some followed the so-called “success formula” espoused in the books more diligently than me.

Yet, I won the “Mr. New Zealand” title. You see, in bodybuilding, no amount of training or diet can change your bone structure or the shape of your muscle bellies. I was lucky to have genetics that lent themselves to success in that sport.

I was the “lucky winner” who got the sponsorships, got featured on the posters, appeared in TV interviews, and was asked to give training seminars. If I chose to write a book on “how to succeed in bodybuilding” it would contain practical advice, but there is no way I could guarantee your success if you followed the same path I did.

In any endeavor, it is the “lucky winners” who write the books, get interviewed in the media, and get asked to give public talks. However, the winners have less to teach us than you might think.

Bodybuilding is not fair. Business is not fair. Life is not fair. No matter how much you want something, how hard you work, or how diligently you follow the so-called “success formula”, your success is never guaranteed. Factors like genetics, country of birth, gender, race, parenting, poverty level, neighborhoods, schools, social contacts, peer group, industry context, macroeconomic environment, competitors, luck, and timing play a far bigger role in our success than the gurus would have us believe.

The gurus peddle simplicity and certainty, in a world where complexity reigns supreme. They also vastly underestimate the role of luck and randomness. As one of my favorite authors, Nassim Taleb said, “Mild success can be explainable by skills and labor. Wild success is attributable to variance.”

Survivorship Bias.

During World War II the Allied bombers flying daylight bombing raids into Germany suffered heavy losses. Crews had a 1 in 4 chance of getting shot down and killed every time they flew. Even the planes that made it back were often riddled with bullet holes. To protect the crews, the initial thinking was to add armor plating to the areas where most of the holes were observed in the planes that limped home (the wings and tail).

The engineers made the same mistake most of us made. They were studying the success stories and trying to learn from the winners. But the data they were observing was biased in that it only contained information that survived, hence the term “survivorship bias”.

What they needed to do was study failures. The planes that got shot down were hit in the cockpit and fuel tanks, but it was hard to figure this out because these planes crashed in Europe where they could not be easily studied.

Survivorship bias is our tendency to glorify the survivors who prevailed, while ignoring the majority who failed. 

Many times “the failures” followed the same success formula as “the winners”, but they were not so lucky.

Don’t believe the hype!

Here are some examples of survivorship bias to get you thinking:

An investing course features testimonials from a handful of happy customers who “got rich” following the investing approach being sold by the company. They don’t mention the thousands of others who failed using the same method.

Mark Zuckerberg, Steve Jobs, and Bill Gates dropped out of college and became billionaires. Does this mean that if you follow their example and drop out of college you are more likely to become a billionaire too? Obviously not. It’s the same as having an uncle who smoked a pack of cigarettes every day and lived to be 90 years old. Does this mean that cigarettes are OK? Of course not. Correlation and causation are not the same thing, but people make this mistake all the time.

Jim Collins’ book, “Good to Great” sold nearly 3 million copies and shared the business practices common to 11 so-called “great” firms that had outperformed the stock market up to that point. Does this mean if you follow their example your firm will become great too?

Actually, if you invested in those 11 firms at the date the book was published in 2001 your portfolio would have subsequently underperformed the S&P 500 stock market index, and 2 of those 11 firms went on to financial ruin. These companies “survived” up until the time of Collins’ book publishing, but the practices they followed were not a magic formula that guaranteed their future success. In the meantime, thanks to his best-selling book, Collins went on to become one of the highest-paid business speakers.

Note: I agree with (most of) the principles that Collins promotes, but they fall short of being the guaranteed success formula he claims.  (On a related note, here is my critique of Jim Collins’s “bus analogy”)

The irony of this is not lost on me. I wrote my book, “Business Execution for RESULTS” based on the lessons I learned from coaching small-medium sized businesses. My book was “lucky” enough to win Best Management Book at the 2014 Small Business Book Awards in the USA. I firmly believe in the process I documented for creating and executing your strategy, but I am painfully aware of the role that industry factors, competitors, strategic bets, luck, and timing play when it comes to business outcomes.

When giving keynote presentations, I downplay any delusions of guru-ness by saying, “I encourage you to think for yourself. You don’t have to agree with everything I say.  It’s OK to disagree with me. Today I will share what I do in my work with clients and what I have learned from the most successful business leaders I’ve worked with. But I’m not here to tell you what you should do. Feel free to take this information back to your business and verify it for yourself under real-life conditions and adapt it to meet your needs”   

Another example of survivorship bias occurs with scientific journals and business schools publishing only the studies that “prove” the point the researchers wanted to make. Research and case studies that did not prove the desired point are discarded and never published. We only hear about the studies that “survived”. Just because a study got published, it does not mean you should believe it without question. 

Alarmingly, science is now facing a “replication crisis” where a large percentage of published research findings cannot be replicated or their proof is found to be tenuous at best. The scope of the issue is still unraveling, but it seems especially prevalent in the social sciences.

Let’s face it, there’s no glamour and not much funding available to reproduce or debunk someone else’s published study. This leads to published research making headlines and being left unchallenged and unverified for many years. A meta-analysis of multiple similar studies is always required to validate any scientific finding. That is what the scientific method is all about after all.

Business Thought Leaders?

In my opinion, the term “scientific management” is an oxymoron. Very little of what is taught in business schools could be rightly called science. There are pitifully few rigorous studies, let alone a meta-analysis of multiple similar studies to validate findings.

I stopped reading the latest “New York Times best-seller” business books a long time ago. I find them to be vapid and full of empty platitudes. Typically, they are best sellers because of hype and good marketing, but the mental equivalent of junk food when it comes to long-lasting educational value.

Much of what I took as gospel at the beginning of my reading journey got downgraded into utter BS as I became more educated and discerning.  My recommendation? Never rely on 1 book or guru. Read several books on a subject before “triangulating” and coming to your own conclusions. Test your conclusions to see how they play out in the real world, keep what works, and discard the rest. 

Likewise, very few Ted Talk speakers are worth listening to. A polished Ted Talk speaker can make stupid ideas sound very convincing. I find myself asking: How strong is the evidence for their claims? How many businesses have they built? How many teams have they managed? How many people have they hired? How many people have they had to fire? 

The rap duo Public Enemy sang, “Don’t believe the hype”. Wise words from Chuck D and Flavor Flav.

Don’t get me wrong. I still love reading. I get up early and read for 1 hour every morning before I go to the gym, and before I start work. But I get most of my information from blogs and podcasts now. I read fewer business books these days because it seems like an exercise in diminishing returns. 

If I do choose to read a business book, I seek out those that meet the following criteria:

  • The book has been in print for more than 2 years (even better if it is older than the last economic boom-bust cycle)
  • The author is believable (here is a good test)
  • Well-read business people whom I respect, recommend the book as a “must-read”. They share the key lessons, how the book changed their thinking, and how the concepts still apply today
  • The book has been thoroughly debated and contested by knowledgeable reviewers. There may be some controversies, but the bulk of the author’s message stands up to challenge

Unfortunately, our culture promotes flashy clickbait headlines over boring truths. Hype over substance. Fads over timeless wisdom. Quick fixes over lasting solutions. Short-term impact over long-term consequences. Soundbites over nuanced analysis. Certainty over probability. 

Beware of survivorship bias, and don’t believe the hype! To increase your odds of success, seek principles that have stood the test of time, but always remember, there are no guaranteed formulas for success.  

***

Until next time…
Stephen