Survivorship Bias – Don’t Believe the Hype

Survivorship Bias – Don’t Believe the Hype. 

I remember the first personal development book I ever purchased. It was called “The Magic of Thinking Big” by David Schwartz. I was going through a low point in life and the promise on the book jacket jumped off the bookshelf: “Millions of readers have acquired the secrets of success through The Magic of Thinking Big. Achieve everything you always wanted: financial security, power and influence, the ideal job, satisfying relationships, and a rewarding, happy life.”

Sold! Who wouldn’t want to achieve everything they have always wanted? Up until then, it never occurred to me there was a “formula for success”. What a revelation. All I had to do was learn this magic formula and apply it. It’s a compelling story that we all want to believe.

That book was my gateway to consuming hundreds of books, videos, audios, and courses to help me achieve my goals.

I was a competitive bodybuilder in my twenties (I won the Mr. New Zealand bodybuilding title in 1993) and as I was coming up through the ranks I devoured all the magazines and books by Arnold Schwarzenegger, Joe Weider, Robert Kennedy etc.

Then there were the “personal development gurus”; Brian Tracy, Zig Ziglar, Jim Rohn, Tony Robbins etc, and books on how those “Millionaires next door” came to be that way.

Later on, came the “business gurus”; Jim Collins, Tom Peters, Malcolm Gladwell, Seth Godin etc, the biographies of successful entrepreneurs, and the research from Stanford and Harvard. I tried to devour pretty much all of the best selling books and business magazines you see in the airport bookstores.

It quickly dawned on me that all these gurus, books, and magazines follow a similar approach. Observe what successful people do and try to reverse engineer it. Package it up and sell it as a guaranteed “success formula” that will transform your life.

The success formulas are pretty similar too, whether it is for business, sports, or just life in general: Visualize the future you want. Set lofty goals. Copy the strategies used by the winners. Model the behaviors used by the winners. Be “bold”. Work hard. Put in 10,000 hours. Track your progress. Never quit. Keep believing. Eventually, you will win!

The general principles seem plausible. Of course, learning the core skills and working hard is important in any endeavor. But after a while, it dawned on me. The authors were glossing over a factor that plays a critical role in success.

Life is not fair.

As a novice bodybuilder, I studied and followed the training routines and diets of the champions. I trained harder than most, even to the point of fainting after workouts. I became stronger than most. I ate cleaner foods than most.

But here’s the big lesson, so did everyone else in my peer group. Actually, some of my friends in the gym trained even harder than me. Some were stronger. Some ate cleaner diets. Some followed the so-called “success formula” espoused in the books more diligently than me.

Yet, I won the Mr. NZ title. You see, in bodybuilding, no amount of training or diet can change your bone structure or the shape of your muscle bellies. I was lucky enough to have genetics that lent themselves to success in that sport.

I was the lucky winner who got the sponsorships, featured on the posters, appeared in TV interviews, and was asked to give training seminars. If I chose to write a book on “how to succeed in bodybuilding” it would contain practical advice, sure, but there is no way I could guarantee your success if you followed the same path I did.

In any endeavor it is the “winners” who write the books, get interviewed in the media, and get asked to give public talks. However, the winners have less to teach us than you might think.

Bodybuilding is not fair. Neither is business. Neither is life. No matter how much you want something, how hard you work, or how diligently you follow the “success formula”, success is never guaranteed. Our genetics, country of birth, gender, skin color, parenting, poverty level, neighborhoods, schools, social contacts, luck, and timing play a far bigger role in outcomes than the gurus would have us believe. They peddle simplicity and certainty, in a world where complexity reigns supreme.  

As one of my favorite authors, Nassim Taleb said, “Mild success can be explainable by skills and labor. Wild success is attributable to variance.”

Survivorship Bias.

During World War II the Allied bombers flying daylight bombing raids into Germany suffered heavy losses. Crews had a 1 in 4 chance of getting shot down and killed every time they flew. Even the planes that made it back were often riddled with bullet holes. In order to protect the crews, the initial thinking was to add armor plating to the areas where most of the holes were observed in the planes that limped home (the wings and tail).

The engineers made the same mistake most of us make. They were studying the success stories and trying to learn from the winners. The data they were observing was biased in that it only contained information that survived, hence the term “survivorship bias”.

What they actually needed to do was study failures. The planes that got shot down were being hit in the cockpit and fuel tanks, but it was hard to figure this out because these planes crashed in Europe where they could not be easily observed.

Survivorship bias is our tendency to study the people or companies who “survived” or were victorious in a certain situation while ignoring those that failed. In many endeavors, it is probable that the failures used the exact same formula as the winners, but they still failed.

Don’t believe the hype!

Here are some examples of survivorship bias to get you thinking:

e.g. A real estate investing course features testimonials from a handful of happy customers who “got rich” following the investing approach being sold by the company. They don’t mention the thousands of others who failed using the same method.

e.g. Mark Zuckerberg, Steve Jobs, and Bill Gates dropped out of college and became billionaires. Does this mean that if you follow their example and drop out of college you are more likely to become a billionaire too? Obviously not. It’s the same as having an uncle who smoked a pack of cigarettes every day and lived to be 90 years old. Does this mean that cigarettes are not bad for you? Of course not. Correlation and causation are not the same things, but people make this logic mistake all the time.

e.g. Jim Collins’ book, “Good to Great” sold nearly 3 million copies and shared the business practices common to 11 so-called “great” firms that had outperformed the stock market up to that point. Does this mean if you follow their example your firm will become great too?

Actually, if you invested in those 11 firms at the date the book was published in 2001 your portfolio would have subsequently underperformed the S&P 500 stock market index, and 2 of those 11 firms went on to financial ruin.

Just because these companies “survived” up until the time of Collins’ book publishing, the practices they followed were not some magic formula that guaranteed their future success. In the meantime, thanks to his best-selling book, Collins went on to become one of the highest-paid business speakers. (On a related note, here is my critique of Jim Collins’s “bus analogy”.)

The irony of this is not lost on me. I wrote my book, “Business Execution for RESULTS” based on the lessons I learned from coaching small-medium sized businesses. My book was “lucky” enough to win an award (best Management book at the 2014 small business book awards in the USA). I remain a firm believer in the process I documented for creating and executing your strategy, but I am fully aware that many other factors contribute to business success that cannot be reverse-engineered.

e.g. Scientific journals and business schools publish only the studies that “prove” the point the researchers wanted to make. Research and case studies that did not prove the desired point are discarded and never published. We only hear about the studies that “survived”. Just because a study got published, it does not mean you should believe it without question.

Alarmingly, science is now facing a “replication crisis” where a large and increasing percentage of published research findings cannot be replicated. The scope of the issue is still being unraveled, but it seems especially prevalent in psychology, medicine, pharmaceuticals, economics, marketing, and sports science.

Bodybuilding got me interested in health and nutrition. Over the last 3 decades, I’ve seen doctors and sports coaches swerve wildly from recommending “lots of aerobic exercise, eat complex carbs, and avoid fatty foods” in order to achieve optimal health, to the current recommendation of “high-intensity interval training, eat high fat, and avoid carbs”. What to believe?

Let’s face it, there’s no glamour and not much funding available to reproduce or debunk someone else’s published study. This leads to published research making headlines and being left unchallenged and unverified for many years. A meta-analysis of multiple similar studies is always required to validate any scientific finding. That is what the scientific method is all about after all.

Unfortunately, our culture promotes flashy clickbait headlines over sobering truths. Hype over substance. Quick fixes over lasting solutions. Short-term impact over long-term consequences. Soundbites over nuanced analysis. Certainty over probability. Fads over timeless wisdom.

The more I look into the current scientific replication crisis, the less I believe what I see in the media, and the less I trust what the so-called experts try to tell me or sell me e.g. doctors, economists, nutritionists, business schools, self-help gurus, and business thought leaders. Don’t even get me started on politicians and what passes for “news” and “truth” these days.

Business Thought Leaders?

In my opinion, the term “scientific management” is an oxymoron. Very little of what is taught in business schools could be rightly called science. There are pitifully few rigorous studies, let alone a meta-analysis of multiple similar studies to validate findings.

I stopped reading the latest “New York Times best-seller” business books a long time ago. I find them to be vapid and full of empty platitudes. Typically, they are best sellers because of hype and good marketing, but the mental equivalent of junk food when it comes to long-lasting educational value. Likewise, very few Ted Talk speakers are worth listening to.

The rap duo Public Enemy sang, “Don’t believe the hype”. Wise words from Chuck D and Flavor Flav.

Don’t get me wrong. I love reading (on many subjects). I get up early and read for 1 hour every morning before I go to the gym, and before I start work. I get most of my information from blogs and podcasts now. I read fewer business books these days because it seems like an exercise in diminishing returns, but I am still keen to glean pearls of wisdom wherever I can. 

If I do choose to read a business book, I seek out those that meet the following criteria:

  • The book has been in print for more than 2 years (even better if it is older than the last economic boom-bust cycle)
  • The author is believable (here is a good test)
  • Well-read people whom I respect, recommend the book as a “must-read”. They share the key lessons, how the book changed their thinking, and how the concepts still apply today
  • The book has been thoroughly debated and contested by knowledgeable reviewers. There may be some controversies, but the bulk of the author’s message stands up to challenge

In summary, I’m seeking timeless wisdom. Wisdom that speaks to human nature and how our world really operates. Principles that have stood the test of time, and will likely continue to stand the test of time. Principles that could even be called “anti-fragile” (thanks Nassim). 

Alternatively, if I’m not absorbing timeless wisdom, I enjoy being exposed to novel ways of looking at the world that challenge my paradigms and make me think.


Until next time…