KPI best practices from Amazon

KPI Best Practices from Amazon

I have written a number of articles on Metrics aka Key Performance Indicators aka KPIs. They form a major part of my consulting work and are something I focus on early in my client engagements to get them dialed in.

I enjoyed this article from Cedric Chin, “This is How Amazon Measures Itself“, which summarizes a chapter from the book “Working Backwards“, a book about how Amazon works internally. I was pleased to read that it reinforced the principles I teach my clients, whilst adding additional examples and nuance. I particularly like the way Amazon graphs their metrics using different time periods and perspectives to help analyze the data (see images below)   

***

This is How Amazon Measures Itself

“Amazon divides metrics into two types: controllable input metrics and output metrics. This is more commonly known in the industry as leading indicators and lagging indicators, but Amazon likes to use their own language. But I think ‘controllable input metrics’ is a particularly nice way of putting it: it makes it really clear that a leading indicator is only worth paying attention to if it’s also controllable.”

“First, it defines and tweaks each metric according to a particular metric lifecycle. Second, it presents its metrics in something called a ‘Weekly Business Review’ meeting, or a WBR meeting – which is fractal: top leadership does a full-company WBR every week, followed by every department and operational team on down.”

Amazon’s Metrics Lifecycle – DMAIC

“They run a process improvement method called DMAIC, which they copied from Six Sigma. The acronym stands for: Define, Measure, Analyze, Improve, and Control.”

Define:

“The first thing that Amazon does is to figure out what the correct, controllable set of input metrics are. This is deceptively tricky, and requires a repeated trial and error process.”

Measure:

“First, removing bias in your metrics is incredibly important — and necessary, if you want to uncover the ground truth of your business. Amazon empowers its finance team to uncover and report the unbiased truth. They do this because business unit leaders are incentivized to choose metrics (or tweak metrics!) to make themselves look good.”

“Second, plan to audit your metrics. Amazon requires its metric owners to have a regular process to audit metrics, to ensure the metric is measuring what it’s actually supposed to be measuring. The base assumption here is that, over time, something will cause your metric to drift, and therefore your numbers to skew.”

“Third, take the time and make the investment to instrument your business. The authors make the point that you want to instrument the right thing, whatever that is for the business – and sometimes the right thing is the more difficult thing!”

Analyze:

“This is the stage of the metrics lifecycle where you develop a comprehensive understanding of the underlying drivers behind the metrics.”

“If you observe a metric behaving strangely, it’ll take a bit of time to figure out what’s driving that behavior. Like Toyota, Amazon uses the ‘five whys’ method to get to the bottom of anomalies.”

“For every new metric that you define, there will be a period where you have to develop a deep understanding of how the metric works, what the root causes are, what the natural variances look like, and so on.”

Improve:

“The reason ‘improve’ comes after ‘define, measure, and analyze’ is that now, you’re going to be making changes on a solid foundation of understanding. Amazon has had departments who have attempted to improve their processes without a full define, measure, and analyze loop. This has nearly always resulted in a lot of thrash, with little to no meaningful results.”

“The authors note that if you improve your process over time, it is possible for a previously useful metric to stop yielding useful information. In such cases, it is totally ok to prune it from your dashboards.”

Control:

“Finally, a metric enters the steady-state control phase. This stage is all about ensuring that your processes are operating normally and performance is not degrading over time. In some Amazon teams, metrics are so well controlled and processes are so smooth that the WBR becomes an exception-based meeting instead of a regular meeting discussing each and every metric. People meet solely to discuss anomalies.”

“Another thing that happens in the control stage is that operators may be able to identify processes that may be automated completely. After all, if a process is well understood and the decisions are predictable, then it is likely that the entire process may be replaced with software. Amazon’s forecasting and purchasing are two examples where the processes are now completely automated.”

How Amazon Uses Metrics

“Amazon uses metrics by reviewing them in what is called the ‘weekly business review’ meeting, or the WBR. Metric owners watch metrics daily. They are expected to know what is normal variance and what is an exception, in order to save time during the WBR.”

“Emerging patterns are a key focus. You want trend lines, and you want to know them long before they show up in a quarterly or yearly result.”

“Graphs are usually plotted against a comparable prior period. Metrics make sense when compared against prior periods, so that you have a proper apples-to-apples comparison (for instance, you’ll want to compare holiday periods to a prior holiday period, not to a slow period).”

“Graphs show two or more timelines, for example, trailing 6-week and trailing 12-months. Small but important issues tend to only show up in shorter trend lines; they tend to be smoothed out in longer ones.”

“Metrics are certified accurate by the finance department.”

Running the WBR (Weekly Business Review)

“The weekly cadence guarantees a number of things. It guarantees that managers are aware of issues as quickly as possible. It guarantees that they have continuity from one WBR meeting to the next.”

“Metrics are formatted in a consistent and familiar way. This formatting means that Amazon leaders are able to look at the same set of data every week, with exactly the same format, in exactly the same order, in order to walk away with a holistic end-to-end perspective of the business. Over time, this familiarity results in a shared ability to spot trends, pick out anomalies, and settle into a consistent review rhythm. The WBR should, therefore, become more efficient over time.”

“WBR meetings focus on variances and ignore the expected. WBR time is precious. If things are within expected variances, business owners say “nothing to see here” and move along. The goal of the meeting is to discuss exceptions and what is being done about them.”

“Business owners own metrics and are expected to explain variances. While Amazon’s finance team is responsible for certifying results, the presentation of each metric is solely the responsibility of the business owner in question. The business owner is expected to know their metrics inside and out; by the time they attend the top-level WBR, they should have an explanation (or at least the results of a preliminary investigation!) to explain an anomaly.”

“Business owners who haven’t done their work before the WBR get chewed out. If they don’t know the causes of an anomaly, they are expected to say ‘I don’t know, we’re still analyzing the data and we’ll get back to you.’ Making a guess, or making things up, will also result in a chewing out.”

“Operational and strategic discussions are kept separate. WBR time is precious. It is a tactical operational meeting, not a strategic one. New strategies, product updates, and upcoming product releases are not allowed during the meeting.”

“Amazon also has several interesting practices around its data presentation: Amazon displays weekly and monthly metrics on a single graph. As mentioned above, Amazon displays trailing 6-week and trailing 12-months. The net result of presenting metrics like this is that the graph looks like a ‘zoomed-in’ version of the same data.”

Take this graph, for instance:

In the graph (above), the gray line is the prior year, the black line is the current year
The left graph, those first 6 data points, shows the trailing 6 weeks
The right graph, with 12 data points, shows the entire trailing year month by month
This built-in “zoom” adds clarity by magnifying the most recent data, which the 12-month graph puts into context.

Amazon Watches Year-over-Year Trends

(see graph below):

“The graph compares actual monthly revenue against both planned revenue and prior year revenue. The graph seems like you are beating the plan and growing at a decent clip year over year…”

But:

“Without the YOY Growth % dotted line, you might not notice the current and projected year trends slowly converging”

“Output metrics show results, input metrics provide guidance. The graph above is an example of an output metric. It serves as a good reminder that output metrics are not actionable – for instance, it is not enough to know that YOY growth has decelerated; you also want to know which factors contributed to the deceleration.”

“Just think of a business as a process. It can be a complicated process, but essentially, it spits up outputs like revenue and profit, numbers of customers, and growth rates. To be a good operator, you can’t just focus on those output metrics – you need to identify the controllable input metrics.”

“A lot of people say that Amazon doesn’t really care about profit or growth. I think that the data say otherwise, but what is true is that the main focus is on those input metrics, if you do the things you have control over right, it’s going to yield the desired result in your output metrics. The best operators I’ve seen very clearly understand that if they push these buttons or turn these levers in the right way, they’re going to get the results they want. They understand that process through and through.”

***

For more information on this topic see my articles:

How to Identify Your Metrics (Key Performance Indicators)

Benefits of Having the Right Metrics

The Dark Side of Goal Setting

How to Set SMART Goals the Right Way

***

Until next time…
Stephen