I have helped businesses create, define, and display metrics throughout my career. I have always strived to ensure that they collect and process data correctly. In abstracting reality into a number that can be compared to previous periods or targets, I have emphasised that numbers are never reality itself but rather a representation of it within a specific context and based on a set of assumptions. Metrics can be beneficial, and I support using them to measure our world. However, there are important caveats that need to be addressed.
One such caveat is what Jerry Z. Muller calls “metric fixation” in his book The Tyranny of Metrics. Metric fixation is the belief that standardised metrics are not just enough to make informed decisions but are at odds with intuition, experience, and expertise - you either use metrics or gut feeling; there is nothing in-between. Metric fixation preaches that to be truly objective and data-driven, one must focus solely on the statistics of a problem. It also confuses metrics and performance, leading people to manipulate numbers to get the desired result. This can fuel the need to “torture” a number until it confesses whatever you want.
It is not a book against the use of metrics. Instead, it is against replacing human critical thinking and judgement with poorly designed and insufficient representations of the world.
The book argues that when a metric becomes the target, it ceases to be a good depiction of reality, creating opportunities for gaming, cheating, and a hyper-focus on the trends displayed by the metric. Jerry Z. Muller provides examples from various areas, such as Health, Education, Policy-making, Military and Business, to illustrate how blindly focusing on metrics can lead to unexpected consequences.
“Your Problem is That You Care”
As an analyst, I have lived my fair share of this problem, so several parts of Jerry Z. Muller’s book had me nodding in agreement.
I once worked on a Supply Chain Operations team at a Consumer Goods company. Supply Chain is a fantastic area for analytical-minded folks. There is some degree of standardisation across different industries, and you’ll often find the same KPIs (Key Performance Indicators) being measured everywhere you go. There is an association dedicated to creating and defining Supply Chain best practices, and they even have a dictionary of terminology. All in all, it sounds like a great environment to implement a genuinely data-informed operation.
Unfortunately, that’s not always the case. In low-trust culture companies, the quest for establishing metrics and baselines is often used as an accountability scapegoat, especially if tied directly to performance. If you don’t trust your teams are doing the right thing, you’ll tie them to performance metrics, and if they don’t perform according to the expectations of those numbers, they can have bonuses cut, promotions held, and salaries frozen. This usually leads to a generalised feeling of panic across the organisation. So now, alongside your low trust and lack of accountability, you’ll also have a culture that fears trying anything that can’t be directly tied to their KPIs, regardless of whether they’re actually measuring or achieving anything that matters. Throw in a few elements of data phobia and low Data Literacy levels, and it’s a recipe for disaster.
I worked as a data analyst. My job was to help establish these metrics, acquire the data, tidy it up, and create business dashboards to track them. One day, at the start of our Financial Year, our Director pulled the leadership team and a few analysts into an auditorium and ran a whole-day workshop about performance. In it, he stressed the importance of increasing one of our Key Performance Indicators - OTIF. On-Time In-Full is a typical Supply Chain Metric. In a nutshell, it tracks the number of deliveries that operations managed to get to the client on time and in full. Not more, not less. Not a different product. Not too early and not too late. The baseline recommendation is that a healthy Supply Chain can accomplish 97% of all deliveries On-Time and In-Full.
During the workshop, our Lead Team stressed that concept and its importance in guaranteeing good Customer Service levels. But then they said something that had catastrophic results down the line. They established that until the end of the Financial Year, our objective was to raise our current average OTIF score from around 80% to 95%. That was the main target for our operations teams: to increase the percentage. There was no mention of making sure we didn’t miss orders, or to improve our shipping lead times, or to improve the flow of stock in our distribution centres to make sure orders could always be fulfilled. The focus for the next 12 months was to raise an average percentage. They also tied this target to the performance bonus of a few teams.
A few months down the line, my team’s work had become close to a living hell. Every morning we had to churn out a report with the latest on OTIF. Forget the other reports. OTIF is the main one that must come out daily. Everyone was happy if the OTIF percentage went up from the previous day. We’d see dragons at the gates setting everything on fire if it dropped. Not only that, the requirements for changing the report and the metric started pouring like crazy. And they weren’t reasonable requirements either. In no time, we had a butchered metric. We had extended the tolerance times for what we considered On-Time. We lowered the target from 95% to 90%. We filtered out specific orders that were a bit too likely to fail either criteria. If you think this is fudging numbers, it’s because it is! And it gets worse!
Suddenly, we had a whole team dedicated to the task of Reason Coding. In the name of accountability, we now not only tracked the percentage of deliveries that arrived On-Time and In-Full, but we had a whole list of reasons for the deliveries that didn’t make it. We called them failures. A team was set to spend time scrubbing the complex history of a delivery’s path through operations to establish why it failed. But the reasons weren’t operationally based; they were all tied to different teams. The delivery didn’t fail for arriving too late or for missing a box. It failed because of Manufacturing. Or it failed because of Planning & Scheduling. Or it failed because of Logistics Operations. It was madness, institutionalised, systematised, and delivered daily with colourful bar charts.
At a certain point, the attention turned to us, the Reporting Team. Of course, if we couldn’t pinpoint who was at fault (not what, who), the fault must be us, the Reporting Team. Everyone seemed sure we were reporting it wrong. Well, to be blunt: we were. Not for the lack of trying to report it correctly, but mostly due to the constant pressure to “make the number look good, or we’ll all be doomed!”. We suddenly got tasked with “reinventing OTIF”, which (as someone who still cares about doing the right thing, despite the odds) I took as a call for help to kindly point out what we could do to get out of this mess of our own making.
We tried. Our proposal was to go back to a cleaner, leaner metric - less abstract and more closely tied to supporting metrics like missed orders, deliveries at risk, average shipping times, and stock levels. The idea didn’t go far. In part because everyone was already exhausted from the statistics discussions and in part because driving meaningful change in an environment that has just spent almost a year living and breathing a number is scary, disruptive, and too complicated. It got to a point where my manager signalled us to drop it, as he accepted that the poor analyst in my team, who was running this cursed report, had now been elected as the official scapegoat of all things wrong with OTIF results. It was soul-crushing.
“Your problem is that you care”. This was the end-of-year performance feedback I received from my manager after months of insisting we should make an effort to take lessons learned from all this and try to start fresh. It was gut-wrenching. We could probably have come out the other side better, but apathy took over. Everyone was exhausted, and, in the end, despite all efforts to drive the average percentage of OTIF up, we closed at a level lower than when we had started.
It’s been years, but this experience is still stuck with me, and I think it’s important to share. I am sure someone else out there is going through a similarly dreadful experience right now. I learned heaps from this. And now, I have my head cool enough to learn from this chapter of my career and share, so you may be able to care without it being a problem.
The Tyranny of Metrics
The book goes through similar examples and discusses how ignoring essential human critical input can turn metrics into a tyrannical beast, to which sacrifices must be made in the name of a mythical performance level. It is a book written by a historian, so while it goes in-depth into the origins of the phenomenon of metric fixation and how it became rooted in our society, it doesn’t offer your easy-to-follow step-ified process to fix all ailments. It’s not a bad thing, just atypical from your regular business guidebook.
It can also be a somewhat challenging read, especially if you fall into the cult-like metric-loving spectrum. One of the greatest things about the scientific method is to challenge yourself with alternative perspectives to try and refute your hypothesis. Take this book as part of that thorough experiment. It had me disagreeing with the author at times. I have dedicated my life to calculating and reporting metrics, and there are good things to be taken from them. But it’s good to see what’s on the other side of what I do: the perspective of a non-data person that sees the world from a non-data lens.
At the end of the book, Jerry Z. Muller offers a helpful checklist on how to make good use of metrics. It serves as a good reference guide to critically judge if the application of your metrics is appropriate or if you may be accidentally falling for the traps of metric fixation.
What kind of information are you trying to measure? When the thing you are trying to measure is influenced by the process of measuring, your metric becomes unreliable. It is the case when rewards or punishments are tied to behavioural metrics: people will act to fulfil the metric instead of addressing the process or object of the metric.
How useful is the information? Just because something is easy to measure doesn’t mean it should be measured. The opposite is also true: we tend to avoid more complex metrics and topics even if they’re better representatives of reality and more valuable to drive the change we require.
How useful are more metrics? Hoarding all the data you can and reporting a sea of metrics sounds like a reasonable effort to be more data-informed. Still, the opposite is true: focus on what matters instead of dividing your attention across multiple KPIs and measures that will require more effort than will bring benefits.
What are the costs of not relying upon standardised measurement? This may be challenging, but is human experience, expertise, and the judgement of those more involved in the process more relevant than establishing standardised metrics? This may be the case when dealing with high-urgency situations, where waiting for a metric to be run would result in wasting precious time.
To whom will the measure be made transparent? For example, will crime statistics be used to allocate funds to certain regions, or will it be used to decide whether a precinct gets more bonuses or promotions? Not all measures should be tied to performance to prevent the risk of gaming.
What are the costs of acquiring metrics? And by this, the author means mainly time cost. How long are you and your teams spending discussing the measure at the expense of the time used to examine the object of the measure? Or, in other words, how much time are you spending on the metric that should instead be going to the very thing you wish to measure and improve?
Ask why the people at the top are demanding performance metrics. Achieving a goal aligned with a strategy is an appropriate use of metrics, but often metrics are created for vanity purposes or to micro-manage processes that are inconsequential to strategy. Always ask why.
How and by whom are the measures developed? Measures tend to be less effective the farther they are from the actual process being measured. Are the metrics developed alongside experts, or are they cookie-cutter abstractions that do not translate to the process or object being measured?
Remember that even the best measures can be subject to corruption and goal diversion. Tying performance to a metric instead of an outcome and turning the measure into the target is a surefire way to diverge it from its real purpose.
Quoted straight from the book: “Remember that sometimes, realising the limits of the possible is the beginning of wisdom. Not all problems are soluble, and even fewer are by metrics. It’s not true that everything can be improved by measurement or that everything that can be measured can be improved.”
The one thing I'm afraid I have to disagree with is his take on transparency. I understand the argument for privacy and how it’s necessary to keep our society functioning in a healthy way. I, too, am big on privacy, and I, too, disagree vehemently with the invasion of the deepest areas of our lives in the name of metrics. But the author seems to confuse the line between privacy and secrecy of public information at times. As Noah Chomsky says, “Power prefers darkness. If it’s exposed to light, it evaporates”. I am a staunch supporter of transparency in Public Institutions and government, and I personally see whistleblowing as a driver of much-needed change. There is a part in the book where he argues against it, and it had me feeling that physical manifestation of discomfort and pain we have when our strongest beliefs are challenged: cognitive dissonance. I see the argument's value, though, and I do not believe that disagreeing with one point of the book invalidates its importance. It is still a fantastic read.
Should you read it?
We live in a world of metrics. More and more information is gathered, wrangled, analysed and converted into metrics to assess and define how we live, work, consume, elect our governments and behave as a society. Yet, while extremely useful and influential, metrics are complex and can easily diverge from their original objectives.
The Tyranny of Metrics is a book meant to raise questions and help us put a name to the problem so that we can start thinking about how we can fix it. Metric fixation is an important concept for anyone reading, creating or working with data nowadays. More and more, it dawns on us the need to critically look at what we are doing with all the data available to us, what incentives we are promoting, and what impact we are having in our world and society because of it. I highly recommend it.
Always check your local library first to see if any of the books I recommend are available. If they’re not, consider donating a copy!
Get a copy at your local library | Amazon
If you subscribe to my monthly Newsletter, you’ll get a summary of all recommendations, plus more of my data viz musings.
You can also follow Data Rocks on LinkedIn