On the Quality of KPIs

In this second post on KPIs, I will look at the quality of key performance indicators (KPIs). These are essential tools for measuring an organization’s progress toward its strategic goals. The quality of these indicators is critical as it impacts management’s decision-making and subsequent actions. Let’s have a look!

Quality of the KPI

Another important aspect is the quality of the KPI. A measurement is only useful if it is accurate enough, relevant, and measures what we actually want to know; its complexity is manageable; and its definition is correctly understood by the person using the KPI. In all aspects I have experienced examples of good intentions gone wrong.

Accuracy

Lets start with accuracy: KPIs can be wrong, sometimes by accident, sometimes intentionally. A wrong measurement is sometimes more dangerous than no measurement. For example, I have seen a board presentation with the quarterly performance KPI on profit. These were outstandingly good, and management believed it to be a very good quarter. Unfortunately, the person calculating the data mistakenly summarized the profit over four months (presumably because a “quarter” is somewhat connected to the number four), whereas a quarter year usually includes only three months. No wonder that the “quarterly” profit was 30% higher than expected.

It is estimated that 94% of all spreadsheets contain errors (source: Pak-Lok Poon et al, Spreadsheet quality assurance: a literature review, Frontiers of Computer Science (2024). DOI: 10.1007/s11704-023-2384-6). There are plenty of examples of companies losing billions of dollars due to simple spreadsheet mistakes (source: Million Dollar Mistakes: Real-World Risks of Spreadsheet Errors). In biology, they even renamed some genes due to Excel auto correcting (e.g.,  MARCH1 (for “Membrane Associated Ring-CH-Type Finger 1”), SEPT1, and others into dates.

One great way to see whether a KPI is actually close to the truth is to go to the shop floor, or—in Japanese terms—Go To the Gemba! Go to the shop floor (or wherever the KPI originates) at least sometimes to verify if the KPIs reported to you are reflected in the shop floor performance. If the OEE is supposedly 85%, and the machine is not running, then there may be something amiss.

Relevance

Some KPIs may also give us information that sounds important but is actually misleading, and does not tell us what we want to know. This is especially critical if the targets for the KPI lead to a local optima, but degrading overall plant performance. One example of such a misleading KPI on the shop floor is the measurement of production efficiency based solely on output quantity without accounting for quality. Suppose a manufacturing plant sets a KPI that aims to maximize the number of units produced per hour. While this metric might encourage workers to increase their production speed, it can inadvertently lead to a compromise in quality if employees prioritize quantity over precision. As a result, the plant might experience a spike in defective products, which could ultimately necessitate rework or result in customer dissatisfaction and returns.

While this sounds obvious, it has happened, albeit it is nowadays less common. Equally misleading but still frequently found are KPIs on utilization. Many managers want their machines to run a lot, especially the expensive ones, to get a return on their investment. However, if the customer does not need the products, you are just building up inventory (I have seen this happen, and the company built a second large warehouse to store all the products before they figured out that they were not selling). Even if they are selling, a high utilization automatically causes a high inventory and high lead time. Your machines are running, yes, but your system is clogging up. For more, see my blog post on The Kingman Formula.

Complexity

A manageable complexity also sounds easy, as any spreadsheet can easily do all the math for even the most complex KPI. However, the more calculation you have, the more things can go wrong. My two anti-favorites in terms of complexity are the OEE and the delivery performance, but there are plenty more. Both require quite a few measurements and calculations. This gives lots of opportunity to make errors, intentionally or not. As for intentional number fudging, I even wrote a whole satirical blog post on How to Fudge Your OEE. The delivery performance is also often twisted to make it look good. I had plenty of examples where the supplier measured a 95% on-time-in-full (OTIF) delivery performance, but the internal measurement of the customer receiving the goods showed only 30% delivery performance.

On a side note, the more complex a KPI is, the harder it is to see on the shop floor. Simpler KPIs are easier to verify, while difficult KPIs are easier to manipulate.

Understanding

This manageable complexity is closely related to understanding the KPI. I have my doubts whether every manager understands what is really behind every KPI that is reported to them. For example, if it is a cost-related KPI, what does it include? Is it labor, materials, machines, overhead, etc.? Is it before tax; after tax; earnings before interest taxes, depreciation, and amortization (EBITDA); or anything else. The OEE, utilization, and up time are also often confused, as are cycle times and takt times. If management wants a number, they will get a number, but it is not always clear if the number is understood as intended.

Missing Out on Soft Factors

KPIs tell us a lot. We can’t manage a modern factory without KPIs. However, it would be foolish to believe that everything can be captured in a KPI. Especially in lean, there are many factors that are hard or impossible to measure. For example, increasing fluctuations are bad for your production system. However, nobody can really say exactly how good or bad they are. Similarly, customer satisfaction and employee morale are important, but hard to measure.

Unfortunately, a lot of the world is run by accountants. And anything they can’t measure is zero. Hence, for an accountant there is no value in reducing fluctuations, only cost.

Now, go out, make sure the KPIs you have are actually good, and organize your industry!