Statistical measurements, usually called key performance indicators (KPIs) are found on pretty much every shop floor and in every company. Many management decisions are made based on KPI. Unfortunately, these numbers often are not reliable at all.
Mark Twain popularized the phrase “Lies, damned lies, and statistics.” Winston Churchill famously said, “I only believe in statistics that I doctored myself.” Hence, both men were wary of trusting numbers. You should be too!
Since many people’s careers depend on good KPI, there is a large temptation to fudge the numbers, and hence they are frequently fudged. The higher up you go in hierarchy, the less you can rely on the numbers. Additionally, these KPIs often are not only incorrect, but much waste goes into manipulating these numbers. In this post I will give several different examples of manipulated KPI. In the next post I will discuss the negative effects of this KPI fudging. In a final post I will give some tricks on how to avoid or at least reduce the number fudging.
Common KPI Examples from Industry
OEE and Utilization
Industry is ripe with examples of numbers that have long since lost their meaning. I blogged before about my Top Three Methods on How to Fudge Your OEE. Personally I do not believe any utilization or OEE (Overall Equipment Efficiency) unless I have measured it myself. I regularly see OEEs above 90%, but when I measure them myself they turn out to be 60%. I even have seen some OEEs above 100%, meaning they produced more than the theoretical limit of 100%.
Another popular KPI in industry and therefore a popular KPI to manipulate is delivery performance, measuring the percentage of customer deliveries provided in full on time. Most plants I have seen regularly have delivery performances above 90%, except nobody creating these numbers really believed them. One plant once made the mistake of asking their customer what the delivery performance was on the receiving side. While the plant measured 90%, the customer measured only 30%. Clearly, something was amiss.
Inventory and WIP
Yet another common KIP is inventory, especially if related to lean manufacturing. One company I know measured these inventory levels on a monthly basis, always at the end of the month. It looked like the graph on the left.
At one point they decided to try to measure the inventory levels on a daily basis. Not all inventories but way too many showed a curve similar to the graph on the right. Miraculously, the inventory levels always dropped significantly at the end of the month. Again, was this coincidence, or was this an intentional effort to fudge the numbers? (Hint: It was not a coincidence!)
To continue with popular KPIs, quality is also often measured. One manager producing dishwasher tabs made the mistake of starting to correctly measure the number of broken tabs in a factory where defective products were notoriously under-reported. Defect rates increased by factor of 10. More precisely, the defect rates always were there, but now the reporting matched the actual quality.
Of course, the new numbers destroyed the quality KPI of the entire plant. The problem, however, was solved quickly: The manager in charge was fired, people were sternly reminded about the quality targets, and hence reported quality rates returned to normal. Of course, quality did not improve at all in reality. (Note: Normally I cannot go into that much detail due to confidentiality, but this example is published. See below for the source.)
Probably the most important KPI in industry is cost. Unfortunately, the true cost of a product is very difficult to calculate. While work hours can be measured and calculated easily, how do you measure customer satisfaction if he gets his product on time – or not? Standard bookkeeping often has some assumptions that make a product cheaper or more expensive than it really is, especially when comparing in-house with external suppliers.
I have seen examples where in-house suppliers could calculate their offers with 100% efficiency, but received payments according to their true efficiency. Naturally, the offer of the external supplier could not compete, even if they would have been cheaper at the end. I have also seen the opposite, where the in-house manufacturer had to account all its old and rarely used equipment into the cost, which of course could not compete with the external supplier. Hence they ended up producing all the parts that no external supplier was interested in – the low-quantity, high-variety, and tricky products, which made them even more expensive.
Overarching Meta Performance Indicators
Yet some other companies have an overarching meta performance indicator to measure the performance of their plant. Throughout the years these meta-KPI continuously increased. It looked like the plants of these companies became better every year. Except the bottom line did not improve at all. There was no connection between the trends of the KPI and the money left at the end of the day. People just got better at fudging the numbers.
Some More Examples Outside of Industry
I’m sure you can find many more examples in your own industry. Of course, the problem is not only in industry, but can also be found elsewhere. Governments are also well known for number fudging.
For example, in the US, students are scored using standardized testing. Naturally, there are tons of reports on manipulations where, for example, teachers gave the answers to the students or corrected the tests before they were graded.
Financial and Economic Indicators
Economic growth and financial stability is also a measure of pride for many nations – and some are more proud than what reality would dictate. The growth rates of China are probably inflated. Greece joined the European currency based on highly questionable financial indicators – and now probably wishes they would not have done that. But somebody surely made a career with those numbers.
Production rates are also often manipulated. Socialist and communist nations especially excelled at this. In the Soviet Union, rather than reporting total quantities, the quantities of one shift or one worker was reported.
One particular individual, Aleksei Stakhanov, Hero of Socialist Labor, was reported to have mined 102 tons of coal in one shift, about 14 times his quota. Later he even exceeded that by mining 227 tons of coal in one shift. This was soon followed by similar over-achievements by other workers and named the Stakhanovite movement. During the great famine in China (1958–1961), farming yields were exaggerated up to the point of covering the field four feet deep with its harvest. Of course, neither Aleksei Stakhanov nor the fields in China were ever anywhere near these production rates.
Unemployment is also something that governments prefer not to have, or at least to have as little of as possible. Here, too, it is easier to fudge the numbers than to actually find jobs for people. In Australia, if you work one hour per week you are counted as employed. In Germany, lots of unemployed people are parked in otherwise almost useless “Job Training Programs,” in which (of course) they are no longer counted as unemployed.
These are only a few examples. Depending on which countries or which industry you are familiar with, you probably can easily add many, many more. The key statement is:
Be very, very wary of any kind of numbers
if you have not measured and calculated them yourself!
In my next post I will discuss the negative effects of these fudged KPIs. More importantly, in a third post I will also show some ideas on how to reduce this negative effect of KPIs. In the meantime, go out and organize your industry, but do not believe only the numbers!
- Lies, Damned Lies, and KPI – Part 1: Examples of Fudging
- Lies, Damned Lies, and KPI – Part 2: Effects of Fudging
- Lies, Damned Lies, and KPI – Part 3: Countermeasures
Roser, Christoph. “Richtig Messen – KPIs Zum Nutzen Des Unternehmens Einsetzen.” Yokoten 5, no. 1 (2016): 26–29.
The dishwasher tabs example is from Konsequent – Das Buch zum NTT by Thomas Hochgeschurtz; ikotes, 2009.