Statistical measurements, usually called key performance indicators (KPIs) are found on pretty much every shop floor and in every company. Many management decisions are made based on KPI. Unfortunately, these numbers often are not reliable at all.
Mark Twain popularized the phrase “Lies, damned lies, and statistics.” Winston Churchill famously said, “I only believe in statistics that I doctored myself.” Hence, both men were wary of trusting numbers. You should be too!
Since many people’s careers depend on good KPI, there is a large temptation to fudge the numbers, and hence they are frequently fudged. The higher up you go in hierarchy, the less you can rely on the numbers. Additionally, these KPIs often are not only incorrect, but much waste goes into manipulating these numbers. In this post I will give several different examples of manipulated KPI. In the next post I will discuss the negative effects of this KPI fudging. In a final post I will give some tricks on how to avoid or at least reduce the number fudging.
Common KPI Examples from Industry
OEE and Utilization
Industry is ripe with examples of numbers that have long since lost their meaning. I blogged before about my Top Three Methods on How to Fudge Your OEE. Personally I do not believe any utilization or OEE (Overall Equipment Efficiency) unless I have measured it myself. I regularly see OEEs above 90%, but when I measure them myself they turn out to be 60%. I even have seen some OEEs above 100%, meaning they produced more than the theoretical limit of 100%.
Another popular KPI in industry and therefore a popular KPI to manipulate is delivery performance, measuring the percentage of customer deliveries provided in full on time. Most plants I have seen regularly have delivery performances above 90%, except nobody creating these numbers really believed them. One plant once made the mistake of asking their customer what the delivery performance was on the receiving side. While the plant measured 90%, the customer measured only 30%. Clearly, something was amiss.
Inventory and WIP
Yet another common KIP is inventory, especially if related to lean manufacturing. One company I know measured these inventory levels on a monthly basis, always at the end of the month. It looked like the graph on the left.
At one point they decided to try to measure the inventory levels on a daily basis. Not all inventories but way too many showed a curve similar to the graph on the right. Miraculously, the inventory levels always dropped significantly at the end of the month. Again, was this coincidence, or was this an intentional effort to fudge the numbers? (Hint: It was not a coincidence!)
To continue with popular KPIs, quality is also often measured. One manager producing dishwasher tabs made the mistake of starting to correctly measure the number of broken tabs in a factory where defective products were notoriously under-reported. Defect rates increased by factor of 10. More precisely, the defect rates always were there, but now the reporting matched the actual quality.
Of course, the new numbers destroyed the quality KPI of the entire plant. The problem, however, was solved quickly: The manager in charge was fired, people were sternly reminded about the quality targets, and hence reported quality rates returned to normal. Of course, quality did not improve at all in reality. (Note: Normally I cannot go into that much detail due to confidentiality, but this example is published. See below for the source.)
Probably the most important KPI in industry is cost. Unfortunately, the true cost of a product is very difficult to calculate. While work hours can be measured and calculated easily, how do you measure customer satisfaction if he gets his product on time – or not? Standard bookkeeping often has some assumptions that make a product cheaper or more expensive than it really is, especially when comparing in-house with external suppliers.
I have seen examples where in-house suppliers could calculate their offers with 100% efficiency, but received payments according to their true efficiency. Naturally, the offer of the external supplier could not compete, even if they would have been cheaper at the end. I have also seen the opposite, where the in-house manufacturer had to account all its old and rarely used equipment into the cost, which of course could not compete with the external supplier. Hence they ended up producing all the parts that no external supplier was interested in – the low-quantity, high-variety, and tricky products, which made them even more expensive.
Overarching Meta Performance Indicators
Yet some other companies have an overarching meta performance indicator to measure the performance of their plant. Throughout the years these meta-KPI continuously increased. It looked like the plants of these companies became better every year. Except the bottom line did not improve at all. There was no connection between the trends of the KPI and the money left at the end of the day. People just got better at fudging the numbers.
Some More Examples Outside of Industry
I’m sure you can find many more examples in your own industry. Of course, the problem is not only in industry, but can also be found elsewhere. Governments are also well known for number fudging.
For example, in the US, students are scored using standardized testing. Naturally, there are tons of reports on manipulations where, for example, teachers gave the answers to the students or corrected the tests before they were graded.
Financial and Economic Indicators
Economic growth and financial stability is also a measure of pride for many nations – and some are more proud than what reality would dictate. The growth rates of China are probably inflated. Greece joined the European currency based on highly questionable financial indicators – and now probably wishes they would not have done that. But somebody surely made a career with those numbers.
Production rates are also often manipulated. Socialist and communist nations especially excelled at this. In the Soviet Union, rather than reporting total quantities, the quantities of one shift or one worker was reported.
One particular individual, Aleksei Stakhanov, Hero of Socialist Labor, was reported to have mined 102 tons of coal in one shift, about 14 times his quota. Later he even exceeded that by mining 227 tons of coal in one shift. This was soon followed by similar over-achievements by other workers and named the Stakhanovite movement. During the great famine in China (1958–1961), farming yields were exaggerated up to the point of covering the field four feet deep with its harvest. Of course, neither Aleksei Stakhanov nor the fields in China were ever anywhere near these production rates.
Unemployment is also something that governments prefer not to have, or at least to have as little of as possible. Here, too, it is easier to fudge the numbers than to actually find jobs for people. In Australia, if you work one hour per week you are counted as employed. In Germany, lots of unemployed people are parked in otherwise almost useless “Job Training Programs,” in which (of course) they are no longer counted as unemployed.
These are only a few examples. Depending on which countries or which industry you are familiar with, you probably can easily add many, many more. The key statement is:
Be very, very wary of any kind of numbers
if you have not measured and calculated them yourself!
In my next post I will discuss the negative effects of these fudged KPIs. More importantly, in a third post I will also show some ideas on how to reduce this negative effect of KPIs. In the meantime, go out and organize your industry, but do not believe only the numbers!
- Lies, Damned Lies, and KPI – Part 1: Examples of Fudging
- Lies, Damned Lies, and KPI – Part 2: Effects of Fudging
- Lies, Damned Lies, and KPI – Part 3: Countermeasures
Roser, Christoph. “Richtig Messen – KPIs Zum Nutzen Des Unternehmens Einsetzen.” Yokoten 5, no. 1 (2016): 26–29.
The dishwasher tabs example is from Konsequent – Das Buch zum NTT by Thomas Hochgeschurtz; ikotes, 2009.
10 thoughts on “Lies, Damned Lies, and KPI – Part 1: Examples of Fudging”
The monthly inventory example is what students sometimes do during “lean games” when they find out that WIP affects their score negatively. Which in a way is great because then we can explain that they are actually losing production capacity and that they are in essence chasing “quarterly report figures”.
Thanks, Leo. I agree, and for me the Lean in Lean production means low inventory (although this is only one aspect of the Toyota production system).
interresting and fun to read. From my personal experience: KPIs are sometimes regarded a great tool when trust is lacking in verbal Statements like “We are on time”. Tis does not tackle the root cause and nothing really is gained as almost every KPI can be doctored. Very few KPIs are immune to fudging and of course that applies to all KPIs that I report to my superiors.
Main reason for this commend however: In Section “Delivery Perfomance” you present the example of 95% reported OTD towards 30% perceived OTD by the customer. The associated Picture shows the numbers 90% to 30%. Judging from your blog on Quality I guess you want to be informed about stuff like this.
Hi Sascha, thanks for the head up – and keep on being careful of KPI’s. Mine are now fixed to 30-90%, and of course these were always the correct numbers … trust my KPI’s … 🙂
All of the above goes in spades at the plant I work at. We do have a board in the meeting room where our production KPIs are posted for employees to see. The main ones are Units Per Man Hour (UPMH) and FTTQ. FTTQ goal is 95% for all lines. All would be well, if not for the fact that the area I usually work at typically has 15%+ rework/scrap rates alone (simply due to the nature of our business). And that’s on a good day. A bad day means double that. So any reported FTTQ over 75% is funny numbers.
Same goes for upmh. One time I got bounced over to another line just to stand around and do nothing for nearly an hour with other guys waiting for the material to be delivered to the dock. Yet when I checked the board the next day, the line met the upmh goal exactly on the spot (as it consistently does, hahahaha).
I genuinely like this article Christophe and I worked with Eli Goldratt for a while and with his organisation for longer. He had a phrase “tell me how you are going to measure me and I’ll tell you how I’m going to behave”. It appears to my eyes to sum up the influence of measures on people.
Andre A provides an excellent example: measuring units per man hour will just create stock and or work in progress. The very opposite of what Lean is truly about: the common sense reduction of lead time
We always have the same phrase we use when talking with management about KPI’s: “Bad metrics drive bad behavior…”
I have a question. What else can we use to calculate efficiency of an equipment if we cannot clearly calculate OEE? OEE doesn’t fit for all type of equipment (example: ovens)
OEE is a good tool, as long as the data basis is valid and correct, or at least the limitations are known. For ovens (or in general batch processes) you can for example measure the ratio of how many breads you could have cooked vs how many you actually cooked.