The Perils of Proportions As Proxies for Performance

Suppose you set a minimum performance standard that you want your employees to do at least 98% of the work they do correctly. So for each person, you select a sample of 50 items of work they have done to check, as a proxy for how they are actually doing in all of their work.

In Worker A's sample less than 98% are correct. So you obviously have a performance problem, right?

Wrong!

If across all of the work they are doing they are actually meeting the standard then statistically there is a 26.4% chance that the sample will show that they aren't meeting the standard! And even if they are getting 99% correct, there is still an 8.9% chance that the sample will show that they aren't meeting the standard! In other words, a false positive.

Now consider Worker B who across all of the work they are doing is only getting 96% of their work right. Then statistically there is a 40% chance that the sample will show that they are meeting the standard! And even if their actual performance is as low as 94%, there is still a 19% chance that the sample will show that they are meeting the standard! In other words, a false negative.

What these two examples show is that if you have a moderately large number of employees then it is almost certain that by sampling like this you would end up wasting time managing the performance of someone who is already achieving the standard you have set, while not managing the performance of someone who isn't.

So what's the answer?

Some might say: increase the sample size. But that is the wrong answer for two reasons.

Firstly, increasing the sample size won't remove the problem, it will only reduce the risk of it happening and for a large employee base the problem would still continue, albeit to a reduced degree.

And secondly, and perhaps more importantly, if you increase the sample size then you have to divert more resources from doing the work to checking the work which reduces productivity and increases your costs.

My answer is different because the above approach is fundamentally flawed: there should be no minimum acceptable standard of error. If your workers process 250,000 items of work a year, then even an error-rate of 1% means that your 2,500 of your customers are experiencing problems of one kind or another. The appropriate standard to aim for is ZERO errors.

But that doesn't mean hitting your employees over the head for every error they make. What it means is having a big enough sample size to capture the most common errors and then making the necessary adjustments, whether this be training staff, improving systems...whatever to eliminate those sources of error.

In other words, checking the quality of work isn't about punishing staff or setting some arbitrary level of acceptable error. It's not as if anyone sets out each day to deliberately make errors.

Instead, it's about looking at the errors that have occurred and taking action to see that they are not repeated. And where one person's error is due to lack of information, misunderstood instructions or lack of training, the chances are that other people are making similar errors which may not have been detected in their particular sample of work. So it's also about sharing more widely the information about what errors are occurring.

In other words, it's not about the past, it's about the future and how your workers can do better in their jobs and your customers can get better service.