The Perils of Proportions As Proxies for Performance

Suppose you set a minimum performance standard that you want your employees to do at least 98% of the work they do correctly. So for each person, you select a sample of 50 items of work they have done to check, as a proxy for how they are actually doing in all of their work.

In Worker A's sample less than 98% are correct. So you obviously have a performance problem, right?

Wrong!

If across all of the work they are doing they are actually meeting the standard then statistically there is a 26.4% chance that the sample will show that they aren't meeting the standard! And even if they are getting 99% correct, there is still an 8.9% chance that the sample will show that they aren't meeting the standard! In other words, a false positive.

Now consider Worker B who across all of the work they are doing is only getting 96% of their work right. Then statistically there is a 40% chance that the sample will show that they are meeting the standard! And even if their actual performance is as low as 94%, there is still a 19% chance that the sample will show that they are meeting the standard! In other words, a false negative.

What these two examples show is that if you have a moderately large number of employees then it is almost certain that by sampling like this you would end up wasting time managing the performance of someone who is already achieving the standard you have set, while not managing the performance of someone who isn't.

So what's the answer?

Some might say: increase the sample size. But that is the wrong answer for two reasons.

Firstly, increasing the sample size won't remove the problem, it will only reduce the risk of it happening and for a large employee base the problem would still continue, albeit to a reduced degree.

And secondly, and perhaps more importantly, if you increase the sample size then you have to divert more resources from doing the work to checking the work which reduces productivity and increases your costs.

My answer is different because the above approach is fundamentally flawed: there should be no minimum acceptable standard of error. If your workers process 250,000 items of work a year, then even an error-rate of 1% means that your 2,500 of your customers are experiencing problems of one kind or another. The appropriate standard to aim for is ZERO errors.

But that doesn't mean hitting your employees over the head for every error they make. What it means is having a big enough sample size to capture the most common errors and then making the necessary adjustments, whether this be training staff, improving systems...whatever to eliminate those sources of error.

In other words, checking the quality of work isn't about punishing staff or setting some arbitrary level of acceptable error. It's not as if anyone sets out each day to deliberately make errors.

Instead, it's about looking at the errors that have occurred and taking action to see that they are not repeated. And where one person's error is due to lack of information, misunderstood instructions or lack of training, the chances are that other people are making similar errors which may not have been detected in their particular sample of work. So it's also about sharing more widely the information about what errors are occurring.

In other words, it's not about the past, it's about the future and how your workers can do better in their jobs and your customers can get better service.

Inert knowledge - when what is 'learned' isn't learned

When a person studies for a coursework degree then in order to obtain the degree they need to demonstrate that they have mastered some minimal proportion of the content of each subject that counts towards the degree. Universities, being the conservative institutions that they are, use things such as essays or exams as proxy measures for whether a student has achieved an acceptable level of mastery. And those teaching such courses have not necessarily put the concepts that they are teaching to work in the real world. (I once argued with regard to a particular university that if what their business school taught was in fact correct then why was it not applied within the University itself or conversely if it wasn't correct then why was it being taught?)

The problem is that what these proxy measures primarily show is that the student is able to repeat back what they have 'learned' and use terminology in an appropriate way. What they don't show is whether the student actually understands what they have learned well enough to be able to recognise when it is applicable, how it must be modified for varying contexts and how it should be applied in a specific context.

And because of this, we find information which has been poorly understood in the first place being misapplied to situations in the real world. When knowledge has only been acquiried superficially and cannot be applied to real problems then it is sometimes referred to as 'inert knowledge' which means effectively 'knowledge which does no work'. However the problem runs deeper than this since misapplication of what has been misunderstood can actually be worse than doing nothing at all, because it can result in costs with no corresponding benefits. And the fact that those doing so have passed a course may give them an unjustified confidence in the correctness of their actions.

Coupled with this is the issue that people are more likely to remember what accords with their preconceptions. So a person with an authoritarian personality will most likely remember those facts, concepts and theories that they were exposed to that support their own world view. In other words, even the little they learn may be biased in a prticular way.

The problem is made worse by the fact that when people gain a degree such as an MBA they may feel that they can now rest on their laurels and simply apply what they have learned. However, reality is dynamic, societies change, the way in which organisations operate change, the external environment or changes in cultural values may throw up challenges unanticipated when the person was studying. So not only may knowledge be inert, it may also become stale and outmoded over time.

While a tertiary education, and particularly a post-graduate education, is supposed to imbue the student with a passion for life-long learning, in many cases the learning stops as soon as the piece of paper is awarded and the person enters the 'real' world, where apparently much of what they learned doesn't actually apply or the person fails to make an appropriate match between what they have learned and particular situations they face. Theorestical models which look so great on paper may not fare so well in the trenches where multiple problems may be inextricably entwined, and what seemed so neat and clean, becomes messy and muddy.

An education is only an education if it makes a difference to behavior, if it exposes us to ideas which we may see merit in (even if we initially disagree with them) and if it fills us with a sense of the dynamic and contingent nature of knowledge so that we don't ever feel that we have reached final certainties, but instead hold our views lightly, ever willing to change them if faced with new information or a new reality.

In other words, an education should endow us with intellectual humility. No-one should think that having a piece of paper turns them into some sort of genius or that the view of a person without that piece of paper are somehow inferior.

After all, virtually none of the richest business people in the world have MBA's or other tertiary qualifications.  A credential only points to what you may have learned sometime in the past, not what your value to a business is now. Credentials are great, especially for opening doors for someone just entering the world of business but what matters in the end is achievement.