The problem of prejudice in hiring, compensation and other decision-making is currently high on the business agenda and is the subject of a range of solutions with varying degrees of effectiveness.
But a new book, Noise, psychologist Daniel Kahneman of Princeton University, professor Olivier Sibony at HEC and Harvard lawyer Cass Sunstein, offers a whole different problem that, according to the authors, has gone under the radar.
Noise, in this case, refers to the plethora of mistakes that creep into decision making and are hard to spot because there are so many and diverse. Speaking to Quartz from Paris, Sibony explained what noise is in an organization and what to do about it.
Quartz: Why is the notion of noise important for organizations? What problem do you think its identification can solve?
Olivier Sibony: The noise is the fact that in an organization, where judgments are expected to be identical, there are differences between people, or sometimes between two different occasions when the same person wears the same judgement. To take an example, hiring decisions or performance review decisions usually depend on who makes the decision, as opposed to who is the subject of the decision.
Organizations should be concerned about this, as they often rely on one person to make the decision – and if they rely on more than one person, they usually don’t have the right measures in place to ensure they are take advantage of this potential diversity. .
You don’t want your decisions to be the result of a lottery, and essentially the ânoiseâ is a lottery. And because it’s a lottery, it creates a lot of errors. If you make âloudâ hiring decisions, you’re not hiring the best people. If you do loud performance reviews, you are not rewarding the best people and sending the right signals to the underperformers. It’s a matter of credibility, it’s a matter of fairness, and it’s a matter of accuracy in your decisions.
You make a clear distinction in your book between bias and noise. Can you explain it?
Bias is a great explanation for errors, it’s a big culprit. We can actually point the finger at it and say, “I was not hired because of the gender bias of the person reviewing me.” These biases explain many errors in HR decisions.
The problem, however, is that there are not just biases, there are also random errors. If you tend to hire more men than women, you probably aren’t hiring the top men either. law men, and when you hire women, you might not be hiring the right women.
Whether you have, or don’t have, a bias is a separate issue of whether there is variability in who you hire depending on who makes the decision, or what time of day it is. is. And when you look at how likely these kinds of decisions are to change in who makes them, or even change in the context in which they are made, it is clear that influences that shouldn’t be at work. a role to do play a game. What we’re trying to do is raise the noise profile, because in many cases it’s actually more important than the bias.
What are some of the practical things organizations can do about noise?
Correcting a bias is like curing a disease: you know what the disease is, you know what the symptoms are, and you push in the opposite direction. Noise control, on the other hand, needs to be prophylactic in nature, because you don’t know in which direction you are going to make mistakes. It’s about changing the process by which you make decisions, to prevent sources of noise from seeping into your decision-making process.
Let’s take a simple measurement example: If you step on your scale in the morning and your scale is a little noisy – when you step on it several times in a row, it gives you different readings – you intuitively know that if you take the average of several readings, you will get a more accurate estimate of your weight than if you take the first reading. Thus, the average of several independent measures of the same thing, or the average of independent judgments of the same problem, such as the quality of a candidate, will reduce noise. You can even tell statistically how much it reduces noise.
The problem with how organizations practice this – for example, when hiring – is that they usually don’t keep independent judgments. In all kinds of ways, sometimes subtle and sometimes not at all subtle, the people involved in the decision-making process influence each other. Let’s say three of us have met the candidate and we get together. The first person walks into the room and says, âWhat a great guy, let’s talk about him. Or: “Interesting candidate, I would like to know what you think of him. Well, if that person is the boss, they’ve already given you their idea of ââhow they look.
These group influences, social influences, do not reduce the noise; they actually increase it. They’re doing it Following Random, Following likely to be different than what another group of people would decide, than if you had had no discussion at all.
So what is the solution ?
Organizations are designed to produce consensus, produce convergence and produce action. If you want to have independent judgments and be able to aggregate them, you need to take special care to keep people in the dark about what other people think so that their judgments remain independent until the moment you decide to put them together, make a final decision.
There are therefore two solutions: aggregating several entries from independent people and structuring the judgments on several dimensions, making sure to evaluate these dimensions independently of one another.
You talk about using rankings rather than ratings. Can you explain a little more?
There is inherently less noise in relative judgments than in absolute judgments. There is less noise when you rank people or things that when you rate people or things.
If I ask you to rate people and you say “very good”, well, “very good” to you may mean something very different than it means to another person. It can mean something very different to the person reading it. âPretty goodâ in England means something very different from âpretty goodâ in the United States. For cultural reasons, and for reasons of interpersonal variability, you might have a lot of misconceptions about the scale a company uses when it says: excellent, very good, OK, etc.
It’s a lot less noisy to say: on this dimension, say, writing quality, our gold standard is Kathy. If you write as well as Kathy, it’s an A. If you were a B, you would write as well as Olivia. To be a C you must have the same writing skills as Tom. And every time we look at someone’s handwriting we ask: is it as good as Kathy? No. Is it as good as Olivia? Yeah, okay, so it should be a B. There’s a lot less noise when you make those kinds of comparisons than when you say, “Yeah, she’s a really good writer.” “
You also talk about using rankings in performance reviews, which sounds terrifying.
What you’re saying is people hate performance reviews, right? They hate them, whether it’s a grade or a ranking. And one of the things they hate about them is that they’re very loud. In fact, most of the research we’ve looked at suggests that in the rating you get, about three-quarters of the variance is noise. Only a quarter of this has something to do with your performance. It is therefore a good question whether, as a company, you want to have an evaluation system. You can do without it. But if you do choose to have one, and if it has consequences, you will probably want to measure something that is not noise, that looks like an individual’s performance.
Now when I say rankings are better than grades, it doesn’t really mean you have to rank your employees. In fact, it’s a practice that companies have at times adopted and it’s quite destructive, for all kinds of reasons. This means that you have to compare the performance of each employee, on each dimension, against a standard that is embodied by a case that you can point your finger at. The case could be someone who left the company five years ago and remains the gold standard. Or the case could be a video thumbnail that was created for the occasion to describe what it’s like to behave like a B in customer service skills in a restaurant. The point is to compare people to a real standard that can be defined in sharp terms, without interpersonal variability, and not just to say “very good” or “ok”, because it is very noisy.
Is there anything in the present moment, with all of its talk of equality, that is good or bad about noise?
The question implicit here is, why haven’t we talked about this for so long? Why don’t we care about it as much as we care about prejudice? And the reason is multiple. First, prejudices are more charismatic, prejudices are sexier. Noise is a statistical observation: it’s abstract, it’s something less easy to get upset about.
The other reason we don’t notice noise is that organizations are pretty good at hiding it. Organizations don’t regularly do what we call a ânoise audit,â which involves asking different people to comment on the same decisions separately and measure their disagreement.
That’s more of a hope I would articulate: Through the concern about prejudice, we become much more sensitive to the importance of making the right decisions and the risk of making the wrong decisions than before. This should get us to tackle noise at the same time, as there are many noise remedies that will reduce bias as well.
This interview has been slightly edited and condensed for clarity.