cover
MORAL SCIENCES FOR THE MODERN WORLD · Book 2 of 25
PART I · LEARNING TO THINK

THE INCENTIVE TRAP

Show me how you're paid, and I will show you what happens next.

“Nobody had to lie. The number simply asked for the wrong thing.”
★ Ages 7+ ★
Portrait of Siri
Siri
Portrait of Charlie
Charlie
Portrait of Warren
Warren
1

The City That Paid For Tails

A museum about how cities learned to stay healthy. Siri had come to see the water pipes.
A museum about how cities learned to stay healthy. Siri had come to see the water pipes.
She never reached the water pipes.
She never reached the water pipes.
“Why is there a photograph of rats?”
1Hanoi, 1902 — a brand-new sewer and a very good plan
Men hired by the city. Their job was rats. The city paid them by the day.
Men hired by the city. Their job was rats. The city paid them by the day.
The city had just built a modern sewer. It was clean, dry, warm and safe.
The city had just built a modern sewer. It was clean, dry, warm and safe.
Above it, an elegant city. Below it, the finest rat home in South-East Asia.
Above it, an elegant city. Below it, the finest rat home in South-East Asia.
Then the inspectors began writing down something odd. Rats in the street. No tails.
Then the inspectors began writing down something odd. Rats in the street. No tails.
EEK!
SIRI'S NOTEBOOK

A rat with no tail. My first three ideas: a disease, a fight, or a predator that grabs the tail and misses the rest. All three of mine were wrong.

Because the city had changed its plan. It stopped paying by the day.
Because the city had changed its plan. It stopped paying by the day.
WORTH REMEMBERING

The new plan was simpler and cheaper. Bring in a rat tail, get a coin. Anyone could do it. No wages, no supervisors, no arguments. You cannot count rats in a sewer. You can count tails on a table.

The tails came in. Thousands of them. Then tens of thousands.
The tails came in. Thousands of them. Then tens of thousands.
THE STORY IN TIME
1902

A modern sewer network opens beneath the city. Rats move in and thrive.

1902

Paid hunters go down. On 12 June alone, 20,114 rats are killed.

1902

The city opens the reward to everybody: a coin for every tail.

1902

Tails arrive in their thousands. Live rats without tails appear in the streets.

1902

Officials find rats being bred on the edge of the city.

Every day the clerk's column grew. Every day the plan looked like it was working.
Every day the clerk's column grew. Every day the plan looked like it was working.
Siri thought she had it.
Siri thought she had it.
“The catchers were cheating. Hire honest ones.”
“Suppose every single one is honest.”
A tail is worth one coin. A living rat is worth every tail it will ever have.
A tail is worth one coin. A living rat is worth every tail it will ever have.
So somebody outside the city built hutches, and fed the rats, and waited.
So somebody outside the city built hutches, and fed the rats, and waited.
WHAT ACTUALLY HAPPENED

The rat farmer was not even breaking the rule. The city had asked for tails. He was producing tails. He was the best supplier the scheme ever had.

Siri wrote down the question instead of the answer.
Siri wrote down the question instead of the answer.
“Why did they think tails meant rats?”
A whole city, working hard, spending money, and getting further from what it wanted.
A whole city, working hard, spending money, and getting further from what it wanted.
2

The Number That Stood In For The Thing

One tray filled up beautifully. The other tray was the one that mattered.
One tray filled up beautifully. The other tray was the one that mattered.
Siri was still sure. Bad people, bad result.
Siri was still sure. Bad people, bad result.
“It's simple. They were dishonest.”
2Take every dishonest person out of the story
Charlie would not let the easy answer stand.
Charlie would not let the easy answer stand.
“Then who bred the rats? A liar?”
“He gave the city exactly what it asked for.”
One coin. One tail. Nobody in this exchange has told a single lie.
One coin. One tail. Nobody in this exchange has told a single lie.
The counting was honest. The clerk was honest. The arithmetic was perfect.
The counting was honest. The clerk was honest. The arithmetic was perfect.
And the rat was still alive, and still hungry, and still busy.
And the rat was still alive, and still hungry, and still busy.
PAUSE & THINK

A city pays one coin for every tail. Catchers bring in 500 tails a day for six months. That is about 90,000 coins and about 90,000 tails. How many fewer rats are there? Work it out before you turn over.

Here is the trouble. What you put on the scale decides what you find out.
Here is the trouble. What you put on the scale decides what you find out.
WORTH REMEMBERING

You cannot answer it. Nothing in the scheme measured rats. If every tail came from a dead rat, there are 90,000 fewer. If every tail came from a rat that was let go, there are more. The count is the same either way. The number could not tell the difference — so nobody could.

Siri went to the board and did what nobody in 1902 had done.
Siri went to the board and did what nobody in 1902 had done.
She wrote the two possible worlds side by side. Both produced the same number.
She wrote the two possible worlds side by side. Both produced the same number.
“The number can't see the difference!”
Ninety thousand coins. Ninety thousand tails. The alley looked exactly the same.
Ninety thousand coins. Ninety thousand tails. The alley looked exactly the same.
In the end the city closed the desk. The sewer kept its residents.
In the end the city closed the desk. The sewer kept its residents.
SIRI'S NOTEBOOK

Nobody chose tails because tails were the point. They chose tails because tails could be counted, carried and checked. That is a completely different reason, and I did not notice it was a different reason.

The arrow hit the target. It always does. The question is who put the target there.
The arrow hit the target. It always does. The question is who put the target there.
And Siri stopped in the corridor, because a much worse thought had arrived.
And Siri stopped in the corridor, because a much worse thought had arrived.
“Then honest people can do this too.”
AH!
3

The Four-Hour Clock

An ordinary hospital. Good doctors, good nurses, a clean floor and a long day.
An ordinary hospital. Good doctors, good nurses, a clean floor and a long day.
And one clock that mattered more than all the others.
And one clock that mattered more than all the others.
TICK
1England, 2004 — nobody waits more than four hours
The promise was a good one. Waits in emergency departments had become frightening.
The promise was a good one. Waits in emergency departments had become frightening.
So a rule was made. Almost everyone must be finished within four hours.
So a rule was made. Almost everyone must be finished within four hours.
“That sounds like a kind rule.”
It was a kind rule. Long waits fell. That part is not in dispute.
It was a kind rule. Long waits fell. That part is not in dispute.
But researchers noticed a shape in the numbers. It was not a gentle shape.
But researchers noticed a shape in the numbers. It was not a gentle shape.
PAUSE & THINK

It is three hours and fifty minutes. Two hundred people are still waiting and the clock is about to fail them. What is the fastest way to hit the target? Not the best way. The fastest.

Siri guessed first, and got it wrong in a very reasonable way.
Siri guessed first, and got it wrong in a very reasonable way.
“More people must get ill late on?”
WORTH REMEMBERING

Move them. Admit them to a ward, and the emergency clock stops. Nothing has been cured. Nobody has lied. The patient may be in exactly the same trouble on the other side of the doors. But the four-hour rule has been kept.

Charlie asked the question that made it complicated again.
Charlie asked the question that made it complicated again.
“So the target failed?”
“Did it? Long waits really did fall.”
When the departure times were plotted, a great spike stood just before the fourth hour.
When the departure times were plotted, a great spike stood just before the fourth hour.
Both things were true at once. Real patients were genuinely helped faster.
Both things were true at once. Real patients were genuinely helped faster.
And a crowd of decisions bunched against a line drawn at two hundred and forty minutes.
And a crowd of decisions bunched against a line drawn at two hundred and forty minutes.
THE STORY IN TIME
2000

England's NHS Plan promises that nobody will wait over four hours.

2004

The target goes live: 98% of patients finished within four hours.

2004–10

Long waits fall sharply. On its own terms, the target works.

2010

The threshold is eased from 98% to 95%.

2012

Researchers publish the spike sitting just before the fourth hour.

Siri wrote down two true sentences that refused to sit together.
Siri wrote down two true sentences that refused to sit together.
“It helped. It also bent things.”
Two ways to hit the same number. From outside, they look identical.
Two ways to hit the same number. From outside, they look identical.
4

The Law With Two Names

Three books. An economist, a psychologist, an anthropologist. Nothing else in common.
Three books. An economist, a psychologist, an anthropologist. Nothing else in common.
Siri read the same idea three times, in three completely different vocabularies.
Siri read the same idea three times, in three completely different vocabularies.
“Somebody here has copied somebody.”
1Three desks, three decades, one wall
So she went looking for who said it first, expecting to catch someone out.
So she went looking for who said it first, expecting to catch someone out.
The dates did not cooperate. Nineteen seventy-five. Nineteen seventy-nine. Nineteen ninety-seven.
The dates did not cooperate. Nineteen seventy-five. Nineteen seventy-nine. Nineteen ninety-seven.
Different countries. Different subjects. Different problems entirely.
Different countries. Different subjects. Different problems entirely.
“Then they all found it separately?”
“What does that tell you about the wall?”
One studied money. One studied schools and social programmes. One studied universities.
One studied money. One studied schools and social programmes. One studied universities.
THE STORY IN TIME
1975

An economist writes that a pattern collapses once it is used for control.

1979

A psychologist writes that indicators corrupt the process they monitor.

1997

An anthropologist gives the idea the short sentence everyone now quotes.

Today

The idea is famous. It is also broken again every single year.

Three people who never met, walking into the same wall from three directions.
Three people who never met, walking into the same wall from three directions.
WORTH REMEMBERING

That is what makes it a law rather than a story. It is not about banks, or hospitals, or rats. It is about what happens to any measurement once somebody's reward depends on it. Change the field and it still happens. Change the century and it still happens.

A ruler is perfectly safe. Right up until somebody is paid by the centimetre.
A ruler is perfectly safe. Right up until somebody is paid by the centimetre.
Hold a match under the thermometer. The reading improves. The room does not.
Hold a match under the thermometer. The reading improves. The room does not.
Then Siri checked her own notes, and found she had the famous sentence wrong.
Then Siri checked her own notes, and found she had the famous sentence wrong.
“I've been crediting the wrong person.”
OH!
The economist's version is longer and duller. The famous short one came twenty-two years later.
The economist's version is longer and duller. The famous short one came twenty-two years later.
CHECK THE SOURCE

The neat sentence — when a measure becomes a target, it stops being a good measure — is the anthropologist's, in 1997. The economist wrote something heavier in 1975. Being nearly right about who said it is still being wrong.

An idea that arrives from three directions at once is usually worth trusting.
An idea that arrives from three directions at once is usually worth trusting.
Which left Siri with the sharpest question yet.
Which left Siri with the sharpest question yet.
“If it's this famous, why does it keep happening?”
5

Eight Is Great

A big bank, an ordinary branch, and one number it was extraordinarily good at.
A big bank, an ordinary branch, and one number it was extraordinarily good at.
Rivals sold each household about three products. This bank was aiming at eight.
Rivals sold each household about three products. This bank was aiming at eight.
1The number that everybody could see
Siri's first idea was the obvious one, and most people's first idea too.
Siri's first idea was the obvious one, and most people's first idea too.
“Their products must just be better.”
A current account. A savings account. A card. Insurance. Then four more.
A current account. A savings account. A card. Insurance. Then four more.
Every branch had a daily figure. It went on the board with your name beside it.
Every branch had a daily figure. It went on the board with your name beside it.
If you were short by four at five o'clock, you stayed and made calls.
If you were short by four at five o'clock, you stayed and made calls.
PAUSE & THINK

A branch is measured on accounts opened. It has 1,000 customers and a target of eight products each. That is 8,000 accounts. How many of the 8,000 does the target require anyone to use?

One square for every branch. Filled in every day, in front of everyone.
One square for every branch. Filled in every day, in front of everyone.
WORTH REMEMBERING

None of them. The target stopped at the word opened. An account nobody wanted, nobody used and nobody remembered counted exactly the same as an account that changed somebody's life. The number could not tell those two apart either.

So statements began arriving for accounts that people were sure they had not opened.
So statements began arriving for accounts that people were sure they had not opened.
WHAT THE NUMBER COUNTED · and what it never asked

✔ Accounts opened

Countable today. Reported by the person being paid for it.

✔ Products per household

One tidy figure for a whole bank, in one line.

✖ Customers actually helped

Nobody could count it easily, so nobody counted it.

✖ Accounts actually used

The measure had already stopped at 'opened'.

Charlie did not ask who was to blame. He asked what the number could see.
Charlie did not ask who was to blame. He asked what the number could see.
“Somebody must have known.”
“Everybody knew. What did the number know?”
In 2016 regulators fined the bank $185 million. It was the largest such fine yet.
In 2016 regulators fined the bank $185 million. It was the largest such fine yet.
A later review put the number of possibly unauthorised accounts near three and a half million.
A later review put the number of possibly unauthorised accounts near three and a half million.
And the bank said it had dismissed about 5,300 staff over five years.
And the bank said it had dismissed about 5,300 staff over five years.
THE STORY IN TIME
2011–16

About 5,300 staff are dismissed for sales-practice breaches.

Sept 2016

Regulators impose $185 million in penalties.

Oct 2016

Product sales goals for branch staff are removed altogether.

Jan 2017

Pay is re-linked to customer service and use, not products sold.

Aug 2017

A review raises the estimate to about 3.5 million accounts.

Siri stood outside on the pavement and did the arithmetic she did not want to do.
Siri stood outside on the pavement and did the arithmetic she did not want to do.
“Five thousand three hundred people.”
From the street, through the glass, absolutely nothing looked wrong.
From the street, through the glass, absolutely nothing looked wrong.
6

Five Thousand Three Hundred People

One pin for every branch. Siri looked at the board for a long time.
One pin for every branch. Siri looked at the board for a long time.
Five thousand three hundred people. None of them had ever met each other.
Five thousand three hundred people. None of them had ever met each other.
“They must have been organised.”
1No meeting. No plan. No message from anybody.
She asked Charlie the question straight out. He would not take it.
She asked Charlie the question straight out. He would not take it.
“So whose fault was it?”
“No. That one is yours to carry.”
There was no meeting. There was no message. There was no secret agreement.
There was no meeting. There was no message. There was no secret agreement.
There was a number, every morning, with your name printed next to it.
There was a number, every morning, with your name printed next to it.
And a quiet understanding that missing it, again, would cost you the job.
And a quiet understanding that missing it, again, would cost you the job.
SIRI'S NOTEBOOK

I keep wanting there to be a villain. If there is a villain, I can be safe by not being him. There is no villain here, and that is exactly why it is frightening.

The same pressure, applied in thousands of places, on the same morning.
The same pressure, applied in thousands of places, on the same morning.
WORTH REMEMBERING

A system does not need a conspiracy. It needs one number, one consequence, and enough people. Give five thousand ordinary people the same squeeze and a few hundred will bend in the same direction — not because they planned it together, but because they were each standing under the same weight.

Somewhere, on some evening, somebody decided that four more would not hurt anyone.
Somewhere, on some evening, somebody decided that four more would not hurt anyone.
Siri hated this answer. It did not let anybody off, and it did not blame anybody either.
Siri hated this answer. It did not let anybody off, and it did not blame anybody either.
“So nobody's guilty? That can't be right.”
Nobody said nobody. The people who set the number chose it, and kept it, for years.
Nobody said nobody. The people who set the number chose it, and kept it, for years.
Understanding why something happened is not the same as excusing it.
Understanding why something happened is not the same as excusing it.
WHAT A SYSTEM SUPPLIES · instead of a plot

• A number

Counted daily, reported publicly, attached to your name.

• A consequence

Miss it often enough and you lose the job.

• A small permission

Everyone around you is somehow managing it.

• No conspiracy needed

Thousands of strangers, one identical squeeze.

She walked home carrying a question she could not put down.
She walked home carrying a question she could not put down.
“Would I have opened the four?”
The honest answer is that she does not know. Neither did any of them, beforehand.
The honest answer is that she does not know. Neither did any of them, beforehand.
7

The Mechanic's Hour

A clean workshop. Good tools. Trained people. Nothing here looks like a scandal.
A clean workshop. Good tools. Trained people. Nothing here looks like a scandal.
In 1992 inspectors drove the same cars into different bays of one chain.
In 1992 inspectors drove the same cars into different bays of one chain.
1California, 1992 — the same car, thirty-eight times
Each car's true condition was known in advance. Siri thought she knew the answer already.
Each car's true condition was known in advance. Siri thought she knew the answer already.
“Mechanics just disagree. Cars are complicated.”
In thirty-four of thirty-eight visits, work was recommended that the car did not need.
In thirty-four of thirty-eight visits, work was recommended that the car did not need.
But look closely at what a mechanic is actually being asked.
But look closely at what a mechanic is actually being asked.
This pad is half worn. Change it now, or in a year? Both answers are defensible.
This pad is half worn. Change it now, or in a year? Both answers are defensible.
WHAT CHANGED FIRST

The year before, the chain had put its service advisers on sales quotas and commissions. Their pay now moved with the size of the bill. Nothing about the brake pads changed. Only the person looking at them.

Identical cars. Identical training. Different bills.
Identical cars. Identical training. Different bills.
WORTH REMEMBERING

This is the quiet part. Almost nobody invents damage that is not there — that is a lie, and lies are frightening and rare. What happens instead is smaller. Every doubtful case, every borderline part, every it-would-probably-be-fine starts landing on the same side. Nobody notices doing it.

Siri wanted them punished. Charlie wanted a number from her.
Siri wanted them punished. Charlie wanted a number from her.
“They were cheating customers!”
“How many of them? All of them?”
Thirty-four out of thirty-eight is not a few bad people. It is nearly everybody.
Thirty-four out of thirty-eight is not a few bad people. It is nearly everybody.
And nearly everybody is never a story about character. It is a story about design.
And nearly everybody is never a story about character. It is a story about design.
The strangest part: most customers drove away grateful.
The strangest part: most customers drove away grateful.
THE STORY IN TIME
1991

Service advisers are moved onto sales quotas and commissions.

June 1992

Investigators report unneeded work in 34 of 38 undercover visits.

June 1992

The company ends the quotas and commissions for those advisers.

Sept 1992

About $8 million settles the California case.

Siri wrote one line and underlined it twice.
Siri wrote one line and underlined it twice.
“It doesn't buy lies. It buys close calls.”
Every car on this road contains a hundred judgements that could have gone either way.
Every car on this road contains a hundred judgements that could have gone either way.
8

What You Are Paid To Believe

Siri had assumed something without noticing. That a bent judgement means a liar.
Siri had assumed something without noticing. That a bent judgement means a liar.
Two experts. One set of evidence. Two different, confident, honest answers.
Two experts. One set of evidence. Two different, confident, honest answers.
1The tilt happens before the answer does
Her first explanation was the one everybody reaches for.
Her first explanation was the one everybody reaches for.
“One of them is lying very well.”
But a lie is a decision, made at the end, in full view of yourself.
But a lie is a decision, made at the end, in full view of yourself.
The tilt happens much earlier, where nobody is watching, including you.
The tilt happens much earlier, where nobody is watching, including you.
Charlie walked her back through it, step by step, in the wrong direction.
Charlie walked her back through it, step by step, in the wrong direction.
“So where does it start?”
“Before the answer. What did you look at?”
SIRI'S NOTEBOOK

Which evidence I pick up first. How long I keep looking. When I decide I have looked enough. Nobody audits any of those. And by the time I reach a conclusion, all three have already happened.

The most convincing person in any room is the one who believes it too.
The most convincing person in any room is the one who believes it too.
WORTH REMEMBERING

A liar can be caught. Ask again, ask differently, check the record. Somebody who is sincerely wrong passes every one of those tests, because there is nothing hidden to find. They will look you in the eye. They will be glad to explain. Sincerity is persuasive, and it is not evidence.

Stop searching a minute earlier and the answer changes. Nobody records the minute.
Stop searching a minute earlier and the answer changes. Nobody records the minute.
The evidence that agrees with you is always somehow nearer the front of the drawer.
The evidence that agrees with you is always somehow nearer the front of the drawer.
Then Siri caught herself doing exactly that with her own notes, and went cold.
Then Siri caught herself doing exactly that with her own notes, and went cold.
“I've been doing it all week.”
This one has a name. Incentive-caused bias. She met the thing before she met the name.
This one has a name. Incentive-caused bias. She met the thing before she met the name.
TWO WAYS TO BE WRONG · only one of them is lying

• The liar

Knows the truth and says otherwise. Rare, and catchable.

• The sincerely wrong

Believes it completely. Passes every honesty test.

• Where it starts

Not at the answer. At which evidence gets picked up.

• Why it spreads

It is contagious, and it feels like conviction.

So the question is no longer who is lying. It is who is being paid to be sure.
So the question is no longer who is lying. It is who is being paid to be sure.
Which pointed her straight at a job that exists only to give an honest opinion.
Which pointed her straight at a job that exists only to give an honest opinion.
“Who pays the person who does the judging?”
9

Who Pays The Judge?

Everything about judging rests on one arrangement that nobody thinks about.
Everything about judging rests on one arrangement that nobody thinks about.
A court works because the judge is not paid by the person being judged.
A court works because the judge is not paid by the person being judged.
1The grade that is bought by the graded
Siri tried to picture the opposite, and could not make it stay in her head.
Siri tried to picture the opposite, and could not make it stay in her head.
“Nobody would ever build it that way.”
Investments carry grades. Very safe. Fairly safe. Risky. Investors lean on them heavily.
Investments carry grades. Very safe. Fairly safe. Risky. Investors lean on them heavily.
Since the 1970s, the firm that creates the investment pays for the grade on it.
Since the 1970s, the firm that creates the investment pays for the grade on it.
If it does not like your grade, it can take the next one to a rival.
If it does not like your grade, it can take the next one to a rival.
SIRI'S NOTEBOOK

Wrong again, and wrong in the same way as last time. I went looking for people who knew it was rotten. Chapter eight already told me what to look for instead: people who were paid to be sure, and were.

Nobody in these buildings woke up meaning to break anything.
Nobody in these buildings woke up meaning to break anything.
WORTH REMEMBERING

The pressure is gentle and constant. A borderline deal, a Friday deadline, a client who will walk. Each judgement is defensible on its own. Add up a hundred thousand of them and one agency stamped its top grade on about thirty mortgage deals every working day of 2006.

Charlie asked her to count the villains it would take.
Charlie asked her to count the villains it would take.
“How many people had to be corrupt?”
“That's the question. Try answering it.”
Her answer, eventually: none. Just a great many people leaning very slightly.
Her answer, eventually: none. Just a great many people leaning very slightly.
Of everything that agency graded top in 2006, eighty-three per cent was later cut.
Of everything that agency graded top in 2006, eighty-three per cent was later cut.
The grades had been sincere. They had also been wrong on an enormous scale.
The grades had been sincere. They had also been wrong on an enormous scale.
THE STORY IN TIME
1970s

Rating agencies move to being paid by the firms they rate.

2006

One agency awards its top grade to about 30 mortgage deals a working day.

2007

The downgrades begin, and do not stop.

2008

Investments graded very safe turn out not to have been.

2011

A public inquiry calls the rating failures essential to the collapse.

Siri read the eighty-three per cent twice, then a third time.
Siri read the eighty-three per cent twice, then a third time.
“And they all believed it.”
Which raises an obvious repair, and an unobvious problem with it.
Which raises an obvious repair, and an unobvious problem with it.
10

The Score That Changed The Patient

The obvious repair is sunlight. Publish the results. Let everybody see.
The obvious repair is sunlight. Publish the results. Let everybody see.
From 1990 one American state began publishing heart-surgery survival rates.
From 1990 one American state began publishing heart-surgery survival rates.
1New York, 1990 — put the results on the wall
Siri was delighted. At last, a chapter where the good idea worked.
Siri was delighted. At last, a chapter where the good idea worked.
“This one is obviously right.”
The reported death rates did fall. Genuinely. That is not in dispute either.
The reported death rates did fall. Genuinely. That is not in dispute either.
But surveys of surgeons found something else sitting underneath the improvement.
But surveys of surgeons found something else sitting underneath the improvement.
Many said they had become less willing to operate on the very sickest patients.
Many said they had become less willing to operate on the very sickest patients.
PAUSE & THINK

Hospital A operates on everyone. 90 of 100 survive. Hospital B turns the hardest cases away. 98 of 100 survive. Which hospital is better? And what would you need to know to answer?

There is no obvious answer here. That is the point of asking.
There is no obvious answer here. That is the point of asking.
WORTH REMEMBERING

You cannot tell from the scores. B's number is better and B's patients were easier. A's number is worse and A took the people nobody else would take. The board shows both hospitals a single figure with no label on it. The score cannot say which stone was heavier to lift.

A score you can improve two ways will get improved both ways at once.
A score you can improve two ways will get improved both ways at once.
TWO WAYS A SCORE IMPROVES · and it shows you the sum

• Do the work better

Real. Slow. Difficult. Exactly what was wanted.

• Change who is counted

Also real. Fast. Free. Invisible in the score.

• What the board shows

Both of them added together, with no label.

• Who pays for it

The patients nobody was willing to take on.

Siri swung the other way, hard. Charlie let her, then stopped her.
Siri swung the other way, hard. Charlie let her, then stopped her.
“Then stop measuring anything!”
“And go back to what? Nobody knowing?”
Before the scores were published, bad results were invisible and nothing improved.
Before the scores were published, bad results were invisible and nothing improved.
Measuring badly is dangerous. Measuring nothing is a different, quieter danger.
Measuring badly is dangerous. Measuring nothing is a different, quieter danger.
The repair is not silence. It is measuring the sickness of the patient too.
The repair is not silence. It is measuring the sickness of the patient too.
SIRI'S NOTEBOOK

Both of my answers this chapter were extreme, and both were wrong. Publish everything was wrong. Publish nothing was wrong. The dull answer in the middle is the one that actually helps people, and it is much more work.

She had been looking for a rule. What she had was a job.
She had been looking for a rule. What she had was a job.
“There isn't a clever trick, is there?”
Somewhere, though, somebody had to have designed one of these well.
Somewhere, though, somebody had to have designed one of these well.
11

The Cord Anyone Can Pull

A very large factory. Thousands of parts an hour. Everything moving.
A very large factory. Thousands of parts an hour. Everything moving.
And a cord above the line that any worker at all is allowed to pull.
And a cord above the line that any worker at all is allowed to pull.
1Pay for the thing you actually want
Siri's guess was the one almost every visitor makes.
Siri's guess was the one almost every visitor makes.
“People will pull it for a rest!”
Pull the cord and a lamp lights. Somebody comes. Often that is the end of it.
Pull the cord and a lamp lights. Somebody comes. Often that is the end of it.
If it is not, the line stops. The whole line. For everybody.
If it is not, the line stops. The whole line. For everybody.
And the person who pulled it is not in trouble. Pulling it is the job.
And the person who pulled it is not in trouble. Pulling it is the job.
WHAT IS BEING REWARDED

Every other chapter rewarded an output: tails, accounts, repairs, grades. This one rewards telling somebody early. That is genuinely the thing the factory wants, and it is countable, and it cannot be faked into existence.

The line runs better because stopping it is praised rather than punished.
The line runs better because stopping it is praised rather than punished.
WORTH REMEMBERING

Invert the design and see it. Reward output only, and a fault gets hidden, passed down the line and found by a customer a year later. Reward the pull, and the same fault costs four minutes. The cheapest moment to find a problem is the moment it appears.

Charlie made her say the uncomfortable half out loud.
Charlie made her say the uncomfortable half out loud.
“So incentives aren't the problem?”
“Were they ever? What was, then?”
Incentives built this factory. Incentives are also why it stops when it should.
Incentives built this factory. Incentives are also why it stops when it should.
One wrong bolt in ten thousand, found in four minutes, because finding it pays.
One wrong bolt in ten thousand, found in four minutes, because finding it pays.
A measure people can go home having honestly met is worth a great deal.
A measure people can go home having honestly met is worth a great deal.
WHAT WOULD YOU DO?

You run a help desk. You can pay people for A. calls answered per hour · B. problems actually solved · C. customers saying they would come back · D. nothing extra, just a fair wage. Each one can be bent. Which do you choose, and what goes wrong first?

Siri finally wrote the sentence the right way round.
Siri finally wrote the sentence the right way round.
“The disease isn't paying. It's what you pay for.”
You can pay for what you want — if you are honest about what you want.
You can pay for what you want — if you are honest about what you want.
12

Charlie's Questions

Six cases. A sewer, a clock, a bank, a workshop, a grade, an operating table.
Six cases. A sewer, a clock, a bank, a workshop, a grade, an operating table.
Siri laid them out flat and produced a rule she was rather proud of.
Siri laid them out flat and produced a rule she was rather proud of.
“Simple. Never pay people by numbers.”
1Turn the question upside down
Charlie declined the rule and handed her a harder job instead.
Charlie declined the rule and handed her a harder job instead.
“That's a rule. What's wrong with it?”
“Design a trap. Good people, wrong result.”
This is inversion. Not how does this succeed, but how would I make it fail?
This is inversion. Not how does this succeed, but how would I make it fail?
Siri's trap needed five ingredients, and she found them quickly.
Siri's trap needed five ingredients, and she found them quickly.
One number. Easy to count. Hard to check. Paid straight away. Counted by the person paid.
One number. Easy to count. Hard to check. Paid straight away. Counted by the person paid.
SIRI'S RECIPE FOR A TRAP

1 · Pick one number. 2 · Make sure it is easy to count. 3 · Make sure the real thing is hard to check. 4 · Pay now, long before the consequences show up. 5 · Let the person being paid do the counting.

Then she turned the recipe over and read her six cases against it.
Then she turned the recipe over and read her six cases against it.
WORTH REMEMBERING

Every one of them had all five. Tails: countable, rats uncheckable, paid at the desk, counted by the catcher. Accounts, repairs, grades, survival rates — the same five, every time. She had not invented a trap. She had described the ones that already happened.

Which is useful, because a recipe can be spoiled. Break any one ingredient.
Which is useful, because a recipe can be spoiled. Break any one ingredient.
Let somebody else do the counting. Pay later. Use two numbers that disagree.
Let somebody else do the counting. Pay later. Use two numbers that disagree.
“You don't need all five. Just break one.”
Measure the difficulty as well as the result, and the operating-table trap loosens.
Measure the difficulty as well as the result, and the operating-table trap loosens.
But every case in this book ran for years before anybody stopped it.
But every case in this book ran for years before anybody stopped it.
WHAT WOULD YOU DO?

Your bonus depends on a number that you also report. You may: A. leave it · B. have somebody else measure it · C. be paid on three numbers that pull against each other · D. be paid in five years, once the results are in. Every option costs something. Which cost would you accept?

Charlie had run out of questions, which is how she knew one was left.
Charlie had run out of questions, which is how she knew one was left.
“Why did it take so long each time?”
“Ah. Now you want somebody else.”
The last question of this book turns out to be a question about time.
The last question of this book turns out to be a question about time.
EASY TO COUNT · HARD TO COUNT WHAT WAS COUNTED WHAT WAS WANTED Tails handed in Fewer rats Patients out by hour four Patients cared for Accounts opened Customers helped Repairs sold Cars made safe Deals graded today Grades that hold up
13

The Long Horizon

Siri's answer was that nobody had been watching. It is the usual answer.
Siri's answer was that nobody had been watching. It is the usual answer.
It is also wrong. Plenty of people were watching. They were watching the number.
It is also wrong. Plenty of people were watching. They were watching the number.
1Paid this month. Judged in ten years.
Warren had been on the bench a while, enjoying himself.
Warren had been on the bench a while, enjoying himself.
“Did Charlie send you?”
“He asked me one thing. I answered ten.”
He wanted to know when each reward arrived, and when each bill did.
He wanted to know when each reward arrived, and when each bill did.
The rewards arrived that month. Every single time. In full.
The rewards arrived that month. Every single time. In full.
The bills arrived in three years, or seven, or twelve, and often to somebody else.
The bills arrived in three years, or seven, or twelve, and often to somebody else.
SIRI'S NOTEBOOK

The accounts were opened between 2011 and 2016. The largest bill landed in 2020. A limit on the bank's size stayed on until 2025. The person who opened the four extra accounts had left years before any of it.

The question Warren kept returning to was simply: how far ahead can this number see?
The question Warren kept returning to was simply: how far ahead can this number see?
WORTH REMEMBERING

A reward that pays today for something judged in a decade is a kind of borrowing. It takes the good part now and posts the bill forward. Nobody feels dishonest doing it, because the bill has not arrived yet. The gap between the reward and the consequence is where every trap in this book has lived.

Which suggests two repairs, and neither of them is clever.
Which suggests two repairs, and neither of them is clever.
THE STORY IN TIME
2011–16

The extra accounts are opened, and paid for, month by month.

2016

The penalties arrive: $185 million.

2018

A regulator caps the bank's size until it fixes its controls.

2020

A $3 billion settlement closes the case.

2025

The size cap is lifted, seven years after it was imposed.

Warren's first repair: shorten the gap. Pay when the answer is actually known.
Warren's first repair: shorten the gap. Pay when the answer is actually known.
“That's slower, though.”
“Yes. Cheaper, too, in the end.”
His second was older and stranger: care about something the number cannot pay you for.
His second was older and stranger: care about something the number cannot pay you for.
Money can be earned back. A reputation is much harder to rebuild.
Money can be earned back. A reputation is much harder to rebuild.
He was careful to say he was not giving advice. Only asking about time.
He was careful to say he was not giving advice. Only asking about time.
PAID NOW · JUDGED LATER

• The reward

Arrives this month, in full, to you.

• The consequence

Arrives in years, often to somebody else.

• The gap between

Every trap in this book lives inside it.

• Closing the gap

Pay later, or measure something that answers sooner.

Siri stood and looked a long way out, and finally said something short.
Siri stood and looked a long way out, and finally said something short.
“It isn't the number. It's the delay.”
A safe measure is one whose consequences arrive while the person paid is still there.
A safe measure is one whose consequences arrive while the person paid is still there.
14

What Gets Rewarded Gets Repeated

A tray. A clock. A blank chart. A brake pad. A stamp. A yellow cord.
A tray. A clock. A blank chart. A brake pad. A stamp. A yellow cord.
She picked up the tray again. It meant something quite different now.
She picked up the tray again. It meant something quite different now.
“I thought this was about cheating.”
1One sentence, and it fits on a line
Six cases. Six countries or centuries. Not one of them needed a villain.
Six cases. Six countries or centuries. Not one of them needed a villain.
Charlie asked nothing at all this time, which was its own kind of question.
Charlie asked nothing at all this time, which was its own kind of question.
“You're not going to ask, are you?”
“You already answered it. Say it out loud.”
What gets rewarded gets repeated. Including whatever the reward accidentally rewards.
What gets rewarded gets repeated. Including whatever the reward accidentally rewards.
So the first question about any number is not is it true. It is who is paid by it.
So the first question about any number is not is it true. It is who is paid by it.
SIRI'S NOTEBOOK

Three questions I can ask about any measure, and none of them need an expert. Who is paid by this number? What does it count that I do not want? And when does the bill arrive?

An ordinary room, an ordinary afternoon, and an idea that will not sit still.
An ordinary room, an ordinary afternoon, and an idea that will not sit still.
WORTH REMEMBERING

None of this makes rewards bad. The cord in the factory is a reward, and it works. Rewards built the sewer, the hospital, the workshop and the bank. The care goes into choosing what is being rewarded — because whatever it is, you will get more of it.

A measure and the thing it measures, level. It takes work to keep them there.
A measure and the thing it measures, level. It takes work to keep them there.
Siri said it to the room, and it fitted on one line.
Siri said it to the room, and it fitted on one line.
“Show me how you're paid, and I'll know what happens.”
One path across the hill is wide and worn. It is wide because it was rewarded.
One path across the hill is wide and worn. It is wide because it was rewarded.
Every building down there runs on somebody's number. Most of them are fine.
Every building down there runs on somebody's number. Most of them are fine.
AND THEN SHE NOTICED SOMETHING ELSE

In the bank, thousands agreed. In the workshop, thirty-four of thirty-eight agreed. Among the graders, an entire industry agreed. They agreed because they were all paid the same way — and agreement that large is very easy to mistake for proof.

Siri closed the notebook on one question and opened it on a bigger one.
Siri closed the notebook on one question and opened it on a bigger one.
“Is agreeing the same as being right?”
The door at the end of the corridor was already open.
The door at the end of the corridor was already open.
THE MORAL SCIENCE MAP Book 2 · how a reward becomes a consequence A REWARD IS ATTACHED TO A NUMBER BEHAVIOUR MOVES TOWARD THE NUMBER WHAT IS REWARDED GETS REPEATED THE NUMBER PARTS FROM THE REALITY DECISIONS ARE MADE ON THE NUMBER THE COST LANDS ON EVERYONE ELSE
A glimpse of the next book in the series

NEXT · THE CROWD

Siri can now ask who is paid by a number. Then she notices something harder. A thousand people, certain of the same thing, on the same morning. Is that evidence — or is everybody copying everybody?

Book 3 · Why Does Everyone Believe the Same Thing?