At some of the biggest companies in tech, staff were judged, and then ranked, by how much AI they used. The staff learned fast. They used more AI. A lot of it on nothing.
We wanted to know why that happens, and what to count instead. The reason turns out to be more than a century old, and it starts with rats.
A CENT A RAT TAIL
In 1902, Hanoi, then run by France, was afraid of plague, and plague travels with rats. So the French set a bounty: one cent per rat, according to the history and travel site Atlas Obscura, drawing on historian Michael Vann’s research. To claim it, you handed in the rat’s tail, which spared the town hall a pile of bodies.
Tails poured in. Then officials spotted something odd: tailless rats, scampering about, very much alive. Hanoi’s rat catchers had worked it out. Snip the tail, let the rat go, and it breeds you more tails. Then health inspectors found rat farms in the countryside just outside Hanoi.
Hanoi paid for tails, and tails is what it got, plus a side business in rats.
There’s the whole mistake, in miniature. Pay people for a sign of the thing you want, and they’ll get very good at making signs. Economists call it Goodhart’s law: once a number becomes a target, it stops telling you much. We call it the rat-tail problem.
THE MANDATES
With AI, it began with bosses judging staff on their AI use.
In April 2025, Shopify’s chief executive, Tobi Lütke, wrote to staff that “reflexive AI usage is now a baseline expectation“, and that AI use would be part of performance reviews. Later that month, Luis von Ahn, who runs the language app Duolingo, told staff that AI use would be weighed in their reviews too. Coinbase’s boss, Brian Armstrong, said an engineer who refused to use the company’s AI coding tools was fired, eFinancialCareers reported in March 2026. And at Meta, counts of AI use went into staff performance reviews, according to daily.dev, which later reported that Meta had dropped them.
We think these bosses were right that AI matters. Our quarrel is with one small word: “use”. Judge people on AI use and somebody has to measure it. And the cheapest thing to measure is simply how much got used.
THE LEADERBOARDS
AI use is measured in tokens, which Fortune describes as “the basic units of data or words that AI models process”. Every one of them costs money, so a token count is really a bill.
At Meta, the owner of Facebook, an employee built an internal leaderboard that tracked the tokens used by the company’s more than 85,000 staff, as Fortune reported in April 2026. It ranked the top 250 users and gave them titles. Real ones. “Token Legend”. “Cache Wizard”. Over 30 days, usage on the dashboard passed 60 trillion tokens. By Fortune’s sums, at $5 for every million tokens, the top user alone could have cost Meta more than $1.4 million. Two days after news of the board broke, the whole thing went dark.
Amazon had one too. It was called KiroRank, after Kiro, Amazon’s AI tool for writing code. According to the tech news site CIO, some staff set AI agents (helpers that keep working on a task by themselves, without a person typing) to do unnecessary tasks, just to push their scores up. It was taken offline, and a senior vice president said the computing bills it ran up were too high.
That’s rat farming with a laptop: nobody needed the tasks, only the tokens. And it’s the perfect way to climb a chart that counts usage.
THE U-TURNS
Then three companies backed down: Duolingo, Meta and Amazon.
Duolingo’s von Ahn told a podcast in April 2026 that AI use was no longer part of performance reviews, the Pittsburgh news site WPXI reported. “For a while, it was part of performance reviews, but we decided not to do that,” he said. In September 2026, Meta said it would no longer use token counts or AI adoption dashboards to judge performance, according to daily.dev, a news site for software developers. In late October, CNBC reported that Amazon had told its engineers that token counts and similar numbers are “deeply flawed” ways to measure teams’ AI use. The memo also said: “If your work does not benefit from current AI tooling, that is a finding, not a failing.”
To us, that’s the most important line in the whole story. Under a usage target, saying “AI didn’t help here” sounds like admitting you’re behind. So people don’t say it. They keep the chart looking healthy, and the company loses the one piece of information it actually needs: where AI works and where it doesn’t.
Zoom out, and here’s what we think it means if you manage people. A usage number tells you how much your company is paying for AI. It tells you nothing about what your company is getting for it. If you judge people on the bill, they will run up the bill. And you will have no idea whether the work got any better.
HALF THE TARGET, MORE SALES
Our favourite evidence comes from a study of a company that kept counting, and simply asked for less.
A study posted to arXiv (a site where researchers share papers) in September 2026 looked at a large medical-device company that told roughly 5,000 of its employees to submit at least 200 queries a month to an in-house AI assistant. Query counts bunched right at the target, a sign people asked just enough to hit it. And 31% of the queries were repeated or off-task.
Then the firm lowered the target to 100 queries a month. It did this one branch at a time, which let the researchers compare branches on the new target with branches still on the old one. Total queries fell by about 30%. (Two numbers here look alike: 31% was the share of junk under the old target, and 30% is how much the total fell once the target was halved.) About 90% of that drop was the junk: the repeats and the off-topic queries. The useful queries mostly stayed.
Among sales staff, monthly sales went up by 7%, the researchers estimate, comparing branches on the new target with those still on the old one. The paper is a preprint, so other researchers haven’t checked it yet.
Why? Our guess is simple. People got back the time they had been spending feeding the target, and spent it selling.
THE COBRA THAT NEVER WAS
You might know a snakier version of this tale, about a British bounty on cobras in Delhi.
We love that story. The Juggernaut, a publication about South Asia, calls it “the made-up story of a British incentive to curb Delhi’s snake population”, with “little evidence to support the veracity” of it. The phrase “cobra effect” itself comes from a German economist, Horst Siebert, in a book from 2001.
So even the most famous story about counting the wrong thing got repeated for years because it sounded right, with nobody checking. Which is a fair warning for anyone about to build a dashboard. A number on a screen feels like proof. It’s only as true as what people did to produce it.
Hanoi counted tails and got tails. At Meta and Amazon, staff ran unnecessary tasks just to climb token leaderboards. The medical-device company asked for fewer queries, and its sales staff sold 7% more. Whatever you count, people learn to produce. So count what got done.
If you’re counting how much AI your people use, you’re counting rat tails.
3 things you can do tomorrow
Scrap any target or ranking for how much AI people use. If it’s in a review form, a dashboard or a team chart, take it out.
Make “AI didn’t help here” a good answer. Amazon now calls it “a finding, not a failing”. Say the same to your team, so nobody feels they have to fake it.
Judge AI on a number you already track, like sales, tickets closed or invoices sent. If that number hasn’t moved, the AI isn’t working yet, whatever the usage says.
SOURCES
Tobi Lütke on X: Reflexive AI usage is now a baseline expectation at Shopify (7 Apr 2025)
eFinancialCareers: Coinbase fired an engineer after they refused to use AI coding tools (Mar 2026)
daily.dev: Meta drops AI token usage from engineer performance reviews (Sep 2026)
CIO: Amazon deletes devs’ tokenmaxxing leaderboard to minimize costs (29 May 2026)
WPXI: Duolingo CEO says company backtracked on AI evaluations in performance reviews (Apr 2026)
arXiv 2609.32859: AI usage targets at a medical-device company (Sep 2026)
Checked on 11 November 2026 against the sources below.









