If you lead several teams, someone has probably asked for your AI numbers by now: who uses the tools, how much they use them, how much code they ship. On 27 October, Amazon published an update from its engineering leadership that says what many leaders suspect. Its AI usage metrics are “deeply flawed”, because they are “disconnected from what matters”. Yet the same memo says using them “with care and judgement” is appropriate when people think about their careers. To report AI productivity honestly, put usage beside delivery outcomes and record where it has not helped.
What Amazon’s memo says
The memo is by Terk Terkowitz, vice president and Distinguished Engineer for Worldwide Amazon Stores, and explains an updated role guideline for its software engineers. On metrics, it says:
- In some cases leaders have counted resources going in, or input metrics, “like token usage or adoption of AI tools”. They also count completed work, such as code changes and reviews.
- Those metrics are flawed mainly because they don’t measure the outcomes that count.
- Used “as a target instead of a measure”, they could “drive negative outcomes”. One of the memo’s examples: “tokenmaxxing is at odds with frugality”. That means consuming more tokens just to increase the usage count.
- Even so, Amazon says the input metrics “correlate well with the outputs that matter”, and that using them carefully is appropriate both for careers and for an organisation’s performance.
- It asks engineers and their managers to pay special attention to the 10th percentile of AI adoption (teams not using AI effectively, and why) and the 90th percentile (heavy users, and whether their delivery rises to match).
- And: “If your work does not benefit from current AI tooling, that is a finding, not a failing.”
Two cautions. This is Amazon’s account of its own organisation, not a controlled study. It publishes no sample, methods or data behind the claim that usage tracks outcomes. It also doesn’t say how much weight the metrics carry in reviews.
Why AI productivity reports drift toward usage
Picture judging delivery vans by how much fuel they burn. The thirstiest van looks the hardest working. Nobody checks whether the parcels arrived.
Tokens, the small fragments of text an AI model processes, and tool adoption are the fuel. Finished work reaching customers is the parcel delivery. Lead time, the wait from starting a change to delivering it, helps you track that. Once usage figures influence performance reviews, the risk is that people optimise for the number rather than the outcome.
Imagine a team that finds AI doesn’t help with part of its work and says so. If its leaders treat low usage as failure, an honest finding gets punished. Other teams may learn to keep quiet. That is a risk to watch for, not an outcome the memo demonstrates.
My take
If you’re asked to report your team’s AI usage, report it, but challenge what those numbers are being used to conclude. Low usage is a conversation starter, not evidence of poor performance. The distinction that matters is between encouraging experimentation and mandating activity.
Measuring AI impact honestly takes a report with three parts:
- Usage, labelled as an input. The adoption and token figures you’ve been asked for, with one line on what they can’t tell anyone.
- Outcomes beside it. Each team’s lead time, quality and rework: work repeated or corrected. Compare similar work and note changes in staffing or workload. Before-and-after differences alone don’t show that AI caused them.
- Findings, including where AI hasn’t helped. Amazon’s own memo treats that as a finding, so put it in writing.
Then answer one question for the people reading it: what can your teams now deliver better, faster or more reliably because of AI that they couldn’t before? If the honest answer is “not much yet”, that’s worth knowing too.
Merged and right aren’t the same thing, which is the point of what code review is for now that agents draft so much code. Used and useful aren’t the same thing either. For a way to check whether AI is actually making a team faster, episode 2 of the video series walks through it.
How you’ll know it’s working
Imagine a leader who sends a report like this every month for a quarter (an illustrative example, not a real one). You’d expect the questions coming back to change, from “why is this team’s usage low?” to “what did that team learn?”. Teams would talk openly about where AI didn’t help. And the discussion would settle on lead time, quality and rework, with usage as background.
If usage targets start appearing in people’s objectives anyway, that’s the moment to raise it, with the outcome figures in hand.
For a wider view of where AI is helping or straining delivery across your teams, the free AI Delivery Health Check gives you a score, the area to focus on first and one thing to try this week. What the Health Check covers explains the rest.
