Token Counts and Outcomes: How Amazon's Engineering Memo, Gartner, Revolut and GitHub Copilot's Metrics API Describe Measuring AI Coding Use

Background

Organizations that roll out AI coding tools have to decide what to measure. Amazon’s engineering leadership writes that long-term outcomes are the preferred measure but can be “time delayed, difficult to measure, and hard to isolate from other trends”, which is why input counts such as token usage and tool adoption get used instead. This Analysis compares how five recent sources describe that choice: a role guideline published by Amazon, a Gartner analyst quoted by InfoWorld, an infrastructure lead at the fintech Revolut quoted by The Register, TechCrunch’s report of one high-spend user, and GitHub’s own changelog for its Copilot usage metrics API.

The Herald has covered adjacent pieces before. A Briefing on the Copilot usage metrics API described the new per-client fields. An Analysis of per-token price tiers and a reported spend cap dealt with what tokens cost. A News report on a reported Red Hat spending cap rested on an insider tip to The Register. This Analysis looks at a different question: what the sources say about how to measure the use of coding agents.

Current situation

Amazon: outcomes preferred, input metrics used with care

On October 27, 2026, Amazon published a memo written by Terk Terkowitz, VP/Distinguished Engineer, Worldwide Amazon Stores, which Amazon says was shared with its software engineers. This is a first-party document, and the Herald does not count it as independent. The memo says: “We believe that it is best to measure, and be measured by, long-term outcomes. But these outcomes can be time delayed, difficult to measure, and hard to isolate from other trends.”

The memo says that in some cases Amazon has used input metrics instead, “like token usage or adoption of AI tools, or throughput metrics like code changes and reviews.” It continues: “We are all aware of how deeply flawed these metrics are, primarily because they are disconnected from what matters.” It also says that using them as a target rather than a measure could drive negative outcomes, including increasing costs, and gives the example that “tokenmaxxing is at odds with frugality”.

The memo nonetheless says Amazon has found that these input metrics “correlate well with the outputs that matter” and that teams not adopting AI or increasing output metrics are falling behind. It recommends paying attention to the 10th percentile of AI adoption, to find teams not embracing AI effectively, and the 90th percentile, to see whether heavy users are seeing delivery increases to match. The memo also states: “If your work does not benefit from current AI tooling, that is a finding, not a failing.” The memo cites no data for its correlation claim.

Corey Quinn, writing in The Register on October 31, 2026, offers a critical reading. This is a commentary piece, and its statements below are the author’s. Quinn writes that Amazon shut down KiroRank, an employee-built leaderboard that ranked people by token usage, in May after people created junk tasks to climb the ranking. He argues that the memo takes three different positions within three sentences, and that this lets a manager cite whichever fits. He also writes that at GitHub Universe, which he attended, speakers talked about how agents start roughly 40% of pull requests, and that GitHub bills on a consumption basis while nothing today ties AI spend to output worth the effort.

Gartner: tokens and productivity are not the same thing

InfoWorld reported in June 2026 (the article says it originally appeared on CIO.com) that Gartner expects AI coding token costs to meet or exceed the typical software engineer’s monthly salary within two years. The prediction rests on a global average salary of $2,000 per month, and Gartner senior principal analyst Nitish Tyagi said it does not mean token usage will exceed all salaries.

On measurement, Tyagi said tokenmaxxing “is not directly related to higher productivity gains”, while optimizing token consumption is, according to InfoWorld. InfoWorld also reports that Tyagi sees no “direct relationship” between tokens consumed and productivity gains, and that he says the lines-of-code-written metric no longer applies, with value instead measured in quality, speed and customer satisfaction. Gartner advises token thresholds, automated usage monitoring and escalation policies, and regular token usage reviews in development cycles. Tyagi said AI-assisted development can provide up to 20% productivity gains.

Revolut: budget limits first, then a cost cut and self-hosting

The Register reported on November 3, 2026 on a talk at the AI Engineer conference in New York by David Abitbol, ops and infrastructure lead at Revolut. According to the report, Abitbol described a 60 percent improvement in coding time and a 70 percent improvement in lead time after rolling out AI coding tools to roughly 2,000 engineers. These are the company’s figures as relayed by The Register, and the report does not describe how they were measured.

Abitbol said: “We were highly skeptical about the cost of AI. We were seeing engineers spend $1,000 a month, $10,000 a month with not so much ROI, and so when we launched it we launched it with strict budget limits.” The Register reports that Revolut then moved to self-hosted GLM models, that Revolut says this cut its AI coding inference costs by roughly 90 percent in less than a year, and that he noted the biggest bottleneck was not the code itself. Gains came from letting models handle debugging and Jira and Confluence tickets, and he said: “You no longer have to save your tokens for the end of the quarter or the most difficult task.”

TechCrunch: one very large spend figure

TechCrunch reported that Y Combinator CEO Garry Tan told Walter Isaacson’s podcast The Builders: “I spend about maybe $100[,000] to $200,000 a year on tokens.” Tan also said agents are “too expensive right now” for personal agents to become mainstream. The report gives a spend figure and does not pair it with an output measure.

GitHub: what its metrics API counts

GitHub’s changelog entries describe what vendor-side measurement covers. According to the October 23, 2026 entry, a new totals_by_client array reports request counts, session counts and token usage per client, including tools such as OpenCode, claude_code and codex, so administrators can “base tool support and governance decisions on real usage.” The same entry cautions that request_count “counts model calls, not prompts” and that client is “a best-effort grouping, not a verified identity.”

The October 22, 2026 entry says reports now show the names of an organization’s custom skills, agents, MCP servers, slash commands and plugins, with up to 25 entries per record. It adds that the metrics “can undercount activity” and should be used “to understand trends and adoption rather than for exact accounting.” Both entries are GitHub’s own and are not independent of the product.

Implications

The points below are analysis drawn from the sources above, not reported facts.

Every source pairs input counts with a caveat. Amazon calls token usage and adoption “deeply flawed” while still using them. Gartner, as quoted by InfoWorld, separates token consumption from productivity. GitHub’s changelog entries describe client labels as best-effort and say the metrics can undercount activity. Amazon says its input metrics correlate with outputs, but also says they should be used with care and judgement; none of the cited sources describes a token or request count as sufficient on its own.

The cited outcome measures are organization-specific and unverified here. Revolut’s 60 percent and 70 percent figures are reported secondhand from a conference talk. Amazon’s statement that input metrics correlate with outputs comes without data in the memo. Gartner’s 20 percent figure is a general estimate. The sources do not offer a shared definition of lead time, coding time or delivery that would allow the numbers to be compared.

Cost controls and measurement appear together. Revolut says it began with budget limits. Gartner recommends thresholds and monitoring. Amazon names cost as a risk of treating token usage as a target. In these accounts, the same counts serve as a control on spending and as a signal of adoption, and Quinn’s column argues that using counts as a target invites gaming, citing KiroRank as the example.

Vendor dashboards and internal guidance count different things. GitHub’s API counts requests and tokens by client. Amazon’s memo asks for outcomes such as delivery. Gartner suggests quality, speed and customer satisfaction. The sources describe no tool that joins the two sets of numbers, and Quinn writes that nothing today ties spend to output worth the effort.

What is not known

• Whether Amazon’s reported correlation between input metrics and outputs holds outside the teams it describes; the memo gives no data.
• How Revolut measured its 60 percent and 70 percent improvements; The Register’s report does not say.
• Whether KiroRank was closed for the reason Quinn gives; the cited Amazon memo does not mention it, and the claim rests on the column.
• Whether GitHub’s request and token counts track any outcome measure; the changelog entries make no such claim.

Source note

The Herald counts three independent outlets: The Register (the Revolut report, plus a commentary column), InfoWorld and TechCrunch. Amazon’s memo and GitHub’s changelog entries are first-party. The Register’s two items are counted as one outlet, and the Quinn column is commentary rather than reporting.