Measuring Claude Code Usage Without Fooling Yourself

Someone approved a budget for AI coding tools and now wants to know whether it worked. The honest answer requires being careful about what the usage data actually covers, because the obvious reading of it - count the people with usage, divide by the people you pay for - is wrong often enough to get licences cancelled for no reason.

Why Does Usage Data Alone Mislead You?

Usage reporting answers one question: which identities did the provider record activity for, and what did it cost. It does not answer the question leadership is actually asking, which is whether the tool reached the people it was bought for.

The gap between those two is not theoretical. Running this against our own organization, the provider's Claude Code usage report returned rows for a small minority of the seats we had provisioned across a 35-day window, and several people we knew were using the tool daily - including the organization owner - produced no rows at all. Adoption calculated from usage alone would have read as a failed rollout. The rollout was fine; the reporting was partial.

"An adoption percentage is a fraction. Most AI tool reporting gets the numerator from the provider and then invents the denominator."

What Should You Actually Track?

Four things, in this order. The first two are about the rollout, the second two about whether it changed anything.

"Tokens spent is an activity metric wearing a cost metric's clothes. It tells you the tool was used, not that it helped."

Why Does One Engineer Show Up as Several People?

This surprises everyone the first time. A single engineer can legitimately appear as a subscription seat under their work email, plus a separate identity for every API key they have created for a service. To the provider these are unrelated actors. To your cost report they look like several people, and to a naive headcount they inflate adoption.

The fix is an explicit identity mapping rather than name matching. Name matching is specifically unsafe here: an API key's name is a mutable label, so matching on it lets a key that was deleted and recreated inherit the previous owner's attribution and spend, silently and with no correction path. Mapping should key on identifiers the provider guarantees are stable.

How Do You Prove It Changed Anything?

Adoption is the easy half. The question that decides renewal is whether delivery improved, and that cannot be answered from the AI tool's own telemetry at all - the tool knows how much it was used, not what happened to the work afterwards.

Compare the same delivery measures before and after the rollout, from version control rather than from the assistant. Cycle time and its phases, review turnaround, and the rate at which merged work gets reverted are the useful three: the first two show whether things sped up, the third shows whether that speed cost you stability. Our cycle time breakdown guide covers what to capture.

"The tool can tell you it was used. Only your delivery data can tell you it was worth it."

The common failure is having no before. If a rollout is imminent, take the baseline this week, because the comparison is impossible to reconstruct afterwards and "it feels faster" will not survive a budget review.

On the tool-choice question that usually precedes all of this, our Claude Code vs GitHub Copilot comparison covers the fit differences, and AI code review tools covers the review-side tooling that often lands alongside it.