For folks a bit out of the loop, here’s my take on what the hoopla is about: causes, mitigations, and path forward.
First, “tokenmaxxing” was a distortion of employers encouraging their workforce to use more AI, measured in tokens consumed. It’s akin to letting all faucets run as a poor proxy for promoting employee hydration. I believe it’s since been stamped out in the few companies where it took place.
Second, CTOs, CIOs, and CFOs at companies that have adopted AI at scale are indeed facing ballooning usage costs due to a few trends:
-
Growing user counts — Leaders worked hard to encourage and train employees to use AI. Got to be AI-native!
-
Evolving techniques — From basic tab completions, the state of the art is now tool-using multi-agent systems powered by multi-turn reasoning models generating thousands of lines of code.
-
Terms of service changes — Where tool licenses were once all-you-can-eat, they are now consumption-based. Newer models are also more expensive.
With each dimension growing independently, token volume is up several orders of magnitude and spend has skyrocketed. How high? Per fintech Ramp’s observation of customer data, the top 1% of AI users spend $7K/employee/month, or about the same as a darn good engineer in many countries. The top 10% spend $600/month.
Want to bring costs back down? Controls include model routing, better caching, cost awareness, tool training, and caps.
That said, the trend is clear: as models and techniques improve, we’ll delegate increasingly bigger tasks to more autonomous agents. Token consumption will continue to rise quickly for the foreseeable future.
To right-size the budget, estimate your return on token (ROT).
ROT asks: for every dollar of AI usage, what business outcome did we accelerate or improve?
Fundamentally, “hiring” AI agents by buying tokens is an investment. This means knowing the value of the outcomes achieved by AI, how much faster they happen, and how often they take place.
This is enabled by:
- Data-driven decision-making
- Project value estimation (i.e. cost of delay)
- Outcome measurement (A/B tests, cohort tracking, pre/post analysis)
- End-to-end optimization (value stream mapping, lead-time reduction)
Ironically, those are the same ol’ management habits that were best practices pre-AI. Diligent managers are being rewarded.
At Pearl, we’ve worked hard to build those practices over many years, and we’re refining our visibility and understanding daily. Anyone can. It’s simple but hard, in the same way eating healthy and exercising is.
If there’s interest, I’d happily share the broad strokes of our approach. It’s imperfect, but the ROT has been incredible.