The True Cost Of AI Conversations: It Is Time For Token Economics

Gaurav Aggarwal, Senior Vice President at Onix, Global Head Presales & Solutions Engineering.

getty

​Generative AI entered the enterprise through an interface that looked deceptively simple: a conversation box.

A user asks a question. A model responds. A document gets drafted. A customer receives an answer. From the outside, it can feel almost effortless. But the enterprise is not paying only for the conversation visible on the screen. It is paying for the intelligence consumed behind it.

OpenAI explains that models process text in units called tokens. These are the building blocks of the input a model receives and the output it generates. Depending on the model and the way an application is designed, repeated context and cached inputs can also shape the economics.

For leadership teams, this changes the question. It is no longer simply: “How much does an AI query cost?”

The more useful question is: “How much intelligence does a workflow consume before it creates a measurable business outcome?”

That is the beginning of token economics.

The visible prompt is only the surface.

Consider a simple instruction: “Summarize this contract.”

Behind that request, an AI application may process the agreement, previous negotiation notes, internal policies, system instructions and parts of the conversation history. If retrieval is poorly designed, it may pass far more information to the model than the task requires.

The FinOps Foundation describes this hidden growth as “Context Window Creep.” In many applications, earlier context is repeatedly sent back to the model so that the conversation remains coherent. As the interaction grows, the cost can expand.

The user sees one prompt. The system may be processing an entire context stack.

This distinction becomes even more important as enterprises move from chatbots to agents. A chatbot responds. An agent may retrieve documents, call tools, compare alternatives, validate its output and repeat steps before completing a task.

One visible request can now trigger several model interactions. The enterprise is not paying only for software usage. It is paying for a growing layer of decision-making capacity.

A small workflow made the problem visible.

I saw this clearly while reviewing an AI-led workflow designed to prepare a short briefing before a client conversation.

The expected output was modest: a concise summary of the account, the latest discussion points and a few recommended next steps.

Over time, however, the workflow had accumulated more context. It was pulling earlier meeting notes, background documents, previous email summaries, company information and internal instructions each time a briefing was generated. Every addition appeared reasonable on its own. Together, they made a relatively simple task heavier than it needed to be.

The answer was not to remove useful context indiscriminately. Some client conversations genuinely required a deeper view. The better approach was to separate the essentials from the optional depth: retrieve only the most relevant material by default, reuse recurring context where possible and bring in the larger history only when the situation justified it.

That experience reinforced a practical lesson. AI waste rarely arrives as one dramatic mistake. It often builds gradually through small design decisions that nobody questions because the output still looks fine.

Falling prices can create false comfort.

There is good news: Capable AI models are becoming significantly cheaper to use.

Stanford HAI’s 2025 AI Index reported that the cost of querying a model performing at approximately GPT-3.5 level fell from $20 per million tokens in November 2022 to $0.07 per million tokens by October 2024. That is a reduction of more than 280 times in roughly 18 months.

But lower unit prices do not automatically mean lower total spending.

As AI becomes more affordable, employees use it more frequently, applications process more documents and teams automate more workflows. Agentic systems add another layer: A single task may require several actions before an outcome is delivered.

The real question is not whether tokens will become cheaper. It is whether consumption will grow faster than efficiency.

Go from cost per token to cost per outcome.

The wrong response is to minimize token usage thoughtlessly. A high-consumption workflow can be worthwhile if it resolves customer issues faster, reviews contracts more effectively or helps developers ship better software.

The real risk is waste: using a premium model for routine work, sending an entire document when only a few sections are relevant or allowing an agent to keep calling tools without improving the result.

This is not merely a procurement issue. It is an operating-model issue.

Leadership teams need to connect AI consumption with business value. Cost per token remains useful, but it is not enough. The more meaningful measures will be cost per resolved case, cost per reviewed contract, cost per qualified lead or cost per agent-led task.

This also requires clearer ownership. Finance teams may monitor spend, but product and engineering teams influence consumption through choices around model selection, retrieval design, caching and agent behavior. Business teams, meanwhile, need to define the outcome that makes the spend worthwhile. Token economics works only when these conversations come together.

The objective is not to use the fewest tokens. It is to extract the highest return from every unit of intelligence consumed.

In the AI era, every conversation has a cost. The real competitive advantage will come from knowing which conversations are worth paying for.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?