The Next Hundredfold Improvement In AI Economics Won't Come From Faster Chips

As CTO, Sven Oehme drives DDN’s AI data platform strategy, powering the world’s most demanding AI and HPC environments.

getty

Three years ago, the AI industry was focused on a single question: Can AI work? Today, organizations face a different challenge: Can AI scale economically?

The first phase of AI innovation delivered extraordinary reductions in cost. GPT-class capabilities that once cost tens of dollars per million tokens can now be delivered for a fraction of that amount. While advances in GPU performance contributed to that ​progress, hardware alone does not explain the magnitude of the improvement. Much of the gain came from software innovation. Technologies such as PagedAttention, KV cache optimization and inference frameworks like vLLM increased the amount of useful work organizations could extract from existing infrastructure.

That lesson is particularly relevant as the industry enters its next phase. For several years, organizations raced to acquire GPUs, train larger models and demonstrate new capabilities. As AI moves into production, however, the economics of operating these systems have become just as important as the intelligence they produce. Every interaction consumes compute, power, networking, storage, memory and operational resources, making efficiency a central concern for any organization seeking to deploy AI at scale.

This is one reason cost per token is emerging as one of the most important metrics in AI. Cost per token reflects the efficiency of the entire system, measuring how effectively an organization converts infrastructure investment into useful output. Model loading, data retrieval, cache reuse, storage latency, networking performance and GPU utilization all contribute to the final economics.

What many organizations are discovering is that the limiting factor is increasingly not the model itself, but the infrastructure responsible for feeding it. For years, compute was viewed as the primary constraint. In production environments, however, GPUs often spend valuable time waiting for models to load, checkpoints to restore, retrieval systems to respond or data to move through storage and networking layers. Each delay reduces productivity and increases operational cost.

The industry frequently discusses GPU shortages, yet GPU underutilization may be the more significant challenge. As deployments expand into thousands or even hundreds of thousands of accelerators, even modest inefficiencies can translate into substantial financial impact. The challenge is no longer simply acquiring more compute capacity; it is ensuring that existing compute resources remain productive. This shift is becoming increasingly visible across production environments, and attention is shifting from raw compute performance toward the efficiency of data movement throughout the system.

Historically, storage was viewed as a passive repository. In AI environments, it has become an active component of the execution path. Modern AI systems depend on storage for checkpointing, retrieval-augmented generation, embedding access, model loading, persistent memory layers and, increasingly, KV cache persistence. The performance of the storage layer now directly influences how quickly models load, how efficiently context is retrieved and whether costly accelerators remain fully utilized.

This becomes even more important during inference. Large language models repeatedly access the same context and memory structures across millions of requests. Until recently, much of that work was recomputed unnecessarily, consuming GPU cycles and increasing operational costs. Advances in persistent KV cache architectures have begun to change that equation.

Whether the challenge involves checkpointing during training, retrieval during inference or memory management across distributed AI systems, the greatest economic gains often come from reducing friction throughout the data path rather than continuously adding more compute. Every transfer between storage, memory, networking layers and GPUs introduces latency, overhead, power consumption and cost. In many cases, eliminating unnecessary movement creates greater value than increasing raw compute capacity.

This reality is driving growing interest in persistent caching, intelligent data placement, memory-aware infrastructure design and tighter integration between inference frameworks and data platforms. As AI systems become larger and more distributed, efficiency increasingly depends on how effectively data flows through the environment.

Nvidia CEO Jensen Huang often describes modern AI environments as AI factories, a comparison that highlights an important shift in how organizations should evaluate success. Factories are ultimately measured by output, and in AI environments, those outputs are tokens, intelligence, decisions and business outcomes. The most important question is no longer how many GPUs an organization owns, but how efficiently those GPUs convert power, data and infrastructure investment into business value.

This is why metrics such as cost per token, tokens per watt and GPU utilization are evolving from technical measurements into strategic business indicators. The first wave of AI focused on model intelligence. The second wave focused on scaling compute. The next phase will focus on the economics of production AI and the operational efficiency required to sustain it.

Future breakthroughs will certainly benefit from faster chips. Still, they will also depend on reducing friction across the entire data path—from storage and networking to memory, retrieval, caching and orchestration. Organizations that succeed in the AI era will not necessarily be those with the most compute capacity. They will be the ones who most effectively convert that capacity into measurable business outcomes.

The first hundredfold reduction in AI costs was driven largely by software innovation. I believe the next major leap in AI economics is likely to come from making every GPU, every watt and every byte of data work harder. In the emerging age of AI factories, efficiency is no longer simply an infrastructure concern—it is becoming a fundamental business strategy.​​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?