Pathway Navigates Next Road For AI Foundational Models

AI moves fast. A couple of years ago, the IT trade held perceptions of the “fanciful” artificial intelligence functions that we might one day start to enjoy, if the IT industry ever finally perfected automation technologies. During this previous period of innocence, few people talked about language models (large or otherwise) and retrieval augmented generation was but a glint in the eye of the AI innovators who would rapidly escalate us into the agentic AI tool era of this year’s hype cycle.

What the reality of AI means to the lower substrate technologies that power it is just as profound and important as what’s happening on the surface, at the user interface and inside our applications. Key to AI infrastructures are foundation models, the general core data repositories that every machine learning function starts with.

But even foundational AI models are evolving rapidly; in the next year or so, the sheer volume of these models will explode. We are transitioning into an era in which more intelligent systems, not necessarily larger models, sit at the forefront. The large language model (LLM) market we have all become familiar with will develop into more of an “xLM” market, with the letter x meaning anything, any form, any size, any domain specialism and (almost perhaps) any anything. This is the opinion of Zuzanna Stamirowska, CEO and co-founder of Pathway, a company known for its work with AI data pipelines, many of which are built to serve real-time applications.

AI Model Smörgåsbord

AI models in the immediate future will be large, small, portable, hybrid, remote and domain-specific and use cases will grow more diverse with varying price, security and latency sensitivity levels.

“This next wave of models brings promise of advanced reasoning capabilities, the early stages of which are demonstrated by OpenAI’s o3. While this is exciting and the fastest way to artificial general intelligence, it demands a radical shift in the underlying data infrastructure that supports AI models and the complex challenges that must be navigated,” explained Stamirowska.

She notes that the primary struggle for software and data engineering in this space centralizes on creating architectures that manage structured and unstructured data types, streaming data and real-time updates.

“Evolving AI models need flexibility in data consumption while they must also be able to adhere to rigid non-negotiables like governance and security. This is achieved through two domains of data that must be uniquely managed: training data that is carefully curated, tracked and aligned with changing data governance policies and just-in-time data that must be set up for robustness, cost, latency and governance,” noted CEO Stamirowska, in a London press briefing this month.

Real-time Data Engineering

Looking at what’s happening operationally here, the xLM evolution could potentially put significant pressure on data engineering resources, primarily when an organization relies on static batch data uploads, where batch processing (as opposed to continuous stream processing) involves processing data in discrete chunks or batches, sometimes (but not necessarily) overnight.

Undertaking frequent batch uploads while maintaining meticulous attention to data accuracy demands a team of experts with specialist skills, frequently leading to cost and resource barriers. Growing demand for applications with to-the-moment accuracy exacerbates this challenge.

However, there is a light at the end of the tunnel and the industry is using the term “live AI” to express data engineering that gravitates around faster-moving living data.

What Is Live AI?

“Looking deeper here, we can say that live AI is another area of AI innovation that is developing quickly. Not only do live data feeds enhance model accuracy and enable constant learning and unlearning, but they can also reduce pressure on data engineering teams. By transitioning from static to live pipelines through hybrid systems that leverage batch processing and live data connectors or API-based feeds, the arduous task of plumbing the data pipeline with integrated, transformed data is removed,” detailed Stamirowska.

She advises data engineering teams inside enterprise organizations to assume that most future systems will be at least “real-time-ish” (featuring at least some element or percentage of real-time data immediacy and therefore built for streaming-native application functions in many cases) and so they should design their systems appropriately in preparation for this.

Of course, to implement real-time systems, data infrastructure needs to be robust, and historically, this would have put a heavy burden on data engineering resources. So, a more modern strategy is essential.

“This involves designing a data pipeline from the outset that can automatically integrate, transform and feed data into the xLM without constant manual intervention. Fast-growing tools and cutting-edge data infrastructures allow these feeds to be up and running in hours without an extensive evaluation and training cycle, instantly making data engineers’ lives easier. Agile experimentation becomes more easily attainable, allowing organizations to experiment. This encourages forward-thinking to select tools that will seamlessly accommodate future changes to applications and use cases,” said Stamirowska.

She suggests that by adopting frameworks that enable automation and intelligent data management, engineering teams can move away from repetitive, time-consuming manual tasks and instead focus on innovating to unlock further AI-driven efficiency.

Data Engineers Step Up

As the field of foundational models continues to advance, Stamirowska and team say that the data engineer’s role will develop from protecting data to stewarding data, helping guide strategic decision-making and pipeline innovation.

As we now look to create live or real-time AI systems that themselves are subject to rapid implementation cycles, a new breed of AI models and a new approach to AI-centric data engineering is called for. While many in the IT industry will want to define these practices under the newly-favored label of platform engineering, as long as the wider progression here happens, then forward movement should be guaranteed.