Why Vector Databases Won't Solve Your Enterprise AI Problem

Emma McGrattan is the Chief Technology Officer at Actian.

getty

For decades, enterprise systems were designed around a simple assumption: The consumer of information would always be a human. Humans bring context, judgment and healthy skepticism to every interaction with data. If a dashboard shows negative revenue, someone questions it. If a report looks inconsistent, someone investigates before making a decision.

AI has broken that assumption.

Today's enterprise systems increasingly serve software agents that retrieve information probabilistically, reason across structured and unstructured content, and take action based on what they find. They don't instinctively know when something feels wrong. That shift changes the architectural requirements for enterprise data, and nowhere is it more visible than in the rise of vector databases.

Beyond The Rows And Columns

When the enterprise’s crown jewels—customer records, transactions, product catalogs, financial systems—lived in rows and columns, we knew how to manage, govern and query them. But every organization also has a wealth of unstructured data it wants AI to work with, and the data doesn’t easily fit into tables.

Historically, we simply accepted this problem. If someone needed information, they searched for it manually, asked colleagues or relied on institutional knowledge. Humans are remarkably good at filling in context without even realizing it.

A person searching for information about customer churn understands that they might need retention reports, support tickets, customer surveys and product usage data. Traditional databases don’t think that way. They return exact matches to exact queries.

Vector databases were designed to help bridge that gap. They work by breaking content into chunks that make sense for how it will be searched, such as clauses for a legal contract. Each chunk is then encoded into a key called an embedding, and when a question comes in, the database finds the chunks whose embeddings sit closest to it. Instead of searching for identical words or exact values, they search for information that is semantically similar to the question being asked.

Enterprise architectures were not built to support that retrieval model.

Start With Constraints, Not Technology

Semantic search is easy to demo. Deploying it at enterprise scale is the hard part.

As AI moves from pilot to production, teams tend to start the vector database conversation with vendor comparisons and performance benchmarks. As an engineer, I understand the temptation to start with the technology. But I believe that’s the wrong place to start.

Instead, I encourage teams to begin with the constraints. Unlike preferences, constraints are nonnegotiable—and they narrow or eliminate the architectural choices very quickly.

I’ve seen the consequences of skipping over them. I met a team running a camera system on a ball bearing manufacturing line. When the camera identifies defective bearings, it should stop the line immediately, but their retrieval architecture was built around cloud. Even a well-optimized cloud retrieval takes around 20 milliseconds, too slow for that line. By the time the image traveled to the cloud, a decision came back and the line stopped, they had already manufactured hundreds of defective parts. The physics of a network round-trip don’t bend.

Broadly, there are three places where a vector database can live: in the cloud, on-premises in your own data center or at the edge (on or near the device generating the data). To find the right deployment architecture for each workload you’re building, identify:

1. Latency Requirements: If you need sub-five-millisecond retrieval, certain architectures are eliminated immediately.

2. Data Residency: Regulations such as HIPAA, GDPR and industry-specific requirements often dictate where data can live and what infrastructure it can touch.

3. Corpus Size And Update Frequency: How often the data changes determines how quickly the index becomes a liability. A static collection of documents and a rapidly changing fraud detection system require different approaches.

4. Connectivity: If intermittent connectivity is a normal operating condition rather than an exception, the architecture needs to reflect that reality.

5. Operational Model: Someone has to own infrastructure, updates, governance and maintenance. Organizations often underestimate this requirement.

6. Cost: Cloud, on-premises and edge deployments all have very different cost profiles, and the cheapest option during a pilot may not be the cheapest at scale.

Enterprise AI systems have many distinct workloads. A healthcare organization, for example, might run a HIPAA-regulated patient assistant on-premises while using cloud infrastructure for research workloads built on public data. Data gravity matters too, since medical imaging files are massive. It often makes more sense to bring the vector database to the data than to move the data to the cloud.

These constraints determine your deployment architecture. And your deployment architecture determines everything that comes next.

That's why the goal isn't just finding the highest-performing vector database. It’s ensuring the architecture matches the constraints of each workload. Once you map them, the technology choices tend to narrow themselves.

AI Makes Demos Look Easy

Imagine finding an AI tool that looks exactly like what your data analysts need. You request a demo and tell the vendor you’ll bring 20 people to the workshop. They tell you they can't support that many users. Your enterprise deployment needs to support hundreds or even thousands.

It’s more common than you'd think. AI has made prototypes extraordinarily easy to build and compelling enough to look like finished products. But moving a vector database from demo to production means answering questions many teams never considered during the demo.

• What happens when the underlying data changes?

• What happens when the embedding model changes?

• How do you monitor retrieval quality over time?

• How do you detect drift?

• How do you govern access to sensitive information?

• How do you explain why a particular answer was generated?

Few prototypes can answer these operational questions. If your embedding model changes, you’re re-embedding everything in your vector database. If retrieval accuracy, or recall, starts drifting, you need a defined floor where the system stops and alerts someone. These questions are less about AI than about the fundamentals: metadata, lineage, governance and data quality.

They're the questions that determine whether a system can survive in a complex and always-changing enterprise environment.

Rethinking Information Consumption

Vector databases solve semantic retrieval. They don’t solve governance, context, data quality, deployment architecture or operational resilience. Those remain enterprise architecture problems.

AI has introduced a fundamentally different kind of data consumer, and the assumptions that served us well when humans were the only consumers are no longer sufficient. The new reality is that enterprise systems must be designed not just for people but for machines that retrieve, reason and increasingly act on information. Delivering the right information, with the right context and the right controls, is no longer optional. It's the foundation of enterprise AI.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?