Data Platforms Must Adapt To The Rise Of Conversational AI And LLMs

Mandar Khoje is an Engineering Leader at Moveworks, leading Data Platform teams and based in the San Francisco Bay Area.

The conversational capabilities of large language models (LLMs) are reshaping business workflows, from customer support and knowledge management to advanced analytics and decision making. However, this transformative power introduces a unique set of challenges and risks for the data platforms that underpin these innovations.

As companies increasingly deploy LLMs, the conversational nature of the data they handle requires data platforms to evolve rapidly—not only to harness new opportunities but also to mitigate the inherent risks.

Here are a few trends I've noted during my time in the industry that are worth addressing as you and your business fully embrace data platforms and language models.

Understanding The Shift: From Structured To Conversational Data

Traditional data platforms were designed to handle structured datasets such as sales records, user logs or inventory lists. These platforms excel at organizing, querying and analyzing structured and semi-structured data formats like CSV, JSON, Parquet or Avro. However, the integration of LLMs has introduced a deluge of unstructured conversational data.

This conversational data—derived from chat logs, voice transcripts, customer interactions and internal communications—presents new challenges. It is rich in context but messy, ambiguous and unstructured, often mixing relevant information with noise. Existing data platforms must adapt to accommodate this shift, enhancing their storage, processing and governance capabilities to handle the complexity of conversational data at scale.

Risks To Data Platforms In The Age Of LLMs

The rise of conversational AI has highlighted several new risks for data platforms. Let’s examine the most critical ones:

1. Unstructured Data Challenges: Conversational data does not conform to rigid schemas. Instead, it comes in diverse formats, including text, audio and metadata. This unstructured nature requires data platforms to introduce new capabilities for storage and processing. Without proper tools, companies risk mismanaging or losing valuable insights embedded within this data.

2. Scalability Pressures: Conversational data’s sheer volume and richness amplify the need for scalable storage and compute resources. Traditional data platforms may struggle to process this influx, leading to increased latency or system bottlenecks. Organizations need scalable solutions to meet the growing demand without compromising performance.

3. Sensitive Data Detection And Protection: Conversations often contain sensitive information, such as personal identifiers, financial details or proprietary business data. Inadequate detection mechanisms could result in unintentional exposure, leading to regulatory non-compliance or reputational damage. For example, failing to identify and mask sensitive data during ingestion could result in breaches or misuse.

4. Data Lineage And Provenance: With conversational data flowing across multiple systems and being transformed for various purposes, tracking its lineage becomes increasingly complex. Data platforms must address this challenge to ensure traceability, especially for compliance and auditing purposes.

5. Regulatory And Compliance Complexity: Data privacy regulations such as GDPR, CCPA and HIPAA mandate stringent controls over sensitive data. The conversational nature of LLM data—often interspersed with personal or confidential information—adds layers of complexity for compliance. Failure to adhere to these regulations can result in significant fines and legal consequences.

How Data Platforms Can Evolve

To mitigate these risks and fully leverage the power of LLMs, data platforms must undergo a significant transformation. Here are the key strategies for adaptation:

1. Support For Unstructured Data: Data platforms must evolve to natively support unstructured and semi-structured data. Tools like Apache Hudi, Delta Lake or Snowflake’s semi-structured data capabilities can help companies ingest, store and process conversational data efficiently. Designing flexible data schemas will ensure platforms remain adaptable to the varied formats of conversational inputs.

2. Scalable Infrastructure: To handle the volume and velocity of conversational data, organizations must adopt scalable storage and compute solutions. Cloud-native platforms like Amazon S3 and Google BigQuery can provide elastic scaling to meet fluctuating demand. Additionally, distributed compute engines such as Apache Spark can process massive datasets in near real time.

3. Sensitive Data Detection And Encryption: Data platforms should integrate sensitive data detection tools that use natural language processing (NLP) to identify PII, financial information or other regulated data in conversational inputs. Once detected, this data can be encrypted, masked or redacted based on access controls and compliance requirements.

4. Data Lineage And Observability: Modern data observability platforms like Monte Carlo or Collibra can track the lineage and transformation of conversational data across pipelines. Implementing robust metadata management will ensure that data provenance is traceable, aiding compliance and boosting trust in data-driven decisions.

5. Governance And Access Controls: Fine-grained access controls must be implemented to regulate who can view or modify sensitive conversational data. Role-based access control (RBAC), attribute-based access control (ABAC) and advanced governance frameworks should be prioritized. Integration with centralized governance services ensures consistency across the organization.

Embracing The Opportunity

While the rise of LLMs introduces significant challenges, it also presents unprecedented opportunities for businesses to extract value from conversational data. Companies that proactively invest in modernizing their data platforms may not only mitigate risks but also position themselves to lead in this AI-driven era.

For instance, enhanced data platforms can power advanced analytics, enabling organizations to derive actionable insights from customer conversations, predict trends, and make data-driven decisions faster. By leveraging the full potential of LLMs, businesses can redefine customer experiences, improve operational efficiency and unlock entirely new revenue streams.

Final Thoughts

As conversational AI continues to evolve, the demands on data platforms will grow exponentially. Companies must address these risks head-on, investing in the technologies and strategies necessary to adapt. From supporting unstructured data to implementing real-time governance and compliance mechanisms, the future of data platforms will hinge on their ability to embrace this transformation.

In this new landscape, the organizations that succeed will be those that strike the right balance—harnessing the power of LLMs while safeguarding the integrity, privacy and quality of their data platforms.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?