Your AI Model’s Weakest Link? The Data You Can’t Trace

Akash Pugalia | Chief Digital Officer at TP.

getty

Enterprises trust their AI models because they are trained on hundreds of millions of data points, sailing through benchmarks and accuracy tests. However, even small distortions in these tests can lead to massive differences in real-world scenarios, as a Microsoft blog recently pointed out. ​

In high-stakes environments, like with a medical AI system, a small issue can be the difference between a correct call and a catastrophic one. For instance, a minor error can lead to the wrong medication dosage recommendation or overlooking a warning sign it was trained to catch, all while still passing every test.

This should matter to anyone building or buying AI. For most of the AI's commercial history, the conversation about training data has been dominated by scale, speed and cost. However, provenance—where the data came from and whether it can be trusted—was not scrutinized with the same rigor.

New regulatory requirements are beginning to change that. Meanwhile, as the World Economic Forum notes, not being able to demonstrate the right to use data can lead to legal consequences.

As a result, enterprise AI governance requirements are maturing across enterprises and institutions. The defining question is shifting from "How much does this data cost?" to "Can you prove where it came from, who touched it and that it hasn't been compromised?" ​

From Best Practice To Legal Requirement

Many companies asking this new question are taking a cue from emerging regulations. ​

The European Union's AI Act, for example, mandates that providers of high-risk AI systems maintain documentation of the training, validation and testing datasets, including their origins, scope, collection methodologies and data processing operations. Non-compliance can result in fines of up to €35 million or up to 7% of the organization's total worldwide annual turnover for the preceding financial year.

The act is a turning point for enterprises, one that raises new requirements for them and their data partners, because all documentation becomes relevant during an audit.

If your team or your data partner cannot produce a structured provenance record—meaning who collected the data, under what guidelines, with what annotator credentials, etc.—then your documentation may be insufficient.

The Rise Of The AI Bill Of Materials

One of the emerging trends to meet the data-lineage best practices is the AI ​bill of materials (AI-BOM). While the concept is not a mandate in the EU AI Act, the underlying documentation the act requires closely aligns with what an AI-BOM contains. So, what is an ​AI-BOM?

Let's start with a software bill of materials (SBOM), which is a structured inventory of all components in an application, including libraries, dependencies, versions and licenses. In other words, it makes visible every piece that went into building the software, including the borrowed and third-party parts you didn't write yourself.

An SBOM is mandatory in U.S. federal software procurement after the 2021 Executive Order on Cybersecurity precisely because supply-chain attacks made transparency a national security requirement.

Similarly, an AI-BOM spans every layer of an AI system. It documents base model provenance, fine-tuning checkpoints, collection methodology, date ranges, licensing terms and other details required at both the foundational model and data levels.

In short, every transformation of the data passed through, from filtering and deduplication to augmentation, must be timestamped and attributed.

Evaluation is equally non-negotiable. Benchmark selection rationale, red-team findings and bias audit results, including inconclusive ones, must be documented rather than summarized.

Binding it all together is the chain-of-custody record that includes every handoff between teams, vendors and systems, with verification signatures that make each component's provenance independently confirmable.

At a minimum, an AI-BOM should be able to answer these basic but critical questions:

• Where did this data come from?

• Who collected, labeled, reviewed or modified it?

• Which guidelines and quality controls were applied?

• What transformations did the data go through?

• What evaluation, red-team or bias findings were identified?

• Can the organization defend the system under audit or legal review? ​

Five Requirements Every AI Company Should Demand From Data Partners

As the AI-BOM framework matures and regulatory timelines firm up, the criteria for selecting training data partners must evolve beyond cost-per-annotation and turnaround time. The following five requirements are the new baseline for defensible AI development:

1. Documented Data Provenance: Full source documentation including collection methodology, annotator credentials, guideline versions and data-origin evidence for every dataset delivered.

2. Chain-of-Custody Tracking: Every handoff, transformation, quality check and human review step logged with timestamps, actor identity and outcome.

3. Anomaly Detection and Adversarial Auditing: Active probing for poisoning and distributional drift, not just accuracy measurement.

4. Regulatory-Ready Documentation: Pre-structured documentation packages designed for EU AI Act Article 10 compliance and audit-ready formatting.

5. Domain-Expert Annotators: Verified credentials across healthcare, legal and financial services domains, not general crowdsourcing, with identity management and expertise records available for regulatory review.

The Conversation Has Changed

The pressure for proven data lineage is arriving from several directions at once, so the question most enterprises are asking has moved from "How much does this data cost?" to “Can you prove where this data came from, who touched it, how it was changed and whether it is fit for its intended AI use case?”

The AI companies that will navigate this shift with confidence are the ones able to build provenance into the pipeline from the first annotation as the foundational architecture of every dataset they deliver.

The weakest link in your AI system may not be your model. It may be the data your model was built on, and whether you can prove, under audit, exactly where it came from. ​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?