When AI Knows Too Much: It's Time To Set Your Own Standard For Personal Data

Swapnil Chawande is a Cybersecurity Leader at PG&E.

getty

The companies building deeply personal AI will win or lose on one thing: whether users trust them enough to be honest about how their data is used and what is being shared.

There is a moment every security leader eventually encounters when they realize the system being built knows something genuinely intimate about a person. Not just their preferences or purchase history, but how they think, how they speak under stress and the fears embedded in their word choices.

That moment carries a weight that compliance checklists were never designed to hold.

We have moved past AI as a productivity tool into an era of AI as a relationship, and how personal data is collected, stored and protected is no longer a legal footnote. It is the product.

The Gap Between What AI Learns And What Users Understand

AI-native applications have introduced something traditional security frameworks weren't built for: the intimacy of the data itself. Health AI knows not just your resting heart rate, but your anxiety patterns. Financial AI maps not just your spending, but the fear behind it. Writing AI learns not just your vocabulary, but the insecurities buried in your revisions. Each holds something that, in the wrong hands, wouldn't just be a breach. It would be a betrayal.

The gap between what AI infers and what users understand is widening, and closing it is not a marketing problem. It is a foundational engineering and ethical responsibility.

What I've Seen In The Field

Across large-scale operational engagements, the most persistent failure wasn't technical—it was conceptual. Strong perimeter controls existed, but the behavioral dimension of data never factored into threat modeling. Telemetry gathered for efficiency could reveal how individuals perform under pressure or respond to stress. The question that went unasked wasn't whether the data was secured. It was whether it should have been held at all.

The most dangerous assumption in enterprise security is that a robust vault justifies whatever you put inside it.

That pattern drove a research effort I led, evaluating personal data handling across the AI ecosystem to advise on a client framework. Three observations shaped my thinking.

1. Where Privacy Gets Built In

First, defensible implementations treat privacy as an infrastructure decision, reducing exposure at the processing layer before data ever moves. In the case of Apple Intelligence, for example, much of the processing is designed to occur on-device, with limited reliance on persistent server-side data. That approach raises a useful question for any deployment: Does this data need to leave the user’s immediate environment at all? In many cases, reducing data movement can mitigate risk more effectively than adding governance controls later.​

2. When A Commitment Is Actually Tested​​

​Second, launch-day policies are easy to write, but the harder test comes when circumstances change. Inflection AI illustrates the importance of transparency in how personal AI systems handle user data, with public disclosures outlining how interactions are collected, used and retained. For leaders assessing vendors, the more relevant consideration is which commitments can survive changes in leadership, business model or ownership.​​

​3. Where Control Sits

​The third observation is this: Platforms that give users item-level decisions over their data build more durable trust than those relying on all-or-nothing consent. Zeek, for example, reflects this: Users can configure how individual data inputs—such as memories, knowledge base entries and training examples—are used within their AI persona, rather than consenting to a system that decides on their behalf. The broader implication is that specificity in user control often correlates with stronger, more durable trust—something that should be reflected in how consent frameworks are designed.​​

Four Dimensions That Actually Matter

Keep these dimensions in mind when setting personal data standards:​

1. Purposeful Collection: Only gather what the product requires. In one enterprise AI pilot I reviewed, the system ingested full email threads when subject lines and metadata alone were sufficient. Excess data creates liability with no product benefit.

2. Structural Isolation: If a breach of one user's data could expose another's, the architecture needs to change, not the terms of service. Shared embedding spaces and multi-tenant vector stores are the new shared databases.

3. Legible Consent: Users should understand in plain language what they're sharing, why and what they can take back. I've audited platforms where the actual retention policy was buried in clause 18 of a linked addendum.

4. Reversibility: Ensure there is a genuine, tested deletion path, not a 90-day queue and a support ticket. If you've never tested it end-to-end, you don't have one.

The Business Case The Industry Undersells

Models are commoditizing. The differentiation that compounds is how deeply users let a system into their lives. This is earned through trust, not features.

Three steps follow from this. First, add trust metrics to product reviews. Track whether users are sharing more or less sensitive context over time and treat declining depth as a warning signal.

Second, run a disclosure audit: Can your team explain every category of inference your system draws? If you can't articulate it internally, you can't disclose it credibly.

Third, treat consent as a product surface. Long-term retention belongs to teams whose users can open a settings panel and understand what their data is doing.

The Standard Above The Floor

The EU AI Act, GDPR enforcement and emerging U.S. state legislation are setting minimums. But floors are not ceilings. The companies worth remembering set their own standard above the legal minimum, not because regulators required it, but because they understood what was at stake.

Most teams can describe their encryption scheme in detail. Fewer can answer this: If your most active user asked you to explain every inference your system has drawn, could you? Would they recognize themselves in your answer? That gap is where the next wave of AI trust failures will originate, before the first line of model training, not after the first breach.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?