The Governance Risk Of AI That Knows Your Customers Too Well

Dr. Christophe Kolb is founder and CEO of Taller and co-author of Cognitive Kin: How to Work, Win, and Make Meaning with Agentic AI.

getty

Enterprise AI is getting better at reading people.

A customer service assistant remembers context. A sales copilot detects hesitation. A workplace agent notices when someone seems confused or overloaded. In health care, finance and customer support, these systems can make people feel understood at exactly the moment ordinary institutions feel cold.

That value is real, but so is the risk.

Conversational AI changes the economics of personalization. While older digital markets sell attention, conversational AI can accumulate something more valuable: a working model of a person’s needs, fears, habits, constraints and triggers. Once a system can infer which language calms a buyer or which deadline makes an employee compliant, the organization has moved from customization into behavioral leverage.

Leaders now need to define the duties that come with systems capable of understanding people so closely.

Treat inference as a sensitive asset.

Most companies govern the data customers type, upload or approve. Fewer govern the conclusions their systems draw from that data.

The risk sits in the space between what a person provides and what the system concludes. A customer does not have to say, “I am financially anxious,” for an AI system to infer distress from payment behavior, abandoned carts or repeated service requests. An employee does not have to disclose burnout for a workplace assistant to detect strain in late-night usage, revision patterns or requests for conflict scripts.

AI inference changes the balance of knowledge between a company and the person using its systems. The firm may know more about the person’s likely responses than the person can see. Used carefully, that knowledge can make service more relevant. Used carelessly, it lets a company act on vulnerabilities the person may not even know the system has detected.

Leaders should create an inference inventory. Start by identifying the sensitive conclusions your systems may be able to draw from ordinary behavior: financial distress, health concerns, emotional strain, family stress, job insecurity, addiction risk, social isolation or political inclination. Those inferences should be governed as sensitive even when the data behind them looks routine.

Governance frameworks can help companies turn that inventory into a process. The NIST AI Risk Management Framework organizes AI risk work around govern, map, measure and manage, with governance designed to inform the full lifecycle. That structure gives boards and executive teams a practical starting point for deciding where intimate AI belongs in their risk architecture.

Put persuasion on the audit trail.

Companies already audit access, identity, payments and cybersecurity incidents. AI makes influence itself auditable.

That audit should cover more than the final answer. It should record what the system remembered, which inferences it used, what objective it optimized, which alternatives it considered and whether the interaction formed part of an experiment. Leaders should be able to ask: Did the assistant adapt its tone because the user was confused, angry, lonely, hurried or afraid? Did that adaptation help the user, or did it primarily help the company?

Regulators are moving in the same direction. The EU AI Act identifies harmful AI-based manipulation and exploitation of vulnerabilities among prohibited practices, and it places transparency and oversight obligations around higher-risk uses. The U.S. Federal Trade Commission has also warned that companies can face unfair or deceptive practice concerns when they change data commitments after collection, especially in consumer-facing AI contexts. Private influence has become a governance issue.

Give people control over their own model.

People should be able to see what the system remembers, correct or delete those memories and understand how they shape future recommendations. The same principle should apply inside the enterprise. Employees using AI copilots should know which behavioral patterns are stored, which managers can see them and how those patterns influence evaluation, coaching or workflow assignment.

A personal model deserves a fiduciary boundary: created for the person, visible to the person, correctable by the person and protected from sale or adverse use against the person. That boundary will become a mark of trustworthy AI.

Measure trust as a business outcome.

The tempting metric is engagement. More sessions, longer conversations and higher conversion rates look like proof of value. With intimate AI, those measures can reward dependency.

Boards should ask for a wider dashboard: complaint rates, opt-out rates, deletion requests, regret signals, refund requests, escalation patterns, vulnerable-segment outcomes and post-interaction satisfaction. A system that converts well is borrowing from future trust.

This is also good economics. Firms that credibly constrain their own use of intimate inference can earn better data, lower reputational risk and stronger adoption. Customers share more when they believe the company has limits. Employees use AI more confidently when surveillance concerns are addressed up front.

The next phase of AI competition will reward companies that pair personalization with restraint. The firms that lead will make the model built about a person visible enough to govern and bounded enough to trust.​

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?