​AI Education Data Privacy: The Hidden Liability Most Leaders Are Ignoring

Russell Sarder, CEO & Founder of AI CERTs – advancing global AI certification & education.

getty

Most education leaders still talk about AI data privacy as if it were a records problem. That framing is already outdated. The larger risk begins when AI systems turn ordinary learner activity into inferred identity, intervention signals and decision support that can shape outcomes long after the original educational purpose has passed. In education, that shift matters because federal student privacy guidance already treats metadata and indirect identifiers seriously, while recent U.S. privacy rules are moving toward tighter limits on disclosure, retention and downstream use.

This is the issue many boards are still mispricing. They are focused on the visible risks of AI, such as model quality, cybersecurity and adoption. Those are real concerns. But the more durable exposure sits one layer lower, in the path from data to inference. Once a platform starts converting prompts, hesitation, revision patterns, support requests or time-on-task into judgments about struggle, aptitude, engagement or risk, privacy stops being a notice-and-consent discussion. It becomes a governance issue. The question is no longer just what the institution collected. It is what the system concluded, who can act on it and whether that chain can be explained later.

How Learning Exhaust Becomes Decision Material

That is why the hidden liability in AI education is not data collection. It is a silent classification. Education systems now generate far more than student records. They generate machine-shaped interpretations of learners. A delay before answering may be read as uncertainty. Repeated edits may be read as a struggle. A pattern of support requests may be read as a dependency. None of these labels may appear in official language, but they can still influence recommendations, flags, interventions or performance signals. This is where ordinary learning exhaust becomes decision material. And once that happens, leaders are no longer managing information. They are managing reputational and legal exposure attached to judgments about people.

Why Vendor Compliance Is Not A Liability Shield

The problem is made worse by a dangerous executive assumption: that vendor compliance solves institutional risk. It does not. Department of Education guidance is clear that a provider can receive personally identifiable information under FERPA’s school official exception only if it is performing an institutional function, has a legitimate educational interest, remains under the school’s direct control and uses education records only for authorized purposes. The same guidance states that FERPA still governs the data after disclosure and that the school or district remains responsible for its protection. That is a much narrower and more demanding standard than many procurement teams act on in practice.

This is why contract language matters far more than many leaders assume. If a provider can broaden its terms, extend retention, repurpose metadata or introduce new uses without meaningful institutional control, then the organization may be carrying more risk than its executives realize. The federal guidance is explicit on best practices here as well: request only the minimum data needed, be clear about any data mining, prohibit use beyond the contract, maintain a destruction plan and avoid unilateral terms that make it hard for a school to demonstrate direct control. That is not administrative housekeeping. It is liability management.

The most useful recent case study is Illuminate. In December 2025, the FTC alleged that the company failed to deploy reasonable security measures for student data stored in cloud databases, even after being alerted to vulnerabilities. According to the FTC, a hacker used the credentials of a former employee who had left three and a half years earlier to breach the company’s systems in late 2021, gaining access to the personal data of 10.1 million students, including dates of birth, student records and health-related information. The FTC also alleged that some school districts representing more than 380,000 students were notified nearly two years late. The proposed order goes beyond a generic security fix. It requires deletion of unnecessary data, a public retention schedule and a comprehensive information security program. The lesson is simple: privacy failure is often an operational discipline failure long before it becomes an enforcement headline.

Why 2026 Changes The Conversation

This is also why 2026 feels different. The FTC’s January 2025 COPPA amendments tightened the rules around collection, use, disclosure and retention of children’s personal information. They require separate parental opt-in for targeted advertising and make clear that covered operators cannot retain data indefinitely. In California, the CPPA adopted regulations in July 2025, effective January 1, 2026, that implement requirements for risk assessments, annual cybersecurity audits and consumer rights tied to automated decision-making technology, with additional compliance time for some provisions. The details vary by legal regime, but the direction of travel is unmistakable: regulators are moving from asking whether a company disclosed its practices to asking whether it can defend the design and governance of the system itself.

The Governance Question Boards Should Be Asking Now

So the board-level questions need to change. Not, do we have a privacy notice? Not, has the vendor signed our template? The better questions are:

• What learner metadata are our tools collecting that we still treat as low risk?

• What inferences are our systems generating from that data?

• Which of those inferences influence recommendations, flags or interventions?

• Where are those outputs stored? Who can access them?

• Can they be challenged, corrected or deleted?

If leadership cannot answer those questions clearly, the institution does not yet have AI governance. It has AI exposure.

The hidden liability in AI education is not that institutions are collecting data. It is that their systems may already be manufacturing judgments they cannot easily justify. Leaders who keep treating privacy as a records issue will miss where the real risk is forming. It is forming in the inferences, the retention decisions, the vendor terms and the system behaviors that quietly turn educational activity into durable reputational signals. In the next phase of AI oversight, that is the layer regulators, auditors and boards will care about most.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?