The product is live. Customers are generating prompts, documents, transcripts, tickets, and outputs every day. And someone on the team says the obvious thing: “We’re sitting on the best training set we’ll ever have. Why aren’t we using it?”

The answer is that you might be able to. But “might” is doing a lot of work in that sentence.

This is a different question from who owns AI-generated content and output data. Ownership of outputs is about what comes out of the model. Training rights are about what goes in.

Using customer data to train AI models sits at the intersection of four bodies of law that rarely line up neatly: FTC consumer protection, state and international privacy statutes, sector rules like HIPAA and BIPA, and your own customer contracts. Get it right and your model compounds in value with every new customer. Get it wrong and a regulator can order you to delete the model itself.

Andrew S. Bosin, a two-time SaaS founder who now represents AI and SaaS startups nationwide, sees this question in nearly every AI engagement. This guide walks through how he analyzes it, where the real enforcement risk sits in 2026, and the contract language that keeps training rights defensible.

The Short Answer: It Depends on Three Documents

Whether you can train on a given dataset rarely turns on one rule. It turns on what three documents said at the time the data was collected:

• Your privacy policy and terms of service. What did you tell end users you would do with their data?
• Your customer contract and DPA. What did your B2B customer authorize, and what did they prohibit?
• Your legal role under privacy law. Are you a “service provider” or “processor” acting for the customer, or a business or “controller” acting for yourself?

Then layer on any sector-specific rules (health data, biometrics, children’s data, financial data).

If all three documents point the same direction and permit training, you are in good shape. If any one of them is silent, ambiguous, or contradicts the others, you have a problem that will surface during an enterprise security review, a diligence request, or a regulator’s inquiry.

Risk #1: Quietly Changing Your Terms Won’t Fix the Past

The most common founder instinct is to update the terms of service and move on.

The FTC has addressed this directly. In a February 2024 Technology Blog post titled AI (and other) Companies: Quietly Changing Your Terms of Service Could Be Unfair or Deceptive, the agency warned that adopting more permissive data practices, explicitly including using consumer data for AI training, and disclosing that change only through a surreptitious, retroactive amendment to a privacy policy or terms of service may violate Section 5 of the FTC Act.

The principle is old. It traces back to the FTC’s 2004 action against Gateway Learning. What is new is that the agency has now named AI training as the paradigm example.

2026 made it concrete. On March 30, 2026, the FTC settled with Match Group Americas and OkCupid over allegations that OkCupid gave roughly three million user photos, plus demographic and location data, to an AI company that used them to build facial recognition technology. The privacy policy at the time limited sharing to listed categories of recipients or situations where users got notice and an opt-out. The AI company fit neither.

There was no fine. But the order runs 20 years, with a 10-year compliance and reporting regime, and it is one of the first federal actions to frame undisclosed AI training data sourcing as a deception violation.

It also came only months after the FTC set aside a prior AI-related consent order (Rytr) as unduly burdensome to innovation. The takeaway: even an FTC that is pulling back on some AI theories still draws a hard line at broken privacy promises.

What this means for you:

• Data collected under a policy that did not disclose training generally cannot be swept into training by a later policy update alone.
• A forward-looking change needs prominent notice and, for material changes, affirmative consent, not a “last updated” date at the bottom of a page.
• Keep a dated archive of every version of your privacy policy and terms. You will need it to prove which data was collected under which promise.

Risk #2: Algorithmic Disgorgement — The Remedy That Deletes Your Model

Most legal risks cost money. This one can cost you the product.

“Algorithmic disgorgement” (also called model deletion) is an FTC remedy that requires a company to destroy not only improperly obtained data, but also any models or algorithms built from it. The FTC has imposed it repeatedly:

• Everalbum (2021): a photo-storage app that used customers’ photos to develop facial recognition was ordered to delete the models and algorithms developed from those photos.
• WW International / Kurbo (2022): ordered to delete children’s data collected without verifiable parental consent, along with models built using it.
• Rite Aid (2023–2024): ordered to delete facial recognition data and the models derived from it, and to instruct third parties that received the data to do the same.
• Avast (2024): required to delete models and algorithms derived from browsing data it sold.

Why does this matter so much for startups? Because models are not easily “un-trained.” Removing a slice of training data typically means retraining from scratch. If tainted data fed a base model that then fed fine-tuned variants, a deletion order can cascade through the whole model lineage.

The EU has reached a similar place. The European Data Protection Board’s Opinion 28/2024 (December 2024) confirms that unlawful processing during model development can taint the lawfulness of the model’s later use, and that supervisory authorities can order data deletion and, in serious cases, restrict or prohibit use of the model.

What this means for you: data lineage is not just an engineering nicety. You need to be able to prove which datasets trained which model versions, and that every one of those datasets was lawfully usable for training.

Risk #3: Training Can Change Your Legal Role

This is the risk founders most often miss, and enterprise customers’ privacy counsel most often catch.

Most B2B SaaS and AI companies process customer data as a service provider (CCPA) or processor (GDPR). (For a broader primer on how these regimes apply to software companies, see Andrew’s guide to navigating data privacy laws for SaaS companies.) That status is valuable. It means your customer’s data flowing to you is not a “sale,” and you are acting on the customer’s instructions, not your own.

Training a shared model on that data can put the status at risk.

Under the CCPA. The regulations at Cal. Code Regs. tit. 11, § 7050(a)(3) let a service provider use personal information internally to build or improve the quality of the services it provides to that business, but only if the service provider does not use the personal information to perform services on behalf of another person. A model trained on Customer A’s personal information and then served to Customers B through Z sits uncomfortably close to that line.

If you cross it, you may no longer be a service provider for that data. That can turn the customer’s disclosure to you into a “sale” or “sharing,” triggering opt-out rights and contract obligations your customer never signed up for.

Under the GDPR. A processor that decides to use customer personal data for its own purpose, such as improving a general model, is determining purposes and means. That is controller behavior. As a controller, you need your own lawful basis, and the EDPB’s Opinion 28/2024 makes clear that legitimate interest is not an automatic default for AI training. It requires a documented three-step test (legitimate interest, necessity, balancing), with mitigations such as pseudonymization or synthetic data weighing in your favor.

What this means for you: if you want to train cross-customer models on personal data, your DPA must say so expressly, your privacy disclosures must match, and you should assume enterprise privacy teams will push back.

Risk #4: Sector Rules and California’s New Risk Assessments

Certain data categories carry their own training restrictions regardless of what your terms say.

• Health data (HIPAA). If you sign a Business Associate Agreement, PHI can be used only as the BAA permits. Most BAAs do not permit training general-purpose models. De-identification under HIPAA’s Safe Harbor or Expert Determination methods is the usual path, and it must happen before training.
• Biometric data (BIPA and similar statutes). Illinois BIPA requires informed written consent before collecting biometric identifiers, carries a private right of action, and has produced some of the largest privacy settlements in U.S. history. Voice prints and face geometry are squarely covered.
• Children’s data (COPPA). The Kurbo order shows that models trained on improperly collected children’s data are a deletion target.

California’s 2026 regulations add a documentation layer. The CPPA’s updated CCPA regulations took effect January 1, 2026. Among other things, they require a formal risk assessment before processing personal information to train automated decision-making technology used for significant decisions, or to train facial recognition, emotion recognition, identity verification, or other biometric identification technology. For assessments conducted in 2026 and 2027, summary information must be submitted to the agency by April 1, 2028.

If your product touches hiring, lending, housing, healthcare, education, or biometrics, budget for this work now.

Risk #5: The Enterprise Contract Chain

Even with zero personal data involved, your customer contract may prohibit training outright.

Customer data is almost always defined as the customer’s Confidential Information. Using it to train a model that other customers can query creates a real risk of “regurgitation,” where the model reproduces fragments of one customer’s confidential material in another customer’s output. That is a confidentiality breach, and potentially a trade secret problem, independent of privacy law.

Sophisticated buyers know this. Enterprise MSAs and AI addenda increasingly include an explicit no-training clause, and vendors’ renewal paperwork is now a known pressure point: customers are scrutinizing whether updated terms quietly expand the vendor’s data rights.

There is also a back-to-back problem, one of the key issues in negotiating AI technology vendor contracts. If your product is built on a third-party foundation model, that upstream provider’s terms govern what happens to the data you send it. If you promise your enterprise customer “no training” but your API tier lets the upstream provider train on inputs, you have made a promise you cannot keep. Before you sign a no-training clause, confirm your upstream terms (and your API account settings) match it.

Risk #6: Consent That Doesn’t Reach Everyone in the Data

Some products capture data about people who never agreed to your terms.

Meeting assistants, call recorders, email tools, and customer-support AI routinely process data from third parties: the other people on the call, the sender of the email, the customer on the support chat. Your user’s consent does not automatically extend to them.

The point is being tested now. A federal class action filed in August 2025 against Otter.ai alleges that the company used recorded meeting conversations to train its transcription models without the informed consent of all participants, and challenges whether its de-identification process actually removes confidential information. Those are allegations, not findings. But they show exactly where plaintiffs will aim: a checkbox from one user, many non-consenting participants, and a de-identification claim that is not publicly explained.

What this means for you: if your training data includes people other than your account holder, map whose data it is, which wiretap and recording-consent laws apply, and whether your de-identification process would hold up if a plaintiff’s expert examined it.

The Spectrum of Training Rights

Training rights are not all-or-nothing. In practice, Andrew structures them along a spectrum, and the right position depends on your product, your customer base, and your data.

1. No training. Customer data is used only to deliver the service to that customer. Pre-trained models only. Easiest enterprise sale; no compounding data advantage.
2. Tenant-isolated training. Customer data trains or fine-tunes only that customer’s private model instance. Low cross-customer leakage risk; generally consistent with service-provider and processor status.
3. De-identified and aggregated usage data. You train on telemetry, metadata, or properly de-identified content, not raw customer content. Defensible if “de-identified” actually meets the statutory standard (more on that below).
4. Opt-in contribution. Customers affirmatively choose to contribute data to shared model improvement, often in exchange for pricing or features. Clean consent record; slower data accumulation.
5. Broad training license. Customer data may be used to train and improve models across your customer base. Maximum data advantage; maximum regulatory and sales friction. Generally unsuitable for personal data, regulated data, or enterprise customers.

Many successful AI companies run a tiered model: options 1 or 2 by default for enterprise, option 3 across the board, and option 4 offered as a commercial choice.

A Word on “Aggregated and Anonymized” Clauses

Nearly every SaaS template includes a clause letting the vendor use “aggregated and anonymized data” for any purpose. It is one of the hidden SaaS contract clauses that deserves a second look. Founders often assume this covers model training.

It may not.

• The language predates generative AI. A court reading an ambiguous pre-2023 clause could easily conclude the parties did not contemplate training large models, and ambiguity is typically construed against the drafter, which is you.
• “Anonymized” has a legal meaning. Under the CCPA, “deidentified” information requires reasonable technical measures to prevent reidentification, a public commitment not to reidentify, and contractual prohibitions on recipients reidentifying it. Under the GDPR, anonymization is a high bar, and the EDPB has said AI models are not presumed anonymous.
• Raw content is not “aggregated.” Training on full customer documents or conversations is hard to characterize as using aggregated statistics.

If you intend to train, say so in plain words. Do not rely on a legacy clause to carry the weight.

Sample Clause Language

The following is illustrative language of the kind Andrew uses as a starting point. It is not a substitute for drafting tailored to your product, data flows, and customer base.

Defining the data categories. Separating categories is the foundation of every defensible training position.

“Customer Content” means all data, files, prompts, inputs, and other materials submitted to the Services by or on behalf of Customer or its Authorized Users, and all Outputs generated from them.

“Usage Data” means technical and operational data relating to the performance, configuration, and use of the Services (such as feature usage, latency, error logs, and system metrics) that does not include Customer Content.

“De-identified Data” means data derived from Customer Content that cannot reasonably be used to infer information about, or otherwise be linked to, Customer, any Authorized User, or any identified or identifiable individual, and that Provider (i) maintains using reasonable technical and organizational measures to prevent re-identification, (ii) publicly commits not to re-identify, and (iii) contractually prohibits any recipient from re-identifying.

Enterprise-friendly no-training default with tenant isolation.

Provider will not use Customer Content to train, retrain, fine-tune, or otherwise improve any machine learning or artificial intelligence model that is made available to any person other than Customer. Provider may use Customer Content to train or fine-tune a model instance used exclusively to provide the Services to Customer, provided that such model instance is not made available to, and its weights or parameters are not shared with, any third party. Provider may use Usage Data and De-identified Data to maintain, secure, and improve the Services.

Opt-in contribution program.

If Customer elects in writing (including through an administrative setting in the Services) to participate in Provider’s Model Improvement Program, Customer authorizes Provider to use Customer Content submitted after the date of such election to train and improve Provider’s models generally. Customer may withdraw its election at any time, effective prospectively. Withdrawal will not require Provider to retrain or delete any model trained before the effective date of withdrawal, but Provider will exclude Customer Content submitted after that date from all future training.

Upstream provider flow-down.

Provider will ensure that any third-party model provider that processes Customer Content in connection with the Services is contractually prohibited from using Customer Content to train or improve its models, and will provide written confirmation of such restriction upon Customer’s reasonable request.

Notice what the opt-in clause does: it makes the prospective-only nature of withdrawal explicit. That is the practical answer to “what happens if they change their mind,” and it is far better negotiated up front than argued about after a model has shipped.

Founder Checklist Before You Turn On Training

Before any customer data enters a training pipeline, Andrew recommends working through these questions:

• Inventory. What data categories are in the proposed training set? Personal data? Third-party data? PHI, biometrics, children’s data, or financial data?
• Collection-time promises. What did the privacy policy and terms say when each dataset was collected? Do you have dated copies?
• Customer contracts. Do any MSAs, DPAs, BAAs, or order forms prohibit or limit training? Do any include a no-training clause or a narrow purpose limitation?
• Legal role. Will training change your service-provider or processor status? Do your DPAs authorize it?
• Upstream terms. Do your foundation-model provider’s terms and API settings match the promises you make downstream?
• De-identification. If you rely on de-identification, does your process meet the CCPA and GDPR standards, and is it documented?
• Risk assessment. Does California’s 2026 risk-assessment requirement apply? Does a GDPR legitimate-interest assessment or DPIA?
• Lineage. Can you trace which datasets trained which model versions, so a single bad dataset doesn’t force deletion of everything?
• Disclosure. Are your public statements about training accurate? Marketing claims that overstate privacy protections are their own enforcement risk.

Frequently Asked Questions

Can I use customer data to train my AI model if my terms of service allow it?

Only if those terms were in place, and adequately disclosed, when the data was collected, and only if your customer contracts and privacy-law role also permit it. The FTC has warned that retroactive, low-visibility changes to terms or privacy policies to permit AI training may be unfair or deceptive.

Is using de-identified or anonymized data safe for AI training?

It is much safer, but only if the data genuinely meets the legal standard. The CCPA requires technical safeguards, a public no-reidentification commitment, and contractual restrictions on recipients. The GDPR’s anonymization bar is high, and the EDPB has said AI models are not presumed anonymous. Weak de-identification is a common plaintiff target.

What is algorithmic disgorgement?

It is an FTC remedy requiring a company to delete not only improperly obtained data but also the models and algorithms built with it. The FTC has imposed it in cases including Everalbum, WW/Kurbo, Rite Aid, and Avast.

Does training on customer data affect my status as a CCPA service provider?

It can. California’s regulations allow service providers to use personal information to improve the services they provide to that business, but not to perform services on behalf of other businesses. Training a shared, cross-customer model on one customer’s personal information can jeopardize service-provider status.

My enterprise customer wants a no-training clause. Should I accept it?

Often, yes, at least for raw customer content. Many AI companies accept no-training for shared models while preserving rights to tenant-isolated fine-tuning, Usage Data, and properly De-identified Data. Before signing, confirm that your upstream model provider’s terms match the promise.

If a customer opts out later, do I have to retrain my model?

That depends on what your contract says. Well-drafted opt-in programs make withdrawal prospective only. Without that language, you may face an argument that continued use of a model trained on the customer’s data violates their withdrawal. Regulators, separately, can order model deletion where the original training was unlawful.

Do I need consent from people who are not my customers, like meeting participants?

Potentially. Your account holder’s consent does not automatically cover third parties whose data appears in recordings, emails, or chats. Recording-consent and wiretap laws, biometric statutes, and privacy laws may each apply independently.

Get Your Training Rights Right Before You Train

Training rights are one of the most valuable assets an AI company has, and one of the easiest to lose. The time to structure them is before the first training run, not after a regulator, an enterprise customer, or an acquirer’s counsel asks how your model was built.

Andrew S. Bosin represents AI and SaaS startups nationwide on a fixed-fee basis. As a two-time SaaS founder himself, he drafts training-data provisions, DPAs, enterprise AI addenda, and privacy disclosures that hold up in enterprise security reviews and investor diligence, without slowing down your sales cycle.

Contact Andrew S. Bosin, Esq. for a fixed-fee quote:

Andrew S. Bosin LLC

36 Highland Road, Glen Rock, NJ 07452

📞 (201) 446-9643

📧 andrewbosin@gmail.com

🌐 www.njbusiness-attorney.com

This article is for general informational purposes only and does not constitute legal advice. Reading it does not create an attorney-client relationship. Attorney Advertising.

#AILaw #SaaSLaw #AIStartups #DataPrivacy #AITrainingData #CCPA #GDPR #FTC #StartupLawyer #TechLaw