· 7 min read

"We don't train on your data," one word at a time

Six words that sound like a privacy promise, and the room each one leaves. Why running AI on hardware you own makes the promise unnecessary.

Almost every AI service that wants business customers has the same sentence somewhere on its pricing page, its security page, or its sales deck: "We don't train on your data."

It's meant to end the conversation, and it usually does. For a lot of buyers it settles whether it's safe to paste in contracts, customer records, source code, or the board deck.

It's a narrower statement than it sounds. Each of its six words has a specific meaning, and each one leaves room for something you might not expect. None of this means any particular provider is doing something wrong. It means the sentence, read carefully, promises less than most people hear.

"We"

The first question is who "we" is.

Usually it's the company you signed up with, and only that company. But an AI service isn't one company. It runs on rented infrastructure, sends logs to monitoring tools, passes content through safety filters, and handles support tickets in someone else's help desk. The companies that do that work are called subprocessors. A provider's data processing terms usually list them, the list can be long, and the provider can usually change it by notifying you rather than asking you.

"Quality assurance" deserves a closer look. Many services keep prompts and responses for a while, often for weeks, to watch for abuse, and reserve the right to have people review flagged conversations. Those people aren't always employees. Labeling and review work is commonly contracted out, so someone at a company you've never heard of may read what your staff typed. Reading isn't training, so the promise still holds.

"We" also isn't permanent. Privacy policies almost always say that if the company is acquired, merges, or sells its assets, your information goes with it. In a bankruptcy, customer data is an asset like any other, and it can be sold to pay creditors. The buyer becomes the new "we," and it never made the promise.

Even while "we" stays the same, the promise is about training, not selling. Usage patterns, topic trends, and "anonymized insights" drawn from your data can be packaged into reports and sold or shared with partners without a single model being trained.

"Don't"

"Don't" is present tense. It describes what the company does today, not what it will do next year.

Terms of service change, and most say that continuing to use the service after a change means you accept it. Notice might be an email, or just a new date at the top of a page nobody rereads. AI companies are under constant pressure to find more training data, so a company that doesn't train on customer data today is making that call against its own incentives every quarter.

The harder question is what happens to data you've already sent. If your prompts are still stored when the policy changes, does the new policy reach them? The answer is rarely spelled out.

"Don't" is also not "can't." It's a policy decision, enforced by the provider's own controls and checked mostly by the provider. Deletion works the same way. In at least one lawsuit, a court ordered an AI provider to preserve user conversations it would otherwise have deleted, including ones users had deleted themselves. "We delete it after 30 days" holds only until someone with more authority says otherwise.

"Train"

"Train" has a narrow technical meaning: using data to adjust a model's internal settings, called weights. Plenty of things a provider can do with your data aren't that.

  • Testing new models against real prompts to see whether they're better.
  • Building and tuning the filters that decide what gets flagged, blocked, or sent to a human reviewer.
  • Generating synthetic data that imitates the patterns in real conversations.
  • Product analytics: which features people use, where answers fall short, what people ask about most.
  • "Improving our services," a phrase broad enough to hold everything above.

Some policies also draw a line between their main models and everything else, so the promise covers the flagship model but not the smaller systems around it.

The word to look for in a contract isn't "train." It's "use." Terms that list everything the provider may use your data for, with a short list, are a real commitment. Terms that only say what the provider won't train on leave everything else open.

"On"

"On" decides how close to your data the training has to get before the promise applies.

Training "on" your data suggests feeding your actual prompts and files into a model. A model can also learn from your data at one step removed. A summary of a conversation, labels describing what it was about, an embedding (a list of numbers that captures its meaning), or a made-up conversation written to resemble it are all derived from your data without technically being your data. Whether training on any of those counts as training "on your data" comes down to definitions most people never read.

Feedback is the clearest case. When someone clicks thumbs-up or thumbs-down, or picks the better of two answers, many terms treat that as feedback the provider can use however it likes, and the conversation it was about may come along for context.

"Your"

"Your" depends on who's asking and which plan they're on.

Business and developer plans usually carry the strongest promises. Free and personal plans often train by default unless someone finds the setting and turns it off. If an employee pastes a customer list into a personal account on their phone, the company's contract doesn't cover it, because as far as the provider knows, that was never the company's data. That's probably the most common way business information ends up in training data, and no policy page can prevent it.

"Your" also assumes the data is yours to give. A lot of what goes into AI tools belongs to someone else: customer records, patient notes, employee reviews, a vendor's pricing under NDA. Your agreements with those people may not let you share it with a third party at all, whatever that third party promises to do with it.

Finally, check how the terms define what's yours. It's usually "customer content," meaning what you send in and sometimes what comes back. Information about your account and how you use the service is usually defined as the provider's own data, under looser rules.

"Data"

That leads to the last word. "Data" sounds all-inclusive, but most terms split it into categories with different rules.

Content, meaning prompts, files, and responses, gets the promise. Metadata, such as who sent a request, when, from where, how long it was, and which features were used, often doesn't. Metadata says more than people expect. It can show which projects are active, who's working late, and when a company suddenly started asking a lot of questions about one particular acquisition target.

Aggregated and de-identified data are often carved out entirely, on the theory that once it's anonymized, it isn't yours anymore. De-identification is harder than it sounds, especially with free text, where a single prompt can hold a name, a deal size, and an internal project code.

And whether or not anyone trains on it, anything stored can be breached, subpoenaed, or exposed by a misconfiguration.

The only version of the promise that holds

Put the six words back together and the sentence means something like this: the company you signed with, under today's policy, isn't currently using the part of your account's data it classifies as content to adjust the weights of its main models. That's worth something. It's a long way from "your data is safe with us."

Running AI on hardware you control takes the sentence out of the picture. There's no "we," because there's no third party. "Don't" becomes "can't," because the model runs on a machine with no route to anyone else's servers. Prompts, files, and logs stay where your other sensitive information already lives, under retention and access rules you set.

This is practical now in a way it wasn't a few years ago. Open-weight models, which anyone can download and run, handle a large share of everyday business work well: summarizing documents, drafting and editing, answering questions about internal files, sorting tickets and email, and helping with code. A single well-equipped workstation or server can serve a small team. The cost is a hardware purchase instead of a per-use bill that grows as people rely on it.

Local isn't automatically private, and it isn't the answer to everything.

  • You take on the trust. The machine needs the same care as any server that holds sensitive information: access control, patching, backups, and logs someone actually reads.
  • Check what the software sends home. Some local AI tools collect usage statistics or check for updates. Block outbound traffic from the AI server by default, and "can't" stays true.
  • Know where the model came from. Download models from the original publisher and verify them, as you would any other software.
  • The largest hosted models still lead on the hardest problems. A sensible setup often uses both: local for anything sensitive, and a hosted service for work you'd be comfortable seeing in public.

If you use a hosted service

Hosted AI is the right call for plenty of work. If you use it with business information, get answers to these questions in the contract, not on a marketing page.

  1. Who are your subprocessors, and how will we find out when that list changes?
  2. How long do you keep prompts, files, and responses? Can people read them, and are those people your employees?
  3. Besides training, what can you use our data for? Does that include testing models, building filters, and generating synthetic data?
  4. Does the commitment cover metadata, feedback, and responses? What about aggregated or de-identified data?
  5. What happens to our data if you're acquired or go out of business?
  6. How much notice do we get before the terms change, and do new terms apply to data we've already sent?
  7. Can we turn off retention entirely?

If the answers are vague, assume anything you send could eventually be read by someone outside your company, and keep the sensitive work on hardware you own.

← All posts