OpenAI Will Honour A Shared Robots.txt Signal For AI Training
On 30 October 2026, OpenAI said it will honour one shared robots.txt signal that tells AI companies whether a website can be used for training. It is not switched on yet, and it is a separate choice from being found in AI search.
What OpenAI Announced
On 30 October 2026, OpenAI announced it will honour a common signal in robots.txt, the file a website uses to tell crawlers what they can and cannot do. The signal tells AI companies whether a site’s content can be used to train the foundation models behind generative AI.
The signal uses a shared name instead of one company’s crawler name: User-Agent: __usage_ai-training. It comes out of the AI Preferences working group at the Internet Engineering Task Force (IETF).
Until now, a site owner who wanted to say no to training had to name each AI company’s crawler and keep separate rules for every one. One shared signal is meant to reach every participating provider at once.
What Is Live And What Is Not
- Not live yet. OpenAI says it aims to implement the signal “in the coming months” and will post an update with details when it does.
- Voluntary. OpenAI calls it a voluntary effort. Nothing forces any AI company to honour it.
- Not a finished standard. The technical document behind it is an Internet-Draft, version 00, dated 30 October 2026. The IETF describes drafts like this as work in progress that can be updated or replaced at any time.
- Existing controls still apply. OpenAI says the new signal does not replace its current robots.txt crawler controls.
- Other providers are not named. OpenAI says it is joining other members of the working group in announcing the signal, but its post does not list them. It also says it encourages other AI providers to honour the signal. Treat coverage as OpenAI only until each company confirms for itself.
One detail matters if your site already has rules. Under the draft, a general rule written for every crawler (User-Agent: *) does not count as a training preference. If the shared signal is missing from your file, no preference has been expressed at all.
Being Found And Being Trained On Are Two Choices
These are easy to mix up.
AI search is when ChatGPT answers a question and points to your site. OpenAI runs a separate crawler for that, called OAI-SearchBot. Its crawler documentation says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.
Training is a different job. OpenAI’s GPTBot crawls content that may be used to train its models. OpenAI’s documentation says the settings are independent, so a site can allow search while saying no to training.
The new shared signal sits on the training side. The draft defines it as a statement about how content is used, not an instruction about crawling, and keeps it separate from any rules you set for named crawlers. So the two choices can be made on their own.
What It Means For An Australian Business
If your website’s job is to bring in enquiries, being cited when someone asks ChatGPT for a provider is the part that matters. Blocking the search crawler to stop training would cost you that visibility.
If you publish original work that is your product, like paid courses, research or detailed guides, a firm no on training may suit you.
Plenty of businesses will land in the middle: open to search, closed to training, with a few sections such as a media kit left open. The draft supports path-by-path choices like that.
Nothing changes on your site until the signal is live. OpenAI has not switched it on, and its post names no other provider.
What To Do Next
Make the decision now, so the change takes minutes when the signal goes live.
- Decide your training preference. Yes, no, or no except for named sections of the site. Write it down in one sentence.
- Keep AI search crawlers allowed if you want to be found. Ask whoever manages your site to confirm your robots.txt does not block OAI-SearchBot.
- Hand your web person the line for later. Ask them to add
User-Agent: __usage_ai-training, with rules that match your one sentence, once OpenAI confirms it is live.
If you are not sure what your robots.txt tells AI crawlers right now, get in touch.
Sources
Sam co-founded Echelon AI Solutions and leads transformation strategy, client engagements and growth. He has built and operated businesses across marketing and AI education, and has guided companies in retail, trades, hospitality and professional services through operational change. His focus is making AI earn its place through measurable business performance.