What changed
OpenAI announced support for a shared robots.txt training-preference signal on October 30. Publishers should distinguish discovery from model training.
Why it matters
A publication may want its articles to be searchable while setting a different policy for training. Those choices require reading the actual crawler documentation: similar user-agent names can serve different purposes. Robots rules express crawler instructions; they are not access control for private material. Sensitive content belongs behind authentication.
What to check next
Inventory public content and the crawlers you want to serve. Test robots syntax and confirm important articles remain crawlable. Document the policy for a future site owner and review provider behaviour periodically. Do not assume one directive controls every AI system.
Event dates refer to the original announcement. Publication dates show when this article first appeared on Codorc.
