Discover
Find the URL through a link, sitemap or submission.
Crawling and Indexing
Opening robots.txt is necessary when you want compliant crawlers to enter. It is not sufficient when the server, CDN, firewall, login or JavaScript challenge still refuses the request.
Crawled does not mean indexed, and indexed does not mean ranked
Request
The server returns the public page successfully.
Process
The service decides whether to store it.
Serve
The indexed page may or may not be selected later.
Open to all compliant crawlers
User-agent: *
Allow: /This permits every crawler that follows the Robots Exclusion Protocol.
Allow: /This permits every crawler that follows the Robots Exclusion Protocol.
Named sections are included for clarity. The wildcard rule already permits them.
Search access and training access can be separate
OpenAI identifies OAI-SearchBot for ChatGPT search discovery and GPTBot as a separate control. This package follows the requested fully open policy and permits both.
Test infrastructure, not only files
- Confirm robots.txt and sitemap.xml return HTTP 200.
- Check CDN and WAF bot rules.
- Remove CAPTCHA and login walls from public pages.
- Submit the sitemap to Google and Bing.
- Review server logs for successful crawler responses.
- Use IndexNow after material updates where supported.
Sources used on this page
- Robots Exclusion Protocol RFC — robots.txt standard.
- Google robots.txt introduction — Google crawl controls.
- OpenAI publishers FAQ — OAI-SearchBot and GPTBot.
- Perplexity crawler documentation — Perplexity crawler roles.
- IndexNow documentation — URL-change notifications.