OAI-SearchBot Checklist: Access, Verification and GEO

The OAI-SearchBot checklist

As of August 11, 2026, the practical objective is to make selected public content technically accessible without surrendering separate control over training collection. Complete these checks in order:

1. Choose a policy: decide which public sections should be eligible for ChatGPT Search and which must remain excluded.
2. Set robots.txt: add User-agent: OAI-SearchBot and Allow: /, or use narrower directory rules.
3. Control GPTBot separately: allowing search discovery does not require allowing potential training collection.
4. Test infrastructure: inspect CDN, WAF, hosting, geo, authentication and rate-limit controls.
5. Verify identity: compare request IPs with OpenAI’s published ranges. Never trust the user-agent alone.
6. Return clean responses: priority URLs should resolve through minimal redirects to canonical 200 pages.
7. Expose the answer: place essential definitions, facts and conclusions in initial HTML.
8. Strengthen discovery: use canonical tags, XML sitemaps and internal links from relevant hubs.
9. Measure outcomes: monitor verified requests, citations, ChatGPT referrals, engagement and conversions.

OAI-SearchBot, GPTBot and ChatGPT-User are different controls

Do not apply one blanket rule to every OpenAI agent. Each represents a different access decision. The distinction lets a publisher pursue ChatGPT Search visibility while maintaining a separate training preference.

OpenAI’s publisher guidance says sites should not block OAI-SearchBot if they want content considered for summaries and snippets. It also states that public URLs may sometimes remain discoverable as titles and links through third parties even when crawling is blocked. When exclusion is required, OpenAI recommends permitting crawl access while serving noindex so the directive can be observed.

Configure robots.txt without creating a policy conflict

The broad eligibility rule is User-agent: OAI-SearchBot followed by Allow: /. Place the directives in the root robots.txt file and test the exact production host, protocol and subdomain. A rule on www.example.com does not automatically govern an API host, asset host or separate country domain.

If search visibility is welcome but training collection is not, create distinct groups: allow OAI-SearchBot and disallow GPTBot. Review broader wildcard groups because an earlier or more specific rule may produce an unintended result. Document the business owner, approval date and affected directories so a future deployment does not silently reverse the decision.

Robots.txt is an access convention, not authentication or a secrecy mechanism. Private, paid, regulated or personally identifiable material belongs behind genuine access controls. Conversely, do not block a URL in robots.txt and expect a page-level noindex directive to be seen. The crawler generally needs page access to observe that directive.

Verify that OpenAI can reach the content

A successful robots test is only the first layer. The most common operational mismatch is an allowed crawler receiving a 403, 429, challenge page or connection failure from infrastructure farther downstream. Inspect edge and origin logs for the full request path.

Verification sequence

1. Filter logs for the OAI-SearchBot user-agent, but treat those matches as unverified candidates.
2. Compare source addresses with OpenAI’s currently published IP ranges. Use reverse DNS as supporting evidence, not as the only test.
3. Separate robots.txt requests from requests for HTML, feeds, sitemaps and assets.
4. Calculate response codes and latency for verified requests.
5. Replay representative requests through the same CDN, WAF and origin route.
6. Check whether bot management, geo rules, TLS policy, authentication, redirect chains or rate limits alter the response.

A narrow May 2026 JS SEO Lab experiment recorded ten OAI-SearchBot requests on one new Next.js domain, all for robots.txt and none for content. That result is useful as a warning: a robots request does not prove content crawling. It is not evidence that OpenAI behaves this way across established sites.

Make each eligible page retrievable and citation ready

OAI-SearchBot access is a prerequisite, not a GEO shortcut. A page still needs a clear relationship to the query, reliable evidence and passages that an answer system can extract without reconstructing the entire page.

• State the primary answer near the beginning in concise, self-contained language.
• Use descriptive headings for definitions, comparisons, procedures, exceptions and troubleshooting.
• Put essential text, tables and links in the server response or initial HTML.
• Avoid placing the only substantive answer behind tabs, client-side API calls, infinite scroll or mandatory interaction.
• Use one stable canonical URL and include only canonical, eligible URLs in XML sitemaps.
• Show meaningful authorship, publication or update dates, responsible entities and source attribution.
• Link supporting pages from a topic hub with descriptive anchor text.

Build a hub around the core entity and use spokes for likely query fanout, such as bot identification, robots rules, WAF troubleshooting, GPTBot comparisons, citation tracking and log analysis. Consolidate overlapping pages rather than forcing several weak URLs to compete for the same intent. Refresh decayed facts, broken citations and obsolete screenshots on a documented schedule.

Create evidence that answer systems and people can reuse

Pages earn natural citation demand when they contribute something more useful than a rewritten definition. Strong assets include maintained bot reference pages, original log datasets, response-code benchmarks, policy comparison matrices, reproducible technical experiments and statistics pages with explicit methods.

For competitive research, map which domains are cited across a fixed set of ChatGPT questions, Google AI results and Bing or Copilot answers. Perform link-intersect analysis on those cited sources, identify unlinked brand mentions, and pursue expert contributions or digital PR only where the underlying asset deserves coverage. Comparison pages should disclose evaluation criteria and commercial relationships.

Controlled title and intent tests can improve discoverability, but change one major variable at a time and preserve a query-level baseline. High-volume automated pages, recycled comparisons and unsupported superlatives offer weak long-term value. They also create quality and reputation risk. Never use fabricated evidence, cloaking, hidden text, doorway pages, deceptive redirects or structured data that conflicts with visible content.

Measure eligibility, visibility and business value separately

Crawler volume is an operational metric, not the final outcome. Trakkr observed 575,788 AI crawler visits across 84 brands and 314,501 URLs from June 2025 through February 2026. In that vendor dataset, OAI-SearchBot represented 15.1 percent of observed visits and GPTBot represented 57.2 percent. These are sample-specific observations, not universal market shares.

Maintain a repeatable query panel rather than checking a few favorable examples. Record interface, account state, location, date, wording, cited domains and target URL. Answer composition can vary, so trend direction and repeated inclusion are more informative than a single screenshot.

Diagnostic decision framework

If the crawler receives 200 responses but the returned HTML is an empty application shell, treat that as a retrieval failure even if browser rendering looks correct. Inspect the raw response, canonical, robots meta tag, title, headings and substantive text.

What is proven, accepted and still uncertain

Proven by official guidance or controlled research

OAI-SearchBot is distinct from GPTBot, checks robots.txt and is associated with potential ChatGPT Search inclusion. OpenAI provides separate controls and says placement is not guaranteed. A 2025 UCSD and IMC study tested major AI crawlers and found OAI-SearchBot respected robots.txt in its experiments.

Strong practitioner consensus

Teams should verify IPs, inspect edge logs, serve important content in initial HTML, maintain canonical discipline and measure citations rather than crawl counts alone. Crawloria and other technical practitioners report accidental WAF blocking as a recurring implementation problem.

Still uncertain or variable

OpenAI does not disclose a complete ranking system, crawl schedule or page selection model. The relationship among crawling frequency, citation frequency and freshness is not publicly deterministic. Community reports about citation differences among ChatGPT interfaces and APIs are anecdotal and methodologically inconsistent. Use them to form tests, not to make universal claims.

A practical rollout and build versus buy decision

Week one: approve policy, separate the three OpenAI agents, update robots.txt and create a rollback plan. Week two: verify CDN, WAF and origin behavior, then establish log filters based on published IP ranges. Week three: repair canonical, rendering, sitemap and internal-link defects on the highest-value answer pages. Week four: launch a query panel and referral dashboard, then document the baseline.

A small site with accessible server logs can usually operate this process with existing analytics, a log pipeline and manual citation checks. Larger publishers may benefit from a crawler intelligence or AI visibility platform when they manage many domains, edge providers, query markets or compliance policies. Evaluate tools on verified IP handling, raw-log retention, bot separation, query-level citation history, exports and conversion integration. Do not buy based on a proprietary visibility score alone.

Escalate to technical SEO, security and legal stakeholders when bot access affects licensed content, customer data, contractual restrictions or regulatory duties. The best policy is selective and auditable: expose material intended for public discovery, protect sensitive systems with real access controls, and review performance against measurable business value.