Crawling Checklist: Find, Control and Troubleshoot Search Bot Access

How crawling works

Crawling is the automated discovery and retrieval of URLs. A typical search pipeline discovers a URL through a link, sitemap or prior crawl, checks robots.txt, requests the resource, parses its response, extracts links and may render JavaScript before evaluating indexation eligibility.

Google primarily crawls with Googlebot Smartphone. Its smartphone and desktop crawlers share the Googlebot robots.txt token. Google says its average crawl rate is generally no more than one request every few seconds for a site, although short bursts can be higher.

A fetched page is not necessarily indexed. Search engines can exclude crawled pages because they are duplicate, low value, noncanonical, blocked from indexing or otherwise unsuitable. Conversely, a URL blocked by robots.txt can sometimes remain known and appear without useful page content or a snippet. This distinction prevents a common diagnostic error: repeatedly requesting a recrawl when the real problem is content quality, duplication or indexation control.

The complete crawling checklist

Apply this matrix by template and directory, not merely to a sample homepage. Product variants, location pages, article archives and faceted categories often fail in different ways.

Make important URLs easy to discover

Use standard <a href> links that resolve to final URLs. Google explicitly identifies anchor elements with href attributes as generally crawlable. Navigation triggered only by script events is less dependable. Avoid requiring a site search, form submission, hover action or endless scroll before a bot can reach core pages.

• Link every indexable page from a relevant category, topic hub or parent page.
• Keep XML sitemaps limited to canonical, indexable URLs that return 200.
• Split large sitemaps by content type or directory so failures can be isolated.
• Update last modification values only after meaningful page changes.
• Remove links to redirected, broken, blocked or obsolete destinations.

For content sites, build topic hubs that connect definitions, implementation guides, comparisons, troubleshooting pages and original research. This hub-and-spoke structure supports crawler discovery and helps answer systems understand entity relationships. Consolidate overlapping pages rather than maintaining multiple weak URLs for the same intent. When refreshing decayed content, preserve a stable canonical URL and add links from recently crawled pages.

Align robots.txt, noindex and canonical rules

Decision rule: use robots.txt when a bot should not fetch a path. Use a meta robots or HTTP noindex directive when a fetchable page should not appear in an index. Use a canonical when multiple accessible URLs represent substantially the same primary resource.

Do not block a URL in robots.txt and expect its page-level noindex to be processed reliably. The crawler may be unable to fetch the directive. Google also notes that robots.txt can be cached for up to 24 hours and sometimes longer during retrieval failures, so changes may not take effect immediately.

1. Choose the preferred URL for each duplicate cluster.
2. Link internally to that preferred URL.
3. Return a self-referencing canonical on the preferred page.
4. Redirect retired equivalents when users no longer need them.
5. Keep blocked, noncanonical and noindex URLs out of XML sitemaps.
6. Retest after deployments, migrations and CDN rule changes.

Canonical tags are signals, not a substitute for coherent architecture. Contradictory redirects, links, sitemap entries and canonicals force crawlers to reconcile unnecessary ambiguity.

Control crawl waste without hiding useful pages

Google describes crawl budget through two broad components: crawl capacity and crawl demand. Capacity reflects what a site and crawler can handle. Demand reflects factors such as URL importance, freshness and the crawler’s need to revisit content. Crawl budget is most consequential for very large, rapidly changing or inefficient sites, but crawl traps can harm sites of any size.

Common waste patterns

• Faceted navigation generating many filter combinations
• Sorting, tracking, print and duplicate query parameters
• Session IDs embedded in URLs
• Infinite calendars and unbounded pagination
• Internal search result pages
• Soft 404s that return 200
• Long redirect chains and redirect loops
• Thin tag, author or location archives

Inventory each pattern before blocking it. Some filtered pages can satisfy genuine demand and deserve static, internally linked landing pages. Others should be consolidated, noindexed, removed from navigation or prevented from generating crawlable combinations. A blanket robots.txt block can conceal the scale of duplication without resolving internal links or indexation signals.

Do not chase maximum crawl volume. The objective is a higher share of successful requests to fresh, canonical and commercially or editorially valuable pages.

Test JavaScript, mobile output and server behavior

JavaScript introduces separate crawling, rendering and processing stages. A URL can return an HTML shell successfully yet fail to expose meaningful content or links until rendering. Delayed API calls, consent barriers, unsupported interactions and client-side errors can therefore create partial discovery.

Compare four views: raw HTML, browser-rendered DOM, a search engine inspection render and a smartphone viewport. Confirm that titles, canonical tags, robots directives, headings, primary copy, structured data and internal links remain consistent. Test logged-out sessions and new visitors because bots do not inherit a customer’s cookies or account state.

Google recommends server-side rendering, static rendering or hydration as durable approaches. Dynamic rendering is not its preferred long-term solution. Whatever architecture is selected, the origin should return accurate status codes. An attractive error template with a 200 response can become a soft 404, while intermittent 5xx or 429 responses can reduce reliable access.

During a launch, monitor CDN and web application firewall rules as carefully as robots.txt. A permissive robots file does not override a challenge, rate limit, geographic restriction or user-agent rule enforced upstream.

Diagnose crawling failures with a five-step framework

1. Define the affected set

Group URLs by template, directory, canonical status, publication date and business value. A sitewide conclusion based on a few inspected URLs is rarely actionable.

2. Test discovery and permission

Trace internal links to each URL, validate sitemap membership and evaluate robots.txt for the exact user agent. Then test CDN, firewall, authentication and origin behavior.

3. Verify the response chain

Record every redirect and final status. Check headers, canonical targets, noindex directives, content type and mobile output. Resolve loops, chains and mismatched signals.

4. Compare rendered content

Determine whether essential copy and links exist before and after rendering. Review JavaScript errors and blocked resources. A rendered page should not depend on a click to reveal its primary purpose.

5. Reconcile logs with index reports

Server logs prove whether a declared bot requested a URL, subject to user-agent verification. Search consoles report their own sampled or processed views. A crawler tool models architecture from its starting conditions. Use all three rather than treating any single source as complete.

Measure crawl health and prioritize remediation

Create a weekly dashboard that segments verified search bot requests by directory and template. Useful measures include:

• Share of requests returning 200, 3xx, 4xx, 429 and 5xx
• Share of crawl activity reaching canonical, indexable URLs
• Median time from publication or material update to first crawl
• Recrawl interval for priority templates
• Orphan URL count and average internal click depth
• Redirect chain, soft 404 and duplicate parameter counts
• Submitted sitemap URLs compared with crawled and indexed URLs
• Search bot response time and bytes transferred

Prioritize by impact multiplied by affected URL count, then divided by implementation risk. A broken navigation component affecting every product page outranks a single stale redirect. A parameter trap receiving millions of requests can outrank minor metadata defects.

Use a 30-day implementation sequence: establish log and indexation baselines, repair access and status failures, align canonical signals, remove crawl traps, strengthen internal hubs, resubmit clean sitemaps and measure by template. Recrawl requests can help with a small number of important URLs, but Google states that requests do not guarantee immediate crawling or indexing and processing may take days to weeks.

Crawling for AI search and answer systems

Search crawling, AI training access and retrieval by answer products are related but distinct. A site can permit a traditional search crawler while restricting another user agent, or a CDN can block a crawler despite a permissive robots.txt policy. Maintain an explicit crawler policy that identifies the business purpose of each allowance or restriction.

For retrieval and answer absorption, expose concise definitions, self-contained factual passages, comparison tables and procedures in accessible HTML. Connect claims to primary sources and visible evidence. Stable URLs, descriptive headings and coherent internal links help systems locate the right passage, but no implementation guarantees inclusion in Google AI Overviews, AI Mode, Bing or Copilot, or ChatGPT responses.

Common Crawl illustrates the scale and selectivity of web collection. It reports more than 300 billion pages collected over 15 years and roughly 3 to 5 billion new pages per month. Its December 2025 archive contained 2.16 billion pages, including 783 million newly observed URLs. At that scale, scheduling, freshness and URL prioritization matter. Recent academic work similarly examines efficient crawler scheduling and the distinction between persistent core URLs and volatile shell URLs.

Original datasets, statistics pages, expert contributions and transparent methodology can create natural link demand and make a page more discoverable across search and research ecosystems. These assets still need crawlable links and stable hosting.

What is proven, practiced and still uncertain

Proven by official documentation: crawling and indexing are separate; robots.txt controls fetching rather than guaranteed index removal; standard anchor links are the dependable discovery mechanism; JavaScript can add rendering stages; and recrawl requests provide no timing or indexation guarantee.

Practitioner consensus: server logs are the strongest operational record of bot requests, crawl traps should be evaluated by template, and internal link prominence influences how quickly important URLs are found and revisited. Practitioners also commonly report crawled but not indexed pages despite valid sitemaps. That observation is consistent with the crawl and index distinction, but individual reports do not establish a universal cause.

Still uncertain or implementation-specific: the exact effect of a single crawl optimization on rankings, the recrawl schedule for a particular URL and how each answer system selects sources. Bot-management rules can also produce different outcomes by network, user agent and request pattern. Run controlled releases, preserve before-and-after logs and avoid attributing indexation changes to one edit without a stable comparison set.