IPBot
Get Started
Back to crawler hub
provider

Google Crawlers

See how Googlebot differs from Google's special crawlers and user-triggered fetchers, and how IPBot treats verified versus known-only Google traffic.

What this crawler family is

Google’s crawler estate is broader than the generic Googlebot label most dashboards show. That matters because a search crawler, a product-specific crawler, and a user-triggered fetcher should not automatically inherit the same block or allow decision.

Verification and source data

IPBot uses the source inventory below to decide whether a request is merely known or fully verified.

Swipe sideways to review every verification column.

Crawler Intent Verification CIDRs Freshness
Googlebot
Official source
search Reverse + forward DNS 311
IPv4 167 / IPv6 144
2026-05-22
Google Special Crawlers
Official source
special case Known IP range 266
IPv4 133 / IPv6 133
2026-05-22
Google User-Triggered Fetchers
Official source
user triggered Known IP range 1048
IPv4 524 / IPv6 524
2026-05-22

Whether to allow, block, or separate it

Keep Googlebot policy separate from special crawlers and user-triggered fetchers. Search indexing traffic is not the same as fetches triggered by publisher tools or end-user actions.

robots.txt guidance

Apply standard search indexing controls to Googlebot, but document exceptions for special-case crawlers or fetchers that support previews, validation, or product workflows.

Cloudflare and WAF guidance

Only treat Googlebot as verified after reverse and forward DNS confirm the hostname. Keep special crawlers and user-triggered fetchers in their own rulesets when their business impact differs.

Googlebot

Standard `Googlebot` controls apply through `robots.txt` and search indexing directives.

Safe to treat as a verified crawler when reverse and forward DNS both succeed.

Google Special Crawlers

These are not the same as generic Googlebot indexing traffic; apply product-specific rules where needed.

Keep these separate from search crawler rules because the operational purpose differs.

Google User-Triggered Fetchers

These requests are initiated by a user or publisher workflow and should not be treated as standard search crawling.

Avoid blocking these with the same policy you use for autonomous indexing bots if previews matter.

FAQ

How does IPBot verify Googlebot?

IPBot verifies Googlebot through reverse and forward DNS. The IP must resolve to an allowed Google hostname and that hostname must resolve back to the same IP.

Are all Google crawler families search bots?

No. Googlebot is the main search crawler, but Google also publishes special-case crawler families and user-triggered fetchers with different operational roles.