IPBot
Get Started
Crawler Data Hub

Crawler verification guidance for site owners

Root-level guides for the crawler families IPBot already tracks in its public range inventory, plus the verification workflow that turns raw log lines into operational decisions.

Open Crawler Verify Tool API Reference

Provider guides

Each page explains what the crawler family is, how IPBot verifies it, and how to treat it in `robots.txt`, Cloudflare, and WAF rules.

bing

Bingbot

Verify Bingbot traffic correctly, understand what IPBot can confirm through DNS, and decide how to treat Bing crawling separately from lookalike automation.

Open guide
amazon

Amazon Crawlers

Understand the differences between Amazonbot, Amazon SearchBot, and Amazon's user-triggered fetchers so you can avoid mixing search, crawl, and live-fetch policies.

Open guide
google

Google Crawlers

See how Googlebot differs from Google's special crawlers and user-triggered fetchers, and how IPBot treats verified versus known-only Google traffic.

Open guide
openai

OpenAI Crawlers

Understand the difference between GPTBot, OAI-SearchBot, and ChatGPT-User, how IPBot verifies them, and how site owners should treat each one.

Open guide
perplexity

Perplexity Crawlers

Distinguish PerplexityBot from Perplexity user-triggered fetch traffic and decide whether your policy should preserve one while restricting the other.

Open guide

Tracked crawler catalog

This table is generated from `data/netintel-ranges.jsonl` at build time so the public pages stay aligned with IPBot's crawler source inventory.

Swipe sideways to review the full crawler catalog.

Crawler Intent Verification CIDRs Updated
Amazonbot
Amazon
 training Known IP range 524 total
IPv4 524 / IPv6 0
2026-04-30
Amazon SearchBot
Amazon
 search Known IP range 512 total
IPv4 512 / IPv6 0
2025-11-21
Amazon User-Triggered Fetcher
Amazon
 user triggered Known IP range 1023 total
IPv4 1023 / IPv6 0
2025-11-04
Bingbot
Bing
 search Reverse + forward DNS 28 total
IPv4 28 / IPv6 0
2024-01-03
Googlebot
Google
 search Reverse + forward DNS 311 total
IPv4 167 / IPv6 144
2026-05-22
Google Special Crawlers
Google
 special case Known IP range 266 total
IPv4 133 / IPv6 133
2026-05-22
Google User-Triggered Fetchers
Google
 user triggered Known IP range 1048 total
IPv4 524 / IPv6 524
2026-05-22
ChatGPT-User
OpenAI
 user triggered User-agent + official IP range 247 total
IPv4 247 / IPv6 0
2026-05-21
GPTBot
OpenAI
 training User-agent + official IP range 21 total
IPv4 21 / IPv6 0
2025-10-30
OAI-SearchBot
OpenAI
 search User-agent + official IP range 35 total
IPv4 35 / IPv6 0
2026-01-02
Perplexity-User
Perplexity
 user triggered Known IP range 4 total
IPv4 4 / IPv6 0
2025-10-17
PerplexityBot
Perplexity
 assistant Known IP range 8 total
IPv4 8 / IPv6 0
2025-02-07

Provider breakdown

Amazon

  • Amazonbot
    training · Known IP range
  • Amazon SearchBot
    search · Known IP range
  • Amazon User-Triggered Fetcher
    user triggered · Known IP range

Bing

  • Bingbot
    search · Reverse + forward DNS

Google

  • Googlebot
    search · Reverse + forward DNS
  • Google Special Crawlers
    special case · Known IP range
  • Google User-Triggered Fetchers
    user triggered · Known IP range

OpenAI

  • ChatGPT-User
    user triggered · User-agent + official IP range
  • GPTBot
    training · User-agent + official IP range
  • OAI-SearchBot
    search · User-agent + official IP range

Perplexity

  • Perplexity-User
    user triggered · Known IP range
  • PerplexityBot
    assistant · Known IP range

Comparison and concept guides

concept

Known vs Verified Crawler

Learn the core IPBot distinction between a crawler that is merely known from range intelligence and one that is fully verified from stronger evidence.

Open guide