This repository contains up-to-date lists of IP addresses of known bots and crawlers, useful for whitelisting or filtering network traffic. They can also be used as blacklists.
Do you have any questions or want to receive notifications about important changes or new features in my repositories? Join my Discord server! If you don't use Discord, you can also open an issue on GitHub.
The project is released under the MIT license — you can do whatever you want with it.
If you like this repository, leave a star ⭐. Thank you!
- JSON:
source(string) →sources(array) - CSV: Column
Source→Sources, delimiter,→| - BGPView → RIPEstat: Migrated from BGPView (shut down Nov 26, 2025) to RIPEstat + WHOIS backup
Lists are updated every 6 hours.
| Service | Sources | Downloads |
|---|---|---|
| AhrefsBot | api.ahrefs.com | TXT • CSV • JSON |
| BetterStack | uptime.betterstack.com | TXT • CSV • JSON |
| BingBot | www.bing.com | TXT • CSV • JSON |
| Bunny CDN | IPv4 & IPv6 | TXT • CSV • JSON |
| Censys | RIPEstat & RADB & ARIN | TXT • CSV • JSON |
| Cloudflare | IPv4 & IPv6 | TXT • CSV • JSON |
| DuckDuckBot | duckduckgo.com | TXT • CSV • JSON |
| FacebookBot | RIPEstat & RADB & ARIN | TXT • CSV • JSON |
| Google Special Crawlers | developers.google.com | TXT • CSV • JSON |
| GoogleBot | developers.google.com | TXT • CSV • JSON |
| Modat Scanner | scanner.modat.io | TXT • CSV • JSON |
| OpenAI | GitHub | TXT • CSV • JSON |
| Palo Alto Networks | RADB & ARIN | TXT • CSV • JSON |
| PingdomBot | IPv4 & IPv6 | TXT • CSV • JSON |
| RSS API | rssapi.net | TXT • CSV • JSON |
| Semrush | RIPEstat & RADB | TXT • CSV • JSON |
| Shodan | Custom lists | TXT • CSV • JSON |
| Stripe | stripe.com | TXT • CSV • JSON |
| TelegramBot | core.telegram.org | TXT • CSV • JSON |
| UptimeRobot | uptimerobot.com | TXT • CSV • JSON |
| WebPageTest Bot | www.webpagetest.org | TXT • CSV • JSON |
| YandexBot | yandex.com | TXT • CSV • JSON |
| Kagi | Custom lists | TXT • CSV • JSON |
| NASK PL | RIPEstat & RADB | TXT • CSV • JSON |
https://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.txt
curl -fsS --max-redirs 0 -o all-safe-ips.txt https://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.txthttps://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.csv
curl -fsS --max-redirs 0 -o all-safe-ips.csv https://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.csvhttps://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.json
curl -fsS --max-redirs 0 -o all-safe-ips.json https://raw.githubusercontent.com/sefinek/known-bots-ip-whitelist/main/lists/all-safe-ips.json
