1.Baiduspider
百度爬虫
"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Baiduspider-render
"Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)
2.Googlebot
Googlebot 是谷歌的搜索引擎蜘蛛
"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
3.bingbot
微软
"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"
4.Bytespider
字节跳动旗下头条搜索的爬虫
"Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])
5.Sogou web spider
"Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
6.iaskspider
新浪爱问蜘蛛
"GET //xmlrpc.php HTTP/1.1" 302 15 "http://www.rebatesme.com/" "Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)"
7.Yahoo! Slurp China
"http://rebatesme.com" "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html\xA1\xB1)"
8.msnbot
"msnbot/1.0 ( http://search.msn.com/msnbot.htm)"
标签：5.0,www,Mozilla,网页,常见,爬虫,compatible,http,com From： https://www.cnblogs.com/yangmeichong/p/18384587