首页 > 数据库 >scrapy-redis分布式

scrapy-redis分布式

时间：2022-12-06 19:57:24浏览次数：78  
标签：5.0 like KHTML Mozilla redis scrapy Safari Gecko 分布式

一、简介

  scrapy是一个基于redis的scrapy组件，用于快速实现scrapy项目的分布式数据爬取。

（一）安装redis

  • pip install scrapy_redis

（二）执行流程图

  • 调度器、管道不可以被分布式集群共享

二、中间件的使用

  • 下载中间件（Downloader Middleware）核心方法有3个

（一）设置headers和ip池

1、在middlewares里的MySpiderDownloaderMiddleware类属性中写入列表。

class MySpiderDownloaderMiddleware:
    def __init__(self):
        self.user_agent_list = [
            "Mozilla/5.0 (Linux; U; Android 2.3.6; en-us; Nexus S Build/GRK39F) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Avant Browser/1.2.789rel1 (http://www.avantbrowser.com)",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.0.249.0 Safari/532.5",
            "Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/532.9 (KHTML, like Gecko) Chrome/5.0.310.0 Safari/532.9",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14",
            "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.27 (KHTML, like Gecko) Chrome/12.0.712.0 Safari/534.27",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1",
            "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.120 Safari/535.2",
            "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.36 Safari/535.7",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0 x64; en-US; rv:1.9pre) Gecko/2008072421 Minefield/3.0.2pre",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.0.11) Gecko/2009060215 Firefox/3.0.11 (.NET CLR 3.5.30729)",
            "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB5",
            "Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E)",
            "Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
            "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1",
            "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0a2) Gecko/20110622 Firefox/6.0a2",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1",
            "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre",
            "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0 )",
            "Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90)",
            "Mozilla/5.0 (Windows; U; Windows XP) Gecko MultiZilla/1.6.1.0a",
            "Mozilla/2.02E (Win95; U)",
            "Mozilla/3.01Gold (Win95; I)",
            "Mozilla/4.8 [en] (Windows NT 5.1; U)",
            "Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.4) Gecko Netscape/7.1 (ax)",
            "HTC_Dream Mozilla/5.0 (Linux; U; Android 1.5; en-ca; Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (hp-tablet; Linux; hpwOS/3.0.2; U; de-DE) AppleWebKit/534.6 (KHTML, like Gecko) wOSBrowser/234.40.1 Safari/534.6 TouchPad/1.0",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; sdk Build/CUPCAKE) AppleWebkit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Nexus One Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; htc_bahamas Build/CRB17) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.1-update1; de-de; HTC Desire 1.19.161.5 Build/ERE27) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; de-ch; HTC Hero Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.1; en-us; HTC Legend Build/cupcake) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 1.5; de-de; HTC Magic Build/PLAT-RC33) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1 FirePHP/0.3",
            "Mozilla/5.0 (Linux; U; Android 1.6; en-us; HTC_TATTOO_A3288 Build/DRC79) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 1.0; en-us; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-us; T-Mobile G1 Build/CRB43) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari 525.20.1",
            "Mozilla/5.0 (Linux; U; Android 1.5; en-gb; T-Mobile_G2_Touch Build/CUPCAKE) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Droid Build/FRG22D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Milestone Build/ SHOLS_U2_01.03.1) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.0.1; de-de; Milestone Build/SHOLS_U2_01.14.0) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
            "Mozilla/5.0 (Linux; U; Android 0.5; en-us) AppleWebKit/522  (KHTML, like Gecko) Safari/419.3",
            "Mozilla/5.0 (Linux; U; Android 1.1; en-gb; dream) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
            "Mozilla/5.0 (Linux; U; Android 2.0; en-us; Droid Build/ESD20) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.1; en-us; Nexus One Build/ERD62) AppleWebKit/530.17 (KHTML, like Gecko) Version/4.0 Mobile Safari/530.17",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; Sprint APA9292KT Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-us; ADR6300 Build/FRF91) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 2.2; en-ca; GT-P1000M Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
            "Mozilla/5.0 (Linux; U; Android 3.0.1; fr-fr; A500 Build/HRI66) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13",
            "Mozilla/5.0 (Linux; U; Android 3.0; en-us; Xoom Build/HRI39) AppleWebKit/525.10  (KHTML, like Gecko) Version/3.0.4 Mobile Safari/523.12.2",
            "Mozilla/5.0 (Linux; U; Android 1.6; es-es; SonyEricssonX10i Build/R1FA016) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
            "Mozilla/5.0 (Linux; U; Android 1.6; en-us; SonyEricssonX10i Build/R1AA056) AppleWebKit/528.5  (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1",
        ]
        self.http_ip = [
            "27.42.168.46:55481",
            "121.13.252.58:41564"
        ]
        self.https_ip = []

2、 运用random模块以随机形式在process_request(request, spider)下，将user-agent和ip池里的数据输出。

def process_request(self, request, spider):
    request.headers["User-Agent"] = random.choice(self.user_agent_list)

（二）设置response编码等信息

  • 一般保留原代码内容，可视情况在以下函数内改动。
  • process_response(request, response, spider)

（三）异常处理

  • process_exception(request, exception, spider)
  • 一般用于适配更改协议名，如：http或https。
def process_exception(self, request, exception, spider):
    if request.url.split(":")[0] == "http":
        request.meta["proxy"] = "http://" + random.choice(self.http_id)
    if request.url.split(":")[0] == "https":
        request.meta["proxy"] = "https://" + random.choice(self.https_id)

三、分布式爬虫实现流程

（一）开启redis服务

  • cd到安装路径，输入redis-server.exe redis.windows.conf
  • 如下：

  • 出现下图即为启动成功：

 （二）改写爬虫代码

1、在spider中导入RedisSpider模块

from scrapy_redis.spiders import RedisSpider

2、更改类继承

  • 将类继承改为RedisSpider

3、注释start_urls，添加redis_key = 健

（三）添加配置信息

  • 打开setting.py
  • 将以下信息添加到setting中。
# 1.启用调度将请求存储进redis
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
# 2.确保所有spider通过redis共享相同的重复过滤。
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
# 可选 不清理redis队列，允许暂停/恢复抓取。 允许暂定,redis数据不丢失
SCHEDULER_PERSIST = True
# 3.指定连接到Redis时要使用的主机和端口。
REDIS_HOST = '127.0.0.1'
REDIS_PORT = 6379

（四）启动所有爬虫

  打开cmd启动redis-shell

  • cd 到根目录
  • 输入redis-cli.exe

当执行到这一步时，重新打开两个cmd窗口，cd到项目名，执行爬虫名进入监听状态

  • 输入lpush key url

    key: spider设置的redis_key（如：以上设置的键book）

    url : spider里面的start_urls（复制start_urls即可。）

注意：若作为练习在启动后尽量50秒内关闭，分布式爬取效率高、速度快的同时，也面临着封ip的风险。若代理池充裕可无视此提示！

 

标签：5.0,like,KHTML,Mozilla,redis,scrapy,Safari,Gecko,分布式
From： https://www.cnblogs.com/LoLong/p/16960316.html

相关文章

  • 1.5 HDFS分布式文件系统-hadoop-最全最完整的保姆级的java大数据学习资料
    目录1.5HDFS分布式文件系统1.5.1HDFS简介1.5.2HDFS的重要概念1.5.3HDFS架构1.5HDFS分布式文件系统1.5.1HDFS简介HDFS（全称：HadoopDistributeFileSystem，Hadoop......
  • 详解redis网络IO模型
    前言"redis是单线程的"这句话我们耳熟能详。但它有一定的前提，redis整个服务不可能只用到一个线程完成所有工作，它还有持久化、key过期删除、集群管理等其它模块，redis会通......
  • Redis持久化
    一、持久化概述Redis是内存数据库，数据运行在内存中，内存中的数据断电丢失，为了能够恢复Redis数据，或者防止系统故障，我们需要将Redis中的数据写入到磁盘空间中，即持久化。二、持......
  • Redis高可用、持久化及性能管理
    一、Redis高可用1、概述（1）在web服务器中，高可用是指服务器可以正常访问的时间，衡量的标准是在多长时间内可以提供正常服务（99.9%、99.99%、99.999%等等）（2）但是在Redis语境中，高......
  • Redis主从复制，哨兵模式和集群模式
    一、主从复制1、主从复制-哨兵-集群主从复制：主从复制是高可用Redis的基础，哨兵和集群都是在主从复制基础上实现高可用的。主从复制主要实现了数据的多机备份，以及对于读操......
  • 5种Redis基础数据结构的特征及常见的应用场景
    Redis5种基本数据结构（String、List、Hash、Set、SortedSet）是在开发过程种，操作redis最常用的集中数据结构。Redis官网上找到Redis数据结构非常详细的介绍：RedisData......
  • Redis分布式锁应用
    Redis锁的使用起因：分布式环境下需对并发进行逻辑一致性控制架构：springboot2、RedisIDEA实操先新建RedisLock组件注：释放锁使用lua脚本保持原子性@Component@Slf4j......
  • springboot2 搭建日志收集系统存入mongodb + redis+mq+线程池+xxljobs
    我们看到了高效批量插入mongodb速度还不错，那么我们的系统日志收集怎么做呢。当然当前文件日志收集效果也不错，比如前面博文写的elkf搭建日志收集系统。但我们系统中总是有......
  • 踩坑记录：Redis的lettuce连接池不生效
    踩坑记录：Redis的lettuce连接池不生效一、lettuce客户端lettuce客户端Lettuce和Jedis的都是连接RedisServer的客户端程序。Jedis在实现上是直连redisserver，多线程环......
  • Redis如何模糊匹配Key值
    Redis模糊匹配Key值使用Redis的scan代替Keys指令：publicSet<String>scan(StringmatchKey){Set<String>keys=(Set<String>)redisTemplate.execute((RedisC......

赞助商

阅读排行

网站主页关于我们联系我们网站地图

本网站内容转载自其他媒体，侵权联系[admin##ips99.com]。

Copyright © 2020-2023 IPS99 版权所有 IPS99