1. 爬取图片
-
使用 ImagePipeLines 需要安装 Pillow
-
环境安装:pip install Pillow
爬虫文件:
import scrapy from ..items import DeepimgproItem class ImgSpider(scrapy.Spider): name = 'img' # allowed_domains = ['www.xxx.com'] start_urls = ['https://pic.netbian.com/4kmeinv/'] #通用的url模板 url_model = 'https://pic.netbian.com/4kmeinv/index_%d.html' page_num = 2 def parse(self, response): #解析出了图片的名称和详情页的url li_list = response.xpath('//*[@id="main"]/div[3]/ul/li') for li in li_list: title = li.xpath('./a/b/text()').extract_first() + '.jpg' detail_url = 'https://pic.netbian.com'+li.xpath('./a/@href').extract_first() item = DeepimgproItem() item['title'] = title #需要对详情页的url发起请求,在详情页中获取图片的下载链接 yield scrapy.Request(url=detail_url,callback=self.detail_parse,meta={'item':item}) if self.page_num <= 2: new_url = format(self.url_model%self.page_num) self.page_num += 1 yield scrapy.Request(url=new_url,callback=self.parse) #解析详情页的数据 def detail_parse(self,response): meta = response.meta item = meta['item'] img_src = 'https://pic.netbian.com'+response.xpath('//*[@id="img"]/img/@src').extract_first() item['img_src'] = img_src yield item
管道:
import scrapy from itemadapter import ItemAdapter from scrapy.pipelines.images import ImagesPipeline class DeepimgproPipeline(ImagesPipeline): # def process_item(self, item, spider): # return item def get_media_requests(self, item, info): img_src = item['img_src'] #请求传参,将item中的图片名称传递给file_path方法 #meta会将自身传递给file_path print(item['title'],'保存下载成功!') yield scrapy.Request(url=img_src,meta={'title':item['title']}) def file_path(self, request, response=None, info=None, *, item=None): #返回图片的名称 #接收请求传参过来的数据 title = request.meta['title'] return title def item_completed(self, results, item, info): return item
items文件:
import scrapy class DeepimgproItem(scrapy.Item): # define the fields for your item here like: title = scrapy.Field() img_src = scrapy.Field()
配置文件:
ROBOTSTXT_OBEY = False LOG_LEVEL = 'ERROR' IMAGES_STORE = 'ImgLibs' USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.109 Safari/537.36' CONCURRENT_REQUESTS = 32 ITEM_PIPELINES = { 'deepImgPro.pipelines.DeepimgproPipeline': 300, }
2. 提高scrapy的爬取效率
- 增加并发:
- 默认scrapy开启的并发线程为16个,可以适当进行增加。在settings配置文件中修改CONCURRENT_REQUESTS = 100值为100,并发设置成了为100。
- 降低日志级别:
- 在运行scrapy时,会有大量日志信息的输出,为了减少CPU的使用率。可以设置log输出信息为WORNING或者ERROR即可。在配置文件中编写:LOG_LEVEL = ‘ERROR’
- 禁止cookie:
- 如果不是真的需要cookie,则在scrapy爬取数据时可以禁止cookie从而减少CPU的使用率,提升爬取效率。在配置文件中编写:COOKIES_ENABLED = False
- 禁止重试:
- 对失败的HTTP进行重新请求(重试)会减慢爬取速度,因此可以禁止重试。在配置文件中编写:RETRY_ENABLED = False
- 减少下载超时:
- 如果对一个非常慢的链接进行爬取,减少下载超时可以能让卡住的链接快速被放弃,从而提升效率。在配置文件中进行编写:DOWNLOAD_TIMEOUT = 10 超时时间为10s
标签:配置文件,title,url,self,中间件,scrapy,item,Scrapy From: https://www.cnblogs.com/smile1/p/18058828