Splash与Scrapy结合

Splash与Scrapy结合

scrapy-splash 教程 — splash中文文档 0.1 文档

https://splash-cn-doc.readthedocs.io/zh_CN/latest/scrapy-splash-toturial.html

安装scrapy-splash库

 pip install scrapy-splash==0.8.0

配置splash服务（以下操作全部在settings.py）

使用splash解析，要在配置文件中设置splash服务器地址：

SPLASH_URL = 'http://localhost:8050/' 

将splash middleware添加到DOWNLOADER_MIDDLEWARE中

DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}

Enable SplashDeduplicateArgsMiddleware

SPIDER_MIDDLEWARES = {
 'scrapy_splash.SplashDeduplicateArgsMiddleware': 100
}

这个中间件需要支持cache_args功能; 它允许通过不在磁盘请求队列中多次存储重复的Splash参数来节省磁盘空间。如果使用Splash 2.1+，则中间件也可以通过不将这些重复的参数多次发送到Splash服务器来节省网络流量

配置消息队列所使用的过滤类

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'

配置消息队列需要使用的类

HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'

样例

方法1

import scrapy
from scrapy_splash import SplashRequest
class TxxxxxSpider(scrapy.Spider):
  name = 'tuniu'
  allowed_domains = ['tuniu.com']
  def start_requests(self):
    yield SplashRequest('http://www.xxxxx.com/', args={'wait': 0.5})


  def parse(self, response):
    print(response.text)

方法2

class TuniuSpider(scrapy.Spider):
  name = 'xxxxx2'
  allowed_domains = ['xxxxxx.com']


  def start_requests(self):
    url = 'http://www.xxxxx.com/'
    
    lua='''
     function main(splash,args)
       splash:go(args.url)
       splash:wait(1.5)
       return splash:html()
     end
     '''
    yield SplashRequest(url,callback=self.parse,args={'lua_source':lua},endpoint='execute')
  def parse(self, response):
    with open('a2.html','wb') as f:
      f.write(response.body)