- 安装
scrapy
- pip install scrapy
- 创建项目并创建
spider
,跑起来
- scrapy startproject scrapy_demo1
- cd scrapy_demo1
- scrapy genspider baidu baidu.com
- scrapy crawl baidu
- 报错记录
- AttributeError: module ‘OpenSSL.SSL’ has no attribute ‘SSLv3_METHOD’
- 解决办法:卸载新版,安装旧版即可解决问题
pip uninstall pyOpenSSL==22.1.0
pip install pyOpenSSL==22.0.0
- module ‘lib‘ has no attribute ‘OpenSSL_add_all_algorithms‘
- 解决办法: 安装的 cryptography库与你现在的使用的环境不兼容导致的,可能是因为cryptography的版本太高,需要进行降级
pip install cryptography==38.0.4
- ImportError: cannot import name 'HTTPClientFactory' from 'twisted.web.client' (unknown location)
- 解决办法: Twisted版本太高
pip install Twisted==20.3.0
- 配置日志的级别,不然输出太多,不容易看
### settings
......
#----------自定义配置-------#
LOG_LEVEL = 'WARNING' # 配置日志级别
ROBOTSTXT_OBEY = False # 最好False,否则啥都没有
标签:baidu,解决办法,专题,cryptography,scrapy,Scrapy,install,pip
From: https://www.cnblogs.com/qinganning/p/17554587.html