http代理使用提示
1.http/https网页均可适用
2.scrapy不是python原生库,需要安装才能使用:pip install scrapy
3.在第一级tutorial目录下运行如下命令查看结果:scrapy crawl kdl
middlewares.py
1.middlewares.py中新增ProxyDownloaderMiddleware即代理中间件
2.请注意替换代码中的部分信息:username:用户名,password:密码
#!/usr/bin/env python
#--coding:utf-8--
from scrapy import signals
from.myextend import pro
import random
class ProxyDownloaderMiddleware:
def process_request(self,request,spider):
proxy=random.choice(pro.proxy_list)
#用户名密码认证(私密代理/独享代理)
username="username"
password="password"
request.meta['proxy']="http://%(user)s:%(pwd)s %(proxy)s/"%{"user":username,"pwd":password,"proxy":proxy}
#白名单认证(私密代理/独享代理)
#request.meta['proxy']="http://%(proxy)s/"%{"proxy":proxy}
return None
标签:username,http,Python,request,样例,代理,scrapy,proxy From: https://www.cnblogs.com/huakexiaopeng/p/16801938.html