目录
1 前戏
#介绍:使用requests可以模拟浏览器的请求,比起之前用到的urllib,requests模块的api更加便捷(本质就是封装了urllib3)
#注意:requests库发送请求将网页内容下载下来以后,并不会执行js代码,这需要我们自己分析目标站点然后发起新的request请求
#安装:pip3 install requests
#各种请求方式:常用的就是requests.get()和requests.post()
>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r = requests.post('http://httpbin.org/post', data = {'key':'value'})
>>> r = requests.put('http://httpbin.org/put', data = {'key':'value'})
>>> r = requests.delete('http://httpbin.org/delete')
>>> r = requests.head('http://httpbin.org/get')
>>> r = requests.options('http://httpbin.org/get')
# 建议在正式学习requests前,先熟悉下HTTP协议
2 爬虫介绍
# 爬虫:又称网络蜘蛛,spider,一堆程序,从互联网中抓取数据----》数据清洗---->入库
# 爬虫需要掌握的知识
-抓取数据:发送网络请求(http),获得响应(http响应,响应头,响应体---》真正重要的数据在响应体中)
-python模块:requests,selenium
-清洗数据:解析拿回来的数据---》json,xml,html,二进制
-json解析,xml解析。。。
-python模块:re,json,beautifulsoup4(bs4),lxml,selenium
-入库:存文件,存mysql,redis,mongodb
-python模块:file,pymsql,redis-py,pymongo
# 反扒
-频率限制
-封ip(代理池),封账号(一堆小号:cookie池)
-请求头中带加密信息,referer,user-agent。。。
-响应回来的数据是加密
-验证码反扒(破解验证码---》第三方平台)
-js加密---》压缩---》加密方法其实在前端能看到---》看上去很晦涩
-手机设备:唯一id号
# 搜索引擎都是大爬虫
-百度输入框搜索---》美女---》去百度的数据库搜索----》显示在页面上
-百度一刻不停的在互联网中爬网页,爬完存到它的数据库
-seo优化:免费排中,排靠前
-sem优化:搜出来的靠前的,带广告的是花钱的,买断关键词
#可见即可爬
3 requests模块介绍和快速使用
# requests是模拟发送http请求的模块
-不仅仅可以做爬虫
-后端服务,请求别人服务
# pip3 install requests
###1 使用requests发送get请求
import requests
# res 响应对象,http响应,python包装成了对象,响应头,响应头。。。在res中都会有
res=requests.get('https://www.cnblogs.com/liuqingzheng/p/16005866.html')
print(res.text) # 响应体转成字符串
4 get请求携带参数
# res=requests.get('https://www.cnblogs.com/liuqingzheng/p/16005866.html/?name=lqz&age=19')
# res=requests.get('https://www.cnblogs.com/liuqingzheng/p/16005866.html/',params={'name':"lqz",'age':19})
# res=requests.get('https://www.baidu.com/s?wd=%E5%B8%85%E5%93%A5')
# print(res.text) # 响应体转成字符串
# print(res.url) #https://www.cnblogs.com/liuqingzheng/p/16005866.html?name=lqz&age=19
5 编码和解码
from urllib.parse import quote,unquote
print(unquote('%E5%B8%85%E5%93%A5'))
print(quote("帅哥")) # %E5%B8%85%E5%93%A5
6 携带请求头
携带请求头---》发送get请求,有的网站,拿不到数据,模拟的不像,请求头的数据没有携带
# 请求头:1 User-Agent:客户端类型 2 referer :上一个页面的地址 3 cookie
header={
'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}
res=requests.get('https://dig.chouti.com/',headers=header)
print(res.text)
7 发送post请求,携带数据
header = {
'Referer': 'http://www.aa7a.cn/user.php?&ref=http%3A%2F%2Fwww.aa7a.cn%2F',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}
data = {
'username': '[email protected]',
'password': 'l啊啊大qz123!!!!!',
'captcha': 'xxxx',
'remember': 1,
'ref': ' http://www.aa7a.cn/',
'act': 'act_login'
}
# data,json请求体的数据
# 编码方式是 urlencoded
# res = requests.post('http://www.aa7a.cn/user.php', data=data, headers=header)
# # 编码是json格式
# # res = requests.post('http://www.aa7a.cn/user.php',json=data,headers=header)
# print(res.text)
8 携带cookie两种方式
# cookie = res.cookies
# print(cookie)
# 方式一:使用cookie参数
# res1 = requests.get('http://www.aa7a.cn/',cookies=cookie)
# print('[email protected]' in res1.text)
# 方式二:携带在请求头中
header = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
'Cookie':'deviceId=web.eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqaWQiOiI3MzAyZDQ5Yy1mMmUwLTRkZGItOTZlZi1hZGFmZTkwMDBhMTEiLCJleHBpcmUiOiIxNjYxNjU0MjYwNDk4In0.4Y4LLlAEWzBuPRK2_z7mBqz4Tw5h1WeqibvkBG6GM3I; __snaker__id=ozS67xizRqJGq819; YD00000980905869%3AWM_TID=M%2BzgJgGYDW5FVFVAVQbFGXQ654xCRHj8; _9755xjdesxxd_=32; Hm_lvt_03b2668f8e8699e91d479d62bc7630f1=1688616419; YD00000980905869%3AWM_NI=KpVsBKpke6xW0Ozhu07T2FswNWgn4UaBQdTBY7Z6X6f6CbQtBNZDQJ94vZ3PUgbkpIvoPsM5PprPUE2dtyRVhLPXd2cjrMI7MRtbiJtL2lJGNgPkbgbKwARmuFZz5K2JRDE%3D; YD00000980905869%3AWM_NIKE=9ca17ae2e6ffcda170e2e6eeabc744f4aefca3f03d89e78ab7d54a878e9fb1d468f4b28c83cc5383928daced2af0fea7c3b92a89b09a94c141b4b5abd7fb5df491b689f83cf58cfea7f27d8e89a98fb75fa39e9f85d74a9be996d7db3ea6ea8d83d480829296a5d442fbecbabbb25c97b9ae8cf343b6b1a094d03f94b0beb1d84ff2bcbc9ad15eac94a0b9e73aa1b6beaecc3d8f8ebc83ec42b894fe8dc94d8eb9a08ce16da295b9a2d167b79a969bb841f2969ea5c837e2a3; gdxidpyhxdE=8v%5CAccgIfMa0JDz9H9Ptnc%5Cc3owLo0EZ2IiWyPHAKTiu67%2BrKDu4bwfw5Wcy5as25LX3%2FpCICt2Gdf4%5CJ1WmqzAmbQaTarBZsdITdIu021%5C7VhnnOlNgw%2FcwwB5gYpzyegJoffKg6r2DfKJKKP%2FLokWEI0auXpasnO8CAetcf5ijccoZ%3A1688618634928; token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqaWQiOiJjZHVfNTMyMDcwNzg0NjAiLCJleHBpcmUiOiIxNjkxMjA5NzM4NjA1In0.SZmVRy5HxKkmNTDCSQ5LEX-fkx3KyH0_jt0Wi-GEnLE; Hm_lpvt_03b2668f8e8699e91d479d62bc7630f1=1688617739',
}
data = {
'linkId': '39196098'
}
res = requests.post('https://dig.chouti.com/link/vote', headers=header,data=data)
print(res.text)
标签:www,http,请求,第一天,get,res,基础,爬虫,requests
From: https://www.cnblogs.com/yuezongke/p/17535538.html