首页 > 其他分享 >Spider·爬取小说

Spider·爬取小说

时间:2022-10-27 09:56:24浏览次数:91  
标签:headers text tr Spider 爬取 item div 小说 find

导入资源包

import requests
import bs4

获取链接


url = 'https://www.17k.com/top/refactor/top100/18_popularityListScore/18_popularityListScore_finishBook_top_100_pc.html?TabIndex=1&typeIndex=0'

伪装用户


headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36'}

发送网络请求

req=requests.get(url=url,headers=headers)

将网络请求的响应由二进制转换为可读模式

req.encoding='utf-8'

将爬取到的内容进行装饰

soup=bs4.BeautifulSoup(req.text,"html.parser")	

将小说的序号,分类,名称,作者分别爬取出来

div=soup.find_all('div',attrs={'class':'BOX'})[1]
tr_list=div.find_all('tr')
try:
    for item in tr_list[1:]:
        id=item.find('td').text
        sign=item.find('a').text
        book=item.find('a',attrs={'class':'red'}).text
        author=item.find_all('a')[3].text
        print(id,sign,book,author)


except Exception:
    print('异常')


image

标签:headers,text,tr,Spider,爬取,item,div,小说,find
From: https://www.cnblogs.com/JK8395/p/16831091.html

相关文章