2022-03-15 最新博客爬虫

import requests

url = "https://www.cnblogs.com/AggSite/AggSitePostList"

data = {"CategoryType":"SiteHome",
        "ParentCategoryId":0,
        "CategoryId":808,
        "PageIndex":1,
        "TotalPostCount":4000,
        "ItemListActionName":"AggSitePostList"}

headers = {
"content-type":"application/json; charset=UTF-8",
"user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36",
}

import json

resp = requests.post(url, data=json.dumps(data), headers=headers)

resp

<Response [200]>

resp.headers

{'Date': 'Tue, 15 Mar 2022 11:31:22 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Strict-Transport-Security': 'max-age=2592000; includeSubDomains; preload', 'Content-Encoding': 'gzip'}

html = resp.text

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

articles = soup.find_all("article", class_="post-item")

for article in articles:
#     print(article.prettify())
    link = article.find("a", class_="post-item-title")
    title = link.get_text()
    href = link.get("href")
    
    
    a_links = article.find_all("a")
    icon_views = 0
    icon_comment = 0
    icon_digg = 0
    for a_link in a_links:
        if "icon_views" in str(a_link):
            icon_views = a_link.find("span").get_text()
        if "icon_comment" in str(a_link):
            icon_comment = a_link.find("span").get_text()
        if "icon_digg" in str(a_link):
            icon_digg = a_link.find("span").get_text()
    
    print(title, href, icon_views, icon_comment)

自己动手写Vector【Cherno C++教程】 https://www.cnblogs.com/zhangyi1357/p/16009968.html 0 0
Seastar 教程（三） https://www.cnblogs.com/morningli/p/15961636.html 9 0
.NET CORE 授权 https://www.cnblogs.com/yuxl01/p/15991851.html 47 0
对前端“价值”的理解 https://www.cnblogs.com/HuiTaiLang1216/p/16009474.html 95 0
浅析XML https://www.cnblogs.com/hjk-airl/p/16009344.html 44 0
[翻译] TensorFlow 分布式之论文篇 "Implementation of Control Flow in TensorFlow" https://www.cnblogs.com/rossiXYZ/p/16009379.html 21 0
 iCloud开发: key-value Storage,CloudKit,iCloud Documents https://www.cnblogs.com/songliquan/p/16009342.html 27 0
微服务7：通信之RPC https://www.cnblogs.com/wzh2010/p/15642251.html 53 0
Lua中如何实现类似gdb的断点调试—08支持通过包名称添加断点 https://www.cnblogs.com/logchen/p/16009256.html 29 0
spring源码干货分享-对象创建详细解析（set注入和初始化） https://www.cnblogs.com/w001/p/16009244.html 17 0
sql server数据库如何存储数组，int[]float[]double[]数组存储到数据库方法 https://www.cnblogs.com/zhaimaojun/p/16009113.html 77 2
【图文并茂】 做开发这么久了，还不会搭建服务器Maven私有仓库？这也太Low了吧 https://www.cnblogs.com/binghe001/p/16008737.html 243 0
关于分布式锁的那些事儿 https://www.cnblogs.com/zhaorongbiao/p/16005116.html 282 2
一比一还原axios源码（零）—— 概要 https://www.cnblogs.com/zaking/p/15876002.html 175 0
二进制部署1.23.4版本k8s集群-6-部署Node节点服务 https://www.cnblogs.com/wgh2008/p/16008188.html 58 0
字节一面：go的协程相比线程，轻量在哪？ https://www.cnblogs.com/JulianHuang/p/16008107.html 203 1
Android 12(S) 图形显示系统 - BufferQueue/BLASTBufferQueue之初识（六） https://www.cnblogs.com/roger-yu/p/15773008.html 50 1
HBase海量数据高效入仓解决方案 https://www.cnblogs.com/vivotech/p/16007922.html 178 1
从刘维尔方程到Velocity-Verlet算法 https://www.cnblogs.com/dechinphy/p/liouville.html 83 0
 Spring Bean生命周期，好像人的一生。。 https://www.cnblogs.com/three-fighter/p/16007800.html 244 2