标签：xpath content 请求 artical title request 爬取 headers class
import requests
from lxml import etree
url = "https://duanzixing.com/"
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 Edg/116.0.1938.62',
}
res = requests.get(url, headers=headers)
data = res.content.decode()
tree = etree.HTML(data)
artical_list = tree.xpath('//div[@class="content"]/article[@class="excerpt"]')
for artical in artical_list:
title = artical.xpath('./header/h2/a/text()')[0]
content = artical.xpath('./p[@class="note"]/text()')[0]
print(title, content)
标签：xpath,
content,
请求,
artical,
title,
request,
爬取,
headers,
class
From： https://www.cnblogs.com/khalil12138/p/17663633.html