scrapy爬取后中文乱码,解决word转为html 时cp1252编码问题

解决思路1、

循环暴力寻找编码，但是不如思路3

 def parse(self, response):
        print(response.text[:100])
        body = response.body#直接是bytes,response.text是str
        encodings = ['utf-8', 'gbk', 'gb2312', 'iso-8859-1', 'latin1']#实际可用response.encoding获得
        for encoding in encodings:
            try:
                print(body.decode(encoding)[:100])#decode必须是bytes
            except Exception as e:
                print('decode {0}, error: {1}\n'.format(encoding, e))
                pass

解决思路2、
download minddlewares中有个process_response方法，修改它的encoding即可需要自己组装一个修改了charset的页面response，利用HtmlResponse可以完美解决,实现中文乱码的解决，当然你要在setting.py中启用该download middleware

from scrapy.http import HtmlResponse
 
   def process_response(self,request, response, spider):
        # 修改页面编码
        if response.encoding == 'cp1252':
            response = HtmlResponse(url=response.url, body=response.body, encoding='utf-8')
        return response

解决思路3、

scrapy爬取编码为gb2312的网页时出现中文乱码
python3用库chardet查看编码方式;先用encode编码成bytes,再用decode编码成str

import chardet

txt_b=response.xpath('//title').extract()[0].encode(response.encoding)#对找到的具体内容用encode编码成bytes
print(chardet.detect(txt_b))
txt_str= txt_b.decode(response.encoding,errors='ignore')#对bytes内容用decode解码成str
print(txt_str)