爬虫 202107【JavaPub版】

写于20210711 17:10 北京朝阳区

@[toc]

方法:

首先下载 mitproxy，pip安装方法：

> pip install mitmproxy

基本使用方法：

1. 给本机设置代理ip 127.0.0.1端口8001（为了让所有流量走mitmproxy）具体方法请百度。
2. 启动mitmproxy。

windows：

> mitmdump -p 8001

Linux：

> mitmproxy -p 8001

2.修改chromedriver源码；

3.使用mitmproxy或fiddler做代码注入；

4.使用其他方法拦截发送回浏览器指纹的XHR

runBotDetection = function () {
    var documentDetectionKeys = [
        "__webdriver_evaluate",
        "__selenium_evaluate",
        "__webdriver_script_function",
        "__webdriver_script_func",
        "__webdriver_script_fn",
        "__fxdriver_evaluate",
        "__driver_unwrapped",
        "__webdriver_unwrapped",
        "__driver_evaluate",
        "__selenium_unwrapped",
        "__fxdriver_unwrapped",
    ];

    var windowDetectionKeys = [
        "_phantom",
        "__nightmare",
        "_selenium",
        "callPhantom",
        "callSelenium",
        "_Selenium_IDE_Recorder",
    ];

    for (const windowDetectionKey in windowDetectionKeys) {
        const windowDetectionKeyValue = windowDetectionKeys[windowDetectionKey];
        if (window[windowDetectionKeyValue]) {
            return true;
        }
    };
    for (const documentDetectionKey in documentDetectionKeys) {
        const documentDetectionKeyValue = documentDetectionKeys[documentDetectionKey];
        if (window['document'][documentDetectionKeyValue]) {
            return true;
        }
    };

    for (const documentKey in window['document']) {
        if (documentKey.match(/\$[a-z]dc_/) && window['document'][documentKey]['cache_']) {
            return true;
        }
    }

    if (window['external'] && window['external'].toString() && (window['external'].toString()['indexOf']('Sequentum') != -1)) return true;

    if (window['document']['documentElement']['getAttribute']('selenium')) return true;
    if (window['document']['documentElement']['getAttribute']('webdriver')) return true;
    if (window['document']['documentElement']['getAttribute']('driver')) return true;

    return false;
};

供参考：

1. https://www.zhihu.com/question/50738719/answer/545145218
2. https://www.icode9.com/content-4-131489.html
3. 参考修改指纹：https://mp.weixin.qq.com/s?__biz=Mzg5OTU3NjczMQ==&mid=2247510375&idx=2&sn=5233386f674e9dc8b68c5760d8c6bc42&source=41#wechat_redirect
4. 工具网站：https://bot.sannysoft.com/

探讨and了解：look

更多数据采集思路，欢迎公众号【JavaPub】