首页 > 其他分享 >go爬虫 简单请求

go爬虫 简单请求

时间:2023-06-30 12:55:06浏览次数:37  
标签:return 请求 err resp req 爬虫 Println go fmt

 

demo1.go

 

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
)

func fech(url string) string {
    client := &http.Client{}
    req, _ := http.NewRequest("GET", url, nil)
    req.Header.Set("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36")
    req.Header.Set("cookie", "_ga_3Q0DVSGN10=GS1.1.1686653928.1.1.1686653937.51.0.0; _pbjs_userid_consent_data=3524755945110770; cto_bundle=mZmLSF85dUVNdXdwREVkeFE3b2VUajZ0S2klMkI5d3FreVlFenpJeFEyZ0w4RlpmTVZiaWt6Q2I1TGJkbXE3MXlTQlhPYXJLZEl4eURiT3Jjb3I3T0Vvd2xGdm1WQSUyQmNyeDhIUHlBSEpCM3hMUFVUMml3d2RhWmNXZzlzdiUyQkFXTFZIY1FnMg; cto_bidid=CdhCdl9BTlpmSjhpMzlUdTd2d25vSUZqdUNEczlGQ3J1S2h0ckdKU2drU2N1RTJzaTBzUnV0V1FwTXhUbXM3RFJKMUtVN0NGTzgwT2JwUXBoTHpUcmlNZ2RFZyUzRCUzRA; _ga_C2LFP3RFGH=GS1.1.1687754267.6.1.1687754335.0.0.0; _ga=GA1.2.1648403957.1686653923; Hm_lvt_866c9be12d4a814454792b1fd0fed295=1687143481,1687314851,1687751312,1688099485; .Cnblogs.AspNetCore.Cookies=CfDJ8DuWIYDefEVJtUW7VadsHY_tNh7ao1tXgttuUApjlm5XA7WC5kQXnelD_VS1z17zAe8z8QasViHjo-AIK79_CYAUkqcK7j_6gPGy0uxI4nY7Hw9ATFn7lUmc_Las-dx6rp_Wau0QyuwW6-W0ZtP0AFITAtTb2rh5Hr4LbpMll44MeY3yj4M4ffB2OtGFrNkpz4mIO1_7jngK-iNhKF3Sue74eLG-zLzqmjdxnGyxchyzgvCyhY1bG-i0EOAJuuawZXBcjIBMYEdlNBBKF9ztj5qwLlSVJdoAU5gvruowUcBGHr7uhsOHjYI4aG9jZCMjQE2d-kc2wPDXzqybIhIbRGDe_bwQ1Ge1KLq-pmWufRa5e-trq4IGwoOqoRsFm_Al8oEAhVjd_qlVDwKuS-W72Kb8iy--0BmP4LEHVHxvgvB-VslQ1FAl1luI7SM5EULkoC0c3gjsA-yvP0hwpzjrB6DnnDw4MPF-7oztMVy87YeIDKcGM08w8EUK7XMxLdwnYneSCidFxe-AGmoLpQ6ADwHvAg8cDpilu7tlPEhnz-wCbfhopH2QJtH3KTsRTysxY7UNoPR4EA06gMGSE06tbryxGYGNas29fdpjCLUmIu7L; .AspNetCore.Antiforgery.b8-pDmTq1XM=CfDJ8DuWIYDefEVJtUW7VadsHY9GzlkkyGKRFrnaEtkzRl8yCk-MRYDEAm84RV4g-GOp1poZxh_7ku6UhQJ_CdEowtSPN7Z7UkkKvDP-_v_fpBL4vEhFk8OoQuvzToqnIa2TLK5fFRAyMzZdCit2WIU9fTA; _gid=GA1.2.1595448807.1688099486; __gads=ID=6b54244ffc1be3d3:T=1686653929:RT=1688099487:S=ALNI_MbkkoHnP5-VRm-TUcciquZtVsaGCw; __gpi=UID=00000c12b511e32e:T=1686653929:RT=1688099487:S=ALNI_MbURVj9g0e9qRcg2YiA_kHj_7ha3A; Hm_lpvt_866c9be12d4a814454792b1fd0fed295=1688099495; lotame_domain_check=cnblogs.com; _gat_gtag_UA_476124_1=1")

    resp, err := client.Do(req)
    if err != nil {
        fmt.Println(err)
        return ""
    }
    if resp.StatusCode != 200 {
        fmt.Println("error")
        return ""
    }
    defer resp.Body.Close()

    //resp.Body
    var body []byte

    body, err = ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return ""
    }
    return string(body)
}

func main() {
    url := "https://www.cnblogs.com/angdh/"
    fmt.Println(fech(url))
}

 

标签:return,请求,err,resp,req,爬虫,Println,go,fmt
From: https://www.cnblogs.com/angdh/p/17516425.html

相关文章

  • iis部署.netcore项目不允许put 和post,delete请求
    在webconfig中添加红色标记部分<?xmlversion="1.0"encoding="utf-8"?><configuration><system.webServer><modulesrunAllManagedModulesForAllRequests="true"><removename="WebDAVModule"/></......
  • MATLAB仿真Gough-Stewart并联机器人斯图尔特6自由度并联机器人逆运动学仿真 动力学控
    MATLAB仿真Gough-Stewart并联机器人斯图尔特6自由度并联机器人逆运动学仿真动力学控制pid控制1.搭建了六自由度Stewart并联机器人simulinksimscape仿真模型2.建立了逆向运动学仿真输入位置和姿态求解各个杆长3.运用pid控制器进行动力学跟踪控制使用MATLAB进行了Gough-Stewar......
  • docker部署mongodb副本集
    1.规格ip地址端口192.168.1.10127017192.168.1.10227017192.168.1.103270172.部署操作2.1 以下操作分别在三台不同的ECS服务器上操作[root@ecs-ff50-73e8-0002~]#mkdir/path/to-p[root@ecs-ff50-73e8-0002~]#cd/path/to/[root@ecs-ff50-73e8-0002to]#ls[root@ecs-ff50-73......
  • 开发一个通用的python爬虫代码
    以下是一个通用的Python爬虫代码框架,可以作为起点来开发各种不同的爬虫程序:importrequestsfrombs4importBeautifulSoupdefget_html(url):response=requests.get(url)returnresponse.textdefparse_html(html):soup=BeautifulSoup(html,"html.parser......
  • mongodb数据库操作备忘
    db.SMU_REALDATA_PARAMETERS.update({},{$rename:{"interfaceID":"INTERFACEID"}},false,true); 修改field,字段名,不是修改值哟。 网上搜索mongodb语言和CheatSheet手册。https://www.mongodb.com/developer/products/mongodb/cheat-sheet/navcate工具和下面工具都可以。 ......
  • 爬取大量数据有什么爬虫技巧?
    爬虫数据在许多情况下都是非常有用的,爬虫数据提供了对市场和竞争对手的深入了解,可用于商业智能和市场调研。通过采集关于产品、评论、竞争对手策略等,企业可以做出更明智的决策。爬虫数据可用于构建内容聚合网站或搜索引擎。通过采集各种来源的数据,可以构建一个丰富、多样化的内容库......
  • 处理HTTPGET请求参数中含有数组不识别的问题处理
    如:  在Program中增加 ......
  • Kubernetes编程——client-go基础—— 深入 API Machinery —— REST 映射
    深入APIMachinery——REST映射 GVK与GVR之间的映射关系被称为REST映射。我理解意思是说:在Kubernetes中,RESTMapping(REST映射)用于将GroupVersionKind(GVK)与GroupVersionResource(GVR)之间建立映射关系。......
  • uniapp分页请求
    <template><view>循环请求的数据</view><view> {{isLoading?加载中...':'没有更多了'}} </view></template>exportdefault{data(){return{dataForm:[],//分页数据isLoading:false, //是否加载中pageNu......
  • go-选项卡模式
    packagemainimport"fmt"const(defaultNamestring="张建平"defaultAgeint=27defaultHighint=175)typeUserstruct{NamestringAgeintHighint}typeUserOptionsstruct{Namestring......