【爬蟲】第二章-基本請求庫

txmmy發表於2024-04-05

urllib
- request
- parse
- error
- robotparser
- urlretrieve
requests

urllib

request

介紹：模擬傳送請求
urlopen

import urllib
from urllib import request
import cchardet
# 使用urllib分別抓取http://www.xinhuanet.com/politics/leaders/2021-07/01/c_1127615334.htm和https://www.kanunu8.com/book/3829/的原始碼並輸出。

#加上請求頭
headers={
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}

url1="http://www.xinhuanet.com/politics/leaders/2021-07/01/c_1127615334.htm"
url2="https://www.kanunu8.com/book/3829/"

req1=urllib.request.Request(url1,headers=headers)
req2=urllib.request.Request(url2,headers=headers)

# urlopen(url)中可以是url地址，也可以是Request物件
res1=urllib.request.urlopen(req1)
res2=urllib.request.urlopen(req2)

#檢查網頁編碼型別
#print(cchardet.detect(res1.read()))
#print(cchardet.detect(res2.read()))

#列印原始碼
print(res1.read().decode("utf-8"))
print(res2.read().decode('GB18030'))

三種請求方式的區別：
第一種: urlopen直接放入url且不解碼

request=urllib.request.urlopen(url)
print(request.read())
# 執行結果
b'<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#ffffff"><meta name="description" content="\xe5\x85\xa8\xe7\x90\x83\xe9\xa2\x86\xe5\x85\x88\xe7\x9a\x84\xe4\xb8\xad\xe6\x96\x87\xe6\x90\x9c\xe7\xb

第二種：urlopen直接放入url且解碼

request=urllib.request.urlopen(url)
print(request.read().decode('utf-8'))
# 執行結果
<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#ffffff"><meta name="description" content="全球領先的中文搜尋引擎、致力於讓網民更便捷地獲取資訊，找到所求。百度超過千億的中文網頁資料庫，可以瞬間找到相關的搜尋結果。"><link rel="shortcut icon" href="https://www.baidu.com/favicon.ico" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="百度搜尋" /><link rel="icon" sizes="any" mask href="https://www.baidu.com/favicon.ico"><link rel="dns-prefetch" href="//dss0.bdstatic.com"/><link rel="dns-prefetch" href="//dss1.bdstatic.com"/><link rel="dns-prefetch" href="//ss1.bdstatic.com"/><link rel="dns-prefetch" href="//sp0.baidu.com"/><link rel="dns-prefetch" href="//sp1.baidu.com"/><link rel="dns-prefetch" href="//sp2.baidu.com"/><link rel="dns-prefetch" href="//pss.bdstatic.com"/><link rel="apple-touch-icon-precomposed" href="https://psstatic.cdn.bcebos.com/video/wiseindex/aa6eef91f8b5b1a33b454c401_1660835115000.png"><title>百度一下，你就知道</title><style index="newi" type="text/css">#form .bdsug{top:39px}.bdsug{display:none;position:absolute;width:535px;background:#fff;border:1px solid #ccc!important;_overflow:hidden;box-shadow:1px 1px 3px #ededed;-webkit-box-shadow:1px 1px 3px #ededed;-moz-box-shadow:1px 1px 3px #ededed;-o-box-shadow:1px 1px 3px #ededed}.bdsug li{width:519px;color:#000;font:14px arial;line-height:25px;padding:0 8px;position:relative;cursor:default}.bdsug li.bdsug-s{background:#f0f0f0}.bdsug-store span,.bdsug-store b{color:#7A77C8}.bdsug-store-del{font-size:12px;color:#666;text-decoration:underline;position:absolute;right:8px;top:0;cursor:pointer;display:none}.bdsug-s .bdsug-store-del{display:inline-block}.bdsug-ala{display:inline-block;border-bottom:1px solid #e6e6e6}.bdsug-ala h3{line-height:14px;background:url(//www.baidu.com/img/sug_bd.png?v=09816787.png) no-repeat left center;margin:6px 0 4px;font-size:12px;font-weight:400;color:#7B7B7B;padding-left:20px}.bdsug-ala p{font-size:14px;font-weight:700;padding-left:20px}#m .bdsug .bdsug-direct p{color:#00c;font-weight:700;line-height:34px;padding:0 8px;margin-top:0;cursor:pointer;white-space:nowrap;overflow:hidden}#m .bdsug .bdsug-direct p img{width:16px;height:16px;margin:7px 6px 9px 0;vertical-align:middle}#m .bdsug .bdsug-direct p span{margin-left:8px}#form .bdsug .bdsug-direct{width:auto;padding:0;border-bottom:1px solid #f1f1f1}#form .bdsug .bdsug-direct p i{font-size:12px;line-height:100%;font-style:normal;font-weight:400;color:#fff;background-color:#2b99ff;display:inline;text-align:center;padding:1px 5px;*padding:2px 5px 0;margin-left:8px;overflow:hidden}.bdsug .bdsug-pcDirect{color:#000;font-size:14px;line-height:30px;height:30px;background-color:#f8f8f8}.bdsug .bdsug-pc-direct-tip{position:absolute;right:15px;top:8px;width:55px;height:15px;display:block;background:url(http://pss.bdstatic.com/r/www/cache/static/global/img/pc_direct_42d6311.png) no-repeat 0 0}.bdsug li.bdsug-pcDirect-s{background-color:#f0f0f0}.bdsug .bdsug-pcDirect-is{color:#000;font-size:14px;line-height:22px;background-color:#f5f5f5}.bdsug .bdsug-pc-direct-tip-is{position:absolute;right:15px;top:3px;width:55px;height:15px;display:block;background:url(http://pss.bdstatic.com/r/www/cache/static/global/img/pc_direct_42d6311.png) no-repeat 0 0}.bdsug li.bdsug-pcDirect-is-s{background-color:#f0f0f0}.bdsug .bdsug-pcDirect-s .bdsug-pc-direct-tip,.bdsug .bdsug-pcDirect-is-s .bdsug-pc-direct-tip-is{background-position:0 -15px}.bdsug .bdsug-newicon{color:#929292;opacity:.7;font-size:12px;display:inline-block;line-height:22px;letter-spacing:2px}.bdsug .bdsug-s .bdsug-newicon{opacity:1}.bdsug .bdsug-newicon i{letter-spacing:0;font-style:normal}.bdsug .bdsug-feedback-wrap{display:none}.toggle-underline{text-decoration:none}.toggle-underline:hover{text-decoration:underline}.bdpfmenu,.usermenu{border:1px solid #d1d1d1;position:absolute;width:105px;top:36px;z-index:302;box-shadow:1px 1px 5px #d1d1d1;-webkit-box-shadow:1px 1px 5px #d1d1d1;-moz-box-shadow:1px 1px 5px #d1d1d1;-o-box-shadow:1px 1px 5px #d1d1d1}.bdpfmenu{font-size:12px;background-color:#fff}.bdpfmenu a,.usermenu a{display:block;text-align:left;margin:0!important;padding:0 9px;line-height:26px;text-decoration:none}.briiconsbg{background-repeat:no-repeat;background-size:300px 18px;background-image:url(http://pss.bdstatic.com/r/www/cache/static/home/img/icons_0c37e9b.png);background-image:url(http://pss.bdstatic.com/r/www/cache/static/home/img/icons_809ae65.gif)\9}.bdpfmenu a:link,.bdpfmenu a:visited,#u .usermenu a:link,#u .usermenu a:visited{background:#fff;color:#333}.bdpfmenu a:hover,.bdpfmenu a:active,#u .usermenu a:hover,#u .usermenu a:active{background:#38f;text-decoration:none;color:#fff}.bdpfmenu{width:70px}#wrapper .bdnuarrow{width:0;height:0;font-size:0;line-height:0;display:block;position:absolute;top:-10px;left:50%;margin-left:-5px}#wrapper .bdnuarrow em,#wrapper .bdnuarrow i{width:0;height:0;font-size:0;line-height:0;display:block;position:absolute;border:5px solid transparent;border-style:dashed dashed solid}#wrapper .bdnuarrow em{border-bottom-color:#d8d8d8;top:-1px}#wrapper .bdnuarrow i{border-bottom-color:#fff;top:0}#gxszHead .prefpanelclose{cursor:pointer;width:16px;height:16px;float:right;margin-top:7px;background-position:-248px 0}#gxszHead .prefpanelclose:hover{background-position:-264px 0}.s_ipt::-webkit-input-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt::-moz-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt:-ms-input-placeholder{padding-left:3px;color:#aaa;font-size:13px}.s_ipt::placeholder{padding-left:3px;color:#aaa;font-size:13px}.kw-placeholder{position:absolute;top:0;left:0;color:#aaa;font-size:13px;height:40px;line-height:40px;padding-left:10px;max-width:360px;z-index:99;pointer-events:none}.kw-placeholder.kw-placehlder-high{height:40px;line-height:40px}.kw-placeholder.placeholders-hidden{visibility:hidden}#head_wrapper #form .bdsug-new{width:544px;top:35px;border-radius:0 0 10px 10px;border:2px solid #4E6EF2!important;border-top:0!important;box-shadow:none;font-family:Arial,sans-serif;z-index:1}#head_wrapper.sam_head_wrapper2 #form .bdsug-new{width:545px;z-index:1;border:1px solid #4E6EF2!important;border-top:0!important}#head_wrapper #form .bdsug-new ul{margin:7px 14px 0;padding:8px 0 7px;background:0 0;border-top:2px solid #f5f5f6}#head_wrapper #form .bdsug-new ul li{position:relative;width:auto;padding-left:14px;margin-left:-14px;margin-right:-14px;color:#626675;line-height:28px;background:0 0;font-family:Arial,sans-serif}#head_wrapper #form .bdsug-new ul li .sug-search-icon,#head_wrapper #form .bdsug-new ul li .sug-history-icon{margin-right:4px;color:#222}#head_wrapper #form .bdsug-new ul li span{color:#626675}#head_wrapper #form .bdsug-new ul li b{font-weight:400;color:#222}#head_wrapper #form .bdsug-new .bdsug-store-del{font-size:13px;text-decoration:none;color:#9195A3;right:16px}#head_wrapper #form .bdsug-new .bdsug-store-del:hover{color:#315EFB;cursor:pointer}#head_wrapper #form .bdsug-new ul li:hover,#head_wrapper #form .bdsug-new ul li:hover span,#head_wrapper #form .bdsug-new ul li:hover b{cursor:pointer}.wrapper_new #form .bdsug-new .bdsug-s{background-color:#F5F5F6!important}.wrapper_new #form .sam_search .bdsug-new .bdsug-s{background-color:#F1F3FD!important}#head_wrapper #form .sam_search .bdsug-new .bdsug-s{background-color:#F1F3FD!important}#head .s-down #form .bdsug-new{top:32px}.s-skin-hasbg #head_wrapper #form .bdsug-new{border-color:#4569ff!important;border-top:0!important}.s-skin-hasbg #head_wrapper.s-down #form .bdsug-new{border-color:#4e6ef2!important;border-top:0!important}.s-skin-hasbg #head_wrapper.s-down #form.sam_search .bdsug-new{border-color:rgba(0,0,0,.05)!important;border-top:1px solid rgba(0,0,0,.05)!important;top:54px!important}#head_wrapper #form .bdsug-new .bdsug-s,#head_wrapper #form .bdsug-new .bdsug-s span,#head_wrapper #form .bdsug-new .bdsug-s b{color:#315EFB}#head_wrapper #form .bdsug-new>div span:hover,#head_wrapper #form .bdsug-new>div a:hover{color:#315EFB!important}#head_wrapper #form #kw.new-ipt-focus{border-color:#4e6ef2}#head_wrapper #form .bdsug-new ul li{}#head_wrapper #form .bdsug-new ul li .sug-hot-orange,#head_wrapper #form .bdsug-new ul li .sug-hot-grey,#head_wrapper #form .bdsug-new ul li .sug-hot-blue{display:inline-block;width:12px;height:12px;font-size:12px;line-height:12px;padding:2px;text-align:center;font-weight:500;margin-left:6px;vertical-align:text-bottom}#head_wrapper #form .bdsug-new ul li .sug-hot-orange{display:inline-block;color:#fff;background:#F60;border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag{text-align:center;margin-left:6px;box-sizing:border-box;font-size:12px;line-height:14px;padding:1px 4px;font-weight:500}#head_wrapper #form .bdsug-new ul li .sug-new-tag-grey{color:#858585;border:1px solid rgba(133,133,133,.5);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag-blue{color:#36F;border:1px solid rgba(51,102,255,.4);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-new-tag-orange{color:#F33;border:1px solid rgba(255,51,51,.4);border-radius:4px}#head_wrapper #form .bdsug-new ul li .sug-tag-text{position:sticky;right:-4px;margin-top:-2px;display:inline-block;box-sizing:border-box;margin-left:6px;padding:1px;font-size:12px;line-height:12px;font-weight:500;text-align:center;border-radius:4px;vertical-align:middle;border-width:1px;border-style:solid}#head_wrapper #form .bdsug-new ul li .sug-tag-img{height:18px;max-width:80px;vertical-align:middle;margin-left:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap{display:block;height:40px;padding:7px 0;color:#222}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap:visited{color:#222}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img-wrap{position:relative}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img-wrap::before{content:"";width:38px;height:38px;position:absolute;top:0;right:0;left:0;bottom:0;border:1px solid rgba(0,0,0,.06);border-radius:9px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .left-img{box-sizing:border-box;float:left;width:40px;height:40px;border:1px solid rgba(0,0,0,.06);border-radius:9px;margin-right:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .content{float:left}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .header{margin-bottom:6px;line-height:18px;height:18px;vertical-align:middle}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .name{float:left;font-size:16px;color:#222;font-weight:500}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .tag-img{width:18px;margin-left:4px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .more{line-height:14px;width:240px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .brief,#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .info{font-size:14px;color:#222;font-weight:400}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .brief{margin-right:6px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .right-btn{float:right;margin-top:5px;margin-right:6px;width:60px;height:24px;text-align:center;font-size:14px;color:#36F;line-height:24px;background-image:linear-gradient(111deg,#e8f7ff 0,#edf0ff 100%);border-radius:12px}#head_wrapper #form .bdsug-new ul li .direct-sug-wrap .right-btn:hover{background:#315efb;color:#fff}#head_wrapper #form .bdsug-new ul li:hover .direct-sug-wrap .name{color:#315EFB}#head_wrapper #form .bdsug-new ul li:hover .direct-sug-wrap .brief{color:#222}#head_wrapper.s-down #form .sam-bdsug.bdsug-new{top:52px}#head_wrapper #form .sam-bdsug.bdsug-new{width:100%;box-shadow:0 4px 4px 0 rgba(0,0,0,.1);border:1px solid rgba(0,0,0,.05)!important;border-radius:12px;top:56px}#head_wrapper #form .sam-bdsug.bdsug-new ul{border:0;padding:0 0 7px}#head_wrapper #form .sam-bdsug.bdsug-new ul li{line-height:32px}#head_wrapper #form .sam-bdsug.bdsug-new ul .bdsug-s{background-color:#F1F3FD!important}#head_wrapper #form .sam-bdsug.bdsug-new .bdsug-store-del{right:15px}.sam_search .sam_search_rec,.sam_search .sam_search_soutu{z-index:1;display:none;position:absolute;top:50%;margin-top:-12px;font-size:24px;color:#4E6EF2;height:24px;line-height:24px;width:24px;cursor:pointer;-webkit-transform:translate3d(0,0,0);transform:translate3d(0,0,0);transition:transform .3s ease}.sam_search .sam_search_rec{right:54px}.sam_search .sam_search_soutu{right:14px}.sam_search .sam_search_rec:hover,.sam_search .sam_search_soutu:hover{color:#1D4FFF!important;transform:scale(1.08,1.08)}.sam_search .sam_search_rec_hover,.sam_search .sam_search_soutu_hover{background:#626675;border-radius:8px;height:32px;width:76px;text-align:center;line-height:32px;font-size:13px;color:#FFF;position:absolute;z-index:2;top:50px}.sam_search .sam_search_rec_hover:before,.sam_search .sam_search_soutu_hover:before{content:'';border:4px solid transparent;border-bottom:4px solid #626675;position:absolute;left:50%;top:-8px;margin-left:-4px}.sam_search .sam_search_rec_hover{right:29px}.sam_search .sam_search_soutu_hover{display:none;right:-12px}</style><style type="text/css" index="superbase">blockquote,body,button,dd,dl,dt,fieldset,form,h1,h2,h3,h4,h5,h6,hr,input,legend,li,ol,p,pre,td,textarea,th,ul{margin:0;padding:0}
html{color:#000;overflow-y:scroll;overflow:-moz-scrollbars}
body,button,input,select,textarea{font-size:12px;font-family:Arial,sans-serif}
h1,h2,h3,h4,h5,h6{font-size:100%}
em{font-style:normal}

第三種：請求Request物件不解碼

req=urllib.request.Request("http://www.baidu.com")
rep=urllib.request.urlopen("http://www.baidu.com")
print(rep.read())
# 執行結果： 同1

第三四：請求Request物件且解碼

req=urllib.request.Request("http://www.baidu.com")
rep=urllib.request.urlopen("http://www.baidu.com")
print(rep.read().decode('utf-8'))
# 執行結果： 同2

parse

用於處理URL，如拆分、解析、合併等，常用函式為等
編碼與解碼(urlencode、parse_qs、quote、unquote),常用於post訪問網頁時需提交資料，這時可以用parse將字典或字串轉換為位元組流編碼
一些Handler處理器：用於處理登陸驗證、cookie、代理設定等，如urlopen相當於Opener
官方文件：
https://docs.python.org/3/library/urllib.request.html#urllib.request.BaseHandler

error

URLError 屬性reason可以返回錯誤原因

登入驗證
網頁彈出需要輸入使用者名稱密碼的提示框時，可使用HTTPBasicAuthHandler

from urllib.request import HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler, build_opener 
from urllib.error import URLError 
username = username 
password ='password' 
url = 'http: //localhost:sooo/' 
p = HTTPPasswordMgrWi thDefaultRealm() 
p.add_password(None,url,username,password) 
auth_handler = HTTPBasicAuthHandler(p) 
opener = build_opener(auth_handler) 
try: 
result = opener.open(url) 
html = result.read().decode('utf 8') 
print(html) 
except URLError as e: 
print(e.reason)

代理設定

from urllib.error import URLError 
from urllib.request import ProxyHandler,build opener 
proxy_handler = ProxyHandler({ 
'http':'http://127.0.0.1:9743', 
'https':'https://127.0.0.1:9743' 
}) 
opener = build_opener(proxy_handler) 
try: 
response = opener.open ('https://www.baidu.com') 
print(response.read().decode ('utf-8')) 
except URLError as e: 
print(e.reason)

獲取cookie

import http.cookiejar,urllib.request
# 宣告一個cookieJar物件
cookie=http.cookiejar.CookieJar()
handler=urllib.request.HTTPCookieProcessor(cookie)
opener=urllib.request.build_opener(handler)
response=opener.open('http://www.baidu.com')
for item in cookie:
	print(item.name+"="+item.value)

#將cookie輸出成檔案格式

cookie.save(ignore_discard=True,ignore_expires=True)
# 執行結果
BAIDUID=57A09ECF30B476A6A85EFB2165C219FE:FG=1
BAIDUID_BFESS=57A09ECF30B476A6A85EFB2165C219FE:FG=1
BIDUPSID=57A09ECF30B476A6A85EFB2165C219FE
H_PS_PSSID=40080_40369_40379_40416_40445_40300_40464_40458_40505_40500_40514_40397_60043_60029_60035_60048_40511_60094
PSTM=1712247775

cookie儲存結果為

將會儲存為MozillaCookieJar格式的cookie，LWPCookieJar 同樣可以讀取和儲存 Cookies ，但是儲存的格式和 MozillaCookieJar 不一樣，它會儲存成 libwww-perl(LWP）格式的 Cookies 檔案
若要儲存為LWP格式的cookie，可以在宣告cookieJar時改為：

cookie =http.cookiejar.LWPCookieJar(filename)

從本地txt檔案中讀取cookie:

cookie = http.cookiejar.LWPCookieJar()
cookie.load('cookies.txt',ignore discard=True, ignore_expires=True)

qs:還有沒有其他方式獲取cookie?

robotparser

用於識別網站robots.txt檔案

urllib.robotparser.RobotFileParser(url=’ ’)

urlretrieve

requests

4.爬蟲 requests庫講解 GET請求 POST請求響應
2019-04-09
爬蟲
python爬蟲請求頭
2020-10-06
Python爬蟲
【0基礎學爬蟲】爬蟲基礎之網路請求庫的使用
2023-03-26
爬蟲
爬蟲 | 非同步請求aiohttp模組
2024-06-16
爬蟲非同步AIHTTP
Python爬蟲（二）——傳送請求
2021-08-27
Python爬蟲
爬蟲中網路請求的那些事之urllib庫
2022-03-19
爬蟲
爬蟲快速入門——Get請求的使用
2020-10-25
爬蟲
Python爬蟲基礎-01-帶有請求引數的爬蟲
2018-06-06
Python爬蟲
爬蟲：HTTP請求與HTML解析（爬取某乎網站）
2021-05-19
爬蟲HTTPHTML網站
【Python3網路爬蟲開發實戰】3-基本庫的使用 1.1-傳送請求
2019-02-22
Python爬蟲
爬蟲中的TCP請求自動切換ip
2021-09-09
爬蟲TCP
利用post請求傳送內容進行爬蟲
2018-04-13
爬蟲
爬蟲基本原理及urllib庫的基本使用
2019-03-21
爬蟲
請求基本流程
2020-04-22
Requests如何在Python爬蟲中實現get請求？
2021-09-11
Python爬蟲
Python爬蟲之Selenium庫的基本使用
2018-11-30
Python爬蟲
Python爬蟲（1.爬蟲的基本概念）
2018-04-20
Python爬蟲
基本的爬蟲工作原理
2023-11-24
爬蟲
Datawhale-爬蟲-Task1（學習get與post請求）
2019-03-01
爬蟲
爬蟲實戰：從HTTP請求獲取資料解析社群
2024-03-20
爬蟲HTTP
Python爬蟲Post請求返回值為-1000
2024-07-16
Python爬蟲
【第二章】深入HTTP請求流程
2020-07-08
HTTP
Python2爬蟲利器：requests庫的基本用法
2021-09-11
Python爬蟲
爬蟲工作原理詳解：從網頁請求到資料提取
2023-11-24
爬蟲網頁
Python網路爬蟲第三彈《爬取get請求的頁面資料》
2018-09-14
Python爬蟲
爬蟲基本功就這？早知道幹爬蟲了
2020-11-21
爬蟲
Java爬蟲系列四：使用selenium-java爬取js非同步請求的資料
2021-10-17
Java爬蟲JS非同步
Python開發技巧：scrapy-redis爬蟲如何傳送POST請求
2021-03-24
PythonRedis爬蟲
爬蟲之requests庫
2022-03-20
爬蟲
【Python學習】爬蟲爬蟲爬蟲爬蟲~
2018-05-03
Python爬蟲
爬蟲01:爬取豆瓣電影TOP 250基本資訊
2020-12-29
爬蟲
Python開發環境詳細配置Anaconda+請求庫+解析庫+資料庫+儲存庫+web庫+爬蟲框架+部署庫---Windows篇
2018-07-20
Python開發環境資料庫Web爬蟲框架Windows
爬蟲需要代理IP的基本要求
2021-09-11
爬蟲
大話爬蟲的基本套路薦
2017-10-09
爬蟲
爬蟲 | 處理cookie的基本方法——session
2024-06-12
爬蟲CookieSession
axios的post請求爬坑
2018-08-05
iOS
Http請求-okhttp3基本用法
2020-08-25
HTTP
全棧 – 7 爬蟲 Http請求和Chrome
2019-02-10
全棧爬蟲HTTPChrome