中信證券抓取（頁碼範圍）

随风小屋發表於2024-08-07

中信證券抓取（頁碼範圍）

建立時間：2024年8月5日

一、完整程式碼

import re
import requests
from lxml import etree

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
}
url1 = 'http://www.cs.ecitic.com/newsite/cpzx/jrcpxxgs/zgcp/index.html'
res = requests.get(url=url1, headers=headers, verify=False)
res.encoding = res.apparent_encoding
total_num = int(re.search('var countPage = (?P<page>\d+)//共多少頁', res.text).group('page'))


def get_data(url):
    url = url.replace('\n', '')
    print(url)
    response = requests.get(url=url,
                            headers=headers, verify=False)
    response.encoding = response.apparent_encoding
    data = response.text
    tree = etree.HTML(data)
    rows = tree.xpath('/html/body/div[4]/div/ul/li')

    for row in rows:
        row_mc = row.xpath('./span[@class="th1"]/text()')
        row_glr = row.xpath('./span[@class="th2"]/text()')
        row_fxpj = row.xpath('./span[@class="th3"]/text()')
        row_je = row.xpath('./span[@class="th4"]/text()')
        row_gs = row.xpath('./span[@class="th5"]/text()')
        sting = f"{row_mc}, {row_glr}, {row_fxpj}, {row_je}, {row_gs},'\n'"

        sting = sting.replace("['", "").replace("']", "").replace("'", "")
        with open('./zxzq/中信證券.txt', 'a+', encoding='utf-8') as f:
            f.write(sting)


start_page = int(input('請輸入需要抓取的頁碼開始：（1開始）'))

end_page = int(input(f'請輸入需要抓取的頁碼結束：（{total_num}結束）'))

for i in range(start_page - 1, end_page):
    if end_page + 1 > total_num:
        print('頁碼超過資料限制！！')
        exit(-1)
    if i == 0:
        url = 'http://www.cs.ecitic.com/newsite/cpzx/jrcpxxgs/zgcp/index.html'
        get_data(url)
    else:
        url = f'http://www.cs.ecitic.com/newsite/cpzx/jrcpxxgs/zgcp/index_{i}.html'
        get_data(url)

1.1 效果

二、學習點

2.1 verify 引數

verify關鍵字引數，在請求的時候不驗證網站的ca證書

2.2 設定編碼

res.encoding = res.apparent_encoding
# 大部分情況可以自動解碼，實在不行可手動設定編碼
res.encoding = 'utf-8'

2.3 運用正規表示式獲取頁碼

total_num = int(re.search('var countPage = (?P<page>\d+)//共多少頁', res.text).group('page'))
# search(pattern, string, flags=0)  掃描字串尋找匹配的模式，返回一個match物件，如果沒有找到匹配則返回None。

頁面原始碼對應的頁碼位置：

2.4 xpath路徑

rows = tree.xpath('/html/body/div[4]/div/ul/li')

for row in rows:
    row_mc = row.xpath('./span[@class="th1"]/text()')
    row_glr = row.xpath('./span[@class="th2"]/text()')
    row_fxpj = row.xpath('./span[@class="th3"]/text()')
    row_je = row.xpath('./span[@class="th4"]/text()')
    row_gs = row.xpath('./span[@class="th5"]/text()')
    sting = f"{row_mc}, {row_glr}, {row_fxpj}, {row_je}, {row_gs},'\n'"

    sting = sting.replace("['", "").replace("']", "").replace("'", "")

先解析到所有的資料，然後在遍歷匹配出來

2.5 分頁邏輯

start_page = int(input('請輸入需要抓取的頁碼開始：（1開始）'))

end_page = int(input(f'請輸入需要抓取的頁碼結束：（{total_num}結束）'))

for i in range(start_page - 1, end_page):
    if end_page + 1 > total_num:
        print('頁碼超過資料限制！！')
        exit(-1)
    if i == 0:
        url = 'http://www.cs.ecitic.com/newsite/cpzx/jrcpxxgs/zgcp/index.html'
        get_data(url)
    else:
        url = f'http://www.cs.ecitic.com/newsite/cpzx/jrcpxxgs/zgcp/index_{i}.html'
        get_data(url)

首頁地址和第二頁地址，第三頁有規律，按照規律即可寫出來

相關文章