Python爬蟲——XPath

weixin_34321977發表於2018-07-28
Python爬蟲

XPath

表示式 描述
nodename 選取此節點的所有子節點
/ 從根節點選取
//xxx 從匹配選擇的當前節點選擇文件中的節點，而不考慮它們的位置
. 選取當前節點
.. 選取當前節點的父節點
@xxx 選取屬性內容
/text() 選取文字內容
starts-with(@屬性名稱，屬性字元相同部分) 以相同字元開始

演示使用HTML內容

html = '''
<!DOCTYPE html>
<html>
<head lang="en">
    <meta charset="UTF-8">
    <title>測試-常規用法</title>
</head>
<body>
<div id="content">
    <ul id="test1">
        <li>這是第一條資訊</li>
        <li>這是第二條資訊</li>
        <li>這是第三條資訊</li>
    </ul>
    <ul id="test2">
        <li>這是第4條資訊</li>
        <li>這是第5條資訊</li>
        <li>這是第6條資訊</li>
    </ul>
    <div id="url">
        <a href="http://www.baidu.com">百度</a>
        <a href="http://www.163.com" title="Netease">網易</a>
    </div>
</div>
</body>
</html>
'''

lxml庫

from lxml import etree
selector = etree.HTML()
content = selector.xpath()

/代表根節點開始的逐層獲取

from lxml import etree
selector = etree.HTML(html)
content = selector.xpath('/html/head/title/text()')
print(content)

獲取所有的li標籤

selector = etree.HTML(html)
content = selector.xpath('//li')
for c in content:
    print(c)

獲取所有title屬性的值

selector = etree.HTML(html)
content = selector.xpath('//@title')
for c in content:
    print(c)

[]獲取列表的index

selector = etree.HTML(html)
content = selector.xpath('//ul[2]/li[1]/text()')
for c in content:
    print(c)
# 這是第4條資訊

獲取所有的li的文字內容，無論在什麼位置

selector = etree.HTML(html)
content = selector.xpath('//li/text()')
for c in content:
    print(c)
143845-7b241687e7949bad.png
image.png

獲取屬性id=url的div下層的a標籤所有href地址

selector = etree.HTML(html)
content = selector.xpath('//div[@id="url"]/a/@href')
for c in content:
    print(c)

獲取屬性class="test1"的ul下層的最後一個li標籤的文字內容

selector = etree.HTML(html)
content = selector.xpath('//ul[@class="test1"]/li[last()]/text()')
print(content)

獲取所有含有id屬性且值為test1的ul下層li的文字內容

selector = etree.HTML(html)
content = selector.xpath('//ul[@id="test1"]/li/text()')
for c in content:
    print(c)

豆瓣讀書250資料抓取

<table width="100%">
    <tbody><tr class="item">
        <td width="100" valign="top">
            <a class="nbg" href="https://book.douban.com/subject/25862578/" onclick="moreurl(this,{i:'1'})">
                <img src="https://img3.doubanio.com/view/subject/m/public/s27264181.jpg" width="90">
            </a>
        </td>
        <td valign="top">
            <div class="pl2">
                <a href="https://book.douban.com/subject/25862578/" onclick="&quot;moreurl(this,{i:'1'})&quot;" title="解憂雜貨店">
                    解憂雜貨店
                </a>
                <br>
                <span style="font-size:12px;">ナミヤ雑貨店の奇蹟</span>
            </div>
            <p class="pl">[日] 東野圭吾 / 李盈春 / 南海出版公司 / 2014-5 / 39.50元</p>
            <div class="star clearfix">
                <span class="allstar45"></span>
                <span class="rating_nums">8.6</span>
                <span class="pl">(
                    294790人評價
                )</span>
            </div>           
            <p class="quote" style="margin: 10px 0; color: #666">
                <span class="inq">一碗精心熬製的東野牌雞湯，拒絕很難</span>
            </p>
        </td>
    </tr>
</tbody></table>
143845-f492cc542d8a8fe9.png
image.png
import requests
from lxml import etree

headers={
    'Accept':'*/*',
    'Accept-Encoding':'gzip, deflate',
    'Accept-Language':'zh-CN,zh;q=0.8',
    'Connection':'keep-alive',
    'Origin':'https://book.douban.com',
    'Referer':'https://book.douban.com/',
    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0'
}
for i in range(0, 10):
    url = 'https://book.douban.com/top250?start={}'.format(i * 25)
    html = requests.get(url, headers=headers).content
    sel = etree.HTML(html)
    tables = sel.xpath('//div[@class="indent"]/table')
    for table in tables:
        item = table.xpath('tr[@class="item"]/td[@valign="top"][2]')
        print(item[0].xpath('div[@class="pl2"]/a/@title')[0])
        print(item[0].xpath('p[@class="pl"]/text()')[0])
        quote = item[0].xpath('p[@class="quote"]/span/text()')
        if quote:
            print([0])

相關文章