BeautifulSoup使用手冊（查詢篇）

夢想家haima發表於2020-10-03

目錄

基本上是按官方文件所寫 https://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/，過完。·BeautifulSoup 內容太多了，用的時候查起來方便一點

開始使用呢

  • 從一個soup物件開始，以下兩種方式生成一個soup物件
from bs4 import BeautifulSoup
soup = BeautifulSoup(open("index.html"))    ##傳入檔案
soup = BeautifulSoup("<html>data</html>")   ##文字
  • 構造soup物件時，可以傳入解析器引數，如果不傳入的話，會以最好的方式去解析

解析器

解析器有如下

html.parser
lxml
xml
html5lib
  • 使用html.parser解析器
soup = BeautifulSoup("<html>data</html>","html.parser")

四種物件

Beautiful Soup將複雜HTML文件轉換成一個複雜的樹形結構,每個節點都是Python物件,所有物件可以歸納為4種 Tag , NavigableString , BeautifulSoup , Comment

tag物件

tag物件，同網頁中的標籤的意思

  • html標籤
soup = BeautifulSoup("<html>data</html>","html.parser")
soup.html
  • a標籤
soup = BeautifulSoup("<a >data</a>","html.parser")
soup.a

標籤名(name)

標籤擁有自己的名字，用可以直接使用標籤呼叫

soup = BeautifulSoup("<html><a >data</a></html>","html.parser")
print(soup.a.name)

**結果為 a **

屬性值(Attributes)

soup = BeautifulSoup("<html><a href='baidu.com'>data</a></html>","html.parser")
print(soup.a['href'])

**結果為： baidu.com **

多值屬性

HTML 4定義了一系列可以包含多個值的屬性.在HTML5中移除了一些,卻增加更多.最常見的多值的屬性是 class (一個tag可以有多個CSS的class). 還有一些屬性 rel , rev , accept-charset , headers , accesskey . 在Beautiful Soup中多值屬性的返回型別是list:

soup = BeautifulSoup("<html><a href='baidu.com' rev='ll' class='night'>data</a></html>","html.parser")
print(soup.a['class'])
print(soup.a['rev'])

**結果為： **

['night']
['ll']

非多值的情況下回返回字串

id_soup = BeautifulSoup('<p id="my id"></p>')
id_soup.p['id']
# 'my id'

屬性賦值，屬性可以賦值，多值的情況如下，陣列回以空格拼接上去

rel_soup = BeautifulSoup('<p>Back to the <a rel="index">homepage</a></p>')
rel_soup.a['rel']
# ['index']
rel_soup.a['rel'] = ['index', 'contents']
print(rel_soup.p)
# <p>Back to the <a rel="index contents">homepage</a></p>

內容

剛剛我們已經得到了tag，並且獲取到了屬性，現在我們來獲取內容。很簡單tag元素呼叫.string

from bs4 import BeautifulSoup
if __name__ == '__main__':
    soup = BeautifulSoup("<a >data</a>", "html.parser")
    print(soup.a.string)
    print(type(soup.a.string))
    print(str(soup.a.string))

**結果為： **

data
<class 'bs4.element.NavigableString'>
data

我們發現這樣得到的內容並不是str型，但檢視bs4.element.NavigableString的原始碼，發現NavigableString的原始碼繼承了str類

class NavigableString(str, PageElement):
省略其他程式碼

官方手冊中如下描述如果想在Beautiful Soup之外使用 NavigableString 物件,需要呼叫 unicode() 方法,將該物件轉換成普通的Unicode字串,否則就算Beautiful Soup已方法已經執行結束,該物件的輸出也會帶有物件的引用地址.這樣會浪費記憶體.而python3中並沒有unicode()，使用str()代替

Comment物件

這種情況下，會產生Comment物件

markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))

結果為：

Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>

我們可以看到這時候.string返回的物件不再是bs4.element.NavigableString，而是Comment

prettify()方法

prettify()方法返回html物件

markup = "<b><!--Hey, buddy. Want to buy a used parser?--></b>"
soup = BeautifulSoup(markup,"html.parser")
comment = soup.b.string
print(comment)
print(type(comment))
print(soup.b.prettify())

結果為：

Hey, buddy. Want to buy a used parser?
<class 'bs4.element.Comment'>
<b>
 <!--Hey, buddy. Want to buy a used parser?-->
</b>

soup = BeautifulSoup("<a href='baidu.com'>data</a>", "html.parser")
print(soup.a.prettify())

結果為：

<a href="baidu.com">
 data
</a>

find_all方法

返回所有<a> 標籤列表

contents屬性

返回元素的所有子元素 也是標籤列表

el = soup.head.contents

children屬性

返回所有子元素的一個迭代器

from bs4 import BeautifulSoup
html='''<!DOCTYPE html>
<html><head><title>The Dormouse's story</title></head>
    <body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
    </body>
</html>'''
if __name__ == '__main__':
    soup = BeautifulSoup(html, "html.parser")
    al=soup.find_all('a')
    print(al)
    print(type(al[0]))
    el = soup.head.contents
    print(el)
    print(type(el[0]))
    chd=soup.head.children
    print(chd)
    for el in soup.head.children:
        print(type(el))

結果為：

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]
<class 'bs4.element.Tag'>
[<title>The Dormouse's story</title>]
<class 'bs4.element.Tag'>
<list_iterator object at 0x0000021EABA5F2B0>
<class 'bs4.element.Tag'>

descendants屬性

contents屬性和children屬性都是直接子元素，而descendants是所有子孫元素（這裡看做孫元素）

from bs4 import BeautifulSoup
if __name__ == '__main__':
    soup = BeautifulSoup("<html><head><title>The Dormouse's story</title></head></html>","html.parser")
    head_tag = soup.head
    for child in head_tag.descendants:
        print(child)

結果為：

<title>The Dormouse's story</title>
The Dormouse's story

兄弟元素

from  bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>
</body>
</html>'''


if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html5lib")
    print(soup.p)
    print(soup.p.next_sibling)

下一個兄弟元素，兄弟元素的理解為同父元素，next_sibling下一個挨著的兄弟元素
結果為：

<p class="title"><b>The Dormouse's story</b></p>
    //空行

解釋一下上面的結果，上面空行是因為第一個p標籤後面有回車符，所以p的下一個兄弟元素並不是第二個p。修改程式碼為如下

next_sibling

from  bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p><p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>
</body>
</html>'''


if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html.parser")
    print(soup.p)
    print(soup.p.next_sibling)

結果為：

<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
    <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
    <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
    <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

previous_siblings

這個代表上一個兄弟元素，和next_sibling同理

next_element

next_element代表下一個元素，和下一個兄弟元素的 next_sibling相比，next_element不排除下一個不是兄弟元素的情況

last_a_tag = soup.find("a", id="link3")
last_a_tag
# <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>

last_a_tag.next_sibling
# '; and they lived at the bottom of a well.'

last_a_tag.next_element
# u'Tillie'

previous_element

previous_element 代表上一個元素，和next_element同理

過濾器find_all(flter)

find_all 方法中的過濾器可以用來過濾tag中的name

字串

soup.find_all('b')
# 結果為 [<b>The Dormouse's story</b>]

正規表示式

import re
for tag in soup.find_all(re.compile("^b")):
    print(tag.name)

匹配到了所有b為開頭的標籤

結果為：

body
b

for tag in soup.find_all(re.compile("t")):
    print(tag.name)
# 結果為下面兩行，標籤名包含t的標籤
# html
# title

列表

soup.find_all(["a", "b"])
# [<b>The Dormouse's story</b>,
#  <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

True/False

沒太大意義和使用價值

for tag in soup.find_all(True):
    print(tag.name)
# html
# head
# title
# body
# p
# b
# p
# a
# a
# a
# p

方法

定義了元素含有class屬性，但不含id屬性的過濾方法

def has_class_but_no_id(tag):
    return tag.has_attr('class') and not tag.has_attr('id')

soup.find_all(has_class_but_no_id)
# 結果如下三行
# [<p class="title"><b>The Dormouse's story</b></p>,
#  <p class="story">Once upon a time there were...</p>,
#  <p class="story">...</p>]

方法還能過濾屬性

def not_lacie(href):
        return href and not re.compile("lacie").search(href)
soup.find_all(href=not_lacie)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

總概

soup.find_all("title")
# [<title>The Dormouse's story</title>]

soup.find_all("p", "title")
# [<p class="title"><b>The Dormouse's story</b></p>]

soup.find_all("a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.find_all(id="link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

import re
soup.find(string=re.compile("sisters"))
# u'Once upon a time there were three little sisters; and their names were\n'

有幾個方法很相似,還有幾個方法是新的,引數中的 string 和 id 是什麼含義? 
為什麼 find_all("p", "title") 返回的是CSS Class為”title”的<p>標籤?

引數列表解讀

find_all( name , attrs , recursive , string , **kwargs )

  • name 根據tag的name篩選
  • attrs 接受字典型別的屬性比如 soup.find_all(attrs={"id": "title"})，閱讀SoupStrainer類的原始碼可以發現如下
 if not isinstance(attrs, dict):
            # Treat a non-dict value for attrs as a search for the 'class'
            # attribute.
            kwargs['class'] = attrs
            attrs = None

這裡可以看出當attrs不為字典，為普通字元實，會把attrs這個引數當做'class'屬性

  • recursive 是否從當前位置遞迴往下查詢，如果不遞迴，只會查詢當前soup文件的子元素
  • string 這裡是通過tag的內容來搜尋，並且返回的是類容，而不是tag型別的元素
  • **kwargs 自動拆包接受屬性值，所以才會有soup.find_all('a',id='title') ，id='title'為**kwargs自動拆包摻入

class_

按照CSS類名搜尋tag的功能非常實用,但標識CSS類名的關鍵字 class 在Python中是保留字,使用 class 做引數會導致語法錯誤.從Beautiful Soup的4.1.1版本開始,可以通過 class_ 引數搜尋有指定CSS類名的tag:

soup.find_all("a", class_="sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

搜尋 class 屬性時也可以通過CSS值完全匹配:

css_soup.find_all("p", class_="body strikeout")
# [<p class="body strikeout"></p>]

limit 引數

限制返回的條數

soup.find_all("a", limit=2)
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

find

同find_all方法類似，唯一區別是find返回一個元素，find_all返回一個列表。方法引數和find_all相同find( name , attrs , recursive , string , **kwargs ) ，使用方式也相同

soup.find_all('title', limit=1)
# [<title>The Dormouse's story</title>]
soup.find('title')
# <title>The Dormouse's story</title>

除了find_all，find之外，還有很多方法類似，如下

find_parents

find_parents( name , attrs , recursive , string , **kwargs )

查詢所有的父輩節點（包括多級的祖先），返回一個列表

find_parent

find_parent( name , attrs , recursive , string , **kwargs )

查詢上一個父輩節點，會一直在文件樹上面查詢，不會只查詢一級別，特別注意

from bs4 import BeautifulSoup
doc='''<!DOCTYPE html>
<html>
<head><title>The Dormouse's story</title></head>
<body>
    <div class='cover'> <div> <p></p>  </div>  </div>
</body>
</html>'''
if __name__ == '__main__':
    soup = BeautifulSoup(doc, "html.parser")
    p=soup.find('p')
    print(p.find_parent("div"))
    print(p.find_parents("div"))

#結果
<div> <p></p> </div>    #文件樹上一個查詢的div
[<div> <p></p> </div>, <div class="cover"> <div> <p></p> </div> </div>]  #所有div

官方案例

a_string = soup.find(string="Lacie")
a_string
# u'Lacie'

a_string.find_parents("a")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

a_string.find_parent("p")
# <p class="story">Once upon a time there were three little sisters; and their names were
#  <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
#  and they lived at the bottom of a well.</p>

a_string.find_parents("p", class="title")
# []

find_next_siblings，find_next_sibling

  • find_next_siblings( name , attrs , recursive , string , **kwargs )
  • find_next_sibling( name , attrs , recursive , string , **kwargs )

這2個方法通過 .next_siblings 屬性對當tag的所有後面解析 [5] 的兄弟tag節點進行迭代，find_next_siblings() 方法返回所有符合條件的後面的兄弟節點, find_next_sibling() 只返回符合條件的後面的第一個tag節點.

find_previous_siblings，find_previous_sibling

  • find_previous_siblings( name , attrs , recursive , string , **kwargs )
  • find_previous_sibling( name , attrs , recursive , string , **kwargs )

和find_next_siblings，find_next_sibling 類似，只是查詢前面的兄弟元素

find_all_next，find_next

  • find_all_next( name , attrs , recursive , string , **kwargs )
  • find_next( name , attrs , recursive , string , **kwargs )

這2個方法通過 .next_elements 屬性對當前tag的之後的 [5] tag和字串進行迭代, find_all_next() 方法返回所有符合條件的節點, find_next() 方法返回第一個符合條件的節點

find_all_previous，find_previous

  • find_all_previous( name , attrs , recursive , string , **kwargs )
  • find_previous( name , attrs , recursive , string , **kwargs )

和find_all_next，find_next類似,向前查詢。這2個方法通過 .previous_elements 屬性對當前節點前面 [5] 的tag和字串進行迭代。

Css選擇器

select

css選擇器的方法為select(css_selector)
目前支援的選擇器如下案例，css選擇器可以參考https://www.w3school.com.cn/cssref/css_selectors.ASP

soup.select("title")
# [<title>The Dormouse's story</title>]

soup.select("p nth-of-type(3)")
# [<p class="story">...</p>]

soup.select("body a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("html head title")
# [<title>The Dormouse's story</title>]


soup.select("head > title")
# [<title>The Dormouse's story</title>]

soup.select("p > a")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie"  id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("p > a:nth-of-type(2)")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]

soup.select("p > #link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("body > a")

soup.select("#link1 ~ .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie"  id="link3">Tillie</a>]

soup.select("#link1 + .sister")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select(".sister")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("[class~=sister]")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select("#link1")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select("a#link2")
# [<a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select("#link1,#link2")
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>]


soup.select('a[href]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href="http://example.com/elsie"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]

soup.select('a[href^="http://example.com/"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
#  <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
#  <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href$="tillie"]')
# [<a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

soup.select('a[href*=".com/el"]')
# [<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>]


multilingual_markup = """
 <p lang="en">Hello</p>
 <p lang="en-us">Howdy, y'all</p>
 <p lang="en-gb">Pip-pip, old fruit</p>
 <p lang="fr">Bonjour mes amis</p>
"""
multilingual_soup = BeautifulSoup(multilingual_markup)
multilingual_soup.select('p[lang|=en]')
# [<p lang="en">Hello</p>,
#  <p lang="en-us">Howdy, y'all</p>,
#  <p lang="en-gb">Pip-pip, old fruit</p>]

select_one

返回查詢到的元素的第一個

相關文章