Python BeautifulSoup 使用

BS4庫簡單使用:

1.最好配合LXML庫，下載：pip install lxml

2.最好配合Requests庫，下載：pip install requests

3.下載bs4：pip install bs4

4.直接輸入pip沒用？解決：環境變數->系統變數->Path->新建：C:Python27Scripts

案例：獲取網站標題

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

import requests

url = “https://www.baidu.com”

response = requests.get(url)

soup = BeautifulSoup(response.content, `lxml`)

print soup.title.text

標籤識別

示例1：

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = “`

<html>

<head><title>The Dormouse`s story</title></head>

<body>

<p class=”title”><b>The Dormouse`s story</b></p>

<p class=”story”>Once upon a time there were three little sisters; and their names were

<a href=”http://example.com/elsie” class=”sister” id=”link1″>Elsie</a>,

<a href=”http://example.com/lacie” class=”sister” id=”link2″>Lacie</a> and

<a href=”http://example.com/tillie” class=”sister” id=”link3″>Tillie</a>;

and they lived at the bottom of a well.</p>

</body>

</html>

“`

soup = BeautifulSoup(html, `lxml`)

# BeautifulSoup中有內建的方法來實現格式化輸出

print(soup.prettify())

# title標籤內容

print(soup.title.string)

# title標籤的父節點名

print(soup.title.parent.name)

# 標籤名為p的內容

print(soup.p)

# 標籤名為p的class內容

print(soup.p[“class”])

# 標籤名為a的內容

print(soup.a)

# 查詢所有的字元a

print(soup.find_all(`a`))

# 查詢id=`link3`的內容

print(soup.find(id=`link3`))

示例2：

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = “`

<html>

<head><title>The Dormouse`s story</title></head>

<body>

<p class=”story”>Once upon a time there were three little sisters; and their names were

<a href=”http://example.com/elsie” class=”sister” id=”link1″>Elsie</a>,

<a href=”http://example.com/lacie” class=”sister” id=”link2″>Lacie</a> and

<a href=”http://example.com/tillie” class=”sister” id=”link3″>Tillie</a>;

and they lived at the bottom of a well.</p>

</body>

</html>

“`

soup = BeautifulSoup(html, `lxml`)

# 將p標籤下的所有子標籤存入到了一個列表中

print (soup.p.contents)

find_all示例:

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = “`

<h4>Hello</h4>

</div>

</ul>

</ul>

</div>

“`

soup = BeautifulSoup(html, `lxml`)

# 查詢所有的ul標籤內容

print(soup.find_all(`ul`))

# 針對結果再次find_all,從而獲取所有的li標籤資訊

for ul in soup.find_all(`ul`):

print(ul.find_all(`li`))

# 查詢id為list-1的內容

print(soup.find_all(attrs={`id`: `list-1`}))

# 查詢class為element的內容

print(soup.find_all(attrs={`class`: `element`}))

# 查詢所有的text=`Foo`的文字

print(soup.find_all(text=`Foo`))

CSS選擇器示例：

# -*- coding:utf-8 -*-

from bs4 import BeautifulSoup

html = “`

<h4>Hello</h4>

</div>

</ul>

</ul>

</div>

“`

soup = BeautifulSoup(html, `lxml`)

# 獲取class名為panel下panel-heading的內容

print(soup.select(`.panel .panel-heading`))

# 獲取class名為ul和li的內容

print(soup.select(`ul li`))

# 獲取class名為element，id為list-2的內容

print(soup.select(`#list-2 .element`))

# 使用get_text()獲取文字內容

for li in soup.select(`li`):

print(li.get_text())

# 獲取屬性的時候可以通過[屬性名]或者attrs[屬性名]

for ul in soup.select(`ul`):

print(ul[`id`])

# print(ul.attrs[`id`])

Python BeautifulSoup 使用

相關文章