玫瑰花變蚊子血,自動化無痕瀏覽器對比測試,新貴PlayWright Vs 老牌Selenium,基於Python3.10

也許每一個男子全都有過這樣的兩個女人，至少兩個。娶了紅玫瑰，久而久之，紅的變了牆上的一抹蚊子血，白的還是床前明月光；娶了白玫瑰，白的便是衣服上沾的一粒飯黏子，紅的卻是心口上一顆硃砂痣。--張愛玲《紅玫瑰與白玫瑰》

Selenium一直都是Python開源自動化瀏覽器工具的王者，但這兩年微軟開源的PlayWright異軍突起，後來者居上，隱隱然有撼動Selenium江湖地位之勢，本次我們來對比PlayWright與Selenium之間的差異，看看曾經的玫瑰花Selenium是否會變成蚊子血。

PlayWright的安裝和使用

PlayWright是由業界大佬微軟(Microsoft)開源的端到端 Web 測試和自動化庫，可謂是大廠背書，功能滿格，雖然作為無頭瀏覽器，該框架的主要作用是測試 Web 應用，但事實上，無頭瀏覽器更多的是用於 Web 抓取目的，也就是爬蟲。

首先終端執行安裝命令：

pip3 install playwright

程式返回：

Successfully built greenlet  
Installing collected packages: pyee, greenlet, playwright  
  Attempting uninstall: greenlet  
    Found existing installation: greenlet 2.0.2  
    Uninstalling greenlet-2.0.2:  
      Successfully uninstalled greenlet-2.0.2  
Successfully installed greenlet-2.0.1 playwright-1.30.0 pyee-9.0.4

目前最新穩定版為1.30.0

隨後可以選擇直接安裝瀏覽器驅動：

playwright install

程式返回：

Downloading Chromium 110.0.5481.38 (playwright build v1045) from https://playwright.azureedge.net/builds/chromium/1045/chromium-mac-arm64.zip  
123.8 Mb [====================] 100% 0.0s  
Chromium 110.0.5481.38 (playwright build v1045) downloaded to /Users/liuyue/Library/Caches/ms-playwright/chromium-1045  
Downloading FFMPEG playwright build v1008 from https://playwright.azureedge.net/builds/ffmpeg/1008/ffmpeg-mac-arm64.zip  
1 Mb [====================] 100% 0.0s  
FFMPEG playwright build v1008 downloaded to /Users/liuyue/Library/Caches/ms-playwright/ffmpeg-1008  
Downloading Firefox 108.0.2 (playwright build v1372) from https://playwright.azureedge.net/builds/firefox/1372/firefox-mac-11-arm64.zip  
69.8 Mb [====================] 100% 0.0s  
Firefox 108.0.2 (playwright build v1372) downloaded to /Users/liuyue/Library/Caches/ms-playwright/firefox-1372  
Downloading Webkit 16.4 (playwright build v1767) from https://playwright.azureedge.net/builds/webkit/1767/webkit-mac-12-arm64.zip  
56.9 Mb [====================] 100% 0.0s  
Webkit 16.4 (playwright build v1767) downloaded to /Users/liuyue/Library/Caches/ms-playwright/webkit-1767

預設會下載Chromium核心、Firefox以及Webkit驅動。

其中使用最廣泛的就是基於Chromium核心的瀏覽器，最負盛名的就是Google的Chrome和微軟自家的Edge。

確保當前電腦安裝了Edge瀏覽器，讓我們小試牛刀一把：

from playwright.sync_api import sync_playwright  
import time  
with sync_playwright() as p:  
    browser = p.chromium.launch(channel="msedge", headless=True)  
    page = browser.new_page()  
    page.goto('http:/v3u.cn')  
    page.screenshot(path=f'./example-v3u.png')
    time.sleep(5)
    browser.close()

這裡匯入sync_playwright模組，顧名思義，同步執行，透過上下文管理器開啟瀏覽器程式。

隨後透過channel指定edge瀏覽器，截圖後關閉瀏覽器程式：

我們也可以指定headless引數為True，讓瀏覽器再後臺執行：

from playwright.sync_api import sync_playwright  
with sync_playwright() as p:  
    browser = p.chromium.launch(channel="msedge", headless=True)  
    page = browser.new_page()  
    page.goto('http:/v3u.cn')  
    page.screenshot(path=f'./example-v3u.png')  
    browser.close()

除了同步模式，PlayWright也支援非同步非阻塞模式：

import asyncio  
from playwright.async_api import async_playwright  
  
async def main():  
    async with async_playwright() as p:  
        browser = await p.chromium.launch(channel="msedge", headless=False)  
        page = await browser.new_page()  
        await page.goto("http://v3u.cn")  
        print(await page.title())  
        await browser.close()  
  
asyncio.run(main())

可以透過原生協程庫asyncio進行呼叫，PlayWright內建函式只需要新增await關鍵字即可，非常方便，與之相比，Selenium原生庫並不支援非同步模式，必須安裝三方擴充套件才可以。

最炫酷的是，PlayWright可以對使用者的瀏覽器操作進行錄製，並且可以轉換為相應的程式碼，在終端執行以下命令：

python -m playwright codegen --target python -o 'edge.py' -b chromium --channel=msedge

這裡透過codegen命令進行錄製，指定瀏覽器為edge，將所有操作寫入edge.py的檔案中：

與此同時，PlayWright也支援移動端的瀏覽器模擬，比如蘋果手機：

from playwright.sync_api import sync_playwright  
with sync_playwright() as p:  
    iphone_13 = p.devices['iPhone 13 Pro']  
    browser = p.webkit.launch(headless=False)  
    page = browser.new_page()  
    page.goto('https://v3u.cn')  
    page.screenshot(path='./v3u-iphone.png')  
    browser.close()

這裡模擬Iphone13pro的瀏覽器訪問情況。

當然了，除了UI功能測試，我們當然還需要PlayWright幫我們乾點髒活累活，那就是爬蟲：

from playwright.sync_api import sync_playwright   
   
def extract_data(entry):   
	name = entry.locator("h3").inner_text().strip("\n").strip()   
	capital = entry.locator("span.country-capital").inner_text()   
	population = entry.locator("span.country-population").inner_text()   
	area = entry.locator("span.country-area").inner_text()   
   
	return {"name": name, "capital": capital, "population": population, "area (km sq)": area}   
   
with sync_playwright() as p:   
	# launch the browser instance and define a new context   
	browser = p.chromium.launch()   
	context = browser.new_context()   
	# open a new tab and go to the website   
	page = context.new_page()   
	page.goto("https://www.scrapethissite.com/pages/simple/")   
	page.wait_for_load_state("load")   
	# get the countries   
	countries = page.locator("div.country")   
	n_countries = countries.count()   
   
	# loop through the elements and scrape the data   
	data = []   
   
	for i in range(n_countries):   
		entry = countries.nth(i)   
		sample = extract_data(entry)   
		data.append(sample)   
   
browser.close()

這裡data變數就是抓取的資料內容：

[   
	{'name': 'Andorra', 'capital': 'Andorra la Vella', 'population': '84000', 'area (km sq)': '468.0'},   
	{'name': 'United Arab Emirates', 'capital': 'Abu Dhabi', 'population': '4975593', 'area (km sq)': '82880.0'},   
	{'name': 'Afghanistan', 'capital': 'Kabul', 'population': '29121286', 'area (km sq)': '647500.0'},   
	{'name': 'Antigua and Barbuda', 'capital': "St. John's", 'population': '86754', 'area (km sq)': '443.0'},   
	{'name': 'Anguilla', 'capital': 'The Valley', 'population': '13254', 'area (km sq)': '102.0'},   
	...   
]

基本上，該有的功能基本都有，更多功能請參見官方文件：https://playwright.dev/python/docs/library

Selenium

Selenium曾經是用於網路抓取和網路自動化的最流行的開源無頭瀏覽器工具之一。在使用 Selenium 進行抓取時，我們可以自動化瀏覽器、與 UI 元素互動並在 Web 應用程式上模仿使用者操作。Selenium 的一些核心元件包括 WebDriver、Selenium IDE 和 Selenium Grid。

關於Selenium的一些基本操作請移玉步至：python3.7爬蟲:使用Selenium帶Cookie登入並且模擬進行表單上傳檔案，這裡不作過多贅述。

如同前文提到的，與Playwright相比，Selenium需要第三方庫來實現非同步併發執行，同時，如果需要錄製動作影片，也需要使用外部的解決方案。

就像Playwright那樣，讓我們使用 Selenium 構建一個簡單的爬蟲指令碼。

首先匯入必要的模組並配置 Selenium 例項，並且透過設定確保無頭模式處於活動狀態option.headless = True：

from selenium import webdriver   
from selenium.webdriver.chrome.service import Service   
from selenium.webdriver.common.by import By   
# web driver manager: https://github.com/SergeyPirogov/webdriver_manager   
# will help us automatically download the web driver binaries   
# then we can use `Service` to manage the web driver's state.   
from webdriver_manager.chrome import ChromeDriverManager   
   
def extract_data(row):   
	name = row.find_element(By.TAG_NAME, "h3").text.strip("\n").strip()   
	capital = row.find_element(By.CSS_SELECTOR, "span.country-capital").text   
	population = row.find_element(By.CSS_SELECTOR, "span.country-population").text   
	area = row.find_element(By.CSS_SELECTOR, "span.country-area").text   
   
	return {"name": name, "capital": capital, "population": population, "area (km sq)": area}   
   
options = webdriver.ChromeOptions()   
options.headless = True   
# this returns the path web driver downloaded   
chrome_path = ChromeDriverManager().install()   
# define the chrome service and pass it to the driver instance   
chrome_service = Service(chrome_path)   
driver = webdriver.Chrome(service=chrome_service, options=options)   
   
url = "https://www.scrapethissite.com/pages/simple"   
   
driver.get(url)   
# get the data divs   
countries = driver.find_elements(By.CSS_SELECTOR, "div.country")   
   
# extract the data   
data = list(map(extract_data, countries))   
   
driver.quit()

資料返回：

[   
	{'name': 'Andorra', 'capital': 'Andorra la Vella', 'population': '84000', 'area (km sq)': '468.0'},   
	{'name': 'United Arab Emirates', 'capital': 'Abu Dhabi', 'population': '4975593', 'area (km sq)': '82880.0'},   
	{'name': 'Afghanistan', 'capital': 'Kabul', 'population': '29121286', 'area (km sq)': '647500.0'},   
	{'name': 'Antigua and Barbuda', 'capital': "St. John's", 'population': '86754', 'area (km sq)': '443.0'},   
	{'name': 'Anguilla', 'capital': 'The Valley', 'population': '13254', 'area (km sq)': '102.0'},   
	...   
]

效能測試

在資料抓取量一樣的前提下，我們當然需要知道到底誰的效能更好，是PlayWright，還是Selenium?

這裡我們使用Python3.10內建的time模組來統計爬蟲指令碼的執行速度。

PlayWright:

import time   
from playwright.sync_api import sync_playwright   
   
def extract_data(entry):   
	name = entry.locator("h3").inner_text().strip("\n").strip()   
	capital = entry.locator("span.country-capital").inner_text()   
	population = entry.locator("span.country-population").inner_text()   
	area = entry.locator("span.country-area").inner_text()   
   
	return {"name": name, "capital": capital, "population": population, "area (km sq)": area}   
   
start = time.time()   
with sync_playwright() as p:   
	# launch the browser instance and define a new context   
	browser = p.chromium.launch()   
	context = browser.new_context()   
	# open a new tab and go to the website   
	page = context.new_page()   
	page.goto("https://www.scrapethissite.com/pages/")   
	# click to the first page and wait while page loads   
	page.locator("a[href='/pages/simple/']").click()   
	page.wait_for_load_state("load")   
	# get the countries   
	countries = page.locator("div.country")   
	n_countries = countries.count()   
   
	data = []   
   
	for i in range(n_countries):   
		entry = countries.nth(i)   
		sample = extract_data(entry)   
		data.append(sample)   
   
browser.close()   
end = time.time()   
   
print(f"The whole script took: {end-start:.4f}")

Selenium:

import time   
from selenium import webdriver   
from selenium.webdriver.chrome.service import Service   
from selenium.webdriver.common.by import By   
# web driver manager: https://github.com/SergeyPirogov/webdriver_manager   
# will help us automatically download the web driver binaries   
# then we can use `Service` to manage the web driver's state.   
from webdriver_manager.chrome import ChromeDriverManager   
   
def extract_data(row):   
	name = row.find_element(By.TAG_NAME, "h3").text.strip("\n").strip()   
	capital = row.find_element(By.CSS_SELECTOR, "span.country-capital").text   
	population = row.find_element(By.CSS_SELECTOR, "span.country-population").text   
	area = row.find_element(By.CSS_SELECTOR, "span.country-area").text   
   
	return {"name": name, "capital": capital, "population": population, "area (km sq)": area}   
   
# start the timer   
start = time.time()   
   
options = webdriver.ChromeOptions()   
options.headless = True   
# this returns the path web driver downloaded   
chrome_path = ChromeDriverManager().install()   
# define the chrome service and pass it to the driver instance   
chrome_service = Service(chrome_path)   
driver = webdriver.Chrome(service=chrome_service, options=options)   
   
url = "https://www.scrapethissite.com/pages/"   
   
driver.get(url)   
# get the first page and click to the link   
first_page = driver.find_element(By.CSS_SELECTOR, "h3.page-title a")   
first_page.click()   
# get the data div and extract the data using beautifulsoup   
countries_container = driver.find_element(By.CSS_SELECTOR, "section#countries div.container")   
countries = driver.find_elements(By.CSS_SELECTOR, "div.country")   
   
# scrape the data using extract_data function   
data = list(map(extract_data, countries))   
   
end = time.time()   
   
print(f"The whole script took: {end-start:.4f}")   
   
driver.quit()

測試結果:

Y軸是執行時間，一望而知，Selenium比PlayWright差了大概五倍左右。

紅玫瑰還是白玫瑰？

不得不承認，Playwright 和 Selenium 都是出色的自動化無頭瀏覽器工具，都可以完成爬蟲任務。我們還不能斷定那個更好一點，所以選擇那個取決於你的網路抓取需求、你想要抓取的資料型別、瀏覽器支援和其他考慮因素：

Playwright 不支援真實裝置，而 Selenium 可用於真實裝置和遠端伺服器。

Playwright 具有內建的非同步併發支援，而 Selenium 需要第三方工具。

Playwright 的效能比 Selenium 高。

Selenium 不支援詳細報告和影片錄製等功能，而 Playwright 具有內建支援。

Selenium 比 Playwright 支援更多的瀏覽器。

Selenium 支援更多的程式語言。

結語

如果您看完了本篇文章，那麼到底誰是最好的無頭瀏覽器工具，答案早已在心間，所謂強中強而立強，只有弱者才害怕競爭，相信PlayWright的出現會讓Selenium變為更好的自己，再接再厲，再創輝煌。