基於 Scrcpy 的遠端除錯方案
基於Scrcpy的遠端除錯方案
前言
感謝STF的開源，讓Android裝置遠端控制變得簡單，STF通過minicap和minitouch實現裝置的顯示和控制。
STF在實際使用中會發現一些棘手的問題
1.電視不支援minitouch
2.新手機比如mi8 mi9不支援minicap
3.Android釋出新版本需要適配minicap
分享一個新的方法，來彌補這些不足
演示效果如下，由於圖片較大（14MB），手機黨請謹慎點選如下連結，建議完全載入完在播放，否則會卡，電腦的錄屏軟體很不給力，渣渣畫質。。。
https://github.com/wenxiaomao1023/scrcpy/blob/master/assets/out.gif
Scrcpy
app目錄 執行在PC端，對於web遠端控制，這部分是不需要的
server目錄 執行在手機端，提供螢幕資料，接收並響應控制事件
Scrcpy對比minicap
1.獲取frame資料方式是一致的（sdk19以上）
2.Scrcpy將frame編碼h264
3.minicap將frame編碼jpeg
Scrcpy處理方式看起來會更好，但是有一個問題，他的設計是將螢幕資料直接發給PC，然後在PC上解碼顯示，這種方式在網頁上卻很不好展示。
調研與嘗試
1.Broadway在前端解碼h264並顯示
2.wfs.js在前端將h264轉成mp4送給h5 MSE實現播放，這種類似直播，B站flv.js那種
以上兩種嘗試都獲得了影像，但個人感覺，以上兩個方案感覺都有坑，還需要大量優化才能脫坑
解決方法
當前摸索出的解決方法，Scrcpy將frame編碼jpeg發給前端然後通過畫布展示，瀏覽器相容好，可行性高，minicap也是這麼做的，修改方式見如下
https://github.com/wenxiaomao1023/scrcpy/commit/46d1c009d8ce559dc1ac1cdfceb234f1c7728498
當前已實現的功能
1.使用ImageReader獲取frame資料，通過libjpeg-turbo編碼jpeg
2.控制幀率，壓縮率，縮放比例，可以減少頻寬佔用，提高流暢性
3.考慮到當前大多是minicap的方案，所以scrcpy返回的螢幕資料格式相容了minicap的資料格式（banner+jpegsize+jpegdata），移植改動會很小
4.最低支援到Android4.4（adb forward命令有變化，見下文，預設埠6612）
5.返回旋轉狀態（為了替換STFService.apk）
6.新增獲取DumpHierarchy資訊（啟動命令加-D引數），獲取介面佈局資訊為錄製Case功能準備
優點
1.德芙般絲滑，手機播放視訊一點不卡，web端展示也很流暢（30 - 50 FPS）
2.支援電視touch
3.支援mi8，mi9等影像展示，不必在適配minicap.so啦，耶！
缺點
1.最低支援android5.0，由於還依賴android.system.Os，若想相容低版本裝置需要配合minicap使用，當前最低可支援到Android4.4
2.流量問題，如果你的網站部署在內網，建議參考此方案，如果部署在公網，建議採用原生的Scrcpy方式
編譯libjpeg-turbo
我已經編好了ARMv7 (32-bit)和ARMv8 (64-bit)
https://github.com/wenxiaomao1023/scrcpy/tree/master/server/libs/libturbojpeg/prebuilt
如果你需要其他平臺，可參考此文件Building libjpeg-turbo for Android部分
https://github.com/libjpeg-turbo/libjpeg-turbo/blob/master/BUILDING.md
如果不需要，可跳過此步驟
編譯Scrcpy程式碼
ninja編譯方式
android sdk裡有ninja，如Android/Sdk/cmake/3.6.4111459/bin/ninja，加到環境變數裡即可，meson需要安裝
如果不想安裝這些，可以往下看，用gradle編譯
git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy
meson x --buildtype release --strip -Db_lto=true
ninja -Cx
編譯後會在scrcpy目錄下生成
x/server/scrcpy-server.jar
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so
gradle編譯方式
git clone https://github.com/wenxiaomao1023/scrcpy.git
cd scrcpy/server
../gradlew assembleDebug
編譯後會在scrcpy目錄下生成
server/build/outputs/apk/debug/server-debug.apk
server/jniLibs/armeabi-v7a/libcompress.so
server/jniLibs/arm64-v8a/libcompress.so
server/build/outputs/apk/debug/server-debug.apk 和 x/server/scrcpy-server.jar 是一樣的，下文中都按scrcpy-server.jar命名方式進行說明
啟動scrcpy-server.jar
# 先看下裝置的abi，
adb shell getprop ro.product.cpu.abi
新版本新增-L引數，LD_LIBRARY_PATH的值等於-L引數的返回值，並追加:/data/local/tmp
# armeabi-v7a
adb push scrcpy/server/jniLibs/armeabi-v7a/libcompress.so /data/local/tmp/
adb push scrcpy/server/libs/libturbojpeg/prebuilt/armeabi-v7a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值為上步-L的返回值加:/data/local/tmp（注意有個英文冒號）
adb shell LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
# arm64-v8a
adb push server/jniLibs/arm64-v8a/libcompress.so /data/local/tmp/
adb push server/libs/libturbojpeg/prebuilt/arm64-v8a/libturbojpeg.so /data/local/tmp/
adb push scrcpy/x/server/scrcpy-server.jar /data/local/tmp/
adb shell chmod 777 /data/local/tmp/scrcpy-server.jar
adb shell CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值為上步-L的返回值加:/data/local/tmp（注意有個英文冒號）
adb shell LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
# Android4.4裝置
# 前幾步同上（檢視abi push libcompress.so libturbojpeg.so，push scrcpy-server.jar，chmod scrcpy-server.jar）
# 不同的地方在於需要指定ANDROID_DATA引數，否則啟動會報錯Dex cache directory isn't writable: /data/dalvik-cache (Permission denied) uid=2000 gid=2000
# 見解決方法https://stackoverflow.com/questions/21757935/running-android-java-based-command-line-utility-from-adb-shell
adb shell mkdir -p /data/local/tmp/dalvik-cache
adb shell ANDROID_DATA=/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server -L
# LD_LIBRARY_PATH的值為上步-L的返回值加:/data/local/tmp（注意有個英文冒號）
adb shell ANDROID_DATA=/data/local/tmp LD_LIBRARY_PATH=???:/data/local/tmp CLASSPATH=/data/local/tmp/scrcpy-server.jar app_process / com.genymobile.scrcpy.Server
app_process / com.genymobile.scrcpy.Server 這個命令可以設定如下引數，建議使用命令如下，壓縮質量60，最高24幀，480P
app_process / com.genymobile.scrcpy.Server -Q 60 -r 24 -P 480
Usage: [-h]
jpeg:
-r <value>: Frame rate (frames/sec).
-P <value>: Display projection (1080, 720, 480, 360...).
-Q <value>: JPEG quality (0-100).
-c: Control only.
-L: Library path.
-D: Dump window hierarchy.
-h: Show help.
啟動app.js
scrcpy-server.jar相容了minicap資料格式，可以直接用minicap的demo app.js看效果
https://github.com/openstf/minicap/tree/master/example
https://github.com/openstf/minicap/blob/master/example/app.js
需要把app.js改一下，多一個連線，修改如下
// 原始程式碼預設的影像socket
var stream = net.connect({
port: 1717
})
// 修改1 加一個控制socket，在預設的net.connect後再net.connect一次（必須）
var controlStream = net.connect({
port: 1717
})
// 修改2 新版本除了返回影像，還會返回旋轉和層級（如果-D），需要容錯處理，遮蔽掉exit()，並加else
if (frameBody[0] !== 0xFF || frameBody[1] !== 0xD8) {
console.error('Frame body does not start with JPG header', frameBody)
// process.exit(1) //遮蔽exit()
}
else { // 新增else
ws.send(frameBody, {
binary: true
})
}
git clone https://github.com/openstf/minicap.git
cd minicap/example
npm install
# 注意這裡要改為localabstract:scrcpy（舊版本）
# 注意這裡要改為tcp:6612（新版本）
adb forward tcp:1717 tcp:6612
node app.js
訪問 http://127.0.0.1:9002
如果有時重新整理會沒效果，只顯示一個紅框，先確認是否執行了adb forward tcp:1717 tcp:6612，如果是概率性重新整理不出影像這是正常的，客戶端連線失敗需要做重連操作，例子裡並沒有，所以簡單解決辦法是重啟下scrcpy-server.jar，然後重新整理網頁
Scrcpy touch
Scrcpy touch的實現可以參考如下實現，當前實現常用的三種事件訊息
// 鍵值 HOME，BACK，MENU等
CONTROL_MSG_TYPE_INJECT_KEYCODE
// 點選和滑動
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
// 滑鼠滾輪滾動
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
後端提供json格式介面
package main
import (
"errors"
"net"
"github.com/qiniu/log"
"bytes"
"encoding/binary"
)
type MessageType int8
const (
CONTROL_MSG_TYPE_INJECT_KEYCODE MessageType = iota
CONTROL_MSG_TYPE_INJECT_TEXT
CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON
CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL
CONTROL_MSG_TYPE_GET_CLIPBOARD
CONTROL_MSG_TYPE_SET_CLIPBOARD
CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE
)
type PositionType struct {
X int32 `json:"x"`
Y int32 `json:"y"`
Width int16 `json:"width"`
Height int16 `json:"height"`
}
type Message struct {
Msg_type MessageType `json:"msg_type"`
// CONTROL_MSG_TYPE_INJECT_KEYCODE
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
// CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
// CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}
type KeycodeMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_keycode_action int8 `json:"msg_inject_keycode_action"`
Msg_inject_keycode_keycode int32 `json:"msg_inject_keycode_keycode"`
Msg_inject_keycode_metastate int32 `json:"msg_inject_keycode_metastate"`
}
type TouchMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_touch_action int8 `json:"msg_inject_touch_action"`
Msg_inject_touch_pointerid int64 `json:"msg_inject_touch_pointerid"`
Msg_inject_touch_position PositionType `json:"msg_inject_touch_position"`
Msg_inject_touch_pressure uint16 `json:"msg_inject_touch_pressure"`
Msg_inject_touch_buttons int32 `json:"msg_inject_touch_buttons"`
}
type ScrollMessage struct {
Msg_type MessageType `json:"msg_type"`
Msg_inject_scroll_position PositionType `json:"msg_inject_scroll_position"`
Msg_inject_scroll_horizontal int32 `json:"msg_inject_scroll_horizontal"`
Msg_inject_scroll_vertical int32 `json:"msg_inject_scroll_vertical"`
}
func drainScrcpyRequests(conn net.Conn, reqC chan Message) error {
for req := range reqC {
var err error
switch req.Msg_type {
case CONTROL_MSG_TYPE_INJECT_KEYCODE:
t := KeycodeMessage{
Msg_type: req.Msg_type,
Msg_inject_keycode_action: req.Msg_inject_keycode_action,
Msg_inject_keycode_keycode: req.Msg_inject_keycode_keycode,
Msg_inject_keycode_metastate: req.Msg_inject_keycode_metastate,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_KEYCODE error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_TEXT:
case CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT:
var pointerid int64 = -1
var pressure uint16 = 65535
var buttons int32 = 1
req.Msg_inject_touch_pointerid = pointerid
req.Msg_inject_touch_pressure = pressure
req.Msg_inject_touch_buttons = buttons
t := TouchMessage{
Msg_type: req.Msg_type,
Msg_inject_touch_action: req.Msg_inject_touch_action,
Msg_inject_touch_pointerid: req.Msg_inject_touch_pointerid,
Msg_inject_touch_position: PositionType{
X: req.Msg_inject_touch_position.X,
Y: req.Msg_inject_touch_position.Y,
Width: req.Msg_inject_touch_position.Width,
Height: req.Msg_inject_touch_position.Height,
},
Msg_inject_touch_pressure: req.Msg_inject_touch_pressure,
Msg_inject_touch_buttons: req.Msg_inject_touch_buttons,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT:
t := ScrollMessage{
Msg_type: req.Msg_type,
Msg_inject_scroll_position: PositionType{
X: req.Msg_inject_scroll_position.X,
Y: req.Msg_inject_scroll_position.Y,
Width: req.Msg_inject_scroll_position.Width,
Height: req.Msg_inject_scroll_position.Height,
},
Msg_inject_scroll_horizontal: req.Msg_inject_scroll_horizontal,
Msg_inject_scroll_vertical: req.Msg_inject_scroll_vertical,
}
buf := &bytes.Buffer{}
err := binary.Write(buf, binary.BigEndian, t)
if err != nil {
log.Debugf("CONTROL_MSG_TYPE_INJECT_SCROLL_EVENT error: %s", err)
log.Debugf("%s",buf.Bytes())
break
}
_, err = conn.Write(buf.Bytes())
case CONTROL_MSG_TYPE_BACK_OR_SCREEN_ON:
case CONTROL_MSG_TYPE_EXPAND_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_COLLAPSE_NOTIFICATION_PANEL:
case CONTROL_MSG_TYPE_GET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_CLIPBOARD:
case CONTROL_MSG_TYPE_SET_SCREEN_POWER_MODE:
default:
err = errors.New("unsupported msg type")
}
if err != nil {
return err
}
}
return nil
}
前端呼叫
let scrcpyKey = (key) => {
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 0,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
ws.send(JSON.stringify({
"msg_type": 0,
"msg_inject_keycode_action": 1,
"msg_inject_keycode_keycode": key,
"msg_inject_keycode_metastate": 0
}))
}
let scrcpyTouchDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 0,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}}));
}
let scrcpyTouchMove = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 2,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
let scrcpyTouchUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 2,
"msg_inject_touch_action": 1,
"msg_inject_touch_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
}
}));
}
//向下滾動
let scrcpyScrollDown = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": -1,
}));
}
//向上滾動
let scrcpyScrollUp = (touch) => {
ws.send(JSON.stringify({
"msg_type": 3,
"msg_inject_scroll_position": {
"x": touch.x, "y": touch.y, "width": touch.w, "height": touch.h
},
"msg_inject_scroll_horizontal": 0,
"msg_inject_scroll_vertical": 1,
}));
}
專案還在開發階段，歡迎反饋問題 : )
收工～
相關文章
- Pycharm遠端除錯PyCharm除錯
- 前端遠端除錯前端除錯
- chrome 遠端除錯Chrome除錯
- 基於NB-IoTeMTC的遠端醫療方案
- PHPSTROM遠端除錯PHP除錯
- pycharm 遠端除錯配置PyCharm除錯
- Spark 1.5.0 遠端除錯Spark除錯
- Mobile Web 除錯指南（2）：遠端除錯Web除錯
- 如何基於SpringBoot+Docker構建公司級別的遠端除錯Spring BootDocker除錯
- Dapr 遠端除錯之 Nocalhost除錯
- pycharm 遠端除錯之二PyCharm除錯
- 遠端除錯 Azure Web App除錯WebAPP
- 用WinDBG遠端除錯程式除錯
- 遠端除錯伺服器python程式碼解決方案除錯伺服器Python
- gdb除錯命令小結_與多檔案除錯_遠端除錯除錯
- WebStorm遠端除錯Node.jsWebORM除錯Node.js
- vs搭建遠端除錯環境除錯
- IDEA、ECLIPSE遠端除錯IdeaEclipse除錯
- VS 遠端除錯 Azure Web App除錯WebAPP
- java Remote Debug（遠端除錯）JavaREM除錯
- VS - 打斷點/本地除錯/遠端除錯 問題斷點除錯
- 構建基於 iOS 模擬器的前端除錯方案iOS前端除錯
- Xdebug+PhpStorm 遠端除錯PHPORM除錯
- Pycharm同步遠端伺服器除錯PyCharm伺服器除錯
- 使用IDEA遠端debug除錯Idea除錯
- vscode遠端除錯c++VSCode除錯C++
- 使用Xdebug進行遠端除錯除錯
- Webkit遠端除錯協議實戰WebKit除錯協議
- 【遠端除錯】Failed to connect to remote VM除錯AIREM
- Pycharm連線遠端伺服器並實現遠端除錯PyCharm伺服器除錯
- 使用Clion優雅的完全遠端自動同步和遠端除錯c++除錯C++
- 【轉貼】將VMware與SoftICE基於網路的遠端除錯功能相結合 (8千字)除錯
- 基於動態IP上的遠端影片監控解決方案(轉)
- 遠端除錯 Android 裝置網頁除錯Android網頁
- phpstorm 遠端除錯 homstead 程式碼PHPORM除錯
- 智慧小程式檔案館——遠端除錯除錯
- 在海思晶片上使用GDB遠端除錯晶片除錯
- 原來 Java 遠端除錯如此簡單Java除錯