對於深度學習初學者來說，JupyterNoteBook的指令碼執行形式顯然更加友好，依託Python語言的跨平臺特性，JupyterNoteBook既可以在本地線下環境執行，也可以線上上伺服器上執行。GoogleColab作為免費GPU算力平臺的執牛耳者，更是讓JupyterNoteBook的指令碼執行形式如虎添翼。
本次我們利用Bert-vits2的最終版Bert-vits2-v2.3和JupyterNoteBook的指令碼來複刻生化危機6的人氣角色艾達王(ada wong)。
本地除錯JupyterNoteBook
眾所周知，GoogleColab雖然可以免費提供GPU讓使用者用於模型訓練和推理，但是每一個JupyterNoteBook檔案指令碼最多隻能執行12小時，隨後就會被限制，所以為了避免浪費寶貴的GPU使用時間，我們可以線上下除錯自己的JupyterNoteBook指令碼，除錯成功後，就可以把指令碼直接上傳到GoogleColab平臺。
首先透過pip命令進行本地安裝：
python3 -m pip install jupyter
隨後執行啟動命令：
jupyter notebook
此時，訪問本地的notebook地址：
隨後選擇檔案-》新建-》Notebook 即可。
輸入筆記內容：
#@title 檢視顯示卡
!nvidia-smi
點選執行單元格：
程式返回：
#@title 檢視顯示卡
!nvidia-smi
Wed Dec 27 12:36:10 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 546.17 Driver Version: 546.17 CUDA Version: 12.3 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4060 ... WDDM | 00000000:01:00.0 Off | N/A |
| N/A 50C P0 20W / 115W | 0MiB / 8188MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
至此，就可以在本地除錯NoteBook了。
安裝ffmpeg
新增單元格：
#@title 安裝ffmpeg
import os, uuid, re, IPython
import ipywidgets as widgets
import time
from glob import glob
from google.colab import output, drive
from IPython.display import clear_output
import os, sys, urllib.request
HOME = os.path.expanduser("~")
pathDoneCMD = f'{HOME}/doneCMD.sh'
if not os.path.exists(f"{HOME}/.ipython/ttmg.py"):
hCode = "https://raw.githubusercontent.com/yunooooo/gcct/master/res/ttmg.py"
urllib.request.urlretrieve(hCode, f"{HOME}/.ipython/ttmg.py")
from ttmg import (
loadingAn,
textAn,
)
loadingAn(name="lds")
textAn("Cloning Repositories...", ty='twg')
!git clone https://github.com/XniceCraft/ffmpeg-colab.git
!chmod 755 ./ffmpeg-colab/install
textAn("Installing FFmpeg...", ty='twg')
!./ffmpeg-colab/install
clear_output()
print('Installation finished!')
!rm -fr /content/ffmpeg-colab
!ffmpeg -version
由於語音轉寫需要ffmpeg的參與，所以需要安裝ffmpeg的最新版本。
程式返回：
Installation finished!
c Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 9 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
configuration: --prefix=/home/ffmpeg-builder/release --pkg-config-flags=--static --extra-libs=-lm --disable-doc --disable-debug --disable-shared --disable-ffprobe --enable-static --enable-gpl --enable-version3 --enable-runtime-cpudetect --enable-avfilter --enable-filters --enable-nvenc --enable-nvdec --enable-cuvid --toolchain=hardened --disable-stripping --enable-opengl --pkgconfigdir=/home/ffmpeg-builder/release/lib/pkgconfig --extra-cflags='-I/home/ffmpeg-builder/release/include -static-libstdc++ -static-libgcc ' --extra-ldflags='-L/home/ffmpeg-builder/release/lib -fstack-protector -static-libstdc++ -static-libgcc ' --extra-cxxflags=' -static-libstdc++ -static-libgcc ' --extra-libs='-ldl -lrt -lpthread' --enable-ffnvcodec --enable-gmp --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libfdk-aac --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libkvazaar --enable-libmp3lame --enable-libopus --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libshine --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtheora --enable-libvidstab --ld=g++ --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg --enable-openssl --enable-zlib --enable-nonfree --extra-libs=-lpthread --enable-pthreads --extra-libs=-lgomp
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
這裡安裝的是最新版ffmpeg version 6.0
克隆程式碼庫
接著克隆程式碼庫：
#@title 克隆程式碼倉庫
!git clone https://github.com/v3ucn/Bert-vits2-V2.3.git
程式返回：
Cloning into 'Bert-vits2-V2.3'...
remote: Enumerating objects: 234, done.
remote: Counting objects: 100% (234/234), done.
remote: Compressing objects: 100% (142/142), done.
remote: Total 234 (delta 80), reused 232 (delta 78), pack-reused 0
Receiving objects: 100% (234/234), 4.16 MiB | 14.14 MiB/s, done.
Resolving deltas: 100% (80/80), done.
安裝專案依賴
隨後進入專案的目錄，安裝依賴：
#@title 安裝所需要的依賴
%cd /content/Bert-vits2-V2.3
!pip install -r requirements.txt
下載必要的模型
新增單元格，下載模型：
#@title 下載必要的模型
!wget -P slm/wavlm-base-plus/ https://huggingface.co/microsoft/wavlm-base-plus/resolve/main/pytorch_model.bin
!wget -P emotional/clap-htsat-fused/ https://huggingface.co/laion/clap-htsat-fused/resolve/main/pytorch_model.bin
!wget -P emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim/ https://huggingface.co/audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim/resolve/main/pytorch_model.bin
!wget -P bert/chinese-roberta-wwm-ext-large/ https://huggingface.co/hfl/chinese-roberta-wwm-ext-large/resolve/main/pytorch_model.bin
!wget -P bert/bert-base-japanese-v3/ https://huggingface.co/cl-tohoku/bert-base-japanese-v3/resolve/main/pytorch_model.bin
!wget -P bert/deberta-v3-large/ https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.bin
!wget -P bert/deberta-v3-large/ https://huggingface.co/microsoft/deberta-v3-large/resolve/main/pytorch_model.generator.bin
!wget -P bert/deberta-v2-large-japanese/ https://huggingface.co/ku-nlp/deberta-v2-large-japanese/resolve/main/pytorch_model.bin
下載底模檔案
接著下載預訓練模型的底模：
#@title 下載底模檔案
!wget -P Data/ada/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/DUR_0.pth
!wget -P Data/ada/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/D_0.pth
!wget -P Data/ada/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/G_0.pth
!wget -P Data/ada/models/ https://huggingface.co/OedoSoldier/Bert-VITS2-2.3/resolve/main/WD_0.pth
注意2.3版本的底模是4個。
切分資料集
接著把艾達王的音訊素材上傳到Data/ada/raw/ada.wav
隨後新建單元格：
#@title 切分資料集
!python3 audio_slicer.py
素材就會被切分。
轉寫和標註
此時我們需要把切片素材轉寫：
#@title 轉寫和標註
!pip install git+https://github.com/openai/whisper.git
!python3 short_audio_transcribe.py
注意這裡單獨安裝whisper，很多人直接用 pip install whisper，其實這不是正確的安裝方式，需要單獨指定安裝源：pip install git+https://github.com/openai/whisper.git，切記，否則會報錯。
執行完畢後會在角色目錄生成轉寫檔案esd.list:
./Data\ada\wavs\ada_0.wav|ada|EN|I do. The kind you like.
./Data\ada\wavs\ada_1.wav|ada|EN|Now where's the amber?
./Data\ada\wavs\ada_10.wav|ada|EN|Leave the girl. She's lost no matter what.
./Data\ada\wavs\ada_11.wav|ada|EN|You walk away now, and who knows?
./Data\ada\wavs\ada_12.wav|ada|EN|Maybe you'll live to meet me again.
./Data\ada\wavs\ada_13.wav|ada|EN|And I might get you that greeting you were looking for.
./Data\ada\wavs\ada_14.wav|ada|EN|How about we continue this discussion another time?
./Data\ada\wavs\ada_15.wav|ada|EN|Sorry, nothing yet.
./Data\ada\wavs\ada_16.wav|ada|EN|But my little helper is creating
./Data\ada\wavs\ada_17.wav|ada|EN|Quite the commotion.
./Data\ada\wavs\ada_18.wav|ada|EN|Everything will work out just fine.
./Data\ada\wavs\ada_19.wav|ada|EN|He's a good boy. Predictable.
./Data\ada\wavs\ada_2.wav|ada|EN|The deal was, we get you out of here when you deliver the amber. No amber, no protection, Louise.
./Data\ada\wavs\ada_20.wav|ada|EN|Nothing personal, Leon.
./Data\ada\wavs\ada_21.wav|ada|EN|Louise and I had an arrangement.
./Data\ada\wavs\ada_22.wav|ada|EN|Don't worry, I'll take good care of it.
./Data\ada\wavs\ada_23.wav|ada|EN|Just one question.
./Data\ada\wavs\ada_24.wav|ada|EN|What are you planning to do with this?
./Data\ada\wavs\ada_25.wav|ada|EN|So, we're talking millions of casualties?
./Data\ada\wavs\ada_26.wav|ada|EN|We're changing course. Now.
./Data\ada\wavs\ada_3.wav|ada|EN|You can stop right there, Leon.
./Data\ada\wavs\ada_4.wav|ada|EN|wouldn't make me use this.
./Data\ada\wavs\ada_5.wav|ada|EN|Would you? You don't seem surprised.
./Data\ada\wavs\ada_6.wav|ada|EN|Interesting.
./Data\ada\wavs\ada_7.wav|ada|EN|Not a bad move
./Data\ada\wavs\ada_8.wav|ada|EN|Very smooth. Ah, Leon.
./Data\ada\wavs\ada_9.wav|ada|EN|You know I don't work and tell.
這裡一共27條切片語音，對應27個轉寫文字，注意語言是英語。
音訊重新取樣
對素材音訊進行重新取樣的操作：
#@title 重新取樣
!python3 resample.py --sr 44100 --in_dir ./Data/ada/raw/ --out_dir ./Data/ada/wavs/
預處理標籤檔案
接著處理轉寫檔案，生成訓練集和驗證集：
#@title 預處理標籤檔案
!python3 preprocess_text.py --transcription-path ./Data/ada/esd.list --t
程式返回：
pytorch_model.bin: 100% 1.32G/1.32G [00:10<00:00, 122MB/s]
spm.model: 100% 2.46M/2.46M [00:00<00:00, 115MB/s]
The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
0it [00:00, ?it/s]
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] /root/nltk_data...
[nltk_data] Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package cmudict to /root/nltk_data...
[nltk_data] Unzipping corpora/cmudict.zip.
100% 27/27 [00:00<00:00, 4457.63it/s]
總重複音訊數：0，總未找到的音訊數:0
訓練集和驗證集生成完成！
生成 BERT 特徵檔案
最後生成bert特徵檔案：
#@title 生成 BERT 特徵檔案
!python3 bert_gen.py --config-path ./Data/ada/configs/config.json
對應27個素材：
100% 27/27 [00:33<00:00, 1.25s/it]
bert生成完畢!, 共有27個bert.pt生成!
模型訓練
萬事俱備，開始訓練：
#@title 開始訓練
!python3 train_ms.py
模型會在models目錄生成，專案預設設定了訓練間隔是50步，可以根據自己的需求修改config.json配置檔案。
模型推理
一般情況下，訓練了50步或者100步左右，可以推理一下檢視效果，然後繼續訓練：
#@title 開始推理
!python3 webui.py
返回：
| numexpr.utils | INFO | NumExpr defaulting to 2 threads.
/usr/local/lib/python3.10/dist-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
| utils | INFO | Loaded checkpoint 'Data/ada/models/G_150.pth' (iteration 25)
推理頁面已開啟!
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://814833a6f477ba151c.gradio.live
點選第二個公網地址進行推理即可。
結語
至此，我們已經完成了基於JupyterNoteBook的資料切分、轉寫、預處理、訓練以及推理流程。最後奉上線上GoogleColab，以饗眾鄉親：
https://colab.research.google.com/drive/1-H1DGG5dTy8u_8vFbq1HACXPX9AAM76s?usp=sharing