快來給你的寵物影片加個表情特效吧

華為雲開發者聯盟發表於2022-12-07

摘要：我們將給貓貼一張卡通臉，給 Elon Musk 貼上小鬍子，給小狗貼上馴鹿角！

本文分享自華為雲社群《影片AI，給你的寵物加個表情特效！》，作者：HWCloudAI。

GAN 監督學習是一種聯合端到端學習判別模型及其 GAN 生成的訓練資料的方法。GANgealing將框架應用於密集視覺對齊問題。受經典 Congealing 方法的啟發，GANgealing 演算法訓練空間變換器將隨機樣本從在未對齊資料上訓練的 GAN 扭曲為共同的、聯合學習的目標模式。目標模式已更新，以使空間轉換器的工作“儘可能簡單”。Spatial Transformer 專門針對 GAN 影像進行訓練，並在測試時自動推廣到真實影像。

我們可以使用它來進行密集跟蹤或建立物鏡。例如，我們將給貓貼一張卡通臉，給 Elon Musk 貼上小鬍子，給小狗貼上馴鹿角！

實驗步驟

1.安裝依賴包

安裝完成之後需要重啟Kernel，重啟之後才會載入新安裝的PyTorch庫

!export CXX=g++
!pip install ninja==1.11.1 ray==2.1.0 plotly==4.14.3 torch==1.10.1 torchvision==0.11.2 moviepy==0.2.3.5 lmdb==0.99

2.下載程式碼

import os
import moxing as mox
if not os.path.exists('gangealing/'):
    mox.file.copy_parallel('obs://weilin/gangealing/', 'gangealing/')

3.進入案例資料夾

cd gangealing/gangealing

model：要檢測的物體，celeba 代表人、dog代表狗、 cat代表貓、 cub代表鳥

pic：要新增的特效圖片

video_name：要新增特效的影片

model = 'cat' #@param ['celeba', 'dog', 'cat', 'cub']
pic = 'ModelArts.png'
video_name = 'demo.mp4'
os.environ['RAW_VIDEO_PATH'] = video_name
!chmod 777 ./ffmpeg
os.environ['FFMPEG_BINARY'] = os.path.join(os.getcwd(), 'ffmpeg')

4.對影片進行抽幀

from pathlib import Path
from utils.download import download_model, download_video
from applications.mixed_reality import run_gangealing_on_video
from applications import load_stn
from glob import glob
video_resolution = "512" #@param [128, 256, 512, 1024, 2048, 4096, 8192]
pad_mode = 'center' #@param ["center", "border"]
os.environ['FFMPEG_BINARY'] = os.path.join(os.getcwd(), 'ffmpeg')
os.environ['VIDEO_SIZE'] = video_size = str(video_resolution)
os.environ['PAD'] = pad_mode
video = Path(os.environ['RAW_VIDEO_PATH']).stem
os.environ['FRAME_PATH'] = f'data/video_frames/{video}'
os.environ['VIDEO_NAME'] = video
video_path = f'data/{video}'
!chmod 777 process_video.sh
!./process_video.sh "$RAW_VIDEO_PATH"
!python prepare_data.py --path "$FRAME_PATH" --out "data/$VIDEO_NAME" --pad "$PAD" --size "$VIDEO_SIZE"

5.為影片新增特效

根據影片的長度和硬體規格，執行此單元需要幾分鐘，您可以在下方監控進度。

fps = 30
batch_size = 1
use_flipping = False
memory_efficient_but_slower = False
if 'cutecat' in video_path:
    fps = 60
class MyDict(): 
 def __init__(self): pass
args = MyDict()
args.real_size = int(video_size)
args.real_data_path = video_path
args.fps = fps
args.batch = batch_size
args.transform = ['similarity', 'flow']
args.flow_size = 128
args.stn_channel_multiplier = 0.5
args.num_heads = 1
args.distributed = False # Colab only uses 1 GPU
args.clustering = False
args.cluster = None
args.objects = True
args.no_flip_inference = not use_flipping
args.save_frames = memory_efficient_but_slower
args.overlay_congealed = False
args.ckpt = model
args.override = False
args.out = 'visuals'
if pic == 'dense tracking':
    args.label_path = f'assets/masks/{model}_mask.png'
 # Feel free to change the parameters below:
    args.resolution = 128
    args.sigma = 1.3
    args.opacity = 0.8
    args.objects = False
else:  # object lense
    args.label_path = f'assets/objects/{model}/{pic}'
    args.resolution = 4 * int(video_size)
    args.sigma = 0.3
    args.opacity = 1.0
    args.objects = True
stn = load_stn(args)
print('Running Spatial Transformer on frames...')
run_gangealing_on_video(args, stn, classifier=None)
print('Preparing videos to be displayed...')
from IPython.display import HTML
from base64 import b64encode
num = len(list(glob(f'{video}_compressed*')))
compressed_name = f'{video}_compressed{num}.mp4'
congealed_compressed_name = f'{video}_compressed_congealed{num}.mp4'
path = f'visuals/video_{video}/propagated.mp4'
congealed_path = f'visuals/video_{video}/congealed.mp4'
os.system(f"ffmpeg -i {path} -vcodec libx264 {compressed_name}")
os.system(f"ffmpeg -i {congealed_path} -vcodec libx264 {congealed_compressed_name}")

6.新增特效前的影片

mp4 = open(video_name,'rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML("""<video width=512 autoplay controls loop><source src="%s" type="video/mp4"></video>""" % (data_url))

7.新增特效後的影片

mp4_1 = open(compressed_name,'rb').read()
data_url_1 = "data:video/mp4;base64," + b64encode(mp4_1).decode()
HTML("""<video width=512 autoplay controls loop><source src="%s" type="video/mp4"></video>""" % (data_url_1))