Flutter實戰圖片元件演進之外接紋理解析

JulyYu發表於2020-07-27

前言

其實Flutter本身已具備載入圖片的能力,Image元件就滿足網路圖片、本地圖片、檔案圖片的載入。那為什麼我們還需要實現其他圖片載入方案呢?其實是因為Flutter圖片元件功能上存在一些缺陷:

  • 圖片快取沒有持久化能力,無網環境下不支援顯示圖片。
  • 檔案圖片與原生環境不共用,導致圖片資原始檔重複。

因此為了滿足日常開發需要和優化點,可以做點什麼讓圖片元件功能達到滿意的效果。接下來從Flutter原生元件再到外接紋理慢慢了解圖片元件功能演進的過程。

Flutter原生圖片元件

Flutter原生圖片支援多種載入形式:

  • Image.network(網路圖片)
  • Image.file (本地圖片)
  • Image.asset (檔案圖片)
  • Image.memory (byte圖片)

圖片載入流程簡要(網路圖片為例)

  • 第一步:網路圖片載入形式以NetworkImage,內部由network_image.NetworkImage構成。
Image.network(
    String src, {
    ......省略不必要程式碼
  }) : image = ResizeImage.resizeIfNeeded(cacheWidth, cacheHeight, NetworkImage(src, scale: scale, headers: headers)),
    ...... 省略不必要程式碼
       super(key: key);
......
const factory NetworkImage(String url, { double scale, Map<String, String> headers }) = network_image.NetworkImage;

複製程式碼
  • 第二步:NetWorkImage實質上繼承於ImageProvider,其他載入形式也是如此。ImageProvider是處理圖片基本抽象類,繼承它的載入類主要實現load方法執行不同形式載入過程。
abstract class ImageProvider<T> {
  const ImageProvider();
  .......
  @protected
  ImageStreamCompleter load(T key, DecoderCallback decode);
  .......
}
複製程式碼
  • 第三步:例如網路形式獲取圖片資料過程通過網路,通過Dart層網路請求HttpClient請求圖片獲取最終資料Uint8List。
class NetworkImage extends image_provider.ImageProvider<image_provider.NetworkImage> implements image_provider.NetworkImage {
    .......
    @override
  ImageStreamCompleter load(image_provider.NetworkImage key, image_provider.DecoderCallback decode) {
    final StreamController<ImageChunkEvent> chunkEvents = StreamController<ImageChunkEvent>();

    return MultiFrameImageStreamCompleter(
      codec: _loadAsync(key as NetworkImage, chunkEvents, decode),
      chunkEvents: chunkEvents.stream,
      scale: key.scale,
      informationCollector: () {
        return <DiagnosticsNode>[
          DiagnosticsProperty<image_provider.ImageProvider>('Image provider', this),
          DiagnosticsProperty<image_provider.NetworkImage>('Image key', key),
        ];
      },
    );
  }
  
  
  Future<ui.Codec> _loadAsync(
    NetworkImage key,
    StreamController<ImageChunkEvent> chunkEvents,
    image_provider.DecoderCallback decode,
  ) async {
    try {
      final Uri resolved = Uri.base.resolve(key.url);
      final HttpClientRequest request = await _httpClient.getUrl(resolved);
      headers?.forEach((String name, String value) {
        request.headers.add(name, value);
      });
      final HttpClientResponse response = await request.close();
      if (response.statusCode != HttpStatus.ok) {
        PaintingBinding.instance.imageCache.evict(key);
        throw image_provider.NetworkImageLoadException(statusCode: response.statusCode, uri: resolved);
      }

      final Uint8List bytes = await consolidateHttpClientResponseBytes(
        response,
        onBytesReceived: (int cumulative, int total) {
          chunkEvents.add(ImageChunkEvent(
            cumulativeBytesLoaded: cumulative,
            expectedTotalBytes: total,
          ));
        },
      );
      if (bytes.lengthInBytes == 0)
        throw Exception('NetworkImage is an empty file: $resolved');
      return decode(bytes);
    } finally {
      chunkEvents.close();
    }
  }
}

複製程式碼
  • 第四步:獲取到圖片Uint8List資料之後就是解碼過程。通過DecoderCallback回撥方法得到圖片原始資料之後交付給全域性單例解碼器PaintingBinding.instance.instantiateImageCodec,然後由引擎層C++的instantiateImageCodec處理資料返回可被Flutter層渲染展示Image資料。
 final ImageStreamCompleter completer = PaintingBinding.instance.imageCache.putIfAbsent(
      key,
      () => load(key, PaintingBinding.instance.instantiateImageCodec),
      onError: handleError,
    );
 Future<ui.Codec> instantiateImageCodec(Uint8List bytes, {
    int cacheWidth,
    int cacheHeight,
  }) {
    assert(cacheWidth == null || cacheWidth > 0);
    assert(cacheHeight == null || cacheHeight > 0);
    return ui.instantiateImageCodec(
      bytes,
      targetWidth: cacheWidth,
      targetHeight: cacheHeight,
    );
  }    
String _instantiateImageCodec(Uint8List list, _Callback<Codec> callback, _ImageInfo imageInfo, int targetWidth, int targetHeight)
  native 'instantiateImageCodec';
複製程式碼
  • 第五步:引擎層c++解碼器具體在codec.cc中,呼叫了Skia的SkCodec對圖片資料做處理。經過解碼器內部處理後執行ToDart將ui_codec返回到Dart層。
/// Dart層程式碼
_String _instantiateImageCodec(Uint8List list, _Callback<Codec> callback, _ImageInfo imageInfo, int targetWidth, int targetHeight)
  native 'instantiateImageCodec';
/// c++層程式碼
static void InstantiateImageCodec(Dart_NativeArguments args) {
  UIDartState::ThrowIfUIOperationsProhibited();
  Dart_Handle callback_handle = Dart_GetNativeArgument(args, 1);
  .......省略部分程式碼
  Dart_Handle image_info_handle = Dart_GetNativeArgument(args, 2);

  std::optional<ImageDecoder::ImageInfo> image_info;
  /// 圖片資訊是否為空,不為空做一些處理
  if (!Dart_IsNull(image_info_handle)) {
    auto image_info_results = ConvertImageInfo(image_info_handle, args);
    if (auto value =
            std::get_if<ImageDecoder::ImageInfo>(&image_info_results)) {
      image_info = *value;
    } else if (auto error = std::get_if<std::string>(&image_info_results)) {
      Dart_SetReturnValue(args, tonic::ToDart(*error));
      return;
    }
  }

  sk_sp<SkData> buffer;

  {
    /// 處理圖片資料
    Dart_Handle exception = nullptr;
    tonic::Uint8List list =
        tonic::DartConverter<tonic::Uint8List>::FromArguments(args, 0,
                                                              exception);
    if (exception) {
      Dart_SetReturnValue(args, exception);
      return;
    }
    /// 圖片資料做拷貝
    buffer = MakeSkDataWithCopy(list.data(), list.num_elements());
  }

  if (image_info) {
    const auto expected_size =
        image_info->row_bytes * image_info->sk_info.height();
    if (buffer->size() < expected_size) {
      Dart_SetReturnValue(
          args, ToDart("Pixel buffer size does not match image size"));
      return;
    }
  }
   /// 獲取圖片目標寬高
  const int targetWidth =
      tonic::DartConverter<int>::FromDart(Dart_GetNativeArgument(args, 3));
  const int targetHeight =
      tonic::DartConverter<int>::FromDart(Dart_GetNativeArgument(args, 4));

  std::unique_ptr<SkCodec> codec;
  bool single_frame;
  if (image_info) {
    single_frame = true;
  } else {
    /// 底層解碼器使用的是SkCodec解碼器,Android底層同樣使用的是它。
    codec = SkCodec::MakeFromData(buffer);
    if (!codec) {
      Dart_SetReturnValue(args, ToDart("Could not instantiate image codec."));
      return;
    }
    single_frame = codec->getFrameCount() == 1;
  }
   /// 解碼器同時解碼後得出幀數資訊,判斷圖片是否為動圖
  fml::RefPtr<Codec> ui_codec;

  if (single_frame) {
    ImageDecoder::ImageDescriptor descriptor;
    descriptor.decompressed_image_info = image_info;

    if (targetWidth > 0) {
      descriptor.target_width = targetWidth;
    }
    if (targetHeight > 0) {
      descriptor.target_height = targetHeight;
    }
    descriptor.data = std::move(buffer);

    ui_codec = fml::MakeRefCounted<SingleFrameCodec>(std::move(descriptor));
  } else {
    ui_codec = fml::MakeRefCounted<MultiFrameCodec>(std::move(codec));
  }
  /// 最後將解碼器結果返回到Dart層
  tonic::DartInvoke(callback_handle, {ToDart(ui_codec)});
}
複製程式碼

外接紋理渲染圖片

Flutter中有一個叫做Texture元件,該元件只有唯一入參textureId,寥寥無幾的幾行程式碼就實現外接紋理著實讓人摸不清頭腦。在分析外接紋理原理之前先簡單瞭解外接紋理渲染圖片功能實現。

Texture元件使用

  • Java層通過Channel外掛PluginRegistry.Registrar建立Surface、textureId。
/// 外掛介面獲取texture註冊器
TextureRegistry textureRegistry = registrar.textures();
/// 建立Texture例項
TextureRegistry.SurfaceTextureEntry surfaceTextureEntry = textureRegistry.createSurfaceTexture();
long textureId = surfaceTextureEntry.id();
SurfaceTexture surfaceTexture = surfaceTextureEntry.surfaceTexture();
/// 獲取圖片地址
String url = call.argument("url");
...... 省略圖片請求載入過程
/// 建立Surface例項載入surfaceTexture
Surface surface = new Surface(surfaceTexture);
/// 畫布繪製bitmap 紋理對映
Canvas canvas = surface.lockCanvas(rect);
canvas.drawBitmap(bitmap, null, rect, null);
bitmap.recycle();
surface.unlockCanvasAndPost(canvas);
/// Dart返回textureId
Map<String, Object> maps = new HashMap<>();
maps.put("textureId", textureId);
result.success(maps);
複製程式碼
  • Dart層建立MethodChannel,向Native層傳遞載入圖片路徑。
  static const MethodChannel _channel = const MethodChannel('texture_channel');
  /// 原始載入圖片介面
  static Future<Map> loadTexture({String url}) async {
    var args = <String, dynamic>{
      "url": url,
    };
    return await _channel.invokeMethod("loadTexture", args);
  }
  /// 執行載入
  Map _textureResult = await TexturePlugin.loadTexture(
      url: _uri.toString(),
      width: url.width,
      height: url.height,
    );
   /// 返回Native生成的textureId
   int id = _textureResult['textureId'];
   /// 例項化texture元件 顯示圖片
   Texture( textureId: id);
複製程式碼

程式碼解析

Dart層

從原始碼上可以看到Texture會建立渲染物件TextureBox。TextureBox會去繪製TextureLayer,TextureLayer則通過ui.SceneBuilder向Scene新增紋理,最終是呼叫引擎層SceneBuilder_addTexture方法實現紋理渲染。

  • Texture
class Texture extends LeafRenderObjectWidget {
  const Texture({
    Key key,
    @required this.textureId,
  }) : assert(textureId != null),
       super(key: key);
       
  final int textureId;
  
  @override
  TextureBox createRenderObject(BuildContext context) => TextureBox(textureId: textureId);

  @override
  void updateRenderObject(BuildContext context, TextureBox renderObject) {
    renderObject.textureId = textureId;
  }
}
複製程式碼
  • TextureBox
class TextureBox extends RenderBox {
  TextureBox({ @required int textureId })
    : assert(textureId != null),
      _textureId = textureId;
  ...... 省略程式碼
  @override
  void paint(PaintingContext context, Offset offset) {
    if (_textureId == null)
      return;
    context.addLayer(TextureLayer(
      rect: Rect.fromLTWH(offset.dx, offset.dy, size.width, size.height),
      textureId: _textureId,
    ));
  }
}
複製程式碼
  • TextureLayer
class TextureLayer extends Layer {
  TextureLayer({
    @required this.rect,
    @required this.textureId,
    this.freeze = false,
  }) : assert(rect != null),
       assert(textureId != null);

  ......
  @override
  void addToScene(ui.SceneBuilder builder, [ Offset layerOffset = Offset.zero ]) {
    final Rect shiftedRect = layerOffset == Offset.zero ? rect : rect.shift(layerOffset);
    builder.addTexture(
      textureId,
      offset: shiftedRect.topLeft,
      width: shiftedRect.width,
      height: shiftedRect.height,
      freeze: freeze,
    );
  }
}
複製程式碼
  • SceneBuilder
class SceneBuilder extends NativeFieldWrapperClass2 {
 
 void addTexture(
   int textureId, {
   Offset offset = Offset.zero,
   double width = 0.0,
   double height = 0.0,
   bool freeze = false,
 }) {
   _addTexture(offset.dx, offset.dy, width, height, textureId, freeze);
 }
   /// SceneBuilder_addTexture對應scene_builder.cc下的SceneBuilder::addTexture方法
 void _addTexture(double dx, double dy, double width, double height, int textureId, bool freeze)
     native 'SceneBuilder_addTexture';
    
複製程式碼
  • scene_builder.cc
void SceneBuilder::addTexture(double dx,
                              double dy,
                              double width,
                              double height,
                              int64_t textureId,
                              bool freeze) {
  auto layer = std::make_unique<flutter::TextureLayer>(
      SkPoint::Make(dx, dy), SkSize::Make(width, height), textureId, freeze);
  AddLayer(std::move(layer));
}
複製程式碼
  • texture_layer.cc
/// 建立紋理層物件
TextureLayer::TextureLayer(const SkPoint& offset,
                           const SkSize& size,
                           int64_t texture_id,
                           bool freeze)
    : offset_(offset), size_(size), texture_id_(texture_id), freeze_(freeze) {}
/// 紋理物件繪製方法
void TextureLayer::Paint(PaintContext& context) const {
  TRACE_EVENT0("flutter", "TextureLayer::Paint");
  /// texture物件繪製時會從GetTexture方法的map中找到紋理物件進行繪製。
  std::shared_ptr<Texture> texture =
      context.texture_registry.GetTexture(texture_id_);
  if (!texture) {
    TRACE_EVENT_INSTANT0("flutter", "null texture");
    return;
  }
  texture->Paint(*context.leaf_nodes_canvas, paint_bounds(), freeze_,
                 context.gr_context);
}
複製程式碼

Java層

  • FlutterRenderer
public class FlutterRenderer implements TextureRegistry {

  public FlutterRenderer(@NonNull FlutterJNI flutterJNI) {
    this.flutterJNI = flutterJNI;
    this.flutterJNI.addIsDisplayingFlutterUiListener(flutterUiDisplayListener);
  }
    @Override
  public SurfaceTextureEntry createSurfaceTexture() {
   /// 建立SurfaceTexture
    final SurfaceTexture surfaceTexture = new SurfaceTexture(0);
    surfaceTexture.detachFromGLContext();
    final SurfaceTextureRegistryEntry entry =
        new SurfaceTextureRegistryEntry(nextTextureId.getAndIncrement(), surfaceTexture);
    
    registerTexture(entry.id(), surfaceTexture);
    return entry;
  }
  ///向Native JNI層註冊
  private void registerTexture(long textureId, @NonNull SurfaceTexture surfaceTexture) {
    flutterJNI.registerTexture(textureId, surfaceTexture);
  }
}
複製程式碼
  • FlutterJNI
  public void registerTexture(long textureId, @NonNull SurfaceTexture surfaceTexture) {
    /// 這裡需要注意必須在主執行緒執行否則會報錯
    ensureRunningOnMainThread();
    ensureAttachedToNative();
    nativeRegisterTexture(nativePlatformViewId, textureId, surfaceTexture);
  }
複製程式碼

C++層

  • platform_view_android_jni.cc
{
          .name = "nativeRegisterTexture",
          .signature = "(JJLandroid/graphics/SurfaceTexture;)V",
          .fnPtr = reinterpret_cast<void*>(&RegisterTexture),
}

static void RegisterTexture(JNIEnv* env,
                            jobject jcaller,
                            jlong shell_holder,
                            jlong texture_id,
                            jobject surface_texture) {
  ANDROID_SHELL_HOLDER->GetPlatformView()->RegisterExternalTexture(
      static_cast<int64_t>(texture_id),                        //
      fml::jni::JavaObjectWeakGlobalRef(env, surface_texture)  //
  );
}
複製程式碼
  • platform_view_android.cc
void PlatformViewAndroid::RegisterExternalTexture(
    int64_t texture_id,
    const fml::jni::JavaObjectWeakGlobalRef& surface_texture) {
  RegisterTexture(
      std::make_shared<AndroidExternalTextureGL>(texture_id, surface_texture));
}
複製程式碼
  • texture.cc
// 註冊紋理方法
void TextureRegistry::RegisterTexture(std::shared_ptr<Texture> texture) {
  if (!texture) {
    return;
  }
  // 內部map儲存texture例項
  mapping_[texture->Id()] = texture;
}
// 獲取到紋理物件,從map中提取
std::shared_ptr<Texture> TextureRegistry::GetTexture(int64_t id) {
  auto it = mapping_.find(id);
  return it != mapping_.end() ? it->second : nullptr;
}
複製程式碼
  • android_external_texture_gl.cc
/// 外接紋理物件例項
AndroidExternalTextureGL::AndroidExternalTextureGL(
    int64_t id,
    const fml::jni::JavaObjectWeakGlobalRef& surfaceTexture)
    : Texture(id), surface_texture_(surfaceTexture), transform(SkMatrix::I()) {}

AndroidExternalTextureGL::~AndroidExternalTextureGL() {
  if (state_ == AttachmentState::attached) {
    glDeleteTextures(1, &texture_name_);
  }
}
/// 繪製方法
void AndroidExternalTextureGL::Paint(SkCanvas& canvas,
                                     const SkRect& bounds,
                                     bool freeze,
                                     GrContext* context) {
  if (state_ == AttachmentState::detached) {
    return;
  }
  if (state_ == AttachmentState::uninitialized) {
    glGenTextures(1, &texture_name_);
    Attach(static_cast<jint>(texture_name_));
    state_ = AttachmentState::attached;
  }
  if (!freeze && new_frame_ready_) {
    Update();
    new_frame_ready_ = false;
  }
  GrGLTextureInfo textureInfo = {GL_TEXTURE_EXTERNAL_OES, texture_name_,
                                 GL_RGBA8_OES};
  GrBackendTexture backendTexture(1, 1, GrMipMapped::kNo, textureInfo);
  sk_sp<SkImage> image = SkImage::MakeFromTexture(
      canvas.getGrContext(), backendTexture, kTopLeft_GrSurfaceOrigin,
      kRGBA_8888_SkColorType, kPremul_SkAlphaType, nullptr);
  if (image) {
    SkAutoCanvasRestore autoRestore(&canvas, true);
    canvas.translate(bounds.x(), bounds.y());
    canvas.scale(bounds.width(), bounds.height());
    if (!transform.isIdentity()) {
      SkMatrix transformAroundCenter(transform);

      transformAroundCenter.preTranslate(-0.5, -0.5);
      transformAroundCenter.postScale(1, -1);
      transformAroundCenter.postTranslate(0.5, 0.5);
      canvas.concat(transformAroundCenter);
    }
    canvas.drawImage(image, 0, 0);
  }
}
複製程式碼
  • ①Java層FlutterRenderer建立SurfaceTexture和textureId。
  • ②將urfaceTexture和textureId通過JNI向引擎層註冊
  • ③向引擎註冊過程中通過層層方法最後在texture.cc的TextureRegistry由map以鍵值對形式快取例項物件。
  • ④將需要顯示圖片在SurfaceTexture上離屏渲染。
  • ⑤Java層建立的textureId通過Channel傳遞到Dart層作為Texture元件入參。
  • ⑥Dart的Texture元件接收textureId入參後向下層元件例項化。
  • ⑦在SceneBuilder呼叫addTexture時執行引擎層建立TextureLayer。
  • ⑧最終在texture.cc中TextureRegistry的map根據TextureId獲取SurfaceTexture例項。

Image VS Texture

Image元件和Texture元件實質上都是對RenderBox的實現,已知Flutter渲染樹實際上就是RenderObject合併為Layer最終通過Flutter引擎進行繪製上屏顯示頁面內容。兩者區別在於實現RenderObject的paint有所不同,Image的實現將ui.Image內容繪製在ui.Canvas上,Texture是將TextureLayer新增到ui.Scene中。不同之處就在於Image做了繪製操作,Texture是做新增操作,紋理方案的圖片載入和繪製完全由原生平臺完成。

?紋理完整程式碼看這裡?

參考

相關文章