CMSampleBuffer 分析

CMSampleBuffer的使用

• 通過攝像頭取樣的視訊資料。
• 通過mic採集的音訊資訊。

- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection;

• AVAssetWriter儲存視訊AVAssetWriterInput

- (BOOL)appendSampleBuffer:(CMSampleBufferRef)sampleBuffer;

• 讀取視訊檔案的輸出流AVAssetReaderOutput

- (nullable CMSampleBufferRef)copyNextSampleBuffer CF_RETURNS_RETAINED;

• AVSampleBufferDisplayLayer 直接顯示解碼後的sampleBuffer

- (void)enqueueSampleBuffer:(CMSampleBufferRef)sampleBuffer;

• ARKit ARSessionObserver輸出捕獲的audioSampleBuffer

/**
 This is called when the session outputs a new audio sample buffer.
 @param session The session being run.
 @param audioSampleBuffer The captured audio sample buffer.
 */
- (void)session:(ARSession *)session didOutputAudioSampleBuffer:(CMSampleBufferRef)audioSampleBuffer;

• VTCompressionSession硬體編碼作為VTCompressionOutputCallback輸出

typedef void (*VTCompressionOutputCallback)(
        void * CM_NULLABLE outputCallbackRefCon,
        void * CM_NULLABLE sourceFrameRefCon, 
        OSStatus status, 
        VTEncodeInfoFlags infoFlags,
        CM_NULLABLE CMSampleBufferRef sampleBuffer );

建立壓縮session

注意：

Clients of CMSampleBuffer must explicitly manage the retain count by calling CFRetain and CFRelease, even in processes using garbage collection.

CMSampleBuffer 資料結構

• CMSampleBufferRef 這是一個包含零個或多個解碼後（未解碼）特定媒體型別的樣本（音訊，視訊，多路複用等）。
  API註釋

A reference to a CMSampleBuffer, a CF object containing zero or more compressed (or uncompressed) samples of a particular media type (audio, video, muxed, etc).

• CaptureOutput輸出video CMSampleBuffer,編碼前內容

(lldb) po sampleBuffer
CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    buffer-level attachments:
        Orientation(P) = 1
        {Exif}   (P)  {//省略}

        DPIWidth  (P) = 72
        {TIFF}    (P) {//省略}

        DPIHeight (P) = 72
    formatDescription = {
    mediaType:'vide' 
    mediaSubType:'420f' 
    mediaSpecific: {
        codecType: '420f'       dimensions: 480 x 640 
    } 
    extensions: {{type = immutable dict, count = 6,
entries =>
    0 :{contents = "CVImageBufferYCbCrMatrix"} = {contents = "ITU_R_601_4"}
    1 : {contents = "CVImageBufferTransferFunction"} ={contents = "ITU_R_709_2"}
    2 : {contents = "CVBytesPerRow"} ={value = +724, type = kCFNumberSInt32Type}
    3 :{contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
    5 : {contents = "CVImageBufferColorPrimaries"} = {contents = "ITU_R_709_2"}
    6 : {contents = "CVImageBufferChromaLocationTopField"} = {contents = "Center"}
}
}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {6510691197875/1000000000 = 6510.691}, DTS = {INVALID}, duration = {INVALID}},
    }
    imageBuffer = 0x1c4134c80

(lldb) 

• 硬編碼後的資料,相比編碼前多了sampleAttachmentsArray，顏色空間是'avc1'

CMSampleBuffer 0x11be1fda0 retainCount: 
  1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription =  {
    mediaType:'vide' 
    mediaSubType:'avc1' 
    mediaSpecific: {
        codecType: 'avc1'       dimensions: 480 x 640 
    } 
  2 : {contents = "FormatName"} = H.264
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {65/1000 = 0.065}, DTS = {64/1000 = 0.064}, duration = {INVALID}},
    }
    sampleSizeArray[1] = {
        sampleSize = 212,
    }
    sampleAttachmentsArray[1] = {
        sample 0:
            NotSync    = true
            DependsOnOthers = true
    }
    dataBuffer = 0x1c410ac20

• CaptureOutput輸出音訊，buffer內容

CMSampleBuffer 0x125e41940 retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription =  {
    mediaType:'soun' 
    mediaSubType:'lpcm' 
    mediaSpecific: {
        ASBD: {
            mSampleRate: 44100.000000 
            mFormatID: 'lpcm' 
            mFormatFlags: 0xc 
            mBytesPerPacket: 2 
            mFramesPerPacket: 1 
            mBytesPerFrame: 2 
            mChannelsPerFrame: 1 
            mBitsPerChannel: 16     } 
        cookie: {(null)} 
        ACL: {(null)}
        FormatList Array: {(null)} 
    } 
    extensions: {(null)}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1024
    sampleTimingArray[1] = {
        {PTS = {10713597355917/1000000000 = 10713.597, rounded}, DTS = {INVALID}, duration = {1/44100 = 0.000}},
    }
    sampleSizeArray[1] = {
        sampleSize = 2,
    }
    dataBuffer = 0x1c4304650

• AVAssetReaderOutput讀取MP4檔案sampleBuffer內容

CMSampleBuffer retainCount: 1 allocator: 0x1b6708538
    invalid = NO
    dataReady = YES
    makeDataReadyCallback = 0x0
    makeDataReadyRefcon = 0x0
    formatDescription = {

    mediaType:'vide' 
    mediaSubType:'BGRA' 
    mediaSpecific: {
        codecType: 'BGRA'   dimensions: 640 x 360 
    } 
    extensions: {{type = immutable dict, count = 9,
entries =>
    0 : {contents = "CVFieldCount"} = {value = +1, type = kCFNumberSInt32Type}
    1 : {contents = "CVImageBufferYCbCrMatrix"} = {contents = "YCbCrMatrix#5"}
    2 : {contents = "CVBytesPerRow"} ={value = +2560, type = kCFNumberSInt32Type}
    3 : {contents = "CVPixelAspectRatio"} = {type = immutable dict, count = 2,
entries =>
    1 : {contents = "HorizontalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
    2 : {contents = "VerticalSpacing"} = {value = +1, type = kCFNumberSInt32Type}
}
    4 : {contents = "Version"} = {value = +2, type = kCFNumberSInt32Type}
    5 : {contents = "CVImageBufferColorPrimaries"} = SMPTE_C
    8 : {contents = "CVImageBufferTransferFunction"} = {contents = "TransferFunction#5"}
    10 : {contents = "CVImageBufferChromaLocationBottomField"} = Left
    12 : {contents = "CVImageBufferChromaLocationTopField"} = Left
}
}
}
    sbufToTrackReadiness = 0x0
    numSamples = 1
    sampleTimingArray[1] = {
        {PTS = {0/600 = 0.000}, DTS = {INVALID}, duration = {INVALID}},
    }
    imageBuffer = 0x1c4139280

• 以上可以看出未編碼\解碼後的CMSampleBuffer都包含了imageBuffer資料、formatDescription資訊和sampleTimingArray資訊。錄影多了exif和tiff資訊。

• CVPixelBuffer：編碼前和解碼後的影象資料結構.

• 無論是讀取還是錄製
  output的settings的kCVPixelBufferPixelFormatTypeKey欄位 控制的mediaSubType輸出的色彩空間，轉換色彩空間過程有GPU參與，效能較高。可以按需設定。

CMSampleBuffer的建立及示例

- (void)appendVideoPixelBuffer:(CVPixelBufferRef)pixelBuffer withPresentationTime:(CMTime)presentationTime
{
    CMSampleBufferRef sampleBuffer = NULL;
    CMFormatDescriptionRef outputFormatDescription = NULL;
    CMVideoFormatDescriptionCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, &outputFormatDescription );


    CMSampleTimingInfo timingInfo = {0,};
    timingInfo.duration = kCMTimeInvalid;
    timingInfo.decodeTimeStamp = kCMTimeInvalid;
    timingInfo.presentationTimeStamp = presentationTime;

    OSStatus err = CMSampleBufferCreateForImageBuffer( kCFAllocatorDefault, pixelBuffer, true, NULL, NULL, outputFormatDescription, &timingInfo, &sampleBuffer );
    if ( sampleBuffer ) {
        // do some thing
        CFRelease( sampleBuffer );
    }
    else {
        NSString *exceptionReason = [NSString stringWithFormat:@"sample buffer create failed (%i)", (int)err];
        @throw [NSException exceptionWithName:NSInvalidArgumentException reason:exceptionReason userInfo:nil];
    }
}

• CMSampleBufferCreateReady與CMSampleBufferCreate相同，只是dataReady始終為true，因此不需要傳遞makeDataReadyCallback或refcon。