Android平臺aac谷歌軟解框架和流程、解碼庫學習

yangxi_001發表於2015-08-24
Android谷歌框架

前言：

    在Android系統4.1上面目前aac谷歌軟解已經給了兩套方案，一套是沿用以前的解碼庫，一套是使用Fraunhofer Institute開發出來的aac解碼庫，目前谷歌已經切換到了後者，Fraunhofer提供了一套標準的呼叫介面，谷歌實現的aac軟解component邏輯類SoftAAC2.cpp只要按照這個標準去呼叫，就能完成aac的解碼。

正文

一：SoftAAC2解碼流程

    前面已經有部落格分析了android中是如何載入解碼元件的，這裡不在重點看android framework中的omxcodec類是如何控制解碼元件完成各種解碼中的工作，這裡預設已經載入好了omx.google.aac.decode這個元件，看看SoftAAC2.cpp中是怎樣完成所有邏輯的，跟著打出來的log進行跟蹤

01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 constructor....
首先會進入SoftAAC2的建構函式
01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initPorts....
為這個解碼器初始化兩個埠（輸入和輸出，portIndex分別為0和1），看下port對應的結構體：
    struct PortInfo {
        OMX_PARAM_PORTDEFINITIONTYPE mDef;
        Vector mBuffers;
        List mQueue;
        enum {
            NONE,
            DISABLING,
            ENABLING,
        } mTransition;
    };
port對應的結構體中包含了三部分的內容
1：關於port所有資訊的結構體OMX_PARAM_PORTDEFINITIONTYPE，定義在omx_core.h標準中，需要好好看下；
2：每個port都會對應分配幾塊buffer（對應的結構體為bufferInfo，後面詳解），用於放置解碼前後的資料
3：port也維護了一個狀態
初始化過程就是初始化OMX_PARAM_PORTDEFINITIONTYPEdef變數，最後呼叫addPort加入到一個容器中去管理
void SimpleSoftOMXComponent::addPort(const OMX_PARAM_PORTDEFINITIONTYPE &def) {
    CHECK_EQ(def.nPortIndex, mPorts.size());
    mPorts.push();
    PortInfo *info = &mPorts.editItemAt(mPorts.size() - 1);
    info->mDef = def;
    info->mTransition = PortInfo::NONE;
}  
01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initDecoder before call aacDecoder_open...
01-07 01:39:14.250 E/SoftAAC2(  151):  in SoftAAC2 initDecoder before call aacDecoder_GetStreamInfo...
其次是在initDecoder函式初始化解碼器，呼叫aacDecoder_Open介面獲得解碼庫的控制程式碼mAACDecoder，以及呼叫aacDecoder_GetStreamInfo獲取aac的一些資訊，儲存在結構體CStreamInfo中。
    mAACDecoder = aacDecoder_Open(TT_MP4_ADIF, 1);
    mStreamInfo = aacDecoder_GetStreamInfo(mAACDecoder);

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocating 4 buffers of size 8192 on input port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x40069f48 on input port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x401fd3b8 on input port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x400a7008 on input port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x400a70c0 on input port
為輸入埠分配四個buffer（個數是在初始化port過程中kNumInputBuffers指定的），這四個buffer的作用就是讓omx client往裡面放原始的aac資料，並且呼叫emptyBuffer，告訴解碼器，將這些buffer拿去解碼，這些buffer都是通過共享記憶體實現的，也就是omx client和omx component共享這段記憶體，client往裡面寫，component讀取裡面的資料，看下buffer對應的資料結構：
    struct BufferInfo {
        OMX_BUFFERHEADERTYPE *mHeader;
        bool mOwnedByUs;
    };
包含了兩部分內容：
1：buffer中包含的資料的結構資訊OMX_BUFFERHEADERTYPE，定義在openmax標頭檔案中；
2：buffer當前被誰佔有，client(mOwnedByUs為false)或者component(mOwnedByUs為true)。

01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocating 4 buffers of size 16384 on output port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x401fe6a8 on output port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x404d09c8 on output port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x408bedd8 on output port
01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent useBufferWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent useBuffer....
01-07 01:39:14.270 V/OMXCodec(  151): [OMX.google.aac.decoder] allocated buffer 0x408bef00 on output port
同理，為輸出埠分配四塊buffer，用於client和component共享，這四個buffer用於存放解碼器解碼後的pcm資料，解碼器往裡面寫，omx client讀取裡面的資料，其他和input port對應的buffer一致。

上面八個buffer都是通過android中Shared Memory機制實現，記憶體是在OMXCodec中分配的
    size_t totalSize = def.nBufferCountActual * def.nBufferSize;
    mDealer[portIndex] = new MemoryDealer(totalSize, "OMXCodec");
    for (OMX_U32 i = 0; i < def.nBufferCountActual; ++i) {
        sp mem = mDealer[portIndex]->allocate(def.nBufferSize);
        CHECK(mem.get() != NULL);
        ......
    }

01-07 01:39:14.270 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent GetParameterWrapper..
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent getParameter ....
01-07 01:39:14.270 E/SoftAAC2(  151): huangbangbang in SoftAAC2 internalGetParameter index is 33554433
01-07 01:39:14.270 E/SoftAAC2(  151): huangbangbang in SoftAAC2 internalGetParameter in default...
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent internalGetParameter....
01-07 01:39:14.270 E/SimpleSoftOMXComponent(  151): huangbangbang in internalGetParameter in case OMX_IndexParamPortDefinition....

01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent msgType = 0 (kWhatSendCommand: 0, kWhatEmptyFillThisBuffer: 1, kWhatFillThisBuffer:2)
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onSendcommand.
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onSendcommand in onChangeState.
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onChangeStage state=2, component_cur_state=1
01-07 01:39:14.280 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent notify..
01-07 01:39:14.280 V/OMXCodec(  151): [OMX.google.aac.decoder] onStateChange 2
01-07 01:39:14.280 V/OMXCodec(  151): [OMX.google.aac.decoder] Now Idle.
01-07 01:39:14.280 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent SendCommandWrapper..
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent sendCommand....
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent msgType = 0 (kWhatSendCommand: 0, kWhatEmptyFillThisBuffer: 1, kWhatFillThisBuffer:2)
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onSendcommand.
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onSendcommand in onChangeState.
01-07 01:39:14.280 E/SimpleSoftOMXComponent(  151): huangbangbang in SimpleSoftOMXComponent onChangeStage state=3, component_cur_state=2
01-07 01:39:14.280 E/SoftOMXComponent(  151): huangbangbang in SoftOMXcomponent notify..
01-07 01:39:14.280 V/OMXCodec(  151): [OMX.google.aac.decoder] onStateChange 3
01-07 01:39:14.280 V/OMXCodec(  151): [OMX.google.aac.decoder] Now Executing.
上面的log表示，client呼叫getParameter和setParameter獲取或者設定解碼器的一些引數，changeState改變解碼器當前狀態等函式，解碼器本身也維護了一個狀態機，每個狀態下面需要或者不能做哪些事情是有規定的，可以在openMAX文件中檢視，如下圖，
Android平臺aac谷歌軟解框架和流程、解碼庫學習
    openMAX中定義的component的狀態機
從log中可以看出，我們的aac解碼器狀態機的變化是：
UNLOADED------->LOADED------->IDLE--------->EXECUTING
到達EXECUTING狀態之後，就能夠開始讀寫資料編解碼了

谷歌實現的軟解碼的邏輯類採用了多型機制，因為谷歌繼承的不只是aac decoder一個component，還有很多其他型別檔案的解碼器，所以為了達到程式碼的最大共用，谷歌實現先了兩個父類，對於和具體解碼器無關的一些操作都放在父類中完成，繼承關係如下：
Android平臺aac谷歌軟解框架和流程、解碼庫學習

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled.....
01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in portIndex is 0 mInputBufferCount == 0.....
01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in portIndex is 0 before call aacDecoder_ConfigRaw.....
解碼器，輸入輸出埠，每個埠需要的buffer都分配好了，下面就是真正呼叫解碼庫進行資料的解碼了，真正音訊資料前面都會有一些configure資料，所以來到的第一個input buffer我們沒有去呼叫aacDecoder_DecodeFrame函式直接去解碼，而是呼叫aacDecoder_ConfigRaw 去重新獲取aac的一些資訊，如sampleRate和numChannels，儲存在CStreamInfo結構體中， 如果sampleRate和numChannels引數有變化，可能還需要disable output port，並且重新初始化output port和對應的四個buffer，對於CStreamInfo結構體將在第二部分講解解碼庫的時候進行

01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled.....
01-07 01:39:14.460 E/SoftAAC2(  151):  in SoftAAC2 onQueueFilled in while looper.....
01-07 01:39:14.460 E/SoftAAC2(  151):  in onQueueFilled in while before call aacDecoder_Fill and aacDecoder_DecodeFrame bytesvalid is 416
所有準備工作都做好了之後就開始真正aac資料的解碼了，解碼都是在onQueueFilled函式中進行，剔除一些引數的初始化，主要過程就是下面的while迴圈：
        AAC_DECODER_ERROR decoderErr = AAC_DEC_NOT_ENOUGH_BITS;
        while (bytesValid[0] > 0 && decoderErr == AAC_DEC_NOT_ENOUGH_BITS) {
            aacDecoder_Fill(mAACDecoder,
                            inBuffer,
                            inBufferLength,
                            bytesValid);
            decoderErr = aacDecoder_DecodeFrame(mAACDecoder,
                                                outBuffer,
                                                outHeader->nAllocLen,
                                                0 );
            if (decoderErr == AAC_DEC_NOT_ENOUGH_BITS) {
                ALOGW("Not enough bits, bytesValid %d", bytesValid[0]);
            }
        }
要明白while迴圈裡面aacDecoder_Fill和aacDecoder_DecodeFrame的作用，需要明白解碼庫的buffer機制：
解碼過程中除了前面已經分配的input buffer和output buffer，解碼庫還會分配一個過渡性的decoder-internal input buffer，這個buffer大小又RANSPORTDEC_INBUF_SIZE規定，可以任意設定但必須滿足兩個條件：
1：each input channel requires 768 bytes
2：the whole buffer must be of size 2^n
So for example a stereo decoder:
          TRANSPORTDEC_INBUF_SIZE = 2 768 = 1536 => 2048（選擇2048bytes）

aacDecoder_Fill就是從input buffer往ecoder-internal input buffer裡面拷貝資料，返回的是input buffer中還剩下多少沒有被拷貝的資料，如果input buffer中的資料全部拷貝完畢了，就需要re-fill，下圖是input buffer的變化圖（from aacDecoder.pdf）
Android平臺aac谷歌軟解框架和流程、解碼庫學習
aacDecoder_DecodeFrame用來解碼internal buffer中的資料，如果資料不足以解碼，則返回AAC_DEC_NOT_ENOUGH_BITS，繼續呼叫aacDecoder_Fill填充資料。

這樣解碼一直進行下去，知道OMXCodec中讀取aac原始資料過程中發現已經到該track的結尾，就會給這個mediabuffer打上一個標籤，在onQueueFilled函式中檢測的這個flag，解碼完這一幀之後，就停止解碼，並且做所有的release操作
OMXCodec::drainInputBuffer
    if (signalEOS) {
        flags |= OMX_BUFFERFLAG_EOS;
    } else {
        mNoMoreOutputData = false;
    }
SoftAAC2::onQueueFilled
        if (inHeader->nFlags & OMX_BUFFERFLAG_EOS) {
            ......
        }
最後就是free所有的buffer，disable埠等

二：解碼庫學習   Advanced Audio Coding DecoderLibrary

1，術語：
MPEG-2 and MPEG-4
AAC Low-Complexity (AAC-LC),
High-Eficiency AAC v2 (HE-AAC v2),
AAC Low-Delay (AAC-LD), andAAC Enhanced Low-Delay (AAC-ELD) decoder

2，主要標頭檔案：#include
Android平臺aac谷歌軟解框架和流程、解碼庫學習

3，常用的資料和資料結構：
struct CStreamInfo
    This structure gives information about the currently decoded audio data. All fields are read-only, public attributes are as follows:
• INT sampleRate
• INT frameSize
• INT numChannels
• AUDIO_CHANNEL_TYPE pChannelType
• UCHAR pChannelIndices
• INT aacSampleRate
• INT profile
• AUDIO_OBJECT_TYPE aot
• INT channelConfig
• INT bitRate
• INT aacSamplesPerFrame
• AUDIO_OBJECT_TYPE extAot
• INT extSamplingRate
• UINT flags
• SCHAR epConfig
• INT numLostAccessUnits
• UINT numTotalBytes
• UINT numBadBytes
• UINT numTotalAccessUnits
• UINT numBadAccessUnits

Defines:
• #define IS_INIT_ERROR(err) ( (((err)>=aac_dec_init_error_start) && ((err)<=aac_dec_init_-
error_end)) ? 1 : 0)
• #define IS_DECODE_ERROR(err) ( (((err)>=aac_dec_decode_error_start) && ((err)<=aac_dec_-
decode_error_end)) ? 1 : 0)
• #define IS_OUTPUT_VALID(err) ( ((err) == AAC_DEC_OK) jj IS_DECODE_ERROR(err) )
• #define AACDEC_CONCEAL 1
• #define AACDEC_FLUSH 2
• #define AACDEC_INTR 4
• #define AACDEC_CLRHIST 8

Typedefs:
    typedef struct AAC_DECODER_INSTANCE HANDLE_AACDECODER

Enumerations:
    enum AAC_DECODER_ERROR
    enum AACDEC_PARAM

Functions:
• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_AncDataInit (HANDLE_AACDECODER self, UCHAR buffer,                                                      int size)
    Initialize ancillary data buffer.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_AncDataGet (HANDLE_AACDECODER self, int index,                                                    UCHAR ptr, int size)
    Get one ancillary data element.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_SetParam (const HANDLE_-AACDECODER self, const                                    AACDEC_PARAM param, const INT value)
    Set one single decoder parameter.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_GetFreeBytes (const HANDLE_-AACDECODER self, UINT                                              pFreeBytes)
    Get free bytes inside decoder internal buffer.

• LINKSPEC_H HANDLE_AACDECODER aacDecoder_Open (TRANSPORT_TYPE trans-portFmt, UINT                                                       nrOfLayers)
    Open an AAC decoder instance.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_ConfigRaw (HANDLE_AACDECODER self, UCHAR conf[ ],                                        const UINT length[ ])
    Explicitly configure the decoder by passing a raw AudioSpecificConfig (ASC) or a StreamMuxConfig (SMC), contained in a binary buffer. This is required for MPEG-4 and Raw Packets file format bitstreams as well as for LATM bitstreams with no in-band SMC. If the transport format is LATM with or without LOAS,
    configuration is assumed to be an SMC, for all other file formats an ASC.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_Fill (HANDLE_AACDECODER self,
UCHAR pBuffer[ ], const UINT bufferSize[ ], UINT bytesValid)
    Fill AAC decoder’s internal input buffer with bitstream data from the external input buffer. The function only copies such data as long as the decoder-internal input buffer is not full. So it grabs whatever it can from pBuffer and returns information (bytesValid) so that at a subsequent call of aacDecoder_Fill(), the right position in pBuffer can be determined to grab the next data.

• LINKSPEC_H AAC_DECODER_ERROR aacDecoder_DecodeFrame (HANDLE_-AACDECODER self, INT_PCM                        pTimeData, const INT timeDataSize, const UINT flags)
      decode one audio frame.

• LINKSPEC_H void aacDecoder_Close (HANDLE_AACDECODER self)
                  De-allocate all resources of an AAC decoder instance.

• LINKSPEC_H CStreamInfo aacDecoder_GetStreamInfo (HANDLE_AACDECODER self)
            Get CStreamInfo handle from decoder.

• LINKSPEC_H INT aacDecoder_GetLibInfo (LIB_INFO info)
            Get decoder library info

4：function call sequence

  4.1. Call aacDecoder_Open() to open and retrieve a handle to a new AAC decoder instance.
    aacDecoderInfo = aacDecoder_Open(mpegFileRead_GetTransportType(hDataSrc), nrOfLayers);

  4.2 If out-of-band config data (Audio Specific Config (ASC) or Stream Mux Config (SMC)) is available, call aacDecoder_ConfigRaw() to pass it to the decoder and before the decoding process starts. If this data is not available in advance, the decoder will get it from the bitstream and configure itself while decoding with aacDecoder_DecodeFrame().

  4.3. Begin decoding loop.  do {

  4.4 Read data from bitstream file or stream into a client-supplied input buffer("inBuffer" in main.cpp). If it is very small like just 4, aacDecoder_DecodeFrame() will repeatedly return AAC_DEC_NOT_-ENOUGH_BITS until enough bits were fed by aacDecoder_Fill(). Only read data when this buffer has completely been processed and is then empty. For file-based input execute mpegFileRead_Read() or any other implementation with similar functionality.

  4.5 Call aacDecoder_Fill() to fill the decoder’s internal bitstream input buffer with the client-suppliedexternal bitstream input buffer.
      aacDecoder_Fill(aacDecoderInfo, inBuffer, bytesRead, bytesValid);

  4.6  Call aacDecoder_DecodeFrame() which writes decoded PCM audio data to a client-supplied buffer.It is the client’s responsibility to allocate a buffer which is large enough to hold this output data.
      ErrorStatus = aacDecoder_DecodeFrame(aacDecoderInfo, TimeData, OUT_BUF_SIZE, flags);
   If the bitstream’s configuration (number of channels, sample rate, frame size) is not known in ad-vance, you may call aacDecoder_GetStreamInfo() to retrieve a structure containing this information and then initialize an audio output device. In the example main.cpp, if the number of channels or the sample rate has changed since program start or since the previously decoded frame, the audio output device will be re-initialized. If WAVE file output is chosen, a new WAVE file for each new configuration will be created.

  4.7 Repeat steps 5 to 7 until no data to decode is available anymore, or if an error occured.
   
  4.8 Call aacDecoder_Close() to de-allocate all AAC decoder and transport layer structures.
Android平臺aac谷歌軟解框架和流程、解碼庫學習

5：buffer system
    There are three main buffers in an AAC decoder application. One external input buffer to hold bitstream data from file I/O or elsewhere, one decoder-internal input buffer, and one to hold the decoded output PCM sample data, whereas this output buffer may overlap with the external input buffer.
    The external input buffer is set in the example framework main.cpp and its size is defined by IN_BUF_-SIZE. You may freely choose different sizes here. To feed the data to the decoder-internal input buffer, use the function aacDecoder_Fill(). This function returns important information about how many bytes in the external input buffer have not yet been copied into the internal input buffer (variable bytesValid). Once the external buffer has been fully copied, it can be re-filled again. In case you want to re-fill it when there are
still unprocessed bytes (bytesValid is unequal 0), you would have to additionally perform a memcpy(), so that just means unnecessary computational overhead and therefore we recommend to re-fill the buffer only when bytesValid is 0.

over，go and have lunch

相關文章