Thrift原理分析（二）協議和編解碼

hiekay發表於2018-11-07

協議和編解碼是一個網路應用程式的核心問題之一，客戶端和伺服器通過約定的協議來傳輸訊息(資料)，通過特定的格式來編解碼位元組流，並轉化成業務訊息，提供給上層框架呼叫。

Thrift的協議比較簡單，它把協議和編解碼整合在了一起。抽象類TProtocol定義了協議和編解碼的頂層介面。個人感覺採用抽象類而不是介面的方式來定義頂層介面並不好，TProtocol關聯了一個TTransport傳輸物件，而不是提供一個類似getTransport()的介面，導致抽象類的擴充套件性比介面差。

TProtocol主要做了兩個事情:

1. 關聯TTransport物件

2.定義一系列讀寫訊息的編解碼介面，包括兩類，一類是複雜資料結構比如readMessageBegin, readMessageEnd, writeMessageBegin, writMessageEnd.還有一類是基本資料結構，比如readI32, writeI32, readString, writeString

public abstract class TProtocol {

  /**
   * Transport
   */
  protected TTransport trans_;

　public abstract void writeMessageBegin(TMessage message) throws TException;

  public abstract void writeMessageEnd() throws TException;

  public abstract void writeStructBegin(TStruct struct) throws TException;

  public abstract void writeStructEnd() throws TException;

  public abstract void writeFieldBegin(TField field) throws TException;

  public abstract void writeFieldEnd() throws TException;

  public abstract void writeFieldStop() throws TException;

  public abstract void writeMapBegin(TMap map) throws TException;

  public abstract void writeMapEnd() throws TException;

  public abstract void writeListBegin(TList list) throws TException;

  public abstract void writeListEnd() throws TException;

  public abstract void writeSetBegin(TSet set) throws TException;

  public abstract void writeSetEnd() throws TException;

  public abstract void writeBool(boolean b) throws TException;

  public abstract void writeByte(byte b) throws TException;

  public abstract void writeI16(short i16) throws TException;

  public abstract void writeI32(int i32) throws TException;

  public abstract void writeI64(long i64) throws TException;

  public abstract void writeDouble(double dub) throws TException;

  public abstract void writeString(String str) throws TException;

  public abstract void writeBinary(ByteBuffer buf) throws TException;

  /**
   * Reading methods.
   */

  public abstract TMessage readMessageBegin() throws TException;

  public abstract void readMessageEnd() throws TException;

  public abstract TStruct readStructBegin() throws TException;

  public abstract void readStructEnd() throws TException;

  public abstract TField readFieldBegin() throws TException;

  public abstract void readFieldEnd() throws TException;

  public abstract TMap readMapBegin() throws TException;

  public abstract void readMapEnd() throws TException;

  public abstract TList readListBegin() throws TException;

  public abstract void readListEnd() throws TException;

  public abstract TSet readSetBegin() throws TException;

  public abstract void readSetEnd() throws TException;

  public abstract boolean readBool() throws TException;

  public abstract byte readByte() throws TException;

  public abstract short readI16() throws TException;

  public abstract int readI32() throws TException;

  public abstract long readI64() throws TException;

  public abstract double readDouble() throws TException;

  public abstract String readString() throws TException;

  public abstract ByteBuffer readBinary() throws TException;

  /**
   * Reset any internal state back to a blank slate. This method only needs to
   * be implemented for stateful protocols.
   */
  public void reset() {}
  
  /**
   * Scheme accessor
   */
  public Class<? extends IScheme> getScheme() {
    return StandardScheme.class;
  }

}

所謂協議就是客戶端和伺服器端約定傳輸什麼資料，如何解析傳輸的資料。對於一個RPC呼叫的協議來說，要傳輸的資料主要有:

呼叫方

方法的名稱，包括類的名稱和方法的名稱
方法的引數，包括型別和引數值

3.一些附加的資料，比如附件，超時事件，自定義的控制資訊等等

返回方

呼叫的返回碼
返回值

3.異常資訊

從TProtocol的定義我們可以看出Thrift的協議約定如下事情:

先writeMessageBegin表示開始傳輸訊息了，寫訊息頭。Message裡面定義了方法名，呼叫的型別，版本號，訊息seqId
接下來是寫方法的引數，實際就是寫訊息體。如果引數是一個類，就writeStructBegin
接下來寫欄位，writeFieldBegin, 這個方法會寫接下來的欄位的資料型別和順序號。這個順序號是Thrfit對要傳輸的欄位的一個編碼，從１開始
如果是一個集合就writeListBegin/writeMapBegin，如果是一個基本資料型別，比如int, 就直接writeI32
每個複雜資料型別寫完都呼叫writeXXXEnd，直到writeMessageEnd結束
讀訊息時根據資料型別讀取相應的長度

每個writeXXX都是採用訊息頭+訊息體的方式。

TBinaryProtocol的實現。

writeMessgeBegin方法寫了訊息頭，包括4位元組的版本號和型別資訊，字串型別的方法名，４位元組的序列號seqId
writeFieldBegin，寫了１個位元組的欄位資料型別，和2個位元組欄位的順序號
writeI32，寫了４個位元組的位元組陣列
writeString,先寫４位元組訊息頭表示字串長度，再寫字串位元組
writeBinary,先寫４位元組訊息頭表示位元組陣列長度，再寫位元組陣列內容

6.readMessageBegin時，先讀４位元組版本和型別資訊，再讀字串，再讀４位元組序列號

7.readFieldBegin，先讀1個位元組的欄位資料型別，再讀2個位元組的欄位順序號

readString時，先讀４位元組字串長度，再讀字串內容。字串統一採用UTF-8編碼

 public void writeMessageBegin(TMessage message) throws TException {
    if (strictWrite_) {
      int version = VERSION_1 | message.type;
      writeI32(version);
      writeString(message.name);
      writeI32(message.seqid);
    } else {
      writeString(message.name);
      writeByte(message.type);
      writeI32(message.seqid);
    }
  }

public void writeFieldBegin(TField field) throws TException {
    writeByte(field.type);
    writeI16(field.id);
  }

private byte[] i32out = new byte[4];
  public void writeI32(int i32) throws TException {
    i32out[0] = (byte)(0xff & (i32 >> 24));
    i32out[1] = (byte)(0xff & (i32 >> 16));
    i32out[2] = (byte)(0xff & (i32 >> 8));
    i32out[3] = (byte)(0xff & (i32));
    trans_.write(i32out, 0, 4);
  }

public void writeString(String str) throws TException {
    try {
      byte[] dat = str.getBytes("UTF-8");
      writeI32(dat.length);
      trans_.write(dat, 0, dat.length);
    } catch (UnsupportedEncodingException uex) {
      throw new TException("JVM DOES NOT SUPPORT UTF-8");
    }
  }

public void writeBinary(ByteBuffer bin) throws TException {
    int length = bin.limit() - bin.position();
    writeI32(length);
    trans_.write(bin.array(), bin.position() + bin.arrayOffset(), length);
  }

public TMessage readMessageBegin() throws TException {
    int size = readI32();
    if (size < 0) {
      int version = size & VERSION_MASK;
      if (version != VERSION_1) {
        throw new TProtocolException(TProtocolException.BAD_VERSION, "Bad version in readMessageBegin");
      }
      return new TMessage(readString(), (byte)(size & 0x000000ff), readI32());
    } else {
      if (strictRead_) {
        throw new TProtocolException(TProtocolException.BAD_VERSION, "Missing version in readMessageBegin, old client?");
      }
      return new TMessage(readStringBody(size), readByte(), readI32());
    }
  }

public TField readFieldBegin() throws TException {
    byte type = readByte();
    short id = type == TType.STOP ? 0 : readI16();
    return new TField("", type, id);
  }

public String readString() throws TException {
    int size = readI32();

    if (trans_.getBytesRemainingInBuffer() >= size) {
      try {
        String s = new String(trans_.getBuffer(), trans_.getBufferPosition(), size, "UTF-8");
        trans_.consumeBuffer(size);
        return s;
      } catch (UnsupportedEncodingException e) {
        throw new TException("JVM DOES NOT SUPPORT UTF-8");
      }
    }

    return readStringBody(size);
  }

TProtocol定義了基本的協議資訊，包括傳輸什麼資料，如何解析傳輸的資料的基本方法。

還存在一個問題，就是伺服器端如何知道客戶端傳送過來的資料是怎麼組合的，比如第一個欄位是字串型別，第二個欄位是int。這個資訊是在IDL生成客戶端時生成的程式碼時提供了。Thrift生成的客戶端程式碼提供了讀寫引數的方法，這兩個方式是一一對應的，包括欄位的序號，型別等等。客戶端使用寫引數的方法，伺服器端使用讀引數的方法。

關於IDL生成的客戶端程式碼會在後面的文章具體描述。

下面簡單看一下自動生成的程式碼

方法的呼叫從writeMessageBegin開始，傳送了訊息頭資訊
寫方法的引數，也就是寫訊息體。方法引數由一個統一的介面TBase描述，提供了read和write的統一介面。自動生成的程式碼提供了read, write方法引數的具體實現
寫完結束

 public void write_args(org.apache.thrift.protocol.TProtocol prot) throws org.apache.thrift.TException {
        prot.writeMessageBegin(new org.apache.thrift.protocol.TMessage("handle", org.apache.thrift.protocol.TMessageType.CALL, 0));
        handle_args args = new handle_args();
        args.setIdentity(identity);
        args.setUid(uid);
        args.setSid(sid);
        args.setType(type);
        args.setMessage(message);
        args.setParams(params);
        args.write(prot);
        prot.writeMessageEnd();
      }

public interface TBase<T extends TBase<?,?>, F extends TFieldIdEnum> extends Comparable<T>,  Serializable {

  public void read(TProtocol iprot) throws TException;

  public void write(TProtocol oprot) throws TException;
｝

public static class handle_args <strong>implements org.apache.thrift.TBase</strong><handle_args, handle_args._Fields>, java.io.Serializable, Cloneable   {
    private static final org.apache.thrift.protocol.TStruct STRUCT_DESC = new org.apache.thrift.protocol.TStruct("handle_args");

    private static final org.apache.thrift.protocol.TField IDENTITY_FIELD_DESC = new org.apache.thrift.protocol.TField("identity", org.apache.thrift.protocol.TType.STRING, (short)1);
    private static final org.apache.thrift.protocol.TField UID_FIELD_DESC = new org.apache.thrift.protocol.TField("uid", org.apache.thrift.protocol.TType.I64, (short)2);
    private static final org.apache.thrift.protocol.TField SID_FIELD_DESC = new org.apache.thrift.protocol.TField("sid", org.apache.thrift.protocol.TType.STRING, (short)3);
    private static final org.apache.thrift.protocol.TField TYPE_FIELD_DESC = new org.apache.thrift.protocol.TField("type", org.apache.thrift.protocol.TType.I32, (short)4);
    private static final org.apache.thrift.protocol.TField MESSAGE_FIELD_DESC = new org.apache.thrift.protocol.TField("message", org.apache.thrift.protocol.TType.STRING, (short)5);
    private static final org.apache.thrift.protocol.TField PARAMS_FIELD_DESC = new org.apache.thrift.protocol.TField("params", org.apache.thrift.protocol.TType.MAP, (short)6);

    private static final Map<Class<? extends IScheme>, SchemeFactory> schemes = new HashMap<Class<? extends IScheme>, SchemeFactory>();
    static {
      schemes.put(StandardScheme.class, new handle_argsStandardSchemeFactory());
      schemes.put(TupleScheme.class, new handle_argsTupleSchemeFactory());
    }

    public String identity; // required
    public long uid; // required
    public String sid; // required
    public int type; // required
    public String message; // required
    public Map<String,String> params; // required

    /** The set of fields this struct contains, along with convenience methods for finding and manipulating them. */
    public enum _Fields implements org.apache.thrift.TFieldIdEnum {
      IDENTITY((short)1, "identity"),
      UID((short)2, "uid"),
      SID((short)3, "sid"),
      TYPE((short)4, "type"),
      MESSAGE((short)5, "message"),
      PARAMS((short)6, "params");
｝

//　　自動生成的寫方法引數的方法，按照欄位順序寫，給客戶端程式碼使用
      public void write(org.apache.thrift.protocol.TProtocol oprot, handle_args struct) throws org.apache.thrift.TException {
        struct.validate();

        oprot.writeStructBegin(STRUCT_DESC);
        if (struct.identity != null) {
          oprot.writeFieldBegin(IDENTITY_FIELD_DESC);
          oprot.writeString(struct.identity);
          oprot.writeFieldEnd();
        }
        oprot.writeFieldBegin(UID_FIELD_DESC);
        oprot.writeI64(struct.uid);
        oprot.writeFieldEnd();
        if (struct.sid != null) {
          oprot.writeFieldBegin(SID_FIELD_DESC);
          oprot.writeString(struct.sid);
          oprot.writeFieldEnd();
        }
        oprot.writeFieldBegin(TYPE_FIELD_DESC);
        oprot.writeI32(struct.type);
        oprot.writeFieldEnd();
        if (struct.message != null) {
          oprot.writeFieldBegin(MESSAGE_FIELD_DESC);
          oprot.writeString(struct.message);
          oprot.writeFieldEnd();
        }
｝

<pre name="code" class="java">//　　自動生成的讀方法引數的方法，按照欄位順序讀，給伺服器端程式碼使用

 public void read(org.apache.thrift.protocol.TProtocol iprot, handle_args struct) throws org.apache.thrift.TException {
        org.apache.thrift.protocol.TField schemeField;
        iprot.readStructBegin();
        while (true)
        {
          schemeField = iprot.readFieldBegin();
          if (schemeField.type == org.apache.thrift.protocol.TType.STOP) { 
            break;
          }
          switch (schemeField.id) {
            case 1: // IDENTITY
              if (schemeField.type == org.apache.thrift.protocol.TType.STRING) {
                struct.identity = iprot.readString();
                struct.setIdentityIsSet(true);
              } else { 
                org.apache.thrift.protocol.TProtocolUtil.skip(iprot, schemeField.type);
              }
              break;
            case 2: // UID
              if (schemeField.type == org.apache.thrift.protocol.TType.I64) {
                struct.uid = iprot.readI64();
                struct.setUidIsSet(true);
              } else { 
                org.apache.thrift.protocol.TProtocolUtil.skip(iprot, schemeField.type);
              }
              break;
            case 3: // SID
              if (schemeField.type == org.apache.thrift.protocol.TType.STRING) {
                struct.sid = iprot.readString();
                struct.setSidIsSet(true);
              } else { 
                org.apache.thrift.protocol.TProtocolUtil.skip(iprot, schemeField.type);
              }
              break;
            case 4: // TYPE
              if (schemeField.type == org.apache.thrift.protocol.TType.I32) {
                struct.type = iprot.readI32();
                struct.setTypeIsSet(true);
              } else { 
                org.apache.thrift.protocol.TProtocolUtil.skip(iprot, schemeField.type);
              }
              break;
｝

FTP - 協議原理和狀態碼
2014-11-25
FTP協議
Thrift協議的服務模型
2017-03-08
協議模型
NSCoding協議的編碼與解碼
2016-02-16
協議
DHCP協議和dhcpcd原始碼分析
2014-06-14
協議原始碼
dubbo原始碼解析（三十二）遠端呼叫——thrift協議
2019-01-19
原始碼協議
淺談WebSocket協議、WS協議和WSS協議原理及關係
2019-07-26
Web協議
Thrift原理分析(一)基本概念
2018-11-07
使用NSSecureCoding協議進行物件編解碼
2014-05-12
協議物件
WebSocket原理與實踐(二)---WebSocket協議
2018-03-08
Web協議
TCP協議、演算法和原理
2019-04-01
TCP協議演算法
IIC 協議原理
2017-08-18
協議
ffmpeg分析系列之二(檔案協議)
2010-11-04
協議
Thrift之Protocol原始碼分析
2016-04-23
Protocol原始碼
基於Netty實現Redis協議的編碼解碼器
2018-03-19
NettyRedis協議
tcp/ip協議和opc協議對比詳解
2023-10-20
TCP協議
透視RPC協議：SOFA-BOLT協議原始碼分析
2021-08-07
RPC協議原始碼
MQTT協議（二）
2019-03-06
MQQT協議
刨根問底HTTP和WebSocket協議(二）
2016-08-22
HTTPWeb協議
URL編碼與解碼原理
2018-08-22
二進位制協議 VS 文字協議
2019-05-08
協議
http協議分析
2019-03-27
HTTP協議
FastCGI協議分析
2016-11-07
AST協議
SSH 協議基本原理及 wireshark 抓包分析
2018-09-27
協議
聯眾升級協議分析續篇（二） (轉)
2007-08-17
協議
WireShark——IP協議包分析(Ping分析IP協議包)
2020-06-20
協議
thrift原始碼分析-架構設計
2018-12-20
原始碼架構
Storm-原始碼分析-Thrift的使用
2017-05-02
ORM原始碼
IS-IS協議原理與配置
2020-10-18
協議
網路通訊協議自動轉換之thrift到http
2019-03-01
協議HTTP
如何在 Istio 中支援 Dubbo、Thrift、Redis 以及任何七層協議？
2021-03-08
Redis協議
以太坊原始碼分析(37)eth以太坊協議分析
2018-05-14
原始碼協議
網路協議之:socket協議詳解之Socket和Stream Socket
2022-03-30
協議
IPIDEA帶你瞭解HTTP協議和SOCKS5協議
2023-03-03
IdeaHTTP協議
IM通訊協議專題學習(三)：由淺入深，從根上理解Protobuf的編解碼原理
2022-11-28
協議
Base64 編碼解碼原理
2017-10-11
NEO從原始碼分析看共識協議
2019-01-23
原始碼協議
gRPC 之流式呼叫原理 http2 協議分析（四）
2019-09-01
RPCHTTP協議
wireshark 分析TCP協議
2020-11-23
TCP協議