Redis 物件系統

TuxedoLinux發表於2018-05-26
Redis原始碼剖析和註釋(八)--- 物件系統(redisObject)

Redis 物件系統

1. 介紹

redis中基於雙端連結串列、簡單動態字串(sds)、字典、跳躍表、整數集合、壓縮列表、快速列表等等資料結構實現了一個物件系統,並且實現了5種不同的物件,每種物件都使用了至少一種前面的資料結構,優化物件在不同場合下的使用效率。

2. 物件的系統的實現

redis 3.2版本。所有註釋在github中:物件系統的註釋

2.1 物件的結構

物件結構robj功能:

  • 為5種不同的物件型別提供同一的表示形式。
  • 為不同的物件適用於不同的場景,支援同一種物件型別採用多種的資料結構方式。
  • 支援引用計數,實現物件共享機制。
  • 記錄物件的訪問時間,便於刪除物件。

物件結構定義在redis 3.2版本的server.h

#define LRU_BITS 24
#define LRU_CLOCK_MAX ((1<<LRU_BITS)-1) /* Max value of obj->lru */
#define LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */

typedef struct redisObject {
    //物件的資料型別,佔4bits,共5種型別
    unsigned type:4;        
    //物件的編碼型別,佔4bits,共10種型別
    unsigned encoding:4;

    //least recently used
    //實用LRU演算法計算相對server.lruclock的LRU時間
    unsigned lru:LRU_BITS; /* lru time (relative to server.lruclock) */

    //引用計數
    int refcount;

    //指向底層資料實現的指標
    void *ptr;
} robj;

//type的佔5種型別:
/* Object types */
#define OBJ_STRING 0    //字串物件
#define OBJ_LIST 1      //列表物件
#define OBJ_SET 2       //集合物件
#define OBJ_ZSET 3      //有序集合物件
#define OBJ_HASH 4      //雜湊物件

/* Objects encoding. Some kind of objects like Strings and Hashes can be
 * internally represented in multiple ways. The 'encoding' field of the object
 * is set to one of this fields for this object. */
// encoding 的10種型別
#define OBJ_ENCODING_RAW 0     /* Raw representation */     //原始表示方式,字串物件是簡單動態字串
#define OBJ_ENCODING_INT 1     /* Encoded as integer */         //long型別的整數
#define OBJ_ENCODING_HT 2      /* Encoded as hash table */      //字典
#define OBJ_ENCODING_ZIPMAP 3  /* Encoded as zipmap */          //不在使用
#define OBJ_ENCODING_LINKEDLIST 4 /* Encoded as regular linked list */  //雙端連結串列,不在使用
#define OBJ_ENCODING_ZIPLIST 5 /* Encoded as ziplist */         //壓縮列表
#define OBJ_ENCODING_INTSET 6  /* Encoded as intset */          //整數集合
#define OBJ_ENCODING_SKIPLIST 7  /* Encoded as skiplist */      //跳躍表和字典
#define OBJ_ENCODING_EMBSTR 8  /* Embedded sds string encoding */   //embstr編碼的簡單動態字串
#define OBJ_ENCODING_QUICKLIST 9 /* Encoded as linked list of ziplists */   //由壓縮列表組成的雙向列表-->快速列表
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43

2.2 字串物件的底層實現型別

編碼—encoding 物件—ptr
OBJ_ENCODING_RAW 簡單動態字串實現的字串物件
OBJ_ENCODING_INT 整數值實現的字串物件
OBJ_ENCODING_EMBSTR embstr編碼的簡單動態字串實現的字串物件

2.3 列表物件的底層實現型別

編碼—encoding 物件—ptr
OBJ_ENCODING_QUICKLIST 快速列表實現的列表物件
OBJ_ENCODING_ZIPLIST 壓縮列表實現的列表物件

2.4 集合物件的底層實現型別

編碼—encoding 物件—ptr
OBJ_ENCODING_HT 字典實現的集合物件
OBJ_ENCODING_INTSET 整數集合實現的集合物件

2.5 雜湊物件的底層實現型別

編碼—encoding 物件—ptr
OBJ_ENCODING_ZIPLIST 壓縮列表實現的雜湊物件
OBJ_ENCODING_HT 字典實現的雜湊物件

2.6 有序集合物件的底層實現型別

編碼—encoding 物件—ptr
OBJ_ENCODING_SKIPLIST 跳躍表和字典實現的有序集合物件
OBJ_ENCODING_ZIPLIST 壓縮列表實現的有序集合物件

3. 物件系統的重要操作

3.1建立一個字串物件

  • 編碼為OBJ_ENCODING_RAW
robj *createObject(int type, void *ptr) {   //建立一個物件
    robj *o = zmalloc(sizeof(*o));          //分配空間
    o->type = type;                         //設定物件型別
    o->encoding = OBJ_ENCODING_RAW;         //設定編碼方式為OBJ_ENCODING_RAW
    o->ptr = ptr;                           //設定
    o->refcount = 1;                        //引用計數為1

    /* Set the LRU to the current lruclock (minutes resolution). */
    o->lru = LRU_CLOCK();                   //計算設定當前LRU時間
    return o;
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 編碼為OBJ_ENCODING_EMBSTR
/* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is
 * an object where the sds string is actually an unmodifiable string
 * allocated in the same chunk as the object itself. */
//建立一個embstr編碼的字串物件
robj *createEmbeddedStringObject(const char *ptr, size_t len) {
    robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr8)+len+1);   //分配空間
    struct sdshdr8 *sh = (void*)(o+1);  //o+1剛好就是struct sdshdr8的地址

    o->type = OBJ_STRING;               //型別為字串物件
    o->encoding = OBJ_ENCODING_EMBSTR;  //設定編碼型別OBJ_ENCODING_EMBSTR
    o->ptr = sh+1;                      //指向分配的sds物件,分配的len+1的空間首地址
    o->refcount = 1;                    //設定引用計數
    o->lru = LRU_CLOCK();               //計算設定當前LRU時間

    sh->len = len;                      //設定字串長度
    sh->alloc = len;                    //設定最大容量
    sh->flags = SDS_TYPE_8;             //設定sds的型別
    if (ptr) {                          //如果傳了字串引數
        memcpy(sh->buf,ptr,len);        //將傳進來的ptr儲存到物件中
        sh->buf[len] = '\0';            //結束符標誌
    } else {
        memset(sh->buf,0,len+1);        //否則將物件的空間初始化為0
    }
    return o;
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 兩種字串物件編碼方式的區別
/* Create a string object with EMBSTR encoding if it is smaller than
 * REIDS_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is
 * used.
 *
 * The current limit of 39 is chosen so that the biggest string object
 * we allocate as EMBSTR will still fit into the 64 byte arena of jemalloc. */

//sdshdr8的大小為3個位元組,加上1個結束符共4個位元組
//redisObject的大小為16個位元組
//redis使用jemalloc記憶體分配器,且jemalloc會分配8,16,32,64等位元組的記憶體
//一個embstr固定的大小為16+3+1 = 20個位元組,因此一個最大的embstr字串為64-20 = 44位元組
#define OBJ_ENCODING_EMBSTR_SIZE_LIMIT 44

// 建立字串物件,根據長度使用不同的編碼型別
// createRawStringObject和createEmbeddedStringObject的區別是:
// createRawStringObject是當字串長度大於44位元組時,robj結構和sdshdr結構在記憶體上是分開的
// createEmbeddedStringObject是當字串長度小於等於44位元組時,robj結構和sdshdr結構在記憶體上是連續的
robj *createStringObject(const char *ptr, size_t len) {
    if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT)
        return createEmbeddedStringObject(ptr,len);
    else
        return createRawStringObject(ptr,len);
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23

3.2 字串物件編碼的優化

/* Try to encode a string object in order to save space */
//嘗試優化字串物件的編碼方式以節約空間
robj *tryObjectEncoding(robj *o) {
    long value;
    sds s = o->ptr;
    size_t len;

    /* Make sure this is a string object, the only type we encode
     * in this function. Other types use encoded memory efficient
     * representations but are handled by the commands implementing
     * the type. */
    serverAssertWithInfo(NULL,o,o->type == OBJ_STRING);

    /* We try some specialized encoding only for objects that are
     * RAW or EMBSTR encoded, in other words objects that are still
     * in represented by an actually array of chars. */
    //如果字串物件的編碼型別為RAW或EMBSTR時,才對其重新編碼
    if (!sdsEncodedObject(o)) return o;

    /* It's not safe to encode shared objects: shared objects can be shared
     * everywhere in the "object space" of Redis and may end in places where
     * they are not handled. We handle them only as values in the keyspace. */
    //如果refcount大於1,則說明物件的ptr指向的值是共享的,不對共享物件進行編碼
     if (o->refcount > 1) return o;

    /* Check if we can represent this string as a long integer.
     * Note that we are sure that a string larger than 20 chars is not
     * representable as a 32 nor 64 bit integer. */
    len = sdslen(s);            //獲得字串s的長度

    //如果len小於等於20,表示符合long long可以表示的範圍,且可以轉換為long型別的字串進行編碼
    if (len <= 20 && string2l(s,len,&value)) {
        /* This object is encodable as a long. Try to use a shared object.
         * Note that we avoid using shared integers when maxmemory is used
         * because every object needs to have a private LRU field for the LRU
         * algorithm to work well. */
        if ((server.maxmemory == 0 ||
             (server.maxmemory_policy != MAXMEMORY_VOLATILE_LRU &&
              server.maxmemory_policy != MAXMEMORY_ALLKEYS_LRU)) &&
            value >= 0 &&
            value < OBJ_SHARED_INTEGERS)    //如果value處於共享整數的範圍內
        {
            decrRefCount(o);                //原物件的引用計數減1,釋放物件
            incrRefCount(shared.integers[value]); //增加共享物件的引用計數
            return shared.integers[value];      //返回一個編碼為整數的字串物件
        } else {        //如果不處於共享整數的範圍
            if (o->encoding == OBJ_ENCODING_RAW) sdsfree(o->ptr);   //釋放編碼為OBJ_ENCODING_RAW的物件
            o->encoding = OBJ_ENCODING_INT;     //轉換為OBJ_ENCODING_INT編碼
            o->ptr = (void*) value;             //指標ptr指向value物件
            return o;
        }
    }

    /* If the string is small and is still RAW encoded,
     * try the EMBSTR encoding which is more efficient.
     * In this representation the object and the SDS string are allocated
     * in the same chunk of memory to save space and cache misses. */
    //如果len小於44,44是最大的編碼為EMBSTR型別的字串物件長度
    if (len <= OBJ_ENCODING_EMBSTR_SIZE_LIMIT) {
        robj *emb;

        if (o->encoding == OBJ_ENCODING_EMBSTR) return o;   //將RAW物件轉換為OBJ_ENCODING_EMBSTR編碼型別
        emb = createEmbeddedStringObject(s,sdslen(s)); //建立一個編碼型別為OBJ_ENCODING_EMBSTR的字串物件
        decrRefCount(o);    //釋放之前的物件
        return emb;
    }

    /* We can't encode the object...
     *
     * Do the last try, and at least optimize the SDS string inside
     * the string object to require little space, in case there
     * is more than 10% of free space at the end of the SDS string.
     *
     * We do that only for relatively large strings as this branch
     * is only entered if the length of the string is greater than
     * OBJ_ENCODING_EMBSTR_SIZE_LIMIT. */
    //無法進行編碼,但是如果s的未使用的空間大於使用空間的10分之1
    if (o->encoding == OBJ_ENCODING_RAW &&
        sdsavail(s) > len/10)
    {
        o->ptr = sdsRemoveFreeSpace(o->ptr);    //釋放所有的未使用空間
    }

    /* Return the original object. */
    return o;
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86

3.3 引用計數管理物件

//引用計數加1
void incrRefCount(robj *o) {
    o->refcount++;
}

//引用計數減1
void decrRefCount(robj *o) {
    if (o->refcount <= 0) serverPanic("decrRefCount against refcount <= 0");

    //當引用物件等於1時,在操作引用計數減1,直接釋放物件的ptr和物件空間
    if (o->refcount == 1) {
        switch(o->type) {
        case OBJ_STRING: freeStringObject(o); break;
        case OBJ_LIST: freeListObject(o); break;
        case OBJ_SET: freeSetObject(o); break;
        case OBJ_ZSET: freeZsetObject(o); break;
        case OBJ_HASH: freeHashObject(o); break;
        default: serverPanic("Unknown object type"); break;
        }
        zfree(o);
    } else {
        o->refcount--;  //否則減1
    }
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24

3.4 物件的複製,建立的物件非共享

//返回 複製的o物件的副本的地址,且建立的物件非共享
robj *dupStringObject(robj *o) {
    robj *d;

    serverAssert(o->type == OBJ_STRING);    //一定是OBJ_STRING型別

    switch(o->encoding) {                   //根據不同的編碼型別
    case OBJ_ENCODING_RAW:
        return createRawStringObject(o->ptr,sdslen(o->ptr));        //建立的物件非共享
    case OBJ_ENCODING_EMBSTR:
        return createEmbeddedStringObject(o->ptr,sdslen(o->ptr));   //建立的物件非共享
    case OBJ_ENCODING_INT:                  //整數編碼型別
        d = createObject(OBJ_STRING, NULL); //即使是共享整數範圍內的整數,建立的物件也是非共享的
        d->encoding = OBJ_ENCODING_INT;
        d->ptr = o->ptr;
        return d;
    default:
        serverPanic("Wrong encoding.");
        break;
    }
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21

3.5 物件的解碼操作

將儲存的整數值解碼成字串物件返回回來。

/* Get a decoded version of an encoded object (returned as a new object).
 * If the object is already raw-encoded just increment the ref count. */
//將物件是整型的解碼為字串並返回,如果是字串編碼則直接返回輸入物件,只需增加引用計數
robj *getDecodedObject(robj *o) {
    robj *dec;

    if (sdsEncodedObject(o)) {  //如果是OBJ_ENCODING_RAW或OBJ_ENCODING_EMBSTR型別的物件
        incrRefCount(o);        //增加引用計數,返回一個共享的物件
        return o;
    }
    if (o->type == OBJ_STRING && o->encoding == OBJ_ENCODING_INT) { //如果是整數物件
        char buf[32];

        ll2string(buf,32,(long)o->ptr); //將整數轉換為字串
        dec = createStringObject(buf,strlen(buf));  //建立一個字串物件
        return dec;
    } else {
        serverPanic("Unknown encoding type");
    }
}
  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20

3.6 其他操作

所有註釋在github中:物件系統的註釋

相關文章