字串壓縮（一）之ZSTD

BARBARIANS發表於2022-03-03
字串

前言

　　最近專案上有大量的字串資料需要儲存到記憶體，並且需要儲存至一定時間，於是自然而然的想到了使用字串壓縮演算法對“源串”進行壓縮儲存。由此觸發了對一些優秀壓縮演算法的調研。
　　字串壓縮，我們通常的需求有幾個，一是高壓縮率，二是壓縮速率高，三是解壓速率高。不過高壓縮率與高壓縮速率是魚和熊掌的關係，不可皆得，優秀的演算法一般也是採用壓縮率與效能折中的方案。從壓縮率、壓縮速率、解壓速率考慮，zstd與lz4有較好的壓縮與解壓效能，最終選取zstd與lz4進行調研。

　　zstd是facebook開源的提供高壓縮比的快速壓縮演算法（參考https://github.com/facebook/zstd），很想了解一下它在壓縮與解壓方面的實際表現。

一、zstd壓縮與解壓

　　ZSTD_compress屬於ZSTD的Simple API範疇，只有壓縮級別可以設定。

　　ZSTD_compress函式原型如下：

　　size_t ZSTD_compress(void* dst, size_t dstCapacity, const void* src, size_t srcSize, int compressionLevel)

 

　　ZSTD_decompress函式原型如下：
　　size_t ZSTD_decompress( void* dst, size_t dstCapacity, const void* src, size_t compressedSize);

　　我們先來看看zstd的壓縮與解壓縮示例。

 1 #include <stdio.h>
 2 #include <string.h>
 3 #include <sys/time.h>
 4 #include <malloc.h>
 5 #include <zstd.h>
 6 #include <iostream>
 7 
 8 using namespace std;
 9 
10 int main()
11 {
12     // compress
13     size_t com_space_size;
14     size_t peppa_pig_text_size;
15 
16     char *com_ptr = NULL;
17     char peppa_pig_buf[2048] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining. Can we go out to play?Daddy: Alright, run along you two.Narrator: Peppa loves jumping in muddy puddles.Peppa: I love muddy puddles.Mummy: Peppa. If you jumping in muddy puddles, you must wear your boots.Peppa: Sorry, Mummy.Narrator: George likes to jump in muddy puddles, too.Peppa: George. If you jump in muddy puddles, you must wear your boots.Narrator: Peppa likes to look after her little brother, George.Peppa: George, let's find some more pud dles.Narrator: Peppa and George are having a lot of fun. Peppa has found a lttle puddle. George hasfound a big puddle.Peppa: Look, George. There's a really big puddle.Narrator: George wants to jump into the big puddle first.Peppa: Stop, George. | must check if it's safe for you. Good. It is safe for you. Sorry, George. It'sonly mud.Narrator: Peppa and George love jumping in muddy puddles.Peppa: Come on, George. Let's go and show Daddy.Daddy: Goodness me.Peppa: Daddy. Daddy. Guess what we' ve been doing.Daddy: Let me think... Have you been wa tching television?Peppa: No. No. Daddy.Daddy: Have you just had a bath?Peppa: No. No.Daddy: | know. You've been jumping in muddy puddles.Peppa: Yes. Yes. Daddy. We've been jumping in muddy puddles.Daddy: Ho. Ho. And look at the mess you're in.Peppa: Oooh....Daddy: Oh, well, it's only mud. Let's clean up quickly before Mummy sees the mess.Peppa: Daddy, when we've cleaned up, will you and Mummy Come and play, too?Daddy: Yes, we can all play in the garden.Narrator: Peppa and George are wearing their boots. Mummy and Daddy are wearing their boots.Peppa loves jumping up and down in muddy puddles. Everyone loves jumping up and down inmuddy puddles.Mummy: Oh, Daddy pig, look at the mess you're in. .Peppa: It's only mud.";
18     
19     peppa_pig_text_size = strlen(peppa_pig_buf);
20     com_space_size= ZSTD_compressBound(peppa_pig_text_size);
21     com_ptr = (char *)malloc(com_space_size);
22     if(NULL == com_ptr) {
23         cout << "compress malloc failed" << endl;
24         return -1;
25     }
26 
27     size_t com_size;
28     com_size = ZSTD_compress(com_ptr, com_space_size, peppa_pig_buf, peppa_pig_text_size, ZSTD_fast);
29     cout << "peppa pig text size:" << peppa_pig_text_size << endl;
30     cout << "compress text size:" << com_size << endl;
31     cout << "compress ratio:" << (float)peppa_pig_text_size / (float)com_size << endl << endl;
32 
33 
34     // decompress
35     char* decom_ptr = NULL;
36     unsigned long long decom_buf_size;
37     decom_buf_size = ZSTD_getFrameContentSize(com_ptr, com_size);
38     
39     decom_ptr = (char *)malloc((size_t)decom_buf_size);
40     if(NULL == decom_ptr) {
41         cout << "decompress malloc failed" << endl;
42         return -1;
43     }
44 
45     size_t decom_size;
46     decom_size = ZSTD_decompress(decom_ptr, decom_buf_size, com_ptr, com_size);
47     cout << "decompress text size:" << decom_size << endl;
48 
49     if(strncmp(peppa_pig_buf, decom_ptr, peppa_pig_text_size)) {
50         cout << "decompress text is not equal peppa pig text" << endl;
51     }
52     
53     free(com_ptr);
54     free(decom_ptr);
55     return 0;
56 }

執行結果：

　　從結果可以發現，壓縮之前的peppa pig文字長度為1827，壓縮後的文字長度為759，壓縮率為2.4，解壓後的長度與壓縮前相等。

　　另外，上文提到可以調整ZSTD_compress函式的壓縮級別，zstd的預設級別為ZSTD_CLEVEL_DEFAULT = 3，最小值為0，最大值為ZSTD_MAX_CLEVEL = 22。另外也提供一些策略設定，例如 ZSTD_fast, ZSTD_greedy, ZSTD_lazy, ZSTD_lazy2, ZSTD_btlazy2。壓縮級別越高，壓縮率越高，但是壓縮速率越低。

二、ZSTD壓縮與解壓效能探索

　　上面探索了zstd的基礎壓縮與解壓方法，接下來再摸索一下zstd的壓縮與解壓縮效能。

　　測試方法是，使用ZSTD_compress連續壓縮同一段文字並持續10秒，最後得到每一秒的平均壓縮速率。測試壓縮效能的程式碼示例如下：

 1 #include <stdio.h>
 2 #include <string.h>
 3 #include <sys/time.h>
 4 #include <malloc.h>
 5 #include <zstd.h>
 6 #include <iostream>
 7 
 8 using namespace std;
 9 
10 int main()
11 {
12     int cnt = 0;
13 
14     size_t com_size;
15     size_t com_space_size;
16     size_t peppa_pig_text_size;
17     
18     char *com_ptr = NULL;
19     char peppa_pig_buf[2048] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining. Can we go out to play?Daddy: Alright, run along you two.Narrator: Peppa loves jumping in muddy puddles.Peppa: I love muddy puddles.Mummy: Peppa. If you jumping in muddy puddles, you must wear your boots.Peppa: Sorry, Mummy.Narrator: George likes to jump in muddy puddles, too.Peppa: George. If you jump in muddy puddles, you must wear your boots.Narrator: Peppa likes to look after her little brother, George.Peppa: George, let's find some more pud dles.Narrator: Peppa and George are having a lot of fun. Peppa has found a lttle puddle. George hasfound a big puddle.Peppa: Look, George. There's a really big puddle.Narrator: George wants to jump into the big puddle first.Peppa: Stop, George. | must check if it's safe for you. Good. It is safe for you. Sorry, George. It'sonly mud.Narrator: Peppa and George love jumping in muddy puddles.Peppa: Come on, George. Let's go and show Daddy.Daddy: Goodness me.Peppa: Daddy. Daddy. Guess what we' ve been doing.Daddy: Let me think... Have you been wa tching television?Peppa: No. No. Daddy.Daddy: Have you just had a bath?Peppa: No. No.Daddy: | know. You've been jumping in muddy puddles.Peppa: Yes. Yes. Daddy. We've been jumping in muddy puddles.Daddy: Ho. Ho. And look at the mess you're in.Peppa: Oooh....Daddy: Oh, well, it's only mud. Let's clean up quickly before Mummy sees the mess.Peppa: Daddy, when we've cleaned up, will you and Mummy Come and play, too?Daddy: Yes, we can all play in the garden.Narrator: Peppa and George are wearing their boots. Mummy and Daddy are wearing their boots.Peppa loves jumping up and down in muddy puddles. Everyone loves jumping up and down inmuddy puddles.Mummy: Oh, Daddy pig, look at the mess you're in. .Peppa: It's only mud.";
20 
21     timeval st, et;
22 
23     peppa_pig_text_size = strlen(peppa_pig_buf);
24     com_space_size= ZSTD_compressBound(peppa_pig_text_size);
25 
26     gettimeofday(&st, NULL);
27     while(1) {
28         
29         com_ptr = (char *)malloc(com_space_size);
30         com_size = ZSTD_compress(com_ptr, com_space_size, peppa_pig_buf, peppa_pig_text_size, ZSTD_fast);
31         
32         free(com_ptr);
33         cnt++;
34         
35         gettimeofday(&et, NULL);
36         if(et.tv_sec - st.tv_sec >= 10) {
37             break;
38         }
39     }
40 
41     cout << "comparess per second:" << cnt/10 << " times" << endl;
42     return 0;
43 }

執行結果：

　　結果顯示ZSTD的壓縮效能大概在每秒6-7萬次左右，這個結果其實並不是太理想。需要說明的是壓縮效能與待壓縮文字的長度、字元內容也是有關係的。

　　我們再來探索一下ZSTD的解壓縮效能。與上面的測試方法類似，先對本文進行壓縮，然後連續解壓同一段被壓縮過的資料並持續10秒，最後得到每一秒的平均解壓速率。測試解壓效能的程式碼示例如下：

 1 #include <stdio.h>
 2 #include <string.h>
 3 #include <sys/time.h>
 4 #include <malloc.h>
 5 #include <zstd.h>
 6 #include <iostream>
 7 
 8 using namespace std;
 9 
10 int main()
11 {
12     int cnt = 0;
13 
14     size_t com_size;
15     size_t com_space_size;
16     size_t peppa_pig_text_size;
17 
18     timeval st, et;
19     
20     char *com_ptr = NULL;
21     char peppa_pig_buf[2048] = "Narrator: It is raining today. So, Peppa and George cannot play outside.Peppa: Daddy, it's stopped raining. Can we go out to play?Daddy: Alright, run along you two.Narrator: Peppa loves jumping in muddy puddles.Peppa: I love muddy puddles.Mummy: Peppa. If you jumping in muddy puddles, you must wear your boots.Peppa: Sorry, Mummy.Narrator: George likes to jump in muddy puddles, too.Peppa: George. If you jump in muddy puddles, you must wear your boots.Narrator: Peppa likes to look after her little brother, George.Peppa: George, let's find some more pud dles.Narrator: Peppa and George are having a lot of fun. Peppa has found a lttle puddle. George hasfound a big puddle.Peppa: Look, George. There's a really big puddle.Narrator: George wants to jump into the big puddle first.Peppa: Stop, George. | must check if it's safe for you. Good. It is safe for you. Sorry, George. It'sonly mud.Narrator: Peppa and George love jumping in muddy puddles.Peppa: Come on, George. Let's go and show Daddy.Daddy: Goodness me.Peppa: Daddy. Daddy. Guess what we' ve been doing.Daddy: Let me think... Have you been wa tching television?Peppa: No. No. Daddy.Daddy: Have you just had a bath?Peppa: No. No.Daddy: | know. You've been jumping in muddy puddles.Peppa: Yes. Yes. Daddy. We've been jumping in muddy puddles.Daddy: Ho. Ho. And look at the mess you're in.Peppa: Oooh....Daddy: Oh, well, it's only mud. Let's clean up quickly before Mummy sees the mess.Peppa: Daddy, when we've cleaned up, will you and Mummy Come and play, too?Daddy: Yes, we can all play in the garden.Narrator: Peppa and George are wearing their boots. Mummy and Daddy are wearing their boots.Peppa loves jumping up and down in muddy puddles. Everyone loves jumping up and down inmuddy puddles.Mummy: Oh, Daddy pig, look at the mess you're in. .Peppa: It's only mud.";
22 
23     size_t decom_size;
24     char* decom_ptr = NULL;
25     unsigned long long decom_buf_size;
26 
27     peppa_pig_text_size = strlen(peppa_pig_buf);
28     com_space_size= ZSTD_compressBound(peppa_pig_text_size);
29     com_ptr = (char *)malloc(com_space_size);
30     
31     com_size = ZSTD_compress(com_ptr, com_space_size, peppa_pig_buf, peppa_pig_text_size, 1);
32 
33     gettimeofday(&st, NULL);
34     decom_buf_size = ZSTD_getFrameContentSize(com_ptr, com_size);
35     
36     while(1) {
37     
38         decom_ptr = (char *)malloc((size_t)decom_buf_size);
39         
40         decom_size = ZSTD_decompress(decom_ptr, decom_buf_size, com_ptr, com_size);
41         if(decom_size != peppa_pig_text_size) {
42         
43             cout << "decompress error" << endl;
44             break;
45         }
46         
47         free(decom_ptr);
48         
49         cnt++;
50         gettimeofday(&et, NULL);
51         if(et.tv_sec - st.tv_sec >= 10) {
52             break;
53         }
54     }
55 
56     cout << "decompress per second:" << cnt/10 << " times" << endl;
57     
58     free(com_ptr);
59     return 0;
60 }

執行結果：

 　　結果顯示ZSTD的解壓縮效能大概在每秒12萬次左右，解壓效能比壓縮效能高。

三、zstd的高階用法

　　zstd提供了一個名為PZSTD的壓縮和解壓工具。PZSTD（parallel zstd），並行壓縮的zstd，是一個使用多執行緒對待壓縮文字進行切片分段，且進行並行壓縮的命令列工具。

　　其實高版本（v1.4.0及以上）的zstd也提供了指定多執行緒對文字進行並行壓縮的相關API介面，也就是本小節要介紹的zstd高階API用法。下面我們再來探索一下zstd的多執行緒壓縮使用方法。

　　多執行緒並行壓縮的兩個關鍵API，一個是引數設定API，另一個是壓縮API。

　　引數設定API的原型是：

　　size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, int value)

　　壓縮API的原型是：

　　size_t ZSTD_compress2(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize)

　　下面給出zstd並行壓縮的示例demo，通過ZSTD_CCtx_setParameter設定執行緒數為3，即指定巨集ZSTD_c_nbWorkers為3，通過ZSTD_compress2壓縮相關文字。另外，為了展示zstd確實使用了多執行緒，需要先讀取一個非常大的檔案，作為zstd的壓縮文字源，儘量使zstd執行較長時間。

 1 #include <stdio.h>
 2 #include <string.h>
 3 #include <sys/time.h>
 4 #include <malloc.h>
 5 #include <zstd.h>
 6 #include <iostream>
 7 
 8 using namespace std;
 9 
10 int main()
11 {
12     size_t com_size;
13     size_t com_space_size;
14 
15     FILE *fp = NULL;
16     unsigned int file_len;
17     
18     char *com_ptr = NULL;
19     char *file_text_ptr = NULL;
20 
21     fp = fopen("xxxxxx", "r");
22     if(NULL == fp){
23          cout << "file open failed" << endl;
24          return -1;
25     }
26 
27     fseek(fp, 0, SEEK_END);
28     file_len = ftell(fp);
29     fseek(fp, 0, SEEK_SET);
30     cout << "file length:" << file_len << endl;
31 
32     // malloc space for file content
33     file_text_ptr = (char *)malloc(file_len);
34     if(NULL == file_text_ptr) {
35         cout << "malloc failed" << endl;
36         return -1;
37     }
38 
39     // malloc space for compress space
40     com_space_size = ZSTD_compressBound(file_len);
41     com_ptr = (char *)malloc(com_space_size);
42     if(NULL == com_ptr) {
43         cout << "malloc failed" << endl;
44         return -1;
45     }
46 
47     // read text from source file
48     fread(file_text_ptr, 1, file_len, fp);
49     fclose(fp);
50 
51     ZSTD_CCtx* cctx;
52     cctx = ZSTD_createCCtx();
53 
54     // set multi-thread parameter
55     ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 3);
56     ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, ZSTD_btlazy2);
57 
58     com_size = ZSTD_compress2(cctx, com_ptr, com_space_size, file_text_ptr, file_len);
59     
60     free(com_ptr);
61     free(file_text_ptr);
62     return 0;
63 }

　　執行上述demo，可見zstd確實啟動了3個執行緒對文字進行了並行壓縮。且設定的執行緒數越多，壓縮時間越短，這裡就不詳細展示了，讀者可以自行實驗。

　　需要說明的是，zstd當前預設編譯單執行緒的庫檔案，要實現多執行緒的API呼叫，需要在make的時候指定編譯引數ZSTD_MULTITHREAD。

　　

　　另外，zstd還支援執行緒池的方式，執行緒池的函式原型：

　　POOL_ctx* ZSTD_createThreadPool(size_t numThreads)

　　執行緒池可以避免在多次、連續壓縮場景時頻繁的去建立執行緒、撤銷執行緒產生的非必要開銷，使得算力主要開銷在文字壓縮方面。

四、總結

 　　本篇分享了zstd壓縮與解壓縮使用的基本方法，對壓縮與解壓的效能進行了摸底，最後探索了zstd多執行緒壓縮的使用方法。

　　從壓縮測試來看，zstd的壓縮比其實已經比較好了，比原文所佔用空間縮小了一半以上，當然壓縮比也跟待壓縮文字的內容有關。

　　從效能執行結果來看，zstd的壓縮與解壓效能表現比較勉強，我認為zstd在魚（效能）和熊掌（壓縮比）之間更偏向熊掌一些，不過對一些效能要求不太高的，但是要高壓縮比的場景是比較符合的。

　　多執行緒並行壓縮，在有大文字需要連續多次壓縮的場景下，結合執行緒池可以很好的提升壓縮速率。

 

　　下一篇介紹調研的LZ4壓縮演算法。碼字不易，還請各位技術愛好者登入點個贊呀！

相關文章