Windows 下 MySQL 資料匯入 Redis

_zzh發表於2019-07-03

最近正式開始學習了Redis的使用,於是就在思考mysql的資料要怎麼樣匯入到redis中呢?於是查詢資料:資料連結如下:

  1. https://www.cnblogs.com/tommy-huang/p/4703...
  2. https://blog.csdn.net/songzhiren5560/artic...
  3. https://blog.csdn.net/Michaelwubo/article/...
  4. https://blog.csdn.net/Michaelwubo/article/...

結果如下:
1.sql指令碼,資料就自行新增吧。

CREATE TABLE `xb_cp_name`  (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `Name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '渠道類別與產品類別關係',
  `Description` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '關係描述',
  `Status` tinyint(4) NOT NULL DEFAULT 1 COMMENT '狀態',
  `Sort` int(11) NOT NULL DEFAULT 999 COMMENT '權重狀態',
  `IsDel` tinyint(4) NOT NULL DEFAULT 0 COMMENT '刪除標誌',
  `OperatorID` int(4) NULL DEFAULT NULL COMMENT '操作者ID',
  `UpdateTime` datetime(0) NULL DEFAULT NULL COMMENT '更新時間',
  PRIMARY KEY (`ID`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 8 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;

2.匯入資料到redis指令碼。

SELECT CONCAT(
           "*18\n",
           '$',LENGTH(redis_cmd),'\n',redis_cmd,'\n',
           '$',LENGTH(redis_key),'\n',redis_key,'\n',
           '$',LENGTH(hkey1),'\n',hkey1,'\n','$',LENGTH(hval1),'\n',hval1,'\n',
           '$',LENGTH(hkey2),'\n',hkey2,'\n','$',LENGTH(hval2),'\n',hval2,'\n',
           '$',LENGTH(hkey3),'\n',hkey3,'\n','$',LENGTH(hval3),'\n',hval3,'\n',
           '$',LENGTH(hkey4),'\n',hkey4,'\n','$',LENGTH(hval4),'\n',hval4,'\n',
           '$',LENGTH(hkey5),'\n',hkey5,'\n','$',LENGTH(hval5),'\n',hval5,'\n',
           '$',LENGTH(hkey6),'\n',hkey6,'\n','$',LENGTH(hval6),'\n',hval6,'\n',
           '$',LENGTH(hkey7),'\n',hkey7,'\n','$',LENGTH(hval7),'\n',hval7,'\n',
           '$',LENGTH(hkey8),'\n',hkey8,'\n','$',LENGTH(hval8),'\n',hval8
        )FROM(
           SELECT 'HMSET' AS redis_cmd,
           concat_ws(':','xb_cp_name', ID) AS redis_key,
           'ID' AS hkey1, ID AS hval1,
           'Name' AS hkey2, Name AS hval2,
           'Description' AS hkey3, if(Description is not null, Description, ' ') AS hval3,
           'Status' AS hkey4, Status AS hval4,
           'Sort' AS hkey5, Sort AS hval5,
           'IsDel' AS hkey6, IsDel AS hval6,
           'OperatorID' AS hkey7, OperatorID AS hval7,
           'UpdateTime' AS hkey8, UpdateTime AS hval8
           From xb_cp_name
        )AS t

3.上述解釋

資料顯示 自Redis 2.6以上版本起,Redis支援快速大批次匯入資料,即官網的Redis Mass Insertion,即Pipe傳輸,透過將要匯入的命令轉換為Resp格式,然後透過MySQL的concat()來整理出最終匯入的命令集合,以達到快速匯入的目的。所以有一下幾點需要注意的。

1. 因為RESP協議中的分隔符為在Linux下是\r\n,而在Windows下則為\n
2. 因為我的實際資料Description列中的資料行是為空的。所以根據參考,新增了特殊處理行: 'Description' AS hkey3, if(Description is not null, Description, ' ') AS hval3, 
2. 第一行的 *18\r\n  :  *表示陣列,8表示陣列元素個數, \r\n是規定分隔符.我猜想的就是hkey+hval+redis_cmd+redis_key的數量總和(真的是猜的,因為我讀不太懂)。
3.第二行的  '$',LENGTH(redis_cmd),'\r\n',redis_cmd,'\r\n',     : $表示長字串,LENGTH(redis_cmd)表示字串長度,redis_cmd字串變數,\r\n還是規定字串
4.還有就是注意記事本的編碼問題,如果出現MySQL ERROR 1300 (HY000): Invalid utf8 character string這個東西,那大機率就是字元編碼的問題。記得點選另存為儲存為utf-8編碼。
5.ERR Protocol error: expected '$', got '',這個問題就是第一個問題的錯誤提示
6.最後一個經驗之談,不建議直接複製貼上,真的不建議,不建議。會出現各種莫名其妙的問題。跪了一下午。

4.執行指令碼命令

mysql -h 資料庫伺服器地址 -u使用者名稱 -p密碼 -D資料庫名  --default-character-set=utf8 --skip-column-names --raw < mysql-to-redis.sql | redis-cli.exe --pipe

PS:記得把中文改成對應的引數就可以了,至此就完成啦~

5.因為覺得每次手動替換欄位字串,遇到大表肯定會瘋掉的。所以就自己擼了一個簡單地方法,傳入表名和空欄位就可以生成文字的函式。沒有經過嚴謹的測試,用的是TP框架。如果使用過程中遇到問題還請多多體諒。

public function makeRedisTxt($tableName,$nullColumn=array()){
        $tableColumns = M($tableName)->getDbFields();
        $count = count($tableColumns)*2+2;
        $RedisTxt = "
           SELECT CONCAT(
           '*".$count."\\n',
           '$',LENGTH(redis_cmd),'\\n',redis_cmd,'\\n',
           '$',LENGTH(redis_key),'\\n',redis_key,'\\n',
        ";
        foreach ($tableColumns as $key=> $column){
            $RedisTxt.="'$',LENGTH(hkey".$key."),'\\n',hkey".$key.",'\\n','$',LENGTH(hval".$key."),'\\n',hval".$key.",'\\n',\n";
        }
        $RedisTxt = substr($RedisTxt,0,-7);//去除最後一行的,     ,'\n',
        $RedisTxt.= "
           )FROM(
           SELECT 'HMSET' AS redis_cmd,
           concat_ws(':','xb_".$tableName."', ID) AS redis_key,
        ";
        foreach ($tableColumns as $key=> $column){
            $RedisTxt.="'".$column."' AS hkey".$key.", ".$column." AS hval".$key.",\n";
        }
        $RedisTxt = substr($RedisTxt,0,-2);
        $RedisTxt .= "
            From xb_".$tableName."
            )AS t
        ";

        //組裝空欄位陣列
        $newColumnArr = array();
        if($nullColumn){
            foreach ($nullColumn as $key => $column){
                $newColumnArr[$column] = "if(".$column." is not null, ".$column." , '') ";
            }
        }
        //替換空欄位最後出現的位置為特殊處理行
        if($newColumnArr){
            foreach ($newColumnArr as $col => $colStr){
                $start = strripos($RedisTxt,$col);//獲取最後出現的位置,即開始替換的位置
                $strlen = strlen($col);//獲取所需替換字元長度
                $RedisTxt = substr_replace($RedisTxt,$colStr,$start,$strlen);
            }
        }
        return $RedisTxt;

        /*****************生成結果如下**********************/
//        SELECT CONCAT(
//            '*18\n',
//            '$', LENGTH(redis_cmd), '\n', redis_cmd, '\n',
//            '$', LENGTH(redis_key), '\n', redis_key, '\n',
//            '$', LENGTH(hkey0), '\n', hkey0, '\n', '$', LENGTH(hval0), '\n', hval0, '\n',
//            '$', LENGTH(hkey1), '\n', hkey1, '\n', '$', LENGTH(hval1), '\n', hval1, '\n',
//            '$', LENGTH(hkey2), '\n', hkey2, '\n', '$', LENGTH(hval2), '\n', hval2, '\n',
//            '$', LENGTH(hkey3), '\n', hkey3, '\n', '$', LENGTH(hval3), '\n', hval3, '\n',
//            '$', LENGTH(hkey4), '\n', hkey4, '\n', '$', LENGTH(hval4), '\n', hval4, '\n',
//            '$', LENGTH(hkey5), '\n', hkey5, '\n', '$', LENGTH(hval5), '\n', hval5, '\n',
//            '$', LENGTH(hkey6), '\n', hkey6, '\n', '$', LENGTH(hval6), '\n', hval6, '\n',
//            '$', LENGTH(hkey7), '\n', hkey7, '\n', '$', LENGTH(hval7), '\n', hval7
//        )FROM(
//           SELECT 'HMSET' AS redis_cmd,
//           concat_ws(':', 'xb_cp_name', ID) AS redis_key,
//          'ID' AS hkey0, ID AS hval0,
//          'Name' AS hkey1, Name AS hval1,
//          'Description' AS hkey2, if (Description is not null, Description , '')  AS hval2,
//          'Status' AS hkey3, Status AS hval3,
//          'Sort' AS hkey4, Sort AS hval4,
//          'IsDel' AS hkey5, IsDel AS hval5,
//          'OperatorID' AS hkey6, OperatorID AS hval6,
//          'UpdateTime' AS hkey7, UpdateTime AS hval7
//           From xb_cp_name
//        )AS t
    }
本作品採用《CC 協議》,轉載必須註明作者和本文連結

相關文章