最近正式開始學習了Redis的使用,於是就在思考mysql的資料要怎麼樣匯入到redis中呢?於是查詢資料:資料連結如下:
結果如下:
1.sql指令碼,資料就自行新增吧。
CREATE TABLE `xb_cp_name` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`Name` varchar(50) CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL COMMENT '渠道類別與產品類別關係',
`Description` varchar(255) CHARACTER SET utf8 COLLATE utf8_general_ci NULL DEFAULT NULL COMMENT '關係描述',
`Status` tinyint(4) NOT NULL DEFAULT 1 COMMENT '狀態',
`Sort` int(11) NOT NULL DEFAULT 999 COMMENT '權重狀態',
`IsDel` tinyint(4) NOT NULL DEFAULT 0 COMMENT '刪除標誌',
`OperatorID` int(4) NULL DEFAULT NULL COMMENT '操作者ID',
`UpdateTime` datetime(0) NULL DEFAULT NULL COMMENT '更新時間',
PRIMARY KEY (`ID`) USING BTREE
) ENGINE = InnoDB AUTO_INCREMENT = 8 CHARACTER SET = utf8 COLLATE = utf8_general_ci ROW_FORMAT = Dynamic;
2.匯入資料到redis指令碼。
SELECT CONCAT(
"*18\n",
'$',LENGTH(redis_cmd),'\n',redis_cmd,'\n',
'$',LENGTH(redis_key),'\n',redis_key,'\n',
'$',LENGTH(hkey1),'\n',hkey1,'\n','$',LENGTH(hval1),'\n',hval1,'\n',
'$',LENGTH(hkey2),'\n',hkey2,'\n','$',LENGTH(hval2),'\n',hval2,'\n',
'$',LENGTH(hkey3),'\n',hkey3,'\n','$',LENGTH(hval3),'\n',hval3,'\n',
'$',LENGTH(hkey4),'\n',hkey4,'\n','$',LENGTH(hval4),'\n',hval4,'\n',
'$',LENGTH(hkey5),'\n',hkey5,'\n','$',LENGTH(hval5),'\n',hval5,'\n',
'$',LENGTH(hkey6),'\n',hkey6,'\n','$',LENGTH(hval6),'\n',hval6,'\n',
'$',LENGTH(hkey7),'\n',hkey7,'\n','$',LENGTH(hval7),'\n',hval7,'\n',
'$',LENGTH(hkey8),'\n',hkey8,'\n','$',LENGTH(hval8),'\n',hval8
)FROM(
SELECT 'HMSET' AS redis_cmd,
concat_ws(':','xb_cp_name', ID) AS redis_key,
'ID' AS hkey1, ID AS hval1,
'Name' AS hkey2, Name AS hval2,
'Description' AS hkey3, if(Description is not null, Description, ' ') AS hval3,
'Status' AS hkey4, Status AS hval4,
'Sort' AS hkey5, Sort AS hval5,
'IsDel' AS hkey6, IsDel AS hval6,
'OperatorID' AS hkey7, OperatorID AS hval7,
'UpdateTime' AS hkey8, UpdateTime AS hval8
From xb_cp_name
)AS t
3.上述解釋
資料顯示 自Redis 2.6以上版本起,Redis支援快速大批次匯入資料,即官網的Redis Mass Insertion,即Pipe傳輸,透過將要匯入的命令轉換為Resp格式,然後透過MySQL的concat()來整理出最終匯入的命令集合,以達到快速匯入的目的。所以有一下幾點需要注意的。
1. 因為RESP協議中的分隔符為在Linux下是\r\n,而在Windows下則為\n
2. 因為我的實際資料Description列中的資料行是為空的。所以根據參考,新增了特殊處理行: 'Description' AS hkey3, if(Description is not null, Description, ' ') AS hval3,
2. 第一行的 *18\r\n : *表示陣列,8表示陣列元素個數, \r\n是規定分隔符.我猜想的就是hkey+hval+redis_cmd+redis_key的數量總和(真的是猜的,因為我讀不太懂)。
3.第二行的 '$',LENGTH(redis_cmd),'\r\n',redis_cmd,'\r\n', : $表示長字串,LENGTH(redis_cmd)表示字串長度,redis_cmd字串變數,\r\n還是規定字串
4.還有就是注意記事本的編碼問題,如果出現MySQL ERROR 1300 (HY000): Invalid utf8 character string這個東西,那大機率就是字元編碼的問題。記得點選另存為儲存為utf-8編碼。
5.ERR Protocol error: expected '$', got '',這個問題就是第一個問題的錯誤提示
6.最後一個經驗之談,不建議直接複製貼上,真的不建議,不建議。會出現各種莫名其妙的問題。跪了一下午。
4.執行指令碼命令
mysql -h 資料庫伺服器地址 -u使用者名稱 -p密碼 -D資料庫名 --default-character-set=utf8 --skip-column-names --raw < mysql-to-redis.sql | redis-cli.exe --pipe
PS:記得把中文改成對應的引數就可以了,至此就完成啦~
5.因為覺得每次手動替換欄位字串,遇到大表肯定會瘋掉的。所以就自己擼了一個簡單地方法,傳入表名和空欄位就可以生成文字的函式。沒有經過嚴謹的測試,用的是TP框架。如果使用過程中遇到問題還請多多體諒。
public function makeRedisTxt($tableName,$nullColumn=array()){
$tableColumns = M($tableName)->getDbFields();
$count = count($tableColumns)*2+2;
$RedisTxt = "
SELECT CONCAT(
'*".$count."\\n',
'$',LENGTH(redis_cmd),'\\n',redis_cmd,'\\n',
'$',LENGTH(redis_key),'\\n',redis_key,'\\n',
";
foreach ($tableColumns as $key=> $column){
$RedisTxt.="'$',LENGTH(hkey".$key."),'\\n',hkey".$key.",'\\n','$',LENGTH(hval".$key."),'\\n',hval".$key.",'\\n',\n";
}
$RedisTxt = substr($RedisTxt,0,-7);//去除最後一行的, ,'\n',
$RedisTxt.= "
)FROM(
SELECT 'HMSET' AS redis_cmd,
concat_ws(':','xb_".$tableName."', ID) AS redis_key,
";
foreach ($tableColumns as $key=> $column){
$RedisTxt.="'".$column."' AS hkey".$key.", ".$column." AS hval".$key.",\n";
}
$RedisTxt = substr($RedisTxt,0,-2);
$RedisTxt .= "
From xb_".$tableName."
)AS t
";
//組裝空欄位陣列
$newColumnArr = array();
if($nullColumn){
foreach ($nullColumn as $key => $column){
$newColumnArr[$column] = "if(".$column." is not null, ".$column." , '') ";
}
}
//替換空欄位最後出現的位置為特殊處理行
if($newColumnArr){
foreach ($newColumnArr as $col => $colStr){
$start = strripos($RedisTxt,$col);//獲取最後出現的位置,即開始替換的位置
$strlen = strlen($col);//獲取所需替換字元長度
$RedisTxt = substr_replace($RedisTxt,$colStr,$start,$strlen);
}
}
return $RedisTxt;
/*****************生成結果如下**********************/
// SELECT CONCAT(
// '*18\n',
// '$', LENGTH(redis_cmd), '\n', redis_cmd, '\n',
// '$', LENGTH(redis_key), '\n', redis_key, '\n',
// '$', LENGTH(hkey0), '\n', hkey0, '\n', '$', LENGTH(hval0), '\n', hval0, '\n',
// '$', LENGTH(hkey1), '\n', hkey1, '\n', '$', LENGTH(hval1), '\n', hval1, '\n',
// '$', LENGTH(hkey2), '\n', hkey2, '\n', '$', LENGTH(hval2), '\n', hval2, '\n',
// '$', LENGTH(hkey3), '\n', hkey3, '\n', '$', LENGTH(hval3), '\n', hval3, '\n',
// '$', LENGTH(hkey4), '\n', hkey4, '\n', '$', LENGTH(hval4), '\n', hval4, '\n',
// '$', LENGTH(hkey5), '\n', hkey5, '\n', '$', LENGTH(hval5), '\n', hval5, '\n',
// '$', LENGTH(hkey6), '\n', hkey6, '\n', '$', LENGTH(hval6), '\n', hval6, '\n',
// '$', LENGTH(hkey7), '\n', hkey7, '\n', '$', LENGTH(hval7), '\n', hval7
// )FROM(
// SELECT 'HMSET' AS redis_cmd,
// concat_ws(':', 'xb_cp_name', ID) AS redis_key,
// 'ID' AS hkey0, ID AS hval0,
// 'Name' AS hkey1, Name AS hval1,
// 'Description' AS hkey2, if (Description is not null, Description , '') AS hval2,
// 'Status' AS hkey3, Status AS hval3,
// 'Sort' AS hkey4, Sort AS hval4,
// 'IsDel' AS hkey5, IsDel AS hval5,
// 'OperatorID' AS hkey6, OperatorID AS hval6,
// 'UpdateTime' AS hkey7, UpdateTime AS hval7
// From xb_cp_name
// )AS t
}
本作品採用《CC 協議》,轉載必須註明作者和本文連結