milvus-migration安裝使用

slnngk發表於2024-11-13

環境:
OS:Centos 7
源端和目的端milvus版本:2.4.6
源端ip:192.168.1.134
目的端ip:192.168.1.135

1.下載地址
https://github.com/zilliztech/milvus-migration/releases/tag/v0.1.26

2.安裝
[root@host135 opt]# cd /opt
[root@host135 opt]# mkdir milvus-migration
[root@host135 opt]# mv milvus-migration_Linux_x86_64.tar.gz ./milvus-migration/
[root@host135 milvus-migration]# tar -xvf milvus-migration_Linux_x86_64.tar.gz

解壓後檔案如下:

[root@host135 milvus-migration]# ls -al
total 70296
drwxrwxr-x   2 root   root        195 Nov 13 09:35 .
drwxr-xr-x. 37 root   root       4096 Nov 13 09:34 ..
-rwxr-xr-x   1 oracle docker 55238808 Aug 23 20:00 milvus-migration
-rw-rw-r--   1 root   root   16671683 Nov 13 09:32 milvus-migration_Linux_x86_64.tar.gz
-rw-r--r--   1 oracle docker     9793 Aug 23 19:58 README_1X.md
-rw-r--r--   1 oracle docker     5454 Aug 23 19:58 README_2X.md
-rw-r--r--   1 oracle docker    12359 Aug 23 19:58 README_ES.md
-rw-r--r--   1 oracle docker     7565 Aug 23 19:58 README_FAISS.md
-rw-r--r--   1 oracle docker     3543 Aug 23 19:58 README.md
-rw-r--r--   1 oracle docker    10790 Aug 23 19:58 README_old.md

若我們這裡是milvus 2遷移到milvus 2的話,閱讀README_2X.md

3.從README_2X.md獲取配置檔案內容

dumper:
  worker:
    workMode: milvus2x      # work mode：milvus2x->milvus2x
    reader:
      bufferSize: 500       # Read source data rows in each time read from Source Milvus.

meta:                       # meta part
  mode: config              # 'config' mode means will get meta config from this config file itself.
  version: 2.3.0            #  Source Milvus version
  collection: src_coll_name # migrate data from this source collection

source:                     # source milvus connection info
  milvus2x:
    endpoint: {milvus2x_domain}:{milvus2x_port}
    username: xxxx
    password: xxxxx

target:                    # target milvus collection info
  milvus2x:
    endpoint: {milvus2x_domain}:{milvus2x_port}
    username: xxxx
    password: xxxxx

將如上內容儲存到 migration.yaml,我們這裡建立配置檔案目錄

[root@host135 milvus-migration]#cd /opt/milvus-migration
[root@host135 milvus-migration]#mkdir configs

最後的配置檔案內容如下(我這裡是遷移db_test庫下的表car):

[root@host135 configs]# more migration.yaml 
dumper:
  worker:
    workMode: milvus2x      # work mode：milvus2x->milvus2x
    reader:
      bufferSize: 500       # Read source data rows in each time read from Source Milvus.

meta:                       # meta part
  mode: config              # 'config' mode means will get meta config from this config file itself.
  version: 2.4.6            #  Source Milvus version
  collection: car # migrate data from this source collection

source:                     # source milvus connection info
  milvus2x:
    endpoint: 192.168.1.134:19530
    database: db_test
    username: root
    password: Milvus

target:                    # target milvus collection info
  milvus2x:
    endpoint: 192.168.1.135:19530
    database: db_test
    username: root
    password: Milvus

4.啟動
[root@host135 milvus-migration]# cd /opt/milvus-migration
[root@host135 milvus-migration]#./milvus-migration start

若配置檔案不在configs目錄,可以在命令裡面指定配置檔案
[root@host135 milvus-migration]#./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml


或者在命令列透過-t引數指定某個collection


[root@host135 milvus-migration]#./milvus-migration start -t="tb_test02" -c=/opt/milvus-migration/configs/migration.yaml
這裡指定了-t引數的話,配置檔案裡指定的collection: car引數就不管用了.

5.同步某個庫多個表
指令碼如下:
vi milvus_migration_all_collection.sh

#!/bin/bash

collections=("car" "tb_test02" "CEC_Corpus")

for collection in "${collections[@]}"; do
echo "BatchMigration==> $collection"
/opt/milvus-migration/milvus-migration start -t="$collection" -c=/opt/milvus-migration/configs/migration.yaml
done

注意事項:
1.源端的collection必須要load,否則包如下的錯誤
error="failed to query: collection not loaded

2.目的端不需要提前建立好資料庫,同步後會自動建立資料庫

3.同步後的collection是沒有索引的,當然也沒有load,需要手工建立索引;

相關文章