MySQL online ddl 工具之pt-online-schema-change
MySQL DDL:
DDL是一個令所有MySQL dDBA 詬病的一個功能,因為在MySQL中在對錶進行dDDL時,會鎖表,當表比較小比如小於1W行時,對前端影響較小,當時遇到千萬級別的表,就會影響前端應用對錶的寫操作!
InnoDB引擎是透過以下步驟來進行DDL的:
1、按照原始表(original_table)的表結構和DDL語句,新建一個不可見的臨時表(tmp_table)
2、在原表上加write lock,阻塞所有更新操作(insert、delete、update等)
3、執行insert into tmp_table select * from original_table
4、rename original_table和tmp_table,最後drop original_table
5、釋放 write lock。
可以看見在InnoDB執行DDL的時候,原表是隻能讀不能寫的。為此 perconal 推出一個工具 pt-online-schema-change ,其特點是修改過程中不會造成讀寫阻塞。
工作原理:
如果表有外來鍵,除非使用 --alter-foreign-keys-method 指定特定的值,否則工具不予執行。
1 建立一個和你要執行 alter 操作的表一樣的空表結構。
2 執行表結構修改,然後從原表中的資料到copy到 表結構修改後的表,
3 在原表上建立觸發器將 copy 資料的過程中,在原表的更新操作 更新到新表.
注意:如果表中已經定義了觸發器這個工具就不能工作了。
4 copy 完成以後,用rename table 新表代替原表,預設刪除原表。
用法介紹:
pt-online-schema-change [OPTIONS] DSN
options 可以自行檢視 help,DNS 為你要操作的資料庫和表。這裡有兩個引數需要介紹一下:
--dry-run
這個引數不建立觸發器,不複製資料,也不會替換原表。只是建立和更改新表。
--execute
這個引數的作用和前面工作原理的介紹的一樣,會建立觸發器,來保證最新變更的資料會影響至新表。注意:如果不加這個引數,這個工具會在執行一些檢查後退出。
依賴條件:
操作的表必須有主鍵否則 報如下錯誤:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column vname varchar(20)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T17:51:43 Dropping new table...
2016-01-08T17:51:43 Dropped new table OK.
`yoon`.`yoon` was not altered.
The new table `yoon`.`_yoon_new` does not have a PRIMARY KEY or a unique index which is required for the DELETE trigger.
欄位vname沒有新增成功!
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL DEFAULT '0',
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8
新增主鍵:
mysql> alter table yoon modify actor_id smallint(8) unsigned primary key;
Query OK, 0 rows affected (0.06 sec)
Records: 0 Duplicates: 0 Warnings: 0
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column vname varchar(20)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T17:57:09 Creating triggers...
2016-01-08T17:57:09 Created triggers OK.
2016-01-08T17:57:09 Copying approximately 200 rows...
2016-01-08T17:57:09 Copied rows OK.
2016-01-08T17:57:09 Swapping tables...
2016-01-08T17:57:09 Swapped original and new tables OK.
2016-01-08T17:57:09 Dropping old table...
2016-01-08T17:57:09 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T17:57:09 Dropping triggers...
2016-01-08T17:57:09 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`vname` varchar(20) DEFAULT NULL,
PRIMARY KEY (`actor_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
新增多個欄位:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column aname varchar(20),add column bname varchar(30)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:04:25 Creating triggers...
2016-01-08T18:04:25 Created triggers OK.
2016-01-08T18:04:25 Copying approximately 200 rows...
2016-01-08T18:04:25 Copied rows OK.
2016-01-08T18:04:25 Swapping tables...
2016-01-08T18:04:26 Swapped original and new tables OK.
2016-01-08T18:04:26 Dropping old table...
2016-01-08T18:04:26 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:04:26 Dropping triggers...
2016-01-08T18:04:26 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`vname` varchar(20) DEFAULT NULL,
`aname` varchar(20) DEFAULT NULL,
`bname` varchar(30) DEFAULT NULL,
PRIMARY KEY (`actor_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
刪除欄位:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='drop column aname,drop column bname' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:05:45 Creating triggers...
2016-01-08T18:05:45 Created triggers OK.
2016-01-08T18:05:45 Copying approximately 200 rows...
2016-01-08T18:05:45 Copied rows OK.
2016-01-08T18:05:45 Swapping tables...
2016-01-08T18:05:45 Swapped original and new tables OK.
2016-01-08T18:05:45 Dropping old table...
2016-01-08T18:05:45 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:05:45 Dropping triggers...
2016-01-08T18:05:46 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
新增索引:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add key index_first(first_name)' --execute D=yoon,t=yoon No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:06:38 Creating triggers...
2016-01-08T18:06:38 Created triggers OK.
2016-01-08T18:06:38 Copying approximately 200 rows...
2016-01-08T18:06:38 Copied rows OK.
2016-01-08T18:06:38 Swapping tables...
2016-01-08T18:06:38 Swapped original and new tables OK.
2016-01-08T18:06:38 Dropping old table...
2016-01-08T18:06:38 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:06:38 Dropping triggers...
2016-01-08T18:06:38 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
DDL是一個令所有MySQL dDBA 詬病的一個功能,因為在MySQL中在對錶進行dDDL時,會鎖表,當表比較小比如小於1W行時,對前端影響較小,當時遇到千萬級別的表,就會影響前端應用對錶的寫操作!
InnoDB引擎是透過以下步驟來進行DDL的:
1、按照原始表(original_table)的表結構和DDL語句,新建一個不可見的臨時表(tmp_table)
2、在原表上加write lock,阻塞所有更新操作(insert、delete、update等)
3、執行insert into tmp_table select * from original_table
4、rename original_table和tmp_table,最後drop original_table
5、釋放 write lock。
可以看見在InnoDB執行DDL的時候,原表是隻能讀不能寫的。為此 perconal 推出一個工具 pt-online-schema-change ,其特點是修改過程中不會造成讀寫阻塞。
工作原理:
如果表有外來鍵,除非使用 --alter-foreign-keys-method 指定特定的值,否則工具不予執行。
1 建立一個和你要執行 alter 操作的表一樣的空表結構。
2 執行表結構修改,然後從原表中的資料到copy到 表結構修改後的表,
3 在原表上建立觸發器將 copy 資料的過程中,在原表的更新操作 更新到新表.
注意:如果表中已經定義了觸發器這個工具就不能工作了。
4 copy 完成以後,用rename table 新表代替原表,預設刪除原表。
用法介紹:
pt-online-schema-change [OPTIONS] DSN
options 可以自行檢視 help,DNS 為你要操作的資料庫和表。這裡有兩個引數需要介紹一下:
--dry-run
這個引數不建立觸發器,不複製資料,也不會替換原表。只是建立和更改新表。
--execute
這個引數的作用和前面工作原理的介紹的一樣,會建立觸發器,來保證最新變更的資料會影響至新表。注意:如果不加這個引數,這個工具會在執行一些檢查後退出。
依賴條件:
操作的表必須有主鍵否則 報如下錯誤:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column vname varchar(20)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T17:51:43 Dropping new table...
2016-01-08T17:51:43 Dropped new table OK.
`yoon`.`yoon` was not altered.
The new table `yoon`.`_yoon_new` does not have a PRIMARY KEY or a unique index which is required for the DELETE trigger.
欄位vname沒有新增成功!
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL DEFAULT '0',
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8
新增主鍵:
mysql> alter table yoon modify actor_id smallint(8) unsigned primary key;
Query OK, 0 rows affected (0.06 sec)
Records: 0 Duplicates: 0 Warnings: 0
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column vname varchar(20)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T17:57:09 Creating triggers...
2016-01-08T17:57:09 Created triggers OK.
2016-01-08T17:57:09 Copying approximately 200 rows...
2016-01-08T17:57:09 Copied rows OK.
2016-01-08T17:57:09 Swapping tables...
2016-01-08T17:57:09 Swapped original and new tables OK.
2016-01-08T17:57:09 Dropping old table...
2016-01-08T17:57:09 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T17:57:09 Dropping triggers...
2016-01-08T17:57:09 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`vname` varchar(20) DEFAULT NULL,
PRIMARY KEY (`actor_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
新增多個欄位:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add column aname varchar(20),add column bname varchar(30)' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:04:25 Creating triggers...
2016-01-08T18:04:25 Created triggers OK.
2016-01-08T18:04:25 Copying approximately 200 rows...
2016-01-08T18:04:25 Copied rows OK.
2016-01-08T18:04:25 Swapping tables...
2016-01-08T18:04:26 Swapped original and new tables OK.
2016-01-08T18:04:26 Dropping old table...
2016-01-08T18:04:26 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:04:26 Dropping triggers...
2016-01-08T18:04:26 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
mysql> show create table yoon\G
*************************** 1. row ***************************
Table: yoon
Create Table: CREATE TABLE `yoon` (
`actor_id` smallint(8) unsigned NOT NULL,
`first_name` varchar(45) NOT NULL,
`last_name` varchar(45) NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`vname` varchar(20) DEFAULT NULL,
`aname` varchar(20) DEFAULT NULL,
`bname` varchar(30) DEFAULT NULL,
PRIMARY KEY (`actor_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
刪除欄位:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='drop column aname,drop column bname' --execute D=yoon,t=yoon
No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:05:45 Creating triggers...
2016-01-08T18:05:45 Created triggers OK.
2016-01-08T18:05:45 Copying approximately 200 rows...
2016-01-08T18:05:45 Copied rows OK.
2016-01-08T18:05:45 Swapping tables...
2016-01-08T18:05:45 Swapped original and new tables OK.
2016-01-08T18:05:45 Dropping old table...
2016-01-08T18:05:45 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:05:45 Dropping triggers...
2016-01-08T18:05:46 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
新增索引:
[root@hank-yoon ~]# pt-online-schema-change -u root -pyoon -h127.0.0.1 --alter='add key index_first(first_name)' --execute D=yoon,t=yoon No slaves found. See --recursion-method if host hank-yoon.com has slaves.
Not checking slave lag because no slaves were found and --check-slave-lag was not specified.
Operation, tries, wait:
copy_rows, 10, 0.25
create_triggers, 10, 1
drop_triggers, 10, 1
swap_tables, 10, 1
update_foreign_keys, 10, 1
Altering `yoon`.`yoon`...
Creating new table...
Created new table yoon._yoon_new OK.
Altering new table...
Altered `yoon`.`_yoon_new` OK.
2016-01-08T18:06:38 Creating triggers...
2016-01-08T18:06:38 Created triggers OK.
2016-01-08T18:06:38 Copying approximately 200 rows...
2016-01-08T18:06:38 Copied rows OK.
2016-01-08T18:06:38 Swapping tables...
2016-01-08T18:06:38 Swapped original and new tables OK.
2016-01-08T18:06:38 Dropping old table...
2016-01-08T18:06:38 Dropped old table `yoon`.`_yoon_old` OK.
2016-01-08T18:06:38 Dropping triggers...
2016-01-08T18:06:38 Dropped triggers OK.
Successfully altered `yoon`.`yoon`.
來自 “ ITPUB部落格 ” ,連結:http://blog.itpub.net/28939273/viewspace-1974008/,如需轉載,請註明出處,否則將追究法律責任。
相關文章
- 一個 MySQL 線上 DDL 工具 — pt-online-schema-changeMySql
- mysql之 openark-kit online ddlMySql
- MySQL修改大表工具pt-online-schema-change原理MySql
- MySQL Online DDL詳解MySql
- MySQL DDL執行方式-Online DDL介紹MySql
- MySQL & MariaDB Online DDL 參考指南MySql
- MySQL:pt-online-schema-change原理及注意點(未完)MySql
- MySQL 5.7使用pt-online-schema-change對大表加索引MySql索引
- pt-online-schema-change和XtraBackup的
- MySQL 5.7使用pt-online-schema-change對大表加欄位MySql
- MySQL 執行 Online DDL 操作報錯空間不足?MySql
- MySQL Online DDL導致全域性鎖表案例分析MySql
- 線上改表工具oak-online-alter-table和pt-online-schema-change的使用限制總結
- pt-online-schema-change工作過程介紹
- pt-online-schema-change使用引數說明
- pt-online-schema-change 錯誤集 Wide characterIDE
- MySQL在其版本迭代後Online DDL功能發生了哪些變化?MySql
- TiDB Online DDL 在 TiCDC 中的應用丨TiDB 工具分享TiDB
- MySQL 5.6使用pt-online-schema-change線上修改大表欄位長度MySql
- MySQL 8.0 Reference Manual(讀書筆記81節-- InnoDB and Online DDL (1))MySql筆記
- MySQL 8.0 Reference Manual(讀書筆記82節-- InnoDB and Online DDL (2))MySql筆記
- MySQL 8.0 Reference Manual(讀書筆記83節-- InnoDB and Online DDL (3))MySql筆記
- MySQL 8.0 Reference Manual(讀書筆記84節-- InnoDB and Online DDL (4))MySql筆記
- 記錄一次 Online DDL 操作
- MySQL線上DDL工具 gh-ostMySql
- MySQL之資料定義語言(DDL)MySql
- MySQL(十三)DDL之庫和表的管理MySql
- MySQL DDL操作表MySql
- 對比上次MySQL的DDLMySql
- 04 MySQL 表的基本操作-DDLMySql
- MySQL 線上DDL "gh-ost"MySql
- MySQL - DDL詳解(Data Definition Language)MySql
- mysql DDL時鎖表的排查MySql
- 詳談 MySQL 8.0 原子 DDL 原理MySql
- mysql 原生 線上DDL 的bug .MySql
- MySQL全面瓦解4:資料定義-DDLMySql
- MySQL5.7 InnoDB線上DDL操作MySql
- MySQL的DDL和DML操作語法MySql
- MySQL DDL Waiting for table metadata lock 解決MySqlAI