故障解析丨一次死鎖問題的解決

GreatSQL發表於2024-03-18

背景

業務端遇到報錯為"Deadlock found when trying to get lock; try restarting transaction"則表明有死鎖發生

名稱	配置
資料庫版本	GreatSQL 8.0.26
隔離級別	Read-Commited

innodb status 日誌

greatsql> show engine innodb status\G
*************************** 1. row ***************************
  Type: InnoDB
  Name: 
Status: 
=====================================
2024-01-28 16:55:38 140737023727360 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 14 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 41 srv_active, 0 srv_shutdown, 17830 srv_idle
srv_master_thread log flush and writes: 0
----------
SEMAPHORES
----------
-------------
RW-LATCH INFO
-------------
Total number of rw-locks 132361
OS WAIT ARRAY INFO: reservation count 11180
OS WAIT ARRAY INFO: signal count 11177
RW-shared spins 0, rounds 0, OS waits 0
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------------------
LATEST DETECTED DEADLOCK
------------------------
2024-01-28 16:53:40 140735053358848
*** (1) TRANSACTION:
TRANSACTION 37616, ACTIVE 8 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 2 lock struct(s), heap size 1192, 1 row lock(s), undo log entries 1
MySQL thread id 16, OS thread handle 140737023432448, query id 652 127.0.0.1 root update
insert into info values (50,11)

*** (1) HOLDS THE LOCK(S):
RECORD LOCKS space id 26 page no 5 n bits 80 index uk_name of table `apple`.`info` trx id 37616 lock mode S waiting
Record lock, heap no 7 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 8000000b; asc     ;;
 1: len 4; hex 80000028; asc    (;;


*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 26 page no 5 n bits 80 index uk_name of table `apple`.`info` trx id 37616 lock mode S waiting
Record lock, heap no 7 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 8000000b; asc     ;;
 1: len 4; hex 80000028; asc    (;;


*** (2) TRANSACTION:
TRANSACTION 37615, ACTIVE 24 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1192, 2 row lock(s), undo log entries 2
MySQL thread id 15, OS thread handle 140737024022272, query id 653 127.0.0.1 root update
insert into info values (60,8)

*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 26 page no 5 n bits 80 index uk_name of table `apple`.`info` trx id 37615 lock_mode X locks rec but not gap
Record lock, heap no 7 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 8000000b; asc     ;;
 1: len 4; hex 80000028; asc    (;;


*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 26 page no 5 n bits 80 index uk_name of table `apple`.`info` trx id 37615 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 7 PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 0: len 4; hex 8000000b; asc     ;;
 1: len 4; hex 80000028; asc    (;;

*** WE ROLL BACK TRANSACTION (1)
------------
TRANSACTIONS
------------

檢視錶結構

greatsql> show create table info \G
*************************** 1. row ***************************
       Table: info
Create Table: CREATE TABLE `info` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` int NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_name` (`name`)
) ENGINE=InnoDB AUTO_INCREMENT=61 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
1 row in set (0.01 sec)

梳理 innodb status 日誌

整理如下：

事務	T1	T2
操作	insert into info values (50,11)	insert into info values (60,8)
關聯的物件	表apple.info的唯一索引 uk_name	表apple.info的唯一索引 uk_name
持有的鎖	lock mode S waitingheap no 7 11，40（十六進位制為8，28）	lock_mode X locks rec but not gapheap no 7 11，40（十六進位制為8，28）
等待的鎖	lock mode S waitingheap no 7 11，40（十六進位制為8，28）	lock_mode X locks gap before rec insert intention waitingheap no 7 11，40（十六進位制為8，28）

首先事務T2獲取到了uk_name中記錄11的 lock x,rec not not gap 鎖
事務T1嘗試獲取uk_name中記錄11的lock s, next key lock，由於T2持有了記錄的獨佔鎖，因此被T1堵塞
事務T2嘗試獲取uk_name中記錄11的lock x, gap before rec,insert intention，但被堵塞

獲取業務歷史SQL語句

透過系統表方式

透過performance_schema.threads、performance_schema.events_statements_history、performance_schema.events_statements_history_long等系統表獲取歷史SQL

根據GreatSQL thread id獲得執行緒id

greatsql> select PROCESSLIST_ID,THREAD_ID,THREAD_OS_ID from  performance_schema.threads where processlist_id in (15,16);
+----------------+-----------+--------------+
| PROCESSLIST_ID | THREAD_ID | THREAD_OS_ID |
+----------------+-----------+--------------+
|             15 |        61 |         5714 |
|             16 |        62 |         5719 |
+----------------+-----------+--------------+
2 rows in set (0.00 sec)

根據執行緒id獲得執行緒歷史SQL

greatsql> select THREAD_ID,EVENT_ID,CURRENT_SCHEMA,SQL_TEXT,MESSAGE_TEXT,EVENT_NAME,SOURCE from performance_schema.events_statements_history where thread_id in (61,62) order by THREAD_ID,EVENT_ID;
+-----------+----------+----------------+---------------------------------+--------------------------------------------------------------------+--------------------------+---------------------------------+
| THREAD_ID | EVENT_ID | CURRENT_SCHEMA | SQL_TEXT                        | MESSAGE_TEXT                                                       | EVENT_NAME               | SOURCE                          |
+-----------+----------+----------------+---------------------------------+--------------------------------------------------------------------+--------------------------+---------------------------------+
|        61 |     3762 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     3807 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     3852 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     3897 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     3942 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     3987 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     4032 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        61 |     4077 | apple          | begin                           | NULL                                                               | statement/sql/begin      | init_net_server_extension.cc:94 |
|        61 |     4100 | apple          | insert into info values (40,11) | NULL                                                               | statement/sql/insert     | init_net_server_extension.cc:94 |
|        61 |     4569 | apple          | insert into info values (60,8)  | NULL                                                               | statement/sql/insert     | init_net_server_extension.cc:94 |
|        62 |     3215 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3260 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3305 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3350 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3395 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3440 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3485 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3530 | apple          | NULL                            | NULL                                                               | statement/com/Field List | init_net_server_extension.cc:94 |
|        62 |     3575 | apple          | begin                           | NULL                                                               | statement/sql/begin      | init_net_server_extension.cc:94 |
|        62 |     3598 | apple          | insert into info values (50,11) | Deadlock found when trying to get lock; try restarting transaction | statement/sql/insert     | init_net_server_extension.cc:94 |
+-----------+----------+----------------+---------------------------------+--------------------------------------------------------------------+--------------------------+---------------------------------+
20 rows in set (0.00 sec)

觀察show engine innodb status中的GreatSQL thread id 16和GreatSQL thread id 15
透過performance_schema.threads獲取THREAD_ID
透過performance_schema.events_statements_history獲取THREAD_ID執行的歷史SQL以及執行時間

最終可復現出如下業務SQL：

事務	T1	T2
語句	begin;	begin;
語句		insert into info values (40,11);
語句	insert into info values (50,11);
語句		insert into info values (60,8);

透過解析binlog

$ mysqlbinlog -vv --base64-output=decode-rows bin.000030

SET @@SESSION.GTID_NEXT= 'e319a624-b2ce-11ee-9aac-00163e62ca8a:8696'/*!*/;
# at 10314
#240128 16:52:35 server id 1024  end_log_pos 10390 CRC32 0x59edb313         Query        thread_id=18        exec_time=0        error_code=0
SET TIMESTAMP=1706431955/*!*/;
BEGIN
/*!*/;
# at 10390
#240128 16:52:35 server id 1024  end_log_pos 10442 CRC32 0xc03dea61         Table_map: `apple`.`info` mapped to number 370
# at 10442
#240128 16:52:35 server id 1024  end_log_pos 10486 CRC32 0x670e0c66         Write_rows: table id 370 flags: STMT_END_F
### INSERT INTO `apple`.`info`
### SET
###   @1=30 /* INT meta=0 nullable=0 is_null=0 */
###   @2=30 /* INT meta=0 nullable=0 is_null=0 */
# at 10486
#240128 16:52:35 server id 1024  end_log_pos 10517 CRC32 0xab4e0d89         Xid = 598
COMMIT/*!*/;
# at 10517
#240128 19:22:12 server id 1024  end_log_pos 10596 CRC32 0x4f4cf08e         GTID        last_committed=30        sequence_number=36        rbr_only=yes        original_committed_timestamp=1706440932450590        immediate_commit_timestamp=1706440932450590 transaction_length=378
/*!50718 SET TRANSACTION ISOLATION LEVEL READ COMMITTED*//*!*/;
# original_commit_timestamp=1706440932450590 (2024-01-28 19:22:12.450590 CST)
# immediate_commit_timestamp=1706440932450590 (2024-01-28 19:22:12.450590 CST)
/*!80001 SET @@session.original_commit_timestamp=1706440932450590*//*!*/;
/*!80014 SET @@session.original_server_version=80026*//*!*/;
/*!80014 SET @@session.immediate_server_version=80026*//*!*/;
SET @@SESSION.GTID_NEXT= 'e319a624-b2ce-11ee-9aac-00163e62ca8a:8697'/*!*/;
# at 10596
#240128 16:53:16 server id 1024  end_log_pos 10672 CRC32 0xf222c003         Query        thread_id=15        exec_time=0        error_code=0
SET TIMESTAMP=1706431996/*!*/;
BEGIN
/*!*/;
# at 10672
#240128 16:53:16 server id 1024  end_log_pos 10724 CRC32 0x20cb8c86         Table_map: `apple`.`info` mapped to number 370
# at 10724
#240128 16:53:16 server id 1024  end_log_pos 10768 CRC32 0xd8f53958         Write_rows: table id 370 flags: STMT_END_F
### INSERT INTO `apple`.`info`
### SET
###   @1=40 /* INT meta=0 nullable=0 is_null=0 */
###   @2=11 /* INT meta=0 nullable=0 is_null=0 */
# at 10768
#240128 16:53:40 server id 1024  end_log_pos 10820 CRC32 0x23f22580         Table_map: `apple`.`info` mapped to number 370
# at 10820
#240128 16:53:40 server id 1024  end_log_pos 10864 CRC32 0x182ecdef         Write_rows: table id 370 flags: STMT_END_F
### INSERT INTO `apple`.`info`
### SET
###   @1=60 /* INT meta=0 nullable=0 is_null=0 */
###   @2=8 /* INT meta=0 nullable=0 is_null=0 */
# at 10864
#240128 19:22:12 server id 1024  end_log_pos 10895 CRC32 0x57fd1d3c         Xid = 650
COMMIT/*!*/;
SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/;
DELIMITER ;
# End of log file
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;

根據binlog中部分SET @@SESSION.GTID_NEXT= 'e319a624-b2ce-11ee-9aac-00163e62ca8a:8697'該GTID的事務資訊，可恢復T2，但T1執行的語句由於被回滾了，則不會記錄到binlog，可開啟general log日誌獲取排查

事務	T1	T2
語句	begin;	begin;
語句		insert into info values (40,11);
語句	insert into info values (50,11);
語句		insert into info values (60,8);

分析死鎖

T1、T2開啟了一個事務
隨後T2執行了插入(40,11)的insert語句：insert into info values (40,11)
T1執行了插入(50,11)的insert語句：insert into info values (50,11) 進行唯一性衝突檢查，嘗試獲取LOCK_S
然後T1所在的連線會將T2中的隱式鎖轉換為顯示鎖，此時T2將獲取Lock X, Rec_not_gap。由於T2的Lock X, Rec_not_gap與T1的LOCK S不相容，因此T1被堵塞
隨後，T2又執行了(60,8)的insert語句：insert into info values (60,8) 由於其插入的唯一索引值是8，因此不存在主鍵衝突，直接執行樂觀插入操作。執行樂觀插入時，需要檢查其它事務是否堵塞insert操作。其核心是獲取待插入記錄的下一個值（這裡剛好是10），並獲取該記錄上的所有鎖，與需要新增的鎖判斷是否存在衝突。
T1持有了記錄11的LOCK_S鎖與T2的LOCK_X、LOCK_INSERT_INTENTION不相容，因此T2被T1堵塞
死鎖形成。

解決

• 適當的減少Unique索引

• 避免插入重複的值（唯一索引所在列）

Enjoy GreatSQL 😃

關於 GreatSQL

GreatSQL是適用於金融級應用的國內自主開源資料庫，具備高效能、高可靠、高易用性、高安全等多個核心特性，可以作為MySQL或Percona Server的可選替換，用於線上生產環境，且完全免費併相容MySQL或Percona Server。

相關連結： GreatSQL社群 Gitee GitHub Bilibili

GreatSQL社群：

社群部落格有獎徵稿詳情：https://greatsql.cn/thread-100-1-1.html

技術交流群：

微信：掃碼新增GreatSQL社群助手微信好友，傳送驗證資訊加群。

解決Oracle死鎖問題步驟
2018-06-20
Oracle
Mysql使用kill命令解決死鎖問題
2020-03-31
MySql
死鎖問題排查過程-間隙鎖的復現以及解決
2022-07-10
手把手教你分析解決MySQL死鎖問題
2022-01-16
MySql
記一次 MySQL select for update 死鎖問題
2020-04-03
MySql
Python | 多執行緒死鎖問題的巧妙解決方法
2020-08-25
Python執行緒
MySQL死鎖問題
2022-10-21
MySql
記憶體混亂及解決方法和死鎖問題
2024-08-28
記憶體
MySQL解決死鎖
2024-06-09
MySql
MySQL 死鎖解決
2019-02-28
MySql
線上問題排查：記一次 Redis Cluster Pipeline 導致的死鎖問題
2023-12-04
Redis
MySQL 死鎖問題分析
2020-04-09
MySql
MySQL死鎖系列-線上死鎖問題排查思路
2020-10-19
MySql
面試：什麼是死鎖，如何避免或解決死鎖；MySQL中的死鎖現象，MySQL死鎖如何解決
2024-11-10
面試MySql
一次徹底講清如何處理mysql 的死鎖問題
2024-10-17
MySql
MySQL鎖等待與死鎖問題分析
2021-03-30
MySql
SpringBoot Seata 死鎖問題排查
2023-12-06
Spring Boot
一文讀懂DNS解析故障的排查思路，輕鬆解決解析常見問題
2023-12-15
DNS
故障分析 | MySQL死鎖案例分析
2023-01-05
MySql
解決庫存扣減及訂單建立時防止併發死鎖的問題
2022-05-26
一次詭異的線上資料庫的死鎖問題排查過程
2019-04-09
資料庫
Oracle死鎖一例(ORA-00060)，鎖表導致的業務死鎖問題
2018-07-05
Oracle
Python | 淺談併發鎖與死鎖問題
2020-08-14
Python
MySQL死鎖分析與解決之路
2022-02-16
MySql
解決DNS解析故障的幾種方法
2022-02-11
DNS
併發技術5：死鎖問題
2018-11-16
什麼是死鎖？如何解決死鎖？
2020-08-11
一次死鎖導致CPU異常飄高的整個故障排查過程
2021-04-23
Docker啟動故障問題 no such file or directory解決方法
2022-09-23
Docker
常見的死鎖情況及解決方法
2018-06-06
mysql慢查詢，死鎖解決方案
2018-06-11
MySql
記錄一次解決App崩潰問題的解決方案
2021-11-22
APP
記一次 Composer 問題的解決過程！！
2018-08-07
一次 nginx 返回 302 問題解決
2020-05-24
Nginx
記一次神奇的Mysql死鎖排查
2019-02-27
MySql
用分散式鎖解決併發問題
2019-03-13
分散式
Mysql(MyISAM)的讀寫互斥鎖問題的解決方法
2021-09-09
MySql
AdornerDecorator的CacheMode繫結和windows鎖屏導致TableControl鎖死問題
2024-08-29
Windows