ERROR 1135 (HY000): Can't create a new thread

賀子_DBA時代發表於2017-03-17
Errorthread
收到報錯：
[root@i-iivphroy ~]# mysql -uroot -p********* -h192.168.0.254
ERROR 1135 (HY000): Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
馬上google一番，有人說可能說磁碟空間滿了，經檢視發現果然是磁碟滿了，（這個盤一直空間很緊張（95%），我高度警惕著，天天檢查，可是昨天我執行了個大事務，產生了大量的binlog，給一下子撐爆了）
馬上刪除了幾天前的binlog和一些別的不需要的資料，空間釋放到了80%，再次登入mysql
[root@i-iivphroy ~]# mysql -uroot -p******* -h192.168.0.254
依舊報錯：
ERROR 1135 (HY000): Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
再次google一番，查到下面這個文件：
This error was the bane of my life for a while, and it was very hard to get a definitive answer as to what was causing it, I hope this saves you some trouble.
My website occasionally got large traffic spikes, and at the top of these peaks, I would start to see errors like these:
MySQL error #1135: Can’t create a new thread (errno 11). If you are not out of available memory, you can consult the manual for a possible OS-dependent bug.
I looked in the my.cnf file on the db server and looked at the open files limit, because a process is counted as an open file, but it seemed fine:
[mysqld_safe]
open-files-limit=10240
I also checked that maximum connections was high enough, it was at 2048.
What the open-files-limit in my.cnf files does is it tells the init script to use ulimit to whatever number you put in there.
After a lot of digging around various places, and much frustration, I discovered that by default linux has a hard limit of 1024 open files for all non super-users, so even though I had set a high open-files-limit, it was capped at 1024 by the OS. I also discovered how to raise it;
/etc/security/limits.conf
This file is used by PAM to set things like maximum processes, max open files, memory usage etc and these limits can be set on a per-user basis, so I added these lines:
mysql soft nofile 4096
mysql hard nofile 4096
大體的意思是說，這個報錯的原因：由於：mysql的配置檔案/etc/my.cnf的引數open-files-limit設定的比linux的max user processes的數值大，需要透過修改linux的配置檔案 /etc/security/limits.d/90-nproc.conf來擴大linux系統的限制，也就是這個錯是由於linux的max user processes閾值太小了。
馬上檢視我的相關配置：
mysql的open-files-limit，如下所示：
[root@i-iivphroy ~]# cat /etc/my.cnf
[mysqld_safe]
open-files-limit=85216
linux的max user processes，如下所示紅色部分：
[root@i-iivphroy ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62842
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 62842
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
將檢視果然是前面文件描述的情況，馬上修改max user processes
方法一：
[root@i-iivphroy ~]# cat /etc/security/limits.conf
* soft nofile 65535
* hard nofile 65535
* soft nproc 65535
* hard nproc 65535
其中nofile對應open_files
nproc對應max_user_processes
但是在 6.4之後，如果只修改了該檔案中的nproc，那麼其他非root使用者對應的max_user_processes並不會改變，仍然是1024，這個是因為受到了下面這個檔案的影響
/etc/security/limits.d/90-nproc.conf
修改/etc/security/limits.d/90-nproc.conf將
* soft nproc 1024
修改為：
* soft nproc 65535
或者
修改/etc/security/limits.conf，將
* soft nofile 10240
修改為
  soft nofile 10240
方法二：這樣為每一個執行bash shell的使用者執行此檔案，當bash shell被開啟時,該檔案被讀取。也就是說，當使用者shell執行了bash時，執行這個檔案，如果這個伺服器上有多個使用者，最好是用方法一。
修改了/etc/bashrc，成功了，並且不用重啟。
vi /etc/bashrc
新增 ：
ulimit -u 65535
退出session,從新開session再次ulimit -a 發現已經變化了
[root@i-iivphroy ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62842
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
並且把mysql的open-files-limit改小。
[root@i-iivphroy ~]# cat /etc/my.cnf
[mysqld_safe]
open-files-limit=65000
重啟了mysql服務，問題解決。。。。
原因分析：
連線數太小。（比如centos 6 預設的  max user process只有 1024個。當mysql process大於這個值時 就會出現Can't create a new thread的問題）
連線數超限處理的辦法：
ulimit -a
檢視max user processes 這一項
要是這個值比較的小 當mysql中process的數目超過這個數的時候 就會抱標題相應的錯誤。
一個過程process可以視為一個開啟的檔案
也就是說 下面幾個引數共同控制這mysql的 create a new thread
1） mysql的 /etc/my.cnf
open-files-limit=65535
2）linux 引數 open files和max user processes
[root@S243 ~]# ulimit
unlimited
[root@S243 ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1032207
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 65535
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
總結：本來從來沒有報錯這個錯的mysql資料庫，怎麼突然就被linux的max user processes所限制呢？很顯然是因為我的磁碟滿了，導致mysql無法提供正常服務，但是業務還在不斷的請求資料庫，process在佇列中排隊，等我資料庫可以提供服務的時候，大量的排隊中的process在mysql中產生，導致process資料量急劇增加，超過linux的max user processes的閾值，報出錯誤。

來自 “ ITPUB部落格 ” ，連結：http://blog.itpub.net/29654823/viewspace-2135578/，如需轉載，請註明出處，否則將追究法律責任。

相關文章