Weblogic BEA-002616 java.io.IOException: Too many open files

某業務系統不能訪問，weblogic控制檯登入不了，檢視系統資源發現CPU使用率100%

[root@app ~]# top
top - 11:38:40 up 17 days, 20:41,  1 user,  load average: 7.99, 7.82, 7.63
Tasks: 171 total,   1 running, 170 sleeping,   0 stopped,   0 zombie
Cpu(s):100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16436780k total, 13225904k used,  3210876k free,   275464k buffers
Swap: 16779884k total,        0k used, 16779884k free,  2676416k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5419 root      25   0 10.5g 9.6g  43m S 791.8 61.0   2549:30 java
 7931 root      15   0 12740 1152  820 R  0.3  0.0   0:00.01 top
    1 root      15   0 10348  692  584 S  0.0  0.0   0:01.99 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.08 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.13 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.30 ksoftirqd/1
    6 root      RT  -5     0    0    0 S  0.0  0.0   0:00.17 migration/2
    7 root      34  19     0    0    0 S  0.0  0.0   0:00.30 ksoftirqd/2
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/3
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.28 ksoftirqd/3
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/4
   11 root      34  19     0    0    0 S  0.0  0.0   0:00.28 ksoftirqd/4
   12 root      RT  -5     0    0    0 S  0.0  0.0   0:00.14 migration/5
   13 root      34  19     0    0    0 S  0.0  0.0   0:00.27 ksoftirqd/5
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/6
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.31 ksoftirqd/6
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.70 migration/7
   17 root      34  19     0    0    0 S  0.0  0.0   0:00.21 ksoftirqd/7
   18 root      10  -5     0    0    0 S  0.0  0.0   7:31.65 events/0
   19 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/1
   20 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/2
   21 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 events/3
   22 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/4
   23 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 events/5
   24 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/6
   25 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 events/7
   26 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
  163 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
  174 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
  175 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/1
  176 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/2
  177 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/3
  178 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/4
  179 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/5
  180 root      10  -5     0    0    0 S  0.0  0.0   0:00.04 kblockd/6
  181 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/7

檢視weblogic日誌找到發下資訊:

[root@app1 base_domain]# more weblogic.log
<Dec 15, 2016 9:09:19 AM CST> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 10.138.130.61:7001, failure count: 1, failing for 0 seconds, java.io.IOException: Too many open files>

問題原因是因為對伺服器配置的允許開啟檔案的最大數量比weblogic所需要的數量小而造成的。
解決方法:
1.設定 ulimit -u 65535
2.修改/etc/security/limits.conf在檔案末加上
soft nofile 65535
hard nofile 65535

參考:System Crashes With "java.net.SocketException: Too many open files" Error (Doc ID 1194575.1)