Error creating bean with name 'memcachedClient'...java.lang.OutOfMemoryError

1，Tomcat啟動報錯如下：

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memcachedClient' defined in 

file [/usr/local/apache-tomcat-6.0.37_6500/webapps/trade_service/WEB-INF/classes/META-INF/spring/springContext_memcached.xml]: 

Instantiation of bean failed; nested exception is org.springframework.beans.factory.BeanDefinitionStoreException: Factory method 

[public net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] 

threw exception; nested exception is java.lang.OutOfMemoryError
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod

(ConstructorResolver.java:581)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod

(AbstractAutowireCapableBeanFactory.java:1015)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance

(AbstractAutowireCapableBeanFactory.java:911)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean

(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean

(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:294)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton

(DefaultSingletonBeanRegistry.java:225)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:291)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference

(BeanDefinitionValueResolver.java:322)
        ... 44 more
Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public 

net.rubyeye.xmemcached.MemcachedClient net.rubyeye.xmemcached.XMemcachedClientBuilder.build() throws java.io.IOException] threw 

exception; nested exception is java.lang.OutOfMemoryError
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate

(SimpleInstantiationStrategy.java:169)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod

(ConstructorResolver.java:570)
        ... 53 more
Caused by: java.lang.OutOfMemoryError

,

2，看到建立memcache clent報錯，馬上去memcache伺服器檢視一下，memcache正常執行，記憶體也足夠剩餘。

[root@localhost ~]# tail -f /tmp/memcache_error.log 
<287    0x00 0x00 0x00 0x00
<287    0x00 0x00 0x00 0x00
<287    0x00 0x00 0x00 0x00
>287 Writing bin response:
>287   0x81 0x0b 0x00 0x00
>287   0x00 0x00 0x00 0x00
>287   0x00 0x00 0x00 0x06
>287   0x00 0x00 0x00 0x00
>287   0x00 0x00 0x00 0x00
>287   0x00 0x00 0x00 0x00
...
[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          5938       3368       2570          0        195        296
-/+ buffers/cache:       2876       3061
Swap:         7935          0       7935
[root@localhost ~]# 

問題在哪裡？再看看日誌裡面有誤error或者warning資訊：

[root@localhost ~]# cat  /tmp/memcache_error.log  |grep warning
[root@localhost ~]# cat  /tmp/memcache_error.log  |grep error
[root@localhost ~]#

一點錯誤資訊也沒有，判斷問題不是出在memcache這一塊。

3，去check tomcat伺服器

[root@localhost logs]# free -m
             total       used       free     shared    buffers     cached
Mem:         18022      16943       1078          0        104        675
-/+ buffers/cache:      16162       1859
Swap:        20031       8012      12019
[root@localhost logs]# 

只剩餘1個G的記憶體，去修改一下tomcat的記憶體配置：

[root@localhost bin]# vim catalina.sh

#JAVA_OPTS="-Xms2048m -Xmx2048m -Xss1024K -XX:PermSize=256m -XX:MaxPermSize=768m"
# add by wangqi on 20140614
 JAVA_OPTS="-Xms512m -Xmx512m -Xss128K -XX:PermSize=64m -XX:MaxPermSize=256m"

再重啟tomcat，看到還是報一樣的錯誤。

4，回到應用本身去查問題

跟開發人員一起排查，看到是trade_service啟動了過多的多餘的事務，把tomcat的記憶體耗光了，所以關閉掉多餘的事務，再啟動tomcat，就不會再報錯。

感慨：就跟大部分web應用卡，頁面登入不上去，相應過慢很多是出在效率低下的sql上面，tomcat啟動不起來，記憶體溢位大部分也是在java程式碼上面，大家需要寫程式碼或者sql需要謹慎，且寫且謹慎。