WLS 10.3.0 更新發布應用異常終止處理一例

單位上新上一應用在weblogic控制檯進行應用程式更新發布出現
java.lang.IllegalArgumentException Registered more than one instance異常，然後
weblogic就異常退出.

檢視AdminServer.log日誌可以看到如下資訊:
####<2014-7-10 下午04時28分11秒 CST> <> <> <1404980891082>
####<2014-7-10 下午04時28分21秒 CST> <> <> <1404980901145>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901148>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901148>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901149>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901173> < [CompressingFilter/1.4.6] CompressingFilter is being destroyed...>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901176>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901177>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901177>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901178> weblogic.work.RequestClassRuntimeMBeanImpl@3702476f>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901178> weblogic.work.WorkManagerRuntimeMBeanImpl@12ab76c4>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901190> java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=AdminServer,Name=default,ApplicationRuntime=hninsiis,Type=WorkManagerRuntime new:weblogic.work.WorkManagerRuntimeMBeanImpl@2109f3fb existing weblogic.work.WorkManagerRuntimeMBeanImpl@12ab76c4
 at weblogic.management.jmx.ObjectNameManagerBase.registerObject(ObjectNameManagerBase.java:168)
 at weblogic.management.mbeanservers.internal.WLSObjectNameManager.lookupObjectName(WLSObjectNameManager.java:131)
 at weblogic.management.jmx.modelmbean.WLSModelMBeanFactory.registerWLSModelMBean(WLSModelMBeanFactory.java:87)
 at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.registered(RuntimeMBeanAgent.java:104)
 at weblogic.management.provider.core.RegistrationManagerBase.invokeRegistrationHandlers(RegistrationManagerBase.java:180)
 at weblogic.management.provider.core.RegistrationManagerBase.register(RegistrationManagerBase.java:110)
 at weblogic.management.runtime.RuntimeMBeanDelegate.register(RuntimeMBeanDelegate.java:317)
 at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:255)
 at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
 at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:193)
 at weblogic.work.WorkManagerRuntimeMBeanImpl.(WorkManagerRuntimeMBeanImpl.java:49)
 at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:59)
 at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
 at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
 at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
 at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
 at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
 at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
 at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
 at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
 at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
 at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
 at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
 at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
 at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
 at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
 at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
 at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
>
####<2014-7-10 下午04時28分21秒 CST> <> <> <> <1404980901192> java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=AdminServer,Name=default@hninsiis@null,WorkManagerRuntime=default,ApplicationRuntime=hninsiis,Type=RequestClassRuntime new:weblogic.work.RequestClassRuntimeMBeanImpl@1470f4af existing weblogic.work.RequestClassRuntimeMBeanImpl@3702476f
 at weblogic.management.jmx.ObjectNameManagerBase.registerObject(ObjectNameManagerBase.java:168)
 at weblogic.management.mbeanservers.internal.WLSObjectNameManager.lookupObjectName(WLSObjectNameManager.java:131)
 at weblogic.management.jmx.modelmbean.WLSModelMBeanFactory.registerWLSModelMBean(WLSModelMBeanFactory.java:87)
 at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.registered(RuntimeMBeanAgent.java:104)
 at weblogic.management.provider.core.RegistrationManagerBase.invokeRegistrationHandlers(RegistrationManagerBase.java:180)
 at weblogic.management.provider.core.RegistrationManagerBase.register(RegistrationManagerBase.java:110)
 at weblogic.management.runtime.RuntimeMBeanDelegate.register(RuntimeMBeanDelegate.java:317)
 at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:255)
 at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
 at weblogic.work.RequestClassRuntimeMBeanImpl.(RequestClassRuntimeMBeanImpl.java:32)
 at weblogic.work.WorkManagerRuntimeMBeanImpl.getRequestClassRuntime(WorkManagerRuntimeMBeanImpl.java:86)
 at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:61)
 at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
 at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
 at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
 at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
 at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
 at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
 at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
 at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
 at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
 at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
 at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
 at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
 at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
 at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
 at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
 at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
 at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

從上面的錯誤資訊可以看到
 
這說明是在更新應用程式重新載入
java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=AdminServer,Name=default@hninsiis@null,WorkManagerRuntime=default,ApplicationRuntime=hninsiis,Type=RequestClassRuntime new:weblogic.work.RequestClassRuntimeMBeanImpl@1470f4af existing weblogic.work.RequestClassRuntimeMBeanImpl@3702476f
這個資訊是說在載入應用程式時已經存在一個同名的應用程式，而實際上並不存在同名的應用程式

這時只能使用無所不能的MOS了,在MOS上找到了關於這個錯誤資訊的文件
WLS 10.3.0: java.lang.IllegalArgumentException: Existing Timer Manager Has Different Work Manager (文件 ID 1097661.1)

In this Document
Symptoms
Cause
Solution
References

APPLIES TO:

Oracle Weblogic Server - Version 10.3 to 10.3
Information in this document applies to any platform.
***Checked for relevance on 2-May-2013***
SYMPTOMS

When trying to redeploy an application that uses a foreign JMS queue, the error below is shown. The server needs to be rebooted with every new deployment. Otherwise, the server fails consistently with the error: "Existing timer manager has different work manager."

### <> <> <> <1272310540224> java.lang.IllegalArgumentException: Existing timer manager has different work manager.
TimerManager requested: JMSPoller-TradeAlertsMessageBean
TimerManager obtained: JMSPoller-TradeAlertsMessageBean
WorkManager requested: weblogic.timers.internal.TimerManagerFactoryImpl$WorkManagerExecutor@9c25d2
WorkManager obtained: weblogic.timers.internal.TimerManagerFactoryImpl$WorkManagerExecutor@1770caa>
CAUSE

This issue is caused by a flaw in the WLS code such that all MDB instances/pollers on the same WLS server of the same MDB deployment share the same timer manager for message polling. When one instance is disconnected due to JMS server migration, the timer manager is stopped, which in turn stops other MDB instances on the same server. This issue is addressed by unpublished defect 7669814.

SOLUTION

Patches are available for unpublished defect 7669814:

PATCH INFORMATION
WLS Version Patch Number
10.3.0 Patch 7669814
Fixed in: 10.3.1

To apply one of these patches, click on the link for your WLS version and download the appropriate patch from My Oracle Support. You can also search in My Oracle Support for the patch number for your WLS version: for detailed instructions, please see Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Document 1302053.1. For instructions on how to apply these patches to your system, please see How to Apply WebLogic Server (WLS) Patches Using Smart Update Document 876004.1. For other issues relating to Smart Update, please see Master Note on Troubleshooting Smart Update Issues Document 1230725.1.

Patches are specifically tied to a particular release and maintenance pack of WebLogic Server (WLS). A patch for WLS 10.3.3, for example, very likely would not work on WLS 10.3.5. In a few cases, patches are also specific to particular operating systems. If you think you are experiencing the problem outlined in this note, and you are running a WLS version which is eligible for error correction (see Document 950131.1 for more about the Oracle Error Correction Policy), but your WLS version is not included in the list of patches provided here, please contact support and ask for a version of the patch appropriate for you. Please reference this note as well as this bug number to help speed our service for you.

NOTE: Patches are applied per WLS installation and not per domain. That is, if you apply this patch on one WLS installation, then all of the servers from all the domains in that installation will have this patch. On the other hand, if you have a managed server in another machine in a domain (that is, set up with its own WLS installation), you need to install this patch on that other machine as well. Generally, patches can only be applied while the server is not running because WLS locks the needed files while it is running. If, however, you are able to apply a patch while WLS is running, you must restart WLS before the patch will take effect.

 

從上面的資訊說已經在10.3.1這個版本已經修復這個bug了.但我們的weblogic版本是10.3.3.0
[root@sx-weblogic31 lib]# java -cp weblogic.jar weblogic.version

WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401

Use 'weblogic.version -verbose' to get subsystem information

Use 'weblogic.utils.Versions' to get version information for all modules

 

其實給weblogic打補丁提供了兩種方法
1. Using Smart Update

You can apply the patch using Smart Update with the following steps:

Download the patch from My Oracle Support (MOS). For more details, please refer to Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Note 1302053.1.
Extract the contents from the zip file: you will have a jar file and patch-catalog_xxx.xml. A readme file may also be included.
Copy the files (for example, E5W8.jar and WGQJ.jar) and the patch-catalog_xxx.xml from the zip file to the target machine. You do not need the readme file. Copy the files to the appropriate cache_dir directory for the target system: for example, on Windows, %MIDDLEWARE_HOME%\utils\bsu\cache_dir, or on UNIX, $MIDDLEWARE_HOME/utils/bsu/cache_dir. The directory MW_HOME\utils\bsu\cache_diris created as the default patch download directory when you install Smart Update 3.3.0. (see )

NOTE: Always copy the patch-catalog_xxx.xml file from the downloaded patch to the cache_dir along with the patch itself. Do NOTrename this file. If you see an "unrecognized patch ID" error, please refer to Note 1186923.1 for details on how to resolve it.
Run Smart Update and apply the patches and/or patch sets to the target system. This can be done using the Smart Update GUI or the command-line interface (see ).
a. Smart Update in graphical (GUI) mode

Run the /utils/bsu/bsu script (bsu.sh for UNIX, bsu.cmd for Windows). This will start the Smart Update GUI.
Look for the patches you copied in the "Downloaded Patches" section at the bottom.
Select the "Apply" button for each patch you want to apply. This will validate the patch and apply it to the whole installation.
The following viewlet provides an example of using Smart Update in GUI mode:

 Video - Applying a Patch Using Smart Update in GUI Mode (1:15) Trouble seeing this video?

b. Command-line interface

This is the syntax for the command to view the downloaded patches as below:
./bsu.sh -prod_dir= -patch_download_dir= -status=downloaded -view -verbose
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir= -patchlist= -verbose -install
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8 -verbose -install
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8,WGQJ -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir= -patch_download_dir= -status=applied -verbose -view
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
2. Applying the patch to the classpath manually

You can apply the patch to the system manually by extracting the actual patch and adding it to the classpath on the system:

Extract the actual patch jar file. It will be in the form .jar (for example: E5W8.jar). Inside this jar file is the actual patch jar file, which will be of the form CR326566_92mp3.jar. Extract the latter file for the following steps.
Add the extracted jar file as the first element of the classpath of the Admin server as well as the managed servers in the domain.
If you are starting servers using the WebLogic Server startup script, update the classpath in the startup script like this:
set CLASSPATH=\jars\CR326566_92mp3.jar;%CLASSPATH% (Windows)
CLASSPATH=/jars/CR326566_92mp3.jar:$CLASSPATH (UNIX)
where PATCH_DIR is the directory on the machine where you extracted/saved the patch file.
Similarly, if you are starting servers using Node Manager, add the patch jar to the beginning of the Class Path argument in the Server Start tab for the server(s).
NOTE: Applying the patch to the classpath manually (approach 2) is recommended only for releases prior to WLS 9.1. Smart Update should be used when it is available as it provides patch conflict and dependency checking.
REFERENCES

上面的第一種方法是要連線網路透過GUI介面或者./bsu.sh的命令列介面來進行但是這也是一種使用不了的方法因為很少有人把weblogic伺服器置於連網狀態.
所以還是使用第二種方法先下補丁Patch 7669814,然後將補丁包中的.jar檔案複製到weblogic伺服器主機上然後將.jar包新增到classpath環境變數中.

補丁Patch 7669814包中的檔案為KG6I.jar將其複製到weblogic的lib目錄下
[root@sx-weblogic32 lib]# ls -lrt
total 28
-rw-r----- 1 root root   702 Mar 29  2011 readme.txt
-rw-r--r-- 1 root root 23302 Jun 10  2011 KG6I.jar

下面就需要修改weblogic的啟動指令碼將KG6I.jar檔案載入到classpath環境變數中然後再重新啟動weblogic
在startWebLogic.sh中增加下面的記錄
CLASSPATH=/bea11/user_projects/domains/mydomain/lib/KG6I.jar:$CLASSPATH

在重新啟動weblogic之後再來更新應用程式
####<2014-7-11 上午08時54分05秒 CST> <> <> <> <1405040045244>
####<2014-7-11 上午08時54分05秒 CST> <> <> <> <1405040045438>
####<2014-7-11 上午08時54分05秒 CST> <> <> <> <1405040045474>
####<2014-7-11 上午08時54分05秒 CST> <> <> <> <1405040045476>
####<2014-7-11 上午08時54分05秒 CST> <> <> <> <1405040045730> < [CompressingFilter/1.4.6] CompressingFilter has initialized>
####<2014-7-11 上午08時54分14秒 CST> <> <> <> <1405040054094>
####<2014-7-11 上午08時54分14秒 CST> <> <> <> <1405040054094>
####<2014-7-11 上午08時54分14秒 CST> <> <> <> <1405040054111>