Sample Header Ad - 728x90

Weblogic process just killed suddenly on redhat linux

1 vote
1 answer
5742 views
One of the process on my WAS killed at dawn. 6 process run on this server. But only one process just killed. I don't know how to fix it. This is Weblogic last log that before process killed. /app/orarecp/bea/domains/HRRECP/bin/startWeblogic.sh: line 180: 27066 killed ${JAVA_HOME}/bin/java ${JAVA_VM} {MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava-security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTION} ${PROXY_SETTINGS} ${SERVER_CLASS} And This is startWeblogic shell script. startWeblogic Shell Script: 180 line
And This is start process shell script parts. USER_MEM_ARGS="-Xms3072m -Xmx3072m -Xverbose:gc -Xverboselog:${LOG_HOME}/gclogs/gc.${SERVER_NAME}_date + '%Y%m%d_%H%M%S'.out" JAVA_OPTION="${JAVA_OPTIONS} -Xbootclasspath/p:/app/orarecp/jennifer/agent/lwst.jdk.jar:/app/orarecp/jennifer/agent/lwst.boot.jar:/app/orarecp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/orarecp/jennifer/agent/R11.conf -javaagent:/app/orarecp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom"

server specification is... - CPU: 8 core, 2.4GHz // The time that CPUs utilization was under 15% - Memory: 32Gb + Mem: 32922024k total, 32817492k used, 104580k free, 147464k buffers + Swap: 33551744k total, 148k used, 33551596k free, 15231864k cached
server stack is... - weblogic 11g - jrockit 1.6 - redhat linux 5.8 (64bit)

I wonder why the process is killed. What am I suppose to do? Please answer if you need additional information. All the answers are going to be helpful to me.
Thanks. ---------- Additinal Information. 1. before that Last log But this log is occurs frequently. > java.lang.NullPointerException at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) > at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) 2. another JAVA process (ps -ef | grep java) ps -ef | grep java 3. another Server Java Process >ps -ef | grep java > oraehrp 818 786 0 19:34 pts/1 00:00:00 grep java > root 9689 1 1 2015 ? 1-01:05:58 /usr/java/jdk1.6.0_23/bin/java -Djava.awt.headless=true -jar /app/CJ_agent//cjagent.jar /app/CJ_agent//conf/cjagent.conf > oraehrp 10128 10074 2 2015 ? 1-10:06:08 /usr/java/jrockit1.6/bin/java -jrockit -Xms1024m -Xmx1024m -Dweblogic.Name=EHRREP1 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server > oraehrp 14795 14740 2 2015 ? 1-10:31:48 /usr/java/jrockit1.6/bin/java -jrockit -DAdminServer -Xms1024m -Xmx1024m -verbosegc -Dweblogic.Name=AdminServer -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server > oraehrp 23456 23395 67 18:25 ? 00:46:23 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD11/gclogs/gc.EHRPRD11_20160201_182544.out -Dweblogic.Name=EHRPRD11 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E11.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server > oraehrp 24233 24179 49 18:27 ? 00:33:03 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD12/gclogs/gc.EHRPRD12_20160201_182747.out -Dweblogic.Name=EHRPRD12 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E12.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server > oraehrp 25006 24944 53 18:29 ? 00:34:35 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD13/gclogs/gc.EHRPRD13_20160201_182933.out -Dweblogic.Name=EHRPRD13 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehRP/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E13.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server
Asked by Chanho Jung (13 rep)
Feb 1, 2016, 07:14 AM
Last activity: Mar 17, 2018, 09:27 AM