Showing posts with label Troubleshooting. Show all posts
Showing posts with label Troubleshooting. Show all posts

6/16/14

Thread Dumps

Aften as process admin, we need to take thread dump.
Thread dump gives you what the jvm threads are doing, what are their status and so on info.

1. WebSphere
We can take WebSphere JVM thread by using this command
kill -3 jvmpid

This generates thread dump, by default dumps the threads to jvm/logs/jvmname/natviestdout
You can configure the jvm to generate generate file in your configured path.
2. Tomcat
/path/to/jdk/bin/jstack -l pid > nameoffile.log
You need to take thread dump as the owner of the pid. Better to use the same jdk as tomcat. As the owner of tomcat, you can say "which java" or ps -ef | grep -i java gives which java tomcat process pointed to.

3. WebLogic
kill -3 or jstack can be used.

[There are several ways. These are few]

7/14/13

Datasource password

This is a nice tool to decode the datasource password on the application server.
http://www.sysman.nl/WASDecoder/

7/11/13

java.io.IOException: Async IO operation failed (1), reason: RC: 10054

Here is a good link to solve the issue
java.io.IOException: Async IO operation failed (1), reason: RC: 10054

http://h30499.www3.hp.com/t5/Networking/10054-WSAECONNRESET-Connection-reset-by-peer/td-p/3436931#.Ud3JfaxGagx

4/12/10

Administrative console throws error 500 with java.lang.NullPointerExcetion

Last Sunday, one of my friend got this problem.

Admin Console, when accessed, throws error 500 with java.lang.NullPointerExcetion.
SystemOut.log has the following entries

5ea42ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/logon.jsp: init
5ea42ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/layouts/browser_detection.jsp: init
5ea02ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/console.jsp: init
5ea02ee7 WebGroup E SRVE0026E: [Servlet Error]-[]: java.lang.NullPointerException
at org.apache.jsp._console._jspService(_console.java:172)

I found this technote http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21177893 regarding NullPointerException, which states -
Resolving the problem
To resolve this issue:

1. Clean up the disk space problem.
2. Delete the /wstemp//preferences.xml file.
3. Try to login to administrative console. This should build a new preferences.xml with the correct data and should allow the user to login to administrative console.

My friend followed these steps and resolved the problem.

3/10/10

Installation Issues

Mr.X installed WAS like this on Win 08 server-

Started to launcher.
Selected WAS installer.
The installer had successfully passed OS verification.
Then selected Cell Topology. WAS 6.1 and above gives you an option to install a Cell. Meaning, installer installs Dmgr, installs an appserver, and federates that to dmgr.
Now, with all the needed inputs, the installer started to install binaries, installed successfully dmgr, appserver and even successfully federated everything.

Now, as this guy didnt configured security at the time of installation, wants to do it in the post installation.

He did like this -
After immediately installing, he ran the ivt (installation verification test).
Then went to admin console, kicked in security setup process.

WHAT IS HE MISSING?

The error what he got was - after configuring security, he wanted to bounce the services, bounce dmgr, bounce node agent, bounce server.
server was failing to start.
dmgr cannot recognize server.

What to do?ADMR0111E is one of the errors.

3/9/10

WASService command

Today one of colleague was trying to add a jvm to Windows Service.
This is the syntax for the command -
WASService.exe -add service_name
-serverName server_name
-profilePath server_profile_directory
[-wasHome
app_server_root]
[-configRoot configuration_repository_directory]
[-startArgs additional_start_arguments]
[-stopArgs additional_stop_arguments]
[-userid user_id -password password]
[-logFile service_log_file]
[-logRoot server_log_directory]
[-restart true | false]
[-startType automatic | manual | disabled]

This guy is fighting with some set of parameters. He is executing, command runs, he goes to the services, he finds that specific jvm in the services, but when he starts it, it simply says *started*, upon right clicking on the service, he sees only start option, remaining faded out.

So, whats the problem?
Go through the command options properly -
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/rins_wasservice.html
He was trying to add jvm to win services like this -
WASService.exe add blabla -logFile path/to/a/location/server.log -logRoot path/to/a/location/

yata yata yata

The problem is - one should pass SERVER LOG FILE as logFile and SERVER_LOG_LOCATION not to another location.
The WASService command looks for a file named server_name .pid to determine if the server is running.

So, if you want to add a jvm, by name server1 to Win services, you need to pass server1 log location.

Have Fun

12/12/09

selinux blocks jvm

SELinux is preventing java from loading /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so which requires text relocation.
Detailed DescriptionThe java application attempted to load /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so which requires text relocation. This is a potential security problem. Most libraries do not need this permission. Libraries are sometimes coded incorrectly and request this permission. The SELinux Memory Protection Tests web page explains how to remove this requirement. You can configure SELinux temporarily to allow /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so to use relocation as a workaround, until the library is fixed. Please file a bug report against this package.
Allowing Access
If you trust /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so to run correctly, you can change the file context to textrel_shlib_t. "chcon -t textrel_shlib_t '/software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so'" You must also change the default file context files on the system in order to preserve them even on a full relabel. "semanage fcontext -a -t textrel_shlib_t '/software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so'"

11/28/09

hostname issues

I am still fighting with Portal Express installation. Now, I have changed my strategy and trying to install as silent with responsefile.

-silent
-G licenseAccepted="true"
-W setupTypePanel.selectedSetupTypeId="full"
-W globalInstall.location="/opt/IBM/WebSphere"
-W wasPanel.installChoice="install"

################################################################################
#
# EXISTING WEBSPHERE APPLICATION SERVER INSTALLATION LOCATION
#
# Description: This is the value of the existing WebSphere Application Server
# if you selected to install on an existing instance.
#
# Be sure to follow the convention for specifying path information on your
# platform. For example,
#
# Windows: C:\WebSphere\AppServer
# AIX: /usr/WebSphere/AppServer
# Linux: /opt/WebSphere/AppServer
# Solaris: /opt/WebSphere/AppServer
# iSeries ND: /QIBM/ProdData/WebSphere/AppServer/V7/ND
# iSeries Base: /QIBM/ProdData/WebSphere/AppServer/V7/Base
################################################################################

# -W wasListPanel.existingWasLocation=""

################################################################################
#
# OPTIONAL:
# UNDETECTED WEBSPHERE APPLICATION SERVER INSTALLATION LOCATION (NON-ISERIES)
#
# Description: Sometimes the Portal install program cannot detect an existing
# WebSphere Application Server(WAS). You cannot know this until install runtime
# when the silent install fails because the value of the above parameter
# wasListPanel.existingWasLocation cannot be validated.
#
# When this error occurs, and you are certain that there is a valid WAS at the
# location specified above, uncomment this parameter and enter the same value
# specified above.
#
# Be sure to follow the convention for specifying path information on your
# platform. For example,
#
# Windows: C:\WebSphere\AppServer
# AIX: /usr/WebSphere/AppServer
# Linux: /opt/WebSphere/AppServer
# Solaris: /opt/WebSphere/AppServer
#
################################################################################

# -W was.undetectedWas="/opt/IBM/WebSphere/AppServer"

and so on.
Now, as the responsefile expected, I have installed AppServer. I just wanted run and check, so called startServer.sh server1, to my surprise, it failed.
Stacktrace -
SystemErr R com.ibm.ws.exception.ConfigurationError:javax.naming.ConfigurationException: Cannot get canonical host name for server.

serverindex.xnl has hostname="null".
This is because -
Portal Server expected localhost.localdomain.com as hostname to localhost.localdomain.

I reverted /etc/hosts back to localhost.localdomain
and I have typed in hostname="localhost" in serverindex.xml.
Server started successfully.

Now, to kick start the PortalExpress, I am a kinda stuck, because, portal expects hostname as localhost.localdomain.com.
Portal cannot work with hostname as "localhost.localdomain.com", AppServer cannot run with host name *localhost.localdomain.com*. What a DEADLOCK!!!!!

11/26/09

build failed again

Well!! n-th run, build failed again with same message as posted here - http://javagongura.blogspot.com/2009/11/build-failed-on-line-3121.html

After this step. installation aborts.
Checked the App Server installation log, it was a successful installation.
Though its a failed installation, I tried to start the app server services.
I found some *amazing* things in startServer.log -
[11/25/09 22:31:28:005 EST] 0000000a ManagerAdmin I TRAS0017I: The startup trace state is *=info.
[11/25/09 22:31:28:157 EST] 0000000a AdminTool A ADMU0128I: Starting tool with the wp_profile profile
[11/25/09 22:31:28:159 EST] 0000000a AdminTool A ADMU3100I: Reading configuration for server: WebSphere_Portal
[11/25/09 22:31:32:605 EST] 0000000a WsServerLaunc E ADMU3002E: Exception attempting to process server WebSphere_Portal
[11/25/09 22:31:32:607 EST] 0000000a WsServerLaunc E ADMU3007E: Exception com.ibm.ws.process.exception.InvalidExecutableException: PROC0004E: Executable: [/opt/XYZWEBSPHEREXYZ/AppServer/java/bin/java] does not appear to be a valid executable. Process could not be created.
at com.ibm.ws.process.UnixProcessImpl.create(UnixProcessImpl.java:159)
at com.ibm.ws.process.ProcessFactory.create(ProcessFactory.java:69)
at com.ibm.ws.management.launcher.ProcessLauncher.launch(ProcessLauncher.java:70)
at com.ibm.ws.management.launcher.LaunchCommand.launchProcess(LaunchCommand.java:246)
at com.ibm.ws.management.tools.WsServerLauncher.runTool(WsServerLauncher.java:309)
at com.ibm.ws.management.tools.AdminTool.executeUtility(AdminTool.java:264)
at com.ibm.ws.management.tools.WsServerLauncher.main(WsServerLauncher.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at com.ibm.ws.bootstrap.WSLauncher.main(WSLauncher.java:263)

This line in the above stack trace *Executable: [/opt/XYZWEBSPHEREXYZ/AppServer/java/bin/java] does not appear to be a valid executable. Process could not be created.* amazed me.
Why startServer is trying to look for java in /opt/XYZWEBSPHEREXYZ/AppServer/java/bin/java?

So, I have created folder structure like /opt/XYZWEBSPHEREXYZ/AppServer/java/bin/ and created a softlink to original appserver/java/bin/java.
But, its looking for remaining binaries in the same location.
So, need to create softlink for the entiar installation under /opt/XYZWEBSPHEREXYZ/AppServer/

Lets see...
BTW - Found a technote on this -
http://www-01.ibm.com/support/docview.wss?uid=swg21202564
The technote mentions that, because its a broken installation, you get such an error.

11/24/09

BUILD FAILED on line 3121

(Nov 23, 2009 1:56:17 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: BUILD FAILED
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: /software/IBM/WebSphere/PortalExpress/PortalServer/exp_inst/wp.config/config/includes/wp.config_cfg.xml:46: The following error occurred while executing this line:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: /software/IBM/WebSphere/PortalExpress/AppServer/profiles/wp_profile/ConfigEngine/config/actions/db_cfg_mgr.xml:277: The following error occurred while executing this line:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: /software/IBM/WebSphere/PortalExpress/PortalServer/base/wp.db.impl/config/includes/wp.db.impl_cfg.xml:95: The following error occurred while executing this line:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: /software/IBM/WebSphere/PortalExpress/PortalServer/base/wp.db.impl/config/includes/wp.db.impl_cfg.xml:337: ERROR: Error during the execution of the sql files.
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdErr: Total time: 38 seconds
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdOut:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdOut:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction$OutputWatcher, msg2, StdOut:
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction, msg2, Return code = 1
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction, msg2, Executing command: completed
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ExternalCommandAction, msg2, Completed install step: Please wait while the WebSphere Portal Express database is initialized.
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ScanFileAction, msg1, fileName=/software/IBM/WebSphere/PortalExpress/AppServer/profiles/wp_profile/ConfigEngine/log/ConfigTrace.log
(Nov 23, 2009 1:56:18 PM), PortalExpress.install, com.ibm.wps.install.ScanFileAction, msg1, Found text BUILD FAILED on line 3121
(Nov 23, 2009 1:57:06 PM), PortalExpress.install, com.ibm.wps.install.SwingPopUp, err, Logged message: EJPIC0017E: Portal installation failed. The Portal configuration step completed with errors. Please refer to the log file /software/IBM/WebSphere/PortalExpress/PortalServer/log/wpinstalllog.txt for additional information. The installer will now exit.
(Nov 23, 2009 1:57:06 PM), PortalExpress.install, com.ibm.wps.install.SwingPopUp, msg2, Exiting installer.
What to do?


Follow Up - http://www-01.ibm.com/support/docview.wss?uid=swg21291586

11/23/09

EJPIC0229E

My DB2Express installation on Linux is failing with the above error - EJPIC0229E: The DB2 Express installation step failed.

Just above to this log entry -
error while loading shared libraries: libstdc++.so.5: cannot pen shared object file: No such file or directory.


Follow up -
Found this technote http://www-01.ibm.com/support/docview.wss?rcss=faqtt_1Q09&uid=swg21297575


Follow up -
As mentioned in the above technote, I have downloaded and installed compat-libstdc++-33-3.2.3-61.i386.rpm. I got this from http://rpm.pbone.net/index.php3/stat/4/idpl/3416184/com/compat-libstdc++-33-3.2.3-61.i386.rpm.html_.
DB2Express successfully installed.

Now!! PortalExpress process get past Db2 hickup and installing WAS. Lets see what happens.

11/20/09

EJPIC0067E

I am installing PortalExpress v 6.1.0.0 on RHEL 5.2 on my laptop.

Some tips -
Run install.sh -console

You may encounter EJPIC0067E error.
EJPIC0067E: {0} requires a fully-qualified host name that is recognized by the DNS server. Short host names, loopback addresses, IP addresses and illegal characters such as blanks are not allowed. Enter the host name again.
Explanation: This installation requires a fully-qualified host name. Short host names, loopback addresses, and IP addresses are not allowed.
User Response: Enter the host name again.

To resolve this -
Just check the complete name of your box
uname -a
for me it is
localhost.localdomain
Now -
vi /etc/hosts as root
edit localhost.localdomain to localhost.localdomain.com

10/30/09

6.1.0.2 installer fails to create profile.

Issue -
WAS 6.1.0.2 installer fails to create profile.
log.txt -
Process, com.ibm.ws.install.ni.ismp.installtoolkitbridge.ISMPInstallToolkitBridgeForNIFramework, wrn, Config action failed: 97SInstallInvokeWSProfile - /software/IBM/WebSphere/AppServer/properties/version/nif/config/install/97SInstallInvokeWSProfile.ijc
Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: clearClassCache, percent complete: 25%
Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: os400fp, percent complete: 50%
Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: copyEmbedProductFile, percent complete: 75%
Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: runISCDeployRestore, percent complete: 100%
Process, com.ibm.ws.install.ni.ismp.actions.SettleNIFRegistryAction, msg1, Current install/uninstall process is successful. Process type is: install
Process, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=2
Process, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFPARTIALSUCCESS

trace.txt -
INSTCONFPARTIALSUCCESS: Partially successful install of: zip:///software/tempinst/WAS/was.installconfig.pak
I Current install/uninstall process is successful. Process type is: install
I CWUPI0000I: EXITCODE=2
I INSTCONFPARTIALSUCCESS

Solution -
Install without creating a profile, apply latest fix packs and then create profile.