Thread Dumps

Aften as process admin, we need to take thread dump.
Thread dump gives you what the jvm threads are doing, what are their status and so on info.

1. WebSphere
We can take WebSphere JVM thread by using this command
kill -3 jvmpid

This generates thread dump, by default dumps the threads to jvm/logs/jvmname/natviestdout
You can configure the jvm to generate generate file in your configured path.
2. Tomcat
/path/to/jdk/bin/jstack -l pid > nameoffile.log
You need to take thread dump as the owner of the pid. Better to use the same jdk as tomcat. As the owner of tomcat, you can say "which java" or ps -ef | grep -i java gives which java tomcat process pointed to.

3. WebLogic
kill -3 or jstack can be used.

[There are several ways. These are few]

java core dump

Java Core Dump

Download the file windows_hang.py and copy the file to your <PROFILE_ROOT>\bin directory. If instead copied to <WAS_HOME>\bin, the default server, which may be the deployment manger (dmgr), will be accessed when wsadmin.bat is launched.


NOTE: This script only works for WebSphere Application Server 6.1 and higher.


If you are looking for the older windows_hang.bat that works with older releases of WebSphere Application Server, see the FAQ section.


To launch the script to produce 3 javacores spaced 2 minutes apart, run this command:

wsadmin -lang jython -f windows_hang.py -j -s SERVER_NAME

Replacing SERVER_NAME with your server's name.

If you have to run this number of times, on windows, you can write a batch file and run the batch file as Administrator.

<WAS_HOME>\profiles\<PROFILE_NAME>\bin\wsadmin -lang jython -username <username> -password <password> -f \<path_to_windows_hang.py>\windows_hang.py -j -s SERVER_NAME

ex.,
\"Program Files"\ibm\WebSphere\AppServer\profiles\AppSrv01\bin\wsadmin -lang jython -username blahblah -password blahblah -f \users\bramaraju\desktop\windows_hang.py -j -s server01

http://www-01.ibm.com/support/docview.wss?uid=swg21111364

Java monitoring tools

There are a few tools you can use to monitor and identify performance inhibitors in your Java™ applications.

vmstat

Provides information about various system resources. It reports statistics on kernel threads in the run queue as well as in the wait queue, memory usage, paging space, disk I/O, interrupts, system calls, context switches, and CPU activity.

iostat

Reports detailed disk I/O information.

topas

Reports CPU, network, disk I/O, Workload Manager and process activity.

tprof

Profiles the application to pinpoint any hot routines or methods, which can be considered performance problems.

ps -mo THREAD

Shows to which CPU a process or thread is bound.

Java profilers [-Xrunhprof, Xrunjpa64]

Determines which routines or methods are the most heavily used.

java -verbose:gc

Checks the impact of garbage collection on your application. It reports total time spent doing garbage collection, average time per garbage collection, average memory collected per garbage collection, and average objects collected per garbage collection.

Here are 10 tools useful for Java application performance troubleshooting.

• jconsole comes together with JDK 1.5 and above. It is a Java Monitoring and Management Console - JMX-compliant graphical tool for monitoring a Java virtual machine. It can monitor both local and remote JVMs.

• VisualVM is a visual tool that integrates several existing JDK software tools and lightweight memory and CPU profiling capabilities. This tool is designed for both production and development time use and further enhances the capability of monitoring and performance analysis for the Java SE platform.

• HeapAnalyzer allows the finding of a possible Java™ heap leak area through its heuristic search engine and analysis of the JavaTM heap dump in Java applications. It analyzes Java heap dumps by parsing the Java heap dump, creating directional graphs, transforming them into directional trees, and executing the heuristic search engine.

• PerfAnal is a GUI-based tool for analyzing the performance of applications on the Java 2 Platform. You can use PerfAnal to identify performance problems in your code and locate code that needs tuning.

• JAMon is a free, simple, high performance, thread safe, Java API that allows developers to easily monitor production applications.

• Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.

• GCViewer is a free open source tool to visualize data produced by the Java VM options -verbose:gc and -Xloggc:<file>. It also calculates garbage collection related performance metrics (throughput, accumulated pauses, longest pause, etc.).

 

Xms!=Xmx

Setting the initial Java heap minimum equal to the Java heap maximum is not recommended. Doing so can delay the start of garbage collection until the heap is full. Therefore, the first time the Garbage Collector runs is a very expensive operation.
The heap is also likely to be very fragmented by the time heap compaction is required. This is also a very expensive operation.

http://www-01.ibm.com/support/docview.wss?uid=swg21160795&acss=was121610

major minor version errors

These types of errors are caused by a Java version mismatch between the compiled proxies and your runtime.

"bad class file. class file has wrong version 50.0, should be 48.0."

Unsupported major.minor version 50.0
JSPG0091E: An error occurred at line: 2 in the file: /login.jsp
JSPG0093E: Generated servlet error from file: /login.jsp

Below are a list of some of the version numbers and their corresponding Java runtimes:
Version 50.0 = Java 1.6.x
Version 49.0 = Java 1.5.x
Version 48.0 = Java 1.4.x

The server is expecting 48.0 version of the code, meaning, code built with 1.4.x version of jvm, but, the supplied code was built against version 50.0 meaning 1.6.x version of the jvm.

What one can do to solve this is, if you use RAD kind of tools to build the code, configure build path to have appropriate JRE, clean your builds and re build.

64 bit or 32 bit JVM?

How to know your WAS version is 32 bit or 64 bit?
This is a machine which has WAS 64 bit
$ ./java -d64 -version
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (IBM build 1.5.0_06-erdist-20060404 20060511)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_06-erdist-20060404, mixed mode)
IBM Java ORB build orb50-20060511a (SR2)
XML build XSLT4J Java 2.7.4
XML build IBM JAXP 1.3.5
XML build XML4J 4.4.5

This is a machine which has WAS 32bit
$ ./java -d64 -version
execv(): No such file or directory
Error trying to exec /software/opt/IBM/WebSphere/AppServer/java/jre/bin/sparcv9/java.
Check if file exists and permissions are set correctly.
Failed to start a 64-bit JVM process from a 32-bit JVM.
Verify all necessary J2SE components have been installed.
(Solaris SPARC 64-bit components must be installed after 32-bit components.)

So acid Test is -
goto App Server javahome
/path/to/washome/java/jre/bin
./java -d64 -version

if it is 64 bit, it returns with version info.
otherwise returns with no such file or directory.

Applet Security -

When an app is accessed through https, if there is no trust between app server and browser jre, client jvm may spit SSL HandShake Error. Error may go like this -
Error Occured during session initialization: Error SSL connecting to server:9446:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Its very simple to solve this. One needs to import client certificate into jre/lib/security/cacerts which is the trust store of the JVM.
example -
"C:\Program Files\Java\jre1.5.0_11\bin\keytool" -import -trustcacerts -keystore "C:\Program Files\Java\jre1.5.0_11\lib\security\cacerts" -storepass changeit -noprompt -alias ALIAS_NAME -file "\CERTIFICATE_NAME"