Administrative console throws error 500 with java.lang.NullPointerExcetion

Last Sunday, one of my friend got this problem.

Admin Console, when accessed, throws error 500 with java.lang.NullPointerExcetion.
SystemOut.log has the following entries

5ea42ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/logon.jsp: init
5ea42ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/layouts/browser_detection.jsp: init
5ea02ee7 WebGroup I SRVE0180I: [adminconsole] [/admin] [Servlet.LOG]: /secure/console.jsp: init
5ea02ee7 WebGroup E SRVE0026E: [Servlet Error]-[]: java.lang.NullPointerException
at org.apache.jsp._console._jspService(_console.java:172)

I found this technote http://www-01.ibm.com/support/docview.wss?rs=180&uid=swg21177893 regarding NullPointerException, which states -
Resolving the problem
To resolve this issue:

1. Clean up the disk space problem.
2. Delete the /wstemp//preferences.xml file.
3. Try to login to administrative console. This should build a new preferences.xml with the correct data and should allow the user to login to administrative console.

My friend followed these steps and resolved the problem.

WASService command

Today one of colleague was trying to add a jvm to Windows Service.
This is the syntax for the command -
WASService.exe -add service_name
-serverName server_name
-profilePath server_profile_directory
[-wasHome
app_server_root]
[-configRoot configuration_repository_directory]
[-startArgs additional_start_arguments]
[-stopArgs additional_stop_arguments]
[-userid user_id -password password]
[-logFile service_log_file]
[-logRoot server_log_directory]
[-restart true | false]
[-startType automatic | manual | disabled]

This guy is fighting with some set of parameters. He is executing, command runs, he goes to the services, he finds that specific jvm in the services, but when he starts it, it simply says *started*, upon right clicking on the service, he sees only start option, remaining faded out.

So, whats the problem?
Go through the command options properly -
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.base.doc/info/aes/ae/rins_wasservice.html
He was trying to add jvm to win services like this -
WASService.exe add blabla -logFile path/to/a/location/server.log -logRoot path/to/a/location/

yata yata yata

The problem is - one should pass SERVER LOG FILE as logFile and SERVER_LOG_LOCATION not to another location.
The WASService command looks for a file named server_name .pid to determine if the server is running.

So, if you want to add a jvm, by name server1 to Win services, you need to pass server1 log location.

Have Fun

Migration Notes #2

The migration tools are used as part of a four-step migration process:
1. Run WASPreUpgrade on the existing WebSphere Application Server configuration.
2. Uninstall the existing version of WebSphere Application Server.
3. Install the new version of WebSphere Application Server.
4. Run WASPostUpgrade on the new version.
In the second step, it is recommended that rather than uninstalling the existing version, it be made dormant. By keeping the existing software installed on the machine, it will be much easier to back out of the migration should unexpected problems arise.

The fully automated migration includes the following steps:
1. Migrate the existing Version 5.x deployment manager to Version 6.0 using the migration
tools.
2. Migrate the HTTP servers and HTTP server plug-ins on all Web tier nodes.
3. Individually migrate the application server nodes using the migration tools.

The WASPreUpgrade and WASPostUpgrade tools effectively move the existing configuration into WebSphere Application Server Version 6.0.

Ideal WAS Environment

Migration Notes #1

Why to migrate?
Two important things to my understanding.
For new features.
For maintenance support.
-> New Features
Again two views. Developer view and support view.
ex., My developer called me one day, said, struts framework x.xx version is throwing error on the development. I paid a look into the log and search for the errors in the IBM. Found that, that version of JVM donot support this version of struts. So, what to do? Its the developers call to have the latest copy the JVM which can support this version of struts and apps.
This is what is developer initiated migration.
Admin initiated migration -
Admin may initiate the migration too. Some of the app server errors must have been addressed in the new version of the application server and so the need for migration.
-> Support
If the product is end of the support, there is a need to migrate to supported version.

While migrating, one should need to understand inconsistencies and deprecations between the target and source environments.
In considering the incompatibilities and deprecations, we look at these major areas:
* J2EE differences
* J2SE/JDK differences
* Application Server runtime differences
* Programming Model Extension differences
* Third Party Library differences

-> J2EE Differences
Java Code, as we all know, is forward compatible. Meaning that, code compiled in JDK 1.3.0 runs on 1.4.0, but, may throw some deprecation warnings.
Its a good practice for the developers to address those deprecations in the code. So, when one plans to migrate from x version to y version, the development team should plan to revisit their code.

What you want to know about HTTP session persistence

Comment lines: Erik Burckart: What you want to know about HTTP session persistence -

Just the FAQs

In general, I get a lot of questions from various customers about HTTP session persistence. With Web 2.0 technologies taking off, more and more Web applications are being redesigned, and as they get changed, these same questions come up even more frequently. If you are involved in redesigning any Web applications for Web 2.0, here are answers to some of the questions you might soon be asking.

1. If I don’t need session persistence, can I turn it off?

Yes, you can configure session management to run with in-memory sessions, which is actually the default. You can still use HttpSessions in this mode, but if a failure occurs, the data stored in the HttpSessions will be lost.
.............read the remaining from this technical library article
Comment lines: Erik Burckart: What you want to know about HTTP session persistence

selinux blocks jvm

SELinux is preventing java from loading /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so which requires text relocation.
Detailed DescriptionThe java application attempted to load /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so which requires text relocation. This is a potential security problem. Most libraries do not need this permission. Libraries are sometimes coded incorrectly and request this permission. The SELinux Memory Protection Tests web page explains how to remove this requirement. You can configure SELinux temporarily to allow /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so to use relocation as a workaround, until the library is fixed. Please file a bug report against this package.
Allowing Access
If you trust /software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so to run correctly, you can change the file context to textrel_shlib_t. "chcon -t textrel_shlib_t '/software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so'" You must also change the default file context files on the system in order to preserve them even on a full relabel. "semanage fcontext -a -t textrel_shlib_t '/software/inst/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9thr23.so'"

hostname issues

I am still fighting with Portal Express installation. Now, I have changed my strategy and trying to install as silent with responsefile.

-silent
-G licenseAccepted="true"
-W setupTypePanel.selectedSetupTypeId="full"
-W globalInstall.location="/opt/IBM/WebSphere"
-W wasPanel.installChoice="install"

################################################################################
#
# EXISTING WEBSPHERE APPLICATION SERVER INSTALLATION LOCATION
#
# Description: This is the value of the existing WebSphere Application Server
# if you selected to install on an existing instance.
#
# Be sure to follow the convention for specifying path information on your
# platform. For example,
#
# Windows: C:\WebSphere\AppServer
# AIX: /usr/WebSphere/AppServer
# Linux: /opt/WebSphere/AppServer
# Solaris: /opt/WebSphere/AppServer
# iSeries ND: /QIBM/ProdData/WebSphere/AppServer/V7/ND
# iSeries Base: /QIBM/ProdData/WebSphere/AppServer/V7/Base
################################################################################

# -W wasListPanel.existingWasLocation=""

################################################################################
#
# OPTIONAL:
# UNDETECTED WEBSPHERE APPLICATION SERVER INSTALLATION LOCATION (NON-ISERIES)
#
# Description: Sometimes the Portal install program cannot detect an existing
# WebSphere Application Server(WAS). You cannot know this until install runtime
# when the silent install fails because the value of the above parameter
# wasListPanel.existingWasLocation cannot be validated.
#
# When this error occurs, and you are certain that there is a valid WAS at the
# location specified above, uncomment this parameter and enter the same value
# specified above.
#
# Be sure to follow the convention for specifying path information on your
# platform. For example,
#
# Windows: C:\WebSphere\AppServer
# AIX: /usr/WebSphere/AppServer
# Linux: /opt/WebSphere/AppServer
# Solaris: /opt/WebSphere/AppServer
#
################################################################################

# -W was.undetectedWas="/opt/IBM/WebSphere/AppServer"

and so on.
Now, as the responsefile expected, I have installed AppServer. I just wanted run and check, so called startServer.sh server1, to my surprise, it failed.
Stacktrace -
SystemErr R com.ibm.ws.exception.ConfigurationError:javax.naming.ConfigurationException: Cannot get canonical host name for server.

serverindex.xnl has hostname="null".
This is because -
Portal Server expected localhost.localdomain.com as hostname to localhost.localdomain.

I reverted /etc/hosts back to localhost.localdomain
and I have typed in hostname="localhost" in serverindex.xml.
Server started successfully.

Now, to kick start the PortalExpress, I am a kinda stuck, because, portal expects hostname as localhost.localdomain.com.
Portal cannot work with hostname as "localhost.localdomain.com", AppServer cannot run with host name *localhost.localdomain.com*. What a DEADLOCK!!!!!

6.0 and 6.1 Co-exist topology

You have 6.0.xx installed on a machine. You wanted to install 6.1 on the same machine and migrate 6.0.xx to 6.1 later.
Now, You found that, your 6.1 installation going on smooth, but, you are amazed to see this message at the end of your log.txt
(Sep 11, 2009 4:00:11 PM), Process, com.ibm.ws.install.ni.ismp.installtoolkitbridge.ISMPInstallToolkitBridgeForNIFramework, wrn, Config action failed: 97SInstallInvokeWSProfile - /software/IBM/WebSphere61/AppServer/properties/version/nif/config/install/97SInstallInvokeWSProfile.ijc
(Sep 11, 2009 4:00:11 PM), Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: clearClassCache, percent complete: 25%
(Sep 11, 2009 4:00:11 PM), Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: os400fp, percent complete: 50%
(Sep 11, 2009 4:00:11 PM), Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: copyEmbedProductFile, percent complete: 75%
(Sep 11, 2009 4:00:11 PM), Process, com.ibm.ws.install.ni.ismp.actions.InstallNIFPackage, msg1, Running configuration command: runISCDeployRestore, percent complete: 100%
(Sep 11, 2009 4:04:37 PM), Process, com.ibm.ws.install.ni.ismp.actions.SettleNIFRegistryAction, msg1, Current install/uninstall process is successful. Process type is: install
(Sep 11, 2009 4:04:37 PM), Process, com.ibm.ws.install.ni.ismp.actions.SetExitCodeAction, msg1, CWUPI0000I: EXITCODE=2
(Sep 11, 2009 4:04:37 PM), Process, com.ibm.ws.install.ni.ismp.actions.ISMPLogSuccessMessageAction, msg1, INSTCONFPARTIALSUCCESS


The reason for this is, since you already installed 6.0.xx, you have already used default ports. Now, for the new installation, you need to tell the installer not to look for default ports.

Explore the following technote -


http://www-01.ibm.com/support/docview.wss?rs=180&context=SSEQTP&q1=wrn%2c+Config+action+failed%3a+97SInstallInvokeWSProfile&uid=swg21261120&loc=en_US&cs=utf-8&lang=en

WAS Migration Step By Step

Here is the IBM WAS Tech Library Article on "A quick guide for migrating to WebSphere Application Server V6.1"

http://www.ibm.com/developerworks/websphere/library/techarticles/0608_chalmers/0608_chalmers.html

64 bit or 32 bit JVM?

How to know your WAS version is 32 bit or 64 bit?
This is a machine which has WAS 64 bit
$ ./java -d64 -version
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (IBM build 1.5.0_06-erdist-20060404 20060511)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_06-erdist-20060404, mixed mode)
IBM Java ORB build orb50-20060511a (SR2)
XML build XSLT4J Java 2.7.4
XML build IBM JAXP 1.3.5
XML build XML4J 4.4.5

This is a machine which has WAS 32bit
$ ./java -d64 -version
execv(): No such file or directory
Error trying to exec /software/opt/IBM/WebSphere/AppServer/java/jre/bin/sparcv9/java.
Check if file exists and permissions are set correctly.
Failed to start a 64-bit JVM process from a 32-bit JVM.
Verify all necessary J2SE components have been installed.
(Solaris SPARC 64-bit components must be installed after 32-bit components.)

So acid Test is -
goto App Server javahome
/path/to/washome/java/jre/bin
./java -d64 -version

if it is 64 bit, it returns with version info.
otherwise returns with no such file or directory.

SSL Technotes

Here are some useful technotes on implementing custome SSL on WAS.
Creating Custom Secure Socket Layer (SSL) Key Files for V5.0 using a CA Certificate
Creating Custom Secure Socket Layer (SSL) Key Files for V6.0 using a CA Certificate
Manually Replacing SSL Certificates in V6.1







Note -
Please join in my WebSphere Community in Orkut, WebSphere Zone

Collector Tool

Someone asked me how to run collector tool. IBM Doc says, run the tool not from APPSERVER_INST_PATH/bin, but run from working directory. What exactly is that?

The collector tool gathers information about your WebSphere Application Server installation and packages it in a Java archive (JAR) file that you can send to IBM Customer Support to assist in determining and analyzing your problem. Information in the JAR file includes logs, property files, configuration files, operating system and Java data, and the presence and level of each software prerequisite.

collector command - summary option

WebSphere Application Server products include an enhancement to the collector tool beginning with Version 5.0.2, known as the collector summary option.

The collector summary option helps you communicate with WebSphere Application Server technical staff at IBM Support. Run the collector tool with the -Summary option to produce a lightweight text file and console version of some of the information in the Java archive (JAR) file that the tool produces without the -Summary parameter. You can use the collector summary option to retrieve basic configuration and prerequisite software level information when starting a conversation with IBM Support.

The collector summary option produces version information for the WebSphere Application Server product and the operating system as well as other information. It stores the information in the Collector_Summary.txt file and writes it to the console. You can use the information to answer initial questions from IBM Support or you can send the Collector_Summary.txt file directly to IBM Support.

Collector tool collect key information including ffdc, configuration, logs, and so on and makes a jar file with all the information. So, depending on your configuration and all, jar file may occupy lot of space. So, IBM docs says that, create a folder out of appserver installation path.
Example,
mkdir /tmp/collector
cd /tmp/collector

Then run the script as needed
/appserver/instpath/profiles/profilename/bin/collector.sh
or
/appserver/instpath/profiles/profilename/bin/collector.sh -summery

Symbolic links

Sometimes, in the process of WebSphere, IHS Administration, it may require you to create symbolic links. This is the procedure.

ln is the command to create symbolic links.
$man ln
User Commands ln

NAME
ln - make hard or symbolic links to files

SYNOPSIS
/usr/bin/ln [-fns] source_file [target]

/usr/bin/ln [-fns] source_file... target

/usr/xpg4/bin/ln [-fs] source_file [target]

/usr/xpg4/bin/ln [-fs] source_file... target

DESCRIPTION
In the first synopsis form, the ln utility creates a new
directory entry (link) for the file specified by
source_file, at the destination path specified by target. If
target is not specified, the link is made in the current
directory. This first synopsis form is assumed when the
final operand does not name an existing directory; if more
than two operands are specified and the final is not an
existing directory, an error will result.

In the second synopsis form, the ln utility creates a new
directory entry for each file specified by a source_file
operand, at a destination path in the existing directory
named by target.
..............................


Example -
Developers wanted to have a look at SystemOut.log and SystemErr.log. They dont have uid on the box. So, I need to create a web link to this through HTTP Server.
I did like this.
I have created a document root called logs.
/logs
I created symbolic links to SystemOut.log and SystemErr.log like this
ln -s source destination
This creates a softlink *from* a file to a *file*
$ln -s /path/to/appserver/profiles/profilename/logs/servername/SystemOut.log SystemOut.log
$ln -s /path/to/appserver/profiles/profilename/logs/servername/SystemErr.log SystemErr.log
This created me two files SystemOut.log and SystemErr.log under /logs
if I ls -ltr on /logs, I get this list
SystemOut.log -> /path/to/appserver/profiles/profilename/logs/servername/SystemOut.log
SystemErr.log -> /path/to/appserver/profiles/profilename/logs/servername/SystemErr.log

Now, I have created a html file, say, index.html with hyperlinks like this -
Server Logs
< A HREF="SystemOut.log">SystemOut< / A>< br />
< A HREF="SystemErr.log">SystemErr< / A>

Multiple Authentication Bypass Vulnerabilities within IBM WAS

Multiple Authentication Bypass Vulnerabilities within IBM WebSphere Application Server

OVERVIEW:
Two vulnerabilities in IBM WebSphere Application Server have recently been made public. IBM WebSphere Application Server is a software application server that uses web technologies and can be implemented on many common operating systems. Both vulnerabilities may allow malicious users to bypass authentication required to access a service running on the vulnerable server. Successful exploitation may allow attackers to gain unauthorized access to the service, which may lead to other attacks.

SYSTEMS AFFECTED:
The vulnerability identified by IBM in PK72138 (Bugtraq ID 35594):
* IBM WebSphere Application Server 7.0 1
* IBM WebSphere Application Server 6.1 23
* IBM WebSphere Application Server 6.1 22
* IBM WebSphere Application Server 6.1 21
* IBM WebSphere Application Server 6.1 20
* IBM WebSphere Application Server 6.1 19
* IBM WebSphere Application Server 6.1 18
* IBM WebSphere Application Server 6.1 17
* IBM WebSphere Application Server 6.1 15
* IBM WebSphere Application Server 6.1 13
* IBM WebSphere Application Server 6.1 12
* IBM WebSphere Application Server 6.1 10
* IBM WebSphere Application Server 6.1 .9
* IBM WebSphere Application Server 6.1 .7
* IBM WebSphere Application Server 6.1 .6
* IBM WebSphere Application Server 6.1 .5
* IBM WebSphere Application Server 6.1 .3
* IBM WebSphere Application Server 6.1 .2
* IBM WebSphere Application Server 6.1 .14
* IBM WebSphere Application Server 6.1 .1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 6.1
* IBM WebSphere Application Server 7.0

The vulnerability identified by IBM in PK75992 (Bugtraq ID 35610):
* IBM WebSphere Application Server 7.0 1
* IBM WebSphere Application Server 6.1 21
* IBM WebSphere Application Server 6.1 19
* IBM WebSphere Application Server 6.1 17
* IBM WebSphere Application Server 6.1 15
* IBM WebSphere Application Server 6.0.2 31
* IBM WebSphere Application Server 6.0.2 29
* IBM WebSphere Application Server 6.0.2 27
* IBM WebSphere Application Server 6.0.2 .25
* IBM WebSphere Application Server 7.0

RISK:
Government:
* Large and medium government entities: High
* Small government entities: High

Businesses:
* Large and medium business entities: High
* Small business entities: High

Home users: N/A

DESCRIPTION:
IBM has confirmed the existence of two vulnerabilities that may allow a remote attacker to bypass application server authentication. Exploiting these vulnerabilities could allow an attacker to access restricted services, which may then lead to other attacks. Both vulnerabilities are associated with WS-Security, which is the security implementation within the Java API for XML Web Services (JAX-WS).
The first vulnerability discovered within WS-Security, referenced and addressed by IBM in PK72138, can only be exploited when the security policy is implemented at the 'Operational Level'. When this policy is established, WS-Security does not properly handle inbound requests that lack a SOAPAction or WS-Addressing Action. An attacker can craft a malicious inbound request to exploit this vulnerability. Successful exploitation may allow attackers to bypass certain security restrictions, which may then lead to other attacks.
The second vulnerability, referenced and addressed by IBM in PK75992, arises in the way WS-Security validates the 'UsernameToken' object. It is possible for WS-Security to incorrectly validate these tokens, allowing a malicious user to bypass the authentication process. This may allow a malicious user to conduct further attacks.

RECOMMENDATIONS:
We recommend the following actions be taken:
* Apply appropriate patches provided by IBM to vulnerable systems immediately after appropriate testing.
* Deploy network intrusion detection systems to monitor network traffic for malicious activity.

REFERENCES:

IBM:
http://www-01.ibm.com/support/docview.wss?rs=180&context=SSEQTP&dc=D600&uid=swg21367223&loc=en_US&cs=UTF-8&lang=en&rss=ct180WebSphere

Security Focus:
http://www.securityfocus.com/bid/35594
http://www.securityfocus.com/bid/35610

CVE:
http://www.cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-0903