WAS 6.1 install fails on Power7 Systems - AIX

WebSphere 6.1 installation fails using ./install on Power7 systems

The crash is in

 Module: /usr/WebSphere61/installBinariesWAS61ND/install/JDK/jre.pak/repository/package.java.jre/java/jre/bin/libj9jit23.so

By any chance is this Power 7 machine ? ( please check the details below )

To determine whether your system is using a POWER7 processor, run the following command:

prtconf | grep -i "Processor Type"

If the system's processors are POWER7 processors, then the output will display the following:

Processor Type: PowerPC_POWER7

The processor implementation mode might also be relevant to your product installation.  To determine the processor implementation mode, run the following command:

prtconf | grep -i "Processor Implementation Mode"

The reason I ask is we have know issue with JIT and Power 7 architecture.

http://www-01.ibm.com/support/docview.wss?uid=swg21422150

To fix the issue , please do the following :

1. In the cmd window where running install please do :

export JAVA_COMPILER=NONE

This just disables JIT for installer.   

2.The run your install .

3. Once installed, please upgrade to suggested level as per above document.

If this is WSAS 6.1 please upgrade to 6.1.0.29 (WSAS and JAVA SDK fixpack both)

4.Please unset the JAVA_COMPILER variable.  You can do so either by closing the shell session that ran the installer, or by running the following command:

unset JAVA_COMPILER

Effects of using the JAVA_COMPILER variable

Using the JAVA_COMPILER=none setting as suggested above will cause the Java SDK to turn off the JIT compiler feature in Java during the installation process.  This feature has an issue running on POWER7 processors which is resolved when the appropriate fix pack is applied to the installed product.  Turning off the feature prevents the installer from crashing.

Once the installation is complete, and the product is updated to the appropriate fix pack level, then the setting is no longer needed.  Note that application servers will not use this setting simply because the product was installed in that manner.This has no negative effects on the install process, except it might take slightly longer to install.

If still issue please send us one of the javacores for review and OS level command

prtconf | grep -i "Processor Type"

Disable Trace/Track in Apache HTTPD

Very recent effort to address Security/Vulnerability Violations as noted by our security team scans.

Disable Trace/Track in Apache HTTPD

Introduction

Disabling TRACE and TRACK in Apache for PCI-related vulnerabilities like Web Server HTTP Trace/Track Method Support Cross-Site Tracing Vulnerability is surprisingly quite easy with the Apache web server.  The main thing to keep in mind is understanding that if you are running apache and this vulnerability pops up during a scan, you can be reasonably certain that TRACK is not the problem—TRACE is.

The HTTP TRACK method is something Microsoft cooked up that performs essentially the same thing that TRACE does with the exception that it never got used—except by penetration testers, hackers, worms, and vulnerability scanners.

Validation Steps

If you web server is listening on port 80, by far the easiest (and universal) way to determine whether it is vulnerable or not is using telnet.  Simply open up your telnet application and connect to your web site/web server over port 80, ( telnet <hostname> <port>).  If you are using the Microsoft telnet client, be careful because it doesn't echo back what you were typing in.  Once you connect, type the following:

    TRACE / HTTP/1.0

    Host: <hostname_you_are_testing>

    TestA: Hello

    TestB: World

Press enter twice and if trace is enabled, you should see output similar to the following:

    HTTP/1.1 200 OK

    Server: Apache

    Date: Tue, 04 Aug 2009 20:17:15 GMT

    Content-Type: message/http

    Content-Length: 76

   

    TRACE / HTTP/1.0

    Host: <hostname_you_are_testing>

    TestA: Hello

    TestB: World

Request and Response over telnet for the HTTP TRACK method is identical, for testing purposes, as it is for TRACE.  Simply subsitute TRACK for TRACE.  If you need to test a host that is listening on ssl port 443 (and does not have an HTTP port exposed), use openssl's s_client.  Simply type " openssl s_client -connect <hostname:sslport> ".  You will connect and then you can enter the above request the same as you would for telnet.

If you use Perl, I did put a script together called 'test4trac', which will test a site to see if trace and track are allowable. It can be downloaded from my blog's download page and more information is available at the test4trac information page.

Remediation

TRACE is enabled by default in an apache installation.  There are two ways to remediate.  The first can be used if you are running Apache 1.3.34, 2.0.55, or anything in the 2.2 release.  Simply add the TraceEnable directive into your httpd.conf and set the value to Off.  

The second mechanism involves creating a mod_rewrite rule that will disable http methods, which is also quite popular and works with ANY version of apache that supports mod_rewrite.  The directives below would need to be set, which are written assuming that this is the first time use for mod_rewrite.

The first thing to do is make sure that mod_rewrite is loaded.  If mod_rewrite.so is missing from your apache configuration but you have it installed, (and your install location is /usr/local/apache), then add the following statement to your httpd.conf:

    LoadModule  rewrite_module  "/usr/local/apache/modules/mod_rewrite.so"

Then add the following as well to your httpd.conf file:

    RewriteEngine On

    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)

    RewriteRule .* - [F]

Restart apache, re-run the steps in the Validation section, and with either method, you should receive an HTTP 405-Method Not Allowed status code back.

Note:

I've seen a lot of posts in forums from people that are attempting to validate TRACE is enabled by issuing an HTTP OPTIONS request against a target web server.  This is not a valid test because HTTP OPTIONS reports back the methods that a particular web server may support and not necessarily the HTTP methods that are enabled on a site.  Even disabling the TRACE method will not remove TRACE from the Supported Methods line in an OPTIONS request, so if you see or hear of anyone telling you that you can validate by issuing an OPTIONS call, they are incorrect

RAR /Resource Adapter Install Cell Level WAS

Helped out my team mate in resolving the issue with Resource Adapter configuration to be able to communicate CICS J2C Connection Factories (Mainframe), Thought would share the notes.

* When you use the Install RAR dialog to install a RAR file, the scope you define on the Resource Adapters page has no effect on where the RAR file is installed. You can install RAR files only at the node level, which you specify on the Install RAR page. To set the scope of an RAR file to a specific cluster, or server, after you install the RAR file at each node level, create a new copy of the RAR file with the appropriate cluster or server scope.

* You cannot install resource adapter at cell level. Resource adapters often contain binary shared libraries and there is no distribution mechanism at the cell level for this. The adapters use native path to locate these executables, which could be different on some nodes or platforms. Because of these problems, J2C resource adapters have to be installed to nodes directly and this is enforced by both wasadmin and admin console. When using resource adapater in a cell environment, you should install to the nodes and then configure them at the cell scope.

ServerIOTimeout property in the IHS Plugin

Notes that could help, the below was summarized over digging all around web to overcome a issue that I was troubleshooting, Thought it was useful to post.

* The issue is with the default setting of ServerIOTimeout property in the IHS Plugin component which needed modification and not on IHS instance

* The changes/modification needs manually editing files as this property is not visible and cannot be modified from Admin Console ( Added in next release of WebSphere/IHS (8.0) )

A breif synopsis and info regarding ServerIOTimeout from IBM Technotes ( IBM Tech Note URLs are provided below for further reference)

ServerIOTimeout

Enables the plug-in to set a timeout value, in seconds, for sending requests to and reading responses from the application server. (Further clarification: This is not from client/browser to IHS (Compression Server) but from Plugin component to WebSphere JVM)

If you set the ServerIOTimeout attribute to a positive value, this attempt to contact the server ends when the timeout occurs. However, the server is not considered unavailable and future requests are still sent to the server on which the unavailable timeout occurred.

If you set the ServerIOTimeout attribute to a negative value, the server is considered unavailable whenever a timeout occurs, and no future requests are sent to the server on which the timeout occurred.

If a value is not set for the ServerIOTimeout attribute, the plug-in, by default, uses blocked I/O to write requests to and read responses from the application server, and does not time out the TCP connection. For example, you might specify the following setting:

nov-2011 update

For example, you might specify the following setting:

<Server Name="server1" ServerIOTimeout=300>In this situation, if an application server stops responding to requests, the plug-in waits 300 seconds (5 minutes) before timing out the TCP connection. Setting the ServerIOTimeout attribute to a reasonable value enables the plug-in to timeout the connection sooner, and transfer requests to another application server when possible.

When selecting a value for this attribute, remember that sometimes it might take several minutes for an application server to process a request. Setting the value of the ServerIOTimeout attribute too low might cause the plug-in to send a false server error response to the client.

 The default value for this property is 60.

jul2011 update

 Avoid trouble:  The ServerIOTimeout limits the amount of time the plug-in waits for each individual read or write operation to return. ServerIOTimeout does not represent a timeout for the overall request.

The ServerIOTimeout property is designed to allow requests to time-out instead of waiting indefinitely for a response from the server. In Versions 6.1 and earlier, the serveriotimeout property defaults to 0, which indicates that there is no time limit for requests. This value is not recommended because if the a server never responds, the resources involved in this request might never be freed and eventually the resource pool is exhausted. In some cases, the operating system or adapter resources might break the connection because of inactivity but this is not behavior that should be relied on. You will need to determine the appropriate behavior for your environment if you plan to set the ServerIOTimeout property to 0.

If the ServerIOTimeout property is set to 0, do not expect requests to be retried. A request is not retried until the previous connection is broken and, as explained previously, the connection might not get broken. If you need to use the retry capabilities of the plug-in, either set the serveriotimeout property to a non-zero value, use the operating system settings, or use some other non-plug-in mechanism to ensure that you can guarantee that a connection is broken at some predetermined time.

With Version 7.0, the default value of the ServerIOTimeout property is 60 seconds. This might not be the ideal setting for applications running intensive queries or other nontrivial functions. The ServerIOTimeout value will need to be adjusted if you expect requests to take longer than 60 seconds to be served.

The ServerIOTimeout setting is based on how long the server takes to handle a request; not on a particular URI or application. Therefore, when specifying a value for this property, you should allow for the slowest, longest request time, and then add a little more time to handle peak operation situations.

The Use read/write timeout setting on the administrative console corresponds to the ServerIOTimeout attribute in the plugin-cfg.xml file. The default for the setting and the attribute for Version 8 is 900 seconds. For Version 7, the default is 60 seconds. Before Version 7, the default is 0.

Please note: The methods of adding this property described in below tech notes from IBM from Admin Console do not work, this has been tried and tested during this effort ( that is -> Adding custom property to IHS Server via console)

http://www-01.ibm.com/support/docview.wss?uid=swg1PK26979

PK26979: INFORMATION CENTER DESCRIPTION OF SERVERIOTIMEOUT IN "WEB SERVERPLUG-IN CONFIGURATION PROPERTIES" PAGE IS INCORRECT

http://www-01.ibm.com/support/docview.wss?uid=swg1PK76657

PK76657: APPSERVERPORTPREFERENCE AND SERVERIOTIMEOUT CHANGES ARE NOT BEING PERSISTED TO THE PLUGIN-CFG.XML FILE.

http://www-01.ibm.com/support/docview.wss?uid=swg21460889

Modifying web server plug-in properties from the WebSphere Application Server administrative console

java core dump

Java Core Dump

Download the file windows_hang.py and copy the file to your <PROFILE_ROOT>\bin directory. If instead copied to <WAS_HOME>\bin, the default server, which may be the deployment manger (dmgr), will be accessed when wsadmin.bat is launched.


NOTE: This script only works for WebSphere Application Server 6.1 and higher.


If you are looking for the older windows_hang.bat that works with older releases of WebSphere Application Server, see the FAQ section.


To launch the script to produce 3 javacores spaced 2 minutes apart, run this command:

wsadmin -lang jython -f windows_hang.py -j -s SERVER_NAME

Replacing SERVER_NAME with your server's name.

If you have to run this number of times, on windows, you can write a batch file and run the batch file as Administrator.

<WAS_HOME>\profiles\<PROFILE_NAME>\bin\wsadmin -lang jython -username <username> -password <password> -f \<path_to_windows_hang.py>\windows_hang.py -j -s SERVER_NAME

ex.,
\"Program Files"\ibm\WebSphere\AppServer\profiles\AppSrv01\bin\wsadmin -lang jython -username blahblah -password blahblah -f \users\bramaraju\desktop\windows_hang.py -j -s server01

http://www-01.ibm.com/support/docview.wss?uid=swg21111364

PM20642: FAILED APPLICATION DEPLOYMENT DOES NOT LEAVE THE WORKSPACE CLEANED PROPERLY

Error description

• Business-level application (BLA) code plugs into the  application management deployment framework to create a BLA  and composition unit (CU) for the new application. After it  does that, more tasks added by the product extension run and  fail, and the BLA and CU are left hanging around. The sceanrio  is a failed deployment and the session is then saved. If the  session is discarded or if the deployment succeeds, things  work correctly. So, the problem is that a failed deployment is  not leaving the work space clean. No warning is issued to  notify the user.  

Local fix

Remove any directories or files created, clean work space.  

http://www-01.ibm.com/support/docview.wss?uid=swg1PM20642
CUS folder path
<WAS_HOME>/profiles/<dmgr>\Dmgr01\config\cells\<CellName>\cus
BLAS folder path
<WAS_HOME>/profiles/<dmgr>\Dmgr01\config\cells\<CellName>\blas

--
Thanks and Regards
Bhaskar Ramaraju
http://www.linkedin.com/in/ramarajubhaskar