We had a need to create a new cluster of WebSphere 7 JVMs (Cluster_B) that are identical to an existing cluster (Cluster_A). No problem, an easy task that I’ve done many times before. I proceeded to venture through the WAS console to create a new cluster using the existing Cluster_A_was01 member as a template. The new config was told to create new ports, I clicked through the save buttons, and gave the cluster members a few minutes to ensure they were synced up properly with the new configuration.
Everything worked as expected right up to the point that the server did not start after issuing the start command from the CLI (Command Line Interface).
websphere_01:~> /was/AppServer/profiles/AppServer/bin/startServer.sh Cluster_B
ADMU0116I: Tool information is being logged in file
ADMU0128I: Starting tool with the AppServer profile
ADMU3100I: Reading configuration for server: Cluster_B
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3011E: Server launched but failed initialization. startServer.log,
SystemOut.log(or job log in zOS) and other log files under
should contain failure information.
This is a new server, that was cloned from an existing one, so there could be a conflict of a param that I missed (ports, cookie names, etc.). I look inside the Cluster_B log directory, and there is no SystemOut.log to be found.
websphere_01:~> cd /was/AppServer/profiles/AppServer/logs/Cluster_B
websphere_01:/was/AppServer/profiles/AppServer/logs/Cluster_B> ls -latr
-rw-r–r– 1 websphereUser websphereGroup 0 2014-02-26 15:19 native_stdout.log
-rw-r–r– 1 websphereUser websphereGroup 5 2014-02-26 15:35 Cluster_B.pid
-rw-r–r– 1 websphereUser websphereGroup 1935 2014-02-28 13:26 startServer.log
-rw-r–r– 1 websphereUser websphereGroup 2259 2014-02-28 13:26 native_stderr.log
Note that I tried to start the server, it failed, and told me to look in the SystemOut.log. There is no SystemOut.log listed. I’m now in uncharted waters. I’ve never seen an instance of starting up a new JVM where no SystemOut.log or SystemErr.log is created. Thanks for mutton WebSphere.
After verifying the ports are different from the cloned JVM from Cluster_A, kicking kittens, and other config comparisons, I thought to look at the JVM args, which would be identical to Cluster_A, since it is a clone. I see that AppDynamics is there, and right next them are the bane of the past couple of hours: a check mark next to Debug with the port set to 7777, just like Cluster_A’s debug configuration.
To be sure that the identical debug ports are the issue (and not AppD), I first remove the AppDynamics JVM params and try again. Failure. Next the debug config is removed altogether, and the server boots right up. I changed the debug port on Cluster_B to 7778, reboot, and it again starts right up.
It would have been nice for the WAS server to let me know that there was a debug port conflict, instead of me fumbling around in the dark with no idea of where to start. It would have saved me a couple of hours, and several kicks to kittens.