2014 Blog recap

I made 34 blog posts this year (including this one). For my first full year of maintaining my Living Resume, my goal was to average one post per week, and while I fell short of this goal, I did maintain this blog throughout the year. If I would have made time to complete all of my posts that are in-progress, I would have come pretty close to achieving the 52 post goal.  But then again, that’s just an excuse.  Hopefully I’ll do better next year.

Books of 2014

I read a ton this year, but I did not read as many books as my record setting 2013 of 20 books. I read many more technical articles and Longreads than anything else. While all of that is tough to track, I did make time to read 9 different books:

  1. Hatching Twitter: A True Story of Money, Power, Friendship, and Betrayal
  2. Parenting with Love and Logic: Teaching Children Responsibility
  3. A Higher Call: An Incredible True Story of Combat and Chivalry in the War-Torn Skies of World War II
  4. Exploding the Phone: The Untold Story of the Teenagers and Outlaws who Hacked Ma Bell
  5. Catching Fire (Hunger Games Trilogy, Book 2)
  6. Mockingjay: Movie Tie-In Edition (The Hunger Games)
  7. Sex, Drugs, and Cocoa Puffs: A Low Culture ManifestoChuck Klosterman.
  8. George Washington’s Secret Six: The Spy Ring That Saved the American Revolution
  9. Death to the BCS: Totally Revised and Updated: The Definitive Case Against the Bowl Championship Series eBook: Dan Wetzel, Josh Peter, Jeff Passan: Books

Fix WebSphere OutOfMemory error during deployment

Deployments within the WebSphere v7 environment were randomly failing with OutOfMemory (OOM) errors. We initially thought it was due to a very large EAR file being deployed, but after a while, this theory was inconsistent because OOMs occurred with small and large EARs alike. Within the SystemOut.log, we found this error:

[1/10/14 06:17:51:553 CST] 00000000 AbstractShell E   WASX7120E: Diagnostic information from exception with text "com.ibm.websphere.management.application.client.AppDeploymentException: com.ibm.websphere.management.application.client.AppDeploymentException [Root exception is java.lang.OutOfMemoryError]
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError " follows:

 com.ibm.websphere.management.application.client.AppDeploymentException [Root exception is java.lang.OutOfMemoryError]
    at com.ibm.ws.management.application.client.AppInstallHelper.getAppDeploymentInfoGenericRead(AppInstallHelper.java:520)
    at com.ibm.ws.management.application.client.DefaultBindingHelper.prepareTask(DefaultBindingHelper.java:216)
    at com.ibm.ws.scripting.AdminAppClient.createPreferences(AdminAppClient.java:2885)
    ...
    at org.eclipse.core.launcher.Main.basicRun(Main.java:282)
    at org.eclipse.core.launcher.Main.run(Main.java:981)
    at com.ibm.wsspi.bootstrap.WSPreLauncher.launchEclipse(WSPreLauncher.java:341)
    at com.ibm.wsspi.bootstrap.WSPreLauncher.main(WSPreLauncher.java:111)
Caused by: java.lang.OutOfMemoryError
    at org.objectweb.asm.Type.getDescriptor(Unknown Source)
    at com.ibm.ws.amm.scan.util.info.impl.InfoImpl.getClassName(InfoImpl.java:217)
    at com.ibm.ws.amm.scan.util.info.impl.InfoImpl.getClassInfo(InfoImpl.java:158)
    ...

From the install script, we saw this error:

[1/10/14 06:17:52:941 CST] 00000000 AbstractShell A   WASX7093I: Issuing message: "WASX7017E: Exception received while running file "/websphere/utilities/scripts/installNewApplication.py"; exception information: com.ibm.websphere.management.application.client.AppDeploymentException: com.ibm.websphere.management.application.client.AppDeploymentException [Root exception is java.lang.OutOfMemoryError]
java.lang.OutOfMemoryError: java.lang.OutOfMemoryError

We knew we had to increase the heap size of the deployment manager, but that turned out to be a little less obvious then anticipated.

The Python script used to kickoff the install eventually calls the Profile’s (Dmgr) script: /websphere/AppServer/profiles/Dmgr/bin GenPluginCfg.sh

#!/bin/sh
binDir=`dirname ${0}`
. ${binDir}/setupCmdLine.sh
${WAS_HOME}/bin/GenPluginCfg.sh "$@"

The last line of the profile’s GenPluginCfg.sh calls the cell’s version of GenPluginCfg.sh, which is where the heap can be adjusted as needed (-Xmx):

"$JAVA_HOME/bin/java" -Xmx1024m 
  -Dserver.root="$WAS_HOME" 
  $CLIENTSAS 
  -Dws.ext.dirs="$WAS_EXT_DIRS" 
  $USER_INSTALL_PROP 
  $WAS_LOGGING 
  -classpath "$WAS_CLASSPATH" com.ibm.ws.bootstrap.WSLauncher 
  com.ibm.websphere.plugincfg.generator.PluginConfigGenerator "$WAS_HOME" "$CONFIG_ROOT" "$WAS_CELL" "$WAS_NODE" $@

Don’t forget to backup any script you plan to alter before making updates. No reason to make a fat-finger mistake that causes a cryptic error message and takes forever to figure out.

Stash, Bamboo, Jira. How do they fit together?

My current client has bought into Atlassian products pretty hard this year, and I am loving it. Prior to the migration to Atlassian they had a home-grown solution that was less than ideal for software project management. It is nice when software just “works”. Especially when it is software specifically for software projects, written by a company whose sole purpose is to write software to manage software projects.

The language can get a little confusing if you are new to modern software project management, but at a high level these are the Atlassian products we are using, and how they related to each other:

Jira

JIRA is many things. At its most basic level, it is a way to track “issues” (aka tasks) for a given project, regardless of what type of project it may be: TODOs for support projects, project managers to track Agile projects, and everything in between. At a very high level JIRA allows you to create an “issue” for a given project and assign it to someone with a given priority. An issue can be anything, but is most likely to be a: Bug, Improvement, Task, or some variation of one of these. Issue types are highly customizable, so you are not limited to just these options.

Stash

Stash provides a visual way to manage your git repository, much like GitHub, but for the corporate world.

Bamboo

Bamboo is a Continuous Integration (CI) and build server. It will connect to Stash (i.e. git), and build the deliverable (i.e. artifact). The artifact will likely be an EAR, WAR, or JAR file in the Java world. Bamboo will also provide the deployment to the respective environments (Dev, UAT, Prod, etc.).

Combine them all

One of Jira’s most powerful features is that it allows you to create an issue in JIRA, associate it with a new branch in Stash, and a deployment within Bamboo. So if a user sees an issue with the Order Processing app, he can create an issue for this app and assign it to the project lead for the app. JIRA can then track the developer’s progress on this issue, and all collaboration between interested parties via comments in Jira to ensure the issue is fixed properly. Everyone will see that the build has been performed in Bamboo, and the newest version (e.g. 1.3.12) has fixed a list of issues, including the original issue reported.

There is a lot more available with these Atlassian products, but if interested, dial in to one of their many free webinars to better understand how they can help you.

CPU running high, AppDynmaics help

2014-11-10_AppD_CPU_notification

Still trying to get to the root of this error, but we are at least notified of its existence via AppD and are able to give it time to complete, or just kill it off.

When the issue occurs, we typically use the unix ‘top’ command to see what PID is pegging the CPU, and will stop the WebSphere node, and kill off the PID. The hope is to get AppD to help us track down the runaway Java method that is causing the CPU spike, and fix the issue instead of killing off the symptom.

waspapps02:~> top
top - 16:45:47 up 63 days, 14:26,  1 user,  load average: 3.92, 3.58, 3.52
Tasks: 176 total,   1 running, 175 sleeping,   0 stopped,   0 zombie
Cpu(s): 91.3%us,  8.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.2%si,  0.0%st
Mem:   8129720k total,  8064584k used,    65136k free,    35760k buffers
Swap:  4192956k total,  2240932k used,  1952024k free,   212712k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
**18679 wasadmin  20   0 2966m 1.4g 6516 S  159 18.2 148:32.77 java**
18211 wasadmin  20   0 1852m 1.3g 5904 S   15 16.7  49:43.31 java
18727 wasadmin  20   0 3388m 2.2g 7772 S    3 28.2  60:20.76 java
 5378 wasadmin  20   0 1877m 1.2g 7288 S    1 15.3  55:03.02 java
 8031 root      20   0  208m 3628 2252 S    1  0.0 134:01.03 aex-metricprovi
18541 wasadmin  20   0 1946m 940m 7556 S    1 11.8  27:35.41 java
 3278 wasadmin  20   0  165m  15m 2956 S    0  0.2   3:19.99 splunkd
17419 wasadmin  20   0  8772 1236  852 R    0  0.0   0:00.01 top
    1 root      20   0 10376   88   56 S    0  0.0   0:47.11 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.56 kthreadd
    3 root      RT   0     0    0    0 S    0  0.0   0:09.55 migration/0