Splunk’s Interactive Field Extraction (IFX)

Cognos has some very precarious logs that we have to search through from time to time, and during these times I want to poke my eyes out with a rusty nail. However, Splunk has made this entire process much easier through their search capabilities – especially the IFX interface.

We were able to find a pattern for the PID, SessionID, and RequestID in the logs, but they do not fall into a key-value pair pattern. They are separated by what appear to be spaces in the logs. Splunk’s IFX allowed us to easily go through the data and select which "columns" were for the PID, SessionID, and RequestID. So now when we search through the data, we are automatically given data counts for all of the aforementioned fields.

Their tutorial was extremely easy to follow as well:

http://www.splunk.com/base/Documentation/latest/User/Fieldsextractiontutorial

Splunk savedsearch via the CLI

The syntax of this can be a little tricky:

$SPLUNK_HOME/bin/splunk search ‘|savedsearch “Splunk errors last 24 hours”‘

The primary reason I am looking to run Saved Searches from the Command Line Interface (CLI) is to have a scheduled query kick off a script after an error occurs that will sleep for a few minutes and then run a search similar to the one above to test for a server restart.

So when my WebSphere App Server throws a java.lang.OutOfMemoryError, I want to wait a few minutes and make sure that the server restarted. There may be a better design then what I have put together but so far this is the only way I can think to solve the problem. However, I am still searching for a better solution.

UPDATE

Something else to be aware of is the permissions of the Saved Search.  If you created the Saved Search as yourself (bobsmith), but you try to run it as an admin account from the CLI, then the admin account will not have access unless you give permissions to the admin (or everyone) to run it.