Splunk is a great log file search engine that I have been using for a few years now. Gone are the days of opening up 8 different putty sessions to discover which server a user was on. It is euphoric to search across all servers (even those your team does not own) in order to have a birds-eye view of what is going down in your environment.
Sometimes though, something goes awry and you exceed that license of yours. If you exceed your license max (e.g. 10G of data per day), 5 times in a 30 day rolling period, you lose your ability to search within Splunk. So far, we have been lucky and this only happens a couple of times a year, and our Splunk reps are good about getting us a temp license, which allows us to revive our search capabilities.
Over the years we have come up with a few different ways to help prevent the license violations from occurring in the first place. In summary, we remove all forms of log file entries that contain “Debug” when we hit 85% of our license max (e.g. 8.5G). This is accomplished through a saved search that runs every 30 minutes to check how much data has been processed for the day.
host=splunk* index=_internal group="per_index_thruput" NOT series="_*" NOT series="history" NOT series="summary" | eval mb=kb/1024 | stats sum(mb) as MB_indexed | where MB_indexed > 8500
When the daily indexed total is above 8.5G, then we have Splunk run a script that updates the props.conf to include the transforms.xml stanza that will filter anything containing various forms of the word “Debug”. The data will still be sent from the forwarder, but the settings below will prevent the debug related log entries from being indexed by sending them to the nullQueue. The nullQueue is where you send data to die before it is “indexed”, which counts against your license.
cp SPLUNK_HOME/etc/apps/search/local/props.conf.filterDebug SPLUNK_HOME/etc/apps/search/local/props.conf
REGEX = DEBUG|Debug|debug|.debug
DEST_KEY = queue
FORMAT = nullQueue
We also set this saved search to sleep for 8 hours after the indexed total is found to be greater than 8.5G as to prevent Splunk from being put into a yo-yo cycle of stop and starts since at the end of filterDebugEntries.sh, Splunk is restarted for the changes to take affect.
We have another search that runs every 30 minutes as well, which tests to see if our total daily index is at 95% of our license. If this occurs, a separate, but very similar to the Splunk search above (just replace the 8500 with a 9500), will kick off a different script and then sleep for 8 hours:
cp SPLUNK_HOME/etc/apps/search/local/props.conf.filterEverything SPLUNK_HOME/etc/apps/search/local/props.conf
TRANSFORMS-nullhost = nullhost
DEST_KEY = queue
queue = nullQueue
After the restart, no more data is being indexed since all data from every host is going to the nullQueue. Be careful to schedule the 85% and 95% saved searches to run at different times. You will still be able to search across the data that has been indexed, but you can bask in the glory knowing that you prevented a license violation today.