Hmong means free: August 2012

Thursday, August 30, 2012

How to custom install Splunk Forwarder using Puppet

I've added "How to custom install Splunk Forwarder using Puppet - Part 2" to demonstrate how you could customize inputs.conf file for separate classes of servers. So, be sure to check that out if you had special inputs.conf for different classes/groups of servers. Anyway...

In this how to, I will demonstrate how I installed a second Splunk Forwarder instance using Puppet. (or, if you happen to just want Splunk to be installed in a different path)

By doing this, any new machines you add to your pool of Puppetized machines will get splunk installed and configured automatically.

Since the first Splunk was installed using yum, I could not use the puppet built-in to "ensure" the splunk package is installed. I needed a way to put Splunk in its own directory so it would be running side by side with the first Splunk (The first Splunk belongs to our hosting company).

This is a very basic manifest. It tells Puppet to run the rpm command unless 'rpm -qa | grep' returned a result. That way, Puppet will only install the package once.

First of all, you have to host your RPM somewhere where RPM can get to it. I already setup an internal repo so I just added the Splunk RPM to my repo. That way, I can just pull it using http.

exec { "install_splunkforward":
command => "/bin/rpm -ivh --prefix=/opt/custom/splunk/ http://example.com/myrepo/x86_64/splunkforwarder-4.2.5-113966-linux-2.6-x86_64.rpm",
unless => "/bin/rpm -qa | /bin/grep splunkforwarder-4.2.5-113966",
}

But what about configuring Splunk and starting it? That's easy! You sync all the necessary configurations in the correct order and then, execute splunk start.

Here is a list of necessary splunk files:
inputs.conf = contains a list of files you want splunk to monitor
outputs.conf = tells splunk where to send the files you are monitoring
cert4splunk.p12 = the certificate and private key to ensure splunk uses SSL
passwd = the splunk password file
splunk-launch.conf = splunk config file
web.conf = web settings
puppet-serial.txt = a file that puppet monitors for changes (if this file changes, puppet will sync this file and restart splunk. It's a way of restarting Splunk if you feel like it without making any changes to any of your regular configuration files)

The below is the order I have it set to.

file { "/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf":
  ensure => present,
  source => "puppet:///modules/prepapp/splunk-files/splunk-launch.conf",
  group => "splunk",
  owner => "splunk",
  mode => "644",
  require => Exec["install_splunkforward"],
}

file { "/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf":
  ensure => present,
  source => "puppet:///modules/prepapp/splunk-files/web.conf",
  group => "splunk",
  owner => "splunk",
  mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf"],
}

file { "/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/outputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf"],
}

file { "/opt/custom/splunk/splunkforwarder/etc/passwd":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/passwd",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf"],
}

file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local":
ensure => directory,
owner => splunk,
group => splunk,
mode => 755,
require => File["/opt/custom/splunk/splunkforwarder/etc/passwd"],
}

file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/inputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local"],
}

file { "/etc/pki/tls/private/cert4splunk.p12":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/cert4splunk.p12",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf"],
}

file { "/opt/custom/splunk/splunkforwarder/.puppet-serial.txt":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/puppet-serial.txt",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}

exec { "start_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk start --accept-license",
unless => "/opt/custom/splunk/splunkforwarder/bin/splunk status | /bin/grep 'splunkd is running'",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}

exec { "restart_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk restart --accept-license",
refreshonly => true,
subscribe => File["/opt/custom/splunk/splunkforwarder/.puppet-serial.txt"],
require => Exec["start_splunkforward"],
}

Sunday, August 26, 2012

Graphing JBoss Heap Memory

Update: This was done back in 2012. Today, there is Grafana. I would recommend that you plot your Java memory using Grafana. It is super easy to setup and use. However, I'll leave this post up for historical purposes.

-----------------------------------------------------------------------------------------------------------------

I am running Cacti and I looked around for a way to graph JBoss Heap Memory but I didn't find anything to plug in to Cacti. The closes I found was how to graph JBoss Memory. So, I built one myself.

The results for the Eden space looks like this: (of course, you can do the same thing for Survivor, Permanent and Old space)

First off, you can get Heap Memory by using twiddle.sh like:

/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false

This should produce something like:
Total Memory Pools: 5<blockquote>Pool: Code Cache (Non-heap memory)<blockquote>Peak Usage : init:2555904, used:24166912, committed:24444928, max:50331648 Current Usage : init:2555904, used:24156416, committed:24444928, max:50331648</blockquote> Pool: PS Eden Space (Heap memory)<blockquote>Peak Usage : init:1074266112, used:1384251392, committed:1384251392, max:1384251392 Current Usage : init:1074266112, used:760269616, committed:1345912832, max:1346764800</blockquote> Pool: PS Survivor Space (Heap memory)<blockquote>Peak Usage : init:179044352, used:149028456, committed:350420992, max:350420992 Current Usage : init:179044352, used:0, committed:42991616, max:42991616</blockquote> Pool: PS Old Gen (Heap memory)<blockquote>Peak Usage : init:2864709632, used:1832103216, committed:2864709632, max:2864709632 Current Usage : init:2864709632, used:1320418768, committed:2864709632, max:2864709632</blockquote> Pool: PS Perm Gen (Non-heap memory)<blockquote>Peak Usage : init:23068672, used:228710608, committed:268435456, max:268435456 Current Usage : init:23068672, used:215421104, committed:216006656, max:268435456</blockquote> </blockquote>

This is pretty useless to Cacti. So, what we need to do here is parse it. The cool thing is, the format is already setup for Cacti. We just need to extract what we need and remove the HTML format.

To extract it, I built a parser script from perl to remove the HTML. I pass the HTML format to the parser script. I remove Peak Usage and then I grep for what I want.

/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false | /usr/bin/perl /usr/local/bin/parser2.pl | /bin/grep -v "Peak Usage" | /bin/grep -A1 Eden | /bin/grep -v Eden

The output is:
Current Usage : init:1074266112, used:883869872, committed:1345912832, max:1346764800
(Looks pretty close to what we want)

I still need to remove Current Usage and the commas. So I add /bin/sed 's/Current Usage\ :\ //g' | /bin/sed 's/,//g'

The final results:
init:1074266112 used:883340128 committed:1345912832 max:1346764800
(Looks like Cacti can now use it.)
(see Example 9-1 of the following cacti doc
http://docs.cacti.net/manual:087:3a_advanced_topics.1_data_input_methods)

NOTE:
You probably don't want to run this on the cacti machine, especially if you have many JBoss instances to trend in Cacti because you will be calling too many twiddle.sh instances. Instead, you should probably place it on the JBoss machine if you don't have anywhere else to put it. Then, add the following to /etc/snmp/snmpd.conf... (I made the host $1 and Heap type $2)

extend Eden_mem /bin/bash /usr/local/bin/get_jboss_heap.sh localhost Eden

Restart snmpd and give it a test like this...

snmpget -v 1 -c com_string localhost 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1'
The output is:
NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1 = STRING: init:1074266112 used:141552664 committed:1303904256 max:1355939840

More details on extending snmpd can be found at:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html

The parser script looks like this (for jboss 4)
package HTMLStrip;
use base "HTML::Parser";

sub text {
 my ($self, $text) = @_;
 print $text;
}

my $var = new HTMLStrip;
while (<>) {

$_ =~ s/\<br\>/\<br\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;
 $var->parse($_);
}
$var->eof;

For jboss 5, I found that I had to modify the syntax slightly:
$_ =~ s/\<br\/\>/\<br\/\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;

Hmong means free

Thursday, August 30, 2012

How to custom install Splunk Forwarder using Puppet

Sunday, August 26, 2012

Graphing JBoss Heap Memory

About Me

Blog Archive