How to custom install Splunk Forwarder using Puppet

I've added "How to custom install Splunk Forwarder using Puppet - Part 2" to demonstrate how you could customize inputs.conf file for separate classes of servers. So, be sure to check that out if you had special inputs.conf for different classes/groups of servers. Anyway...

In this how to, I will demonstrate how I installed a second Splunk Forwarder instance using Puppet. (or, if you happen to just want Splunk to be installed in a different path)

By doing this, any new machines you add to your pool of Puppetized machines will get splunk installed and configured automatically.

Since the first Splunk was installed using yum, I could not use the puppet built-in to "ensure" the splunk package is installed. I needed a way to put Splunk in its own directory so it would be running side by side with the first Splunk (The first Splunk belongs to our hosting company).

This is a very basic manifest. It tells Puppet to run the rpm command unless 'rpm -qa | grep' returned a result. That way, Puppet will only install the package once.

First of all, you have to host your RPM somewhere where RPM can get to it. I already setup an internal repo so I just added the Splunk RPM to my repo. That way, I can just pull it using http.

exec { "install_splunkforward":
  command => "/bin/rpm -ivh --prefix=/opt/custom/splunk/ http://example.com/myrepo/x86_64/splunkforwarder-4.2.5-113966-linux-2.6-x86_64.rpm",
  unless  => "/bin/rpm -qa | /bin/grep splunkforwarder-4.2.5-113966",
}


But what about configuring Splunk and starting it? That's easy! You sync all the necessary configurations in the correct order and then, execute splunk start.

Here is a list of necessary splunk files:
inputs.conf               = contains a list of files you want splunk to monitor
outputs.conf             = tells splunk where to send the files you are monitoring
cert4splunk.p12       = the certificate and private key to ensure splunk uses SSL
passwd                     = the splunk password file
splunk-launch.conf  = splunk config file
web.conf                  = web settings
puppet-serial.txt        = a file that puppet monitors for changes (if this file changes, puppet will sync this file and restart splunk. It's a way of restarting Splunk if you feel like it without making any changes to any of your regular configuration files)

The below is the order I have it set to.


    file { "/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/splunk-launch.conf",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => Exec["install_splunkforward"],
  }

    file { "/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/web.conf",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
     require => File["/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf"],
   }


    file { "/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/outputs.conf",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf"],
    }

    file { "/opt/custom/splunk/splunkforwarder/etc/passwd":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/passwd",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf"],
    }

    file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local":
      ensure  => directory,
      owner   => splunk,
      group   => splunk,
      mode    => 755,
      require => File["/opt/custom/splunk/splunkforwarder/etc/passwd"],
    }

    file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/inputs.conf",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local"],
    }

    file { "/etc/pki/tls/private/cert4splunk.p12":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/cert4splunk.p12",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf"],
    }

    file { "/opt/custom/splunk/splunkforwarder/.puppet-serial.txt":
      ensure  => present,
      source  => "puppet:///modules/prepapp/splunk-files/puppet-serial.txt",
      group   => "splunk",
      owner   => "splunk",
      mode    => "644",
      require => File["/etc/pki/tls/private/cert4splunk.p12"],
    }

    exec { "start_splunkforward":
      command => "/opt/custom/splunk/splunkforwarder/bin/splunk start --accept-license",
      unless  => "/opt/custom/splunk/splunkforwarder/bin/splunk status | /bin/grep 'splunkd is running'",
      require => File["/etc/pki/tls/private/cert4splunk.p12"],
    }

    exec { "restart_splunkforward":
      command     => "/opt/custom/splunk/splunkforwarder/bin/splunk restart --accept-license",
      refreshonly => true,
      subscribe   => File["/opt/custom/splunk/splunkforwarder/.puppet-serial.txt"],
      require     => Exec["start_splunkforward"],
    }