Where is the documentation on how to enable LDAP on ajaxplorer? I couldn't find it. I didn't think they had any docs. They suggest looking in the forum.
It was very frustrating to say the least because I didn't even know where to start or if what I was doing was the right thing... but hey, it worked.
Alright, anyway... by now, you should have already successfully extract the zip or used the rpm to install your ajaxplorer. If you haven't, you should read their doc on installing it.
Make sure to log in using the admin/admin account and update the password. I'm not sure if this is even required but sounds like a good thing to do.
Go in to the ./conf dir and you will see bootstrap_plugins.php.
Make a backup copy of it in case you want to quickly revert.
Either delete or comment out the section:
"AUTH_DRIVER" => array(
"NAME" => "serial",
"OPTIONS" => array(
"LOGIN_REDIRECT" => false,
"USERS_FILEPATH" => "AJXP_DATA_PATH/plugins/auth.serial/users.ser",
"AUTOCREATE_AJXPUSER" => false,
"FAST_CHECKS" => false,
"TRANSMIT_CLEAR_PASS" => false )
),
After that, add a new section like:
"AUTH_DRIVER" => array(
"NAME" => "ldap",
"OPTIONS" => array(
"LDAP_URL" => 'ldap://localhost',
"LDAP_PORT" => '389',
"LDAP_DN" => 'dc=company,dc=com',
"LOGIN_REDIRECT" => false,
"AUTOCREATE_AJXPUSER" => false,
"TRANSMIT_CLEAR_PASS" => true)
),
Now, this is a very simple and basic settings just to get you going. As you can see, there's no bind dn or password, and password is being transferred in the clear. Once you get this working though, you can continue to tweak it to your satisfaction. Just do what they say and check the forums.
You can do some fancy things with MASTER/SLAVE auth. It looks like using MASTER/SLAVE will allow you to have multiple authentication. I haven't tried it as I don't need to have more than one auth.
Saturday, November 3, 2012
Wednesday, October 31, 2012
Nagios SNMP check plugin
I was in need of a basic but cool SNMP Nagios plug-in, so I wrote one.
All I wanted to do was send an snmpget and return a numeric value. If the value was >$warn then throw a warning and if it was >$critical, then throw a critical. Pretty simple right?
Well, this one does it and can be extended easily because you can just keep adding more OIDs.
First of all, you need to make sure you install the Perl Net SNMP.
Then, you can use it like this:
command[check_mysnmp.db]=/opt/nrpe/libexec/check_mysnmp.pl localhost db 20 30
This tells the plug-in to connect to the localhost and to use the db option and set the warn threshold to >20 and critical to >30. In my environment, we have a maximum of 40 DB connections in the DB pool. We normally only see anywhere from 5 to 10 DB connections, therefore, I set the warning to 20 and the critical to 30. Once we hit 30, we definitely know there is a connection leak.
You can add as many OIDs as you want. Just keep adding more to the case statement in the plugin and change the OID to yours. Then, set the threshold to what you need.
So for example... Let's say, you have an OID .1.2.3.4.1.12 and this corresponds to the number of errors that has occurred. Let's say that you can tolerate up to 10 errors at any given time but if you start to see 50 to 100 errors, you want to know about it.
You add the following right after case(db):
case(apperr) { $apperr_oid=".1.2.3.4.1.12";
($apperr_result,$apperr_exit)=&check_snmp($apperr_oid);
&time2exit($apperr_exit,$apperr_result); }
Then, in your nrpe config, you would add:
command[check_mysnmp.apperr]=/opt/nrpe/libexec/check_mysnmp.pl localhost apperr 20 30
and here is the wonderful plugin....
#!/usr/bin/perl
#AUTHOR: GOU YANG
#PURPOSE: This is a nagios plugin to check snmp
#pass the host, the oid, the warn threshold and the critical threshold
#
# if we can't make an snmp connection =UNKNOWN (who knows what happened)
# if > warn threshold =WARNING (throw a warning)
# if > critical threshold =CRITICAL (throw a critical)
# if less than warn & critical =OK (must be ok)
#
# set $theoid = pick from the list of case below
# set $thewarn = value before throwing a warning
# set $thecrit = value before throwing a critical
use Switch;
use Net::SNMP;
$thehost=shift;
$type=shift;
$thewarn=shift;
$thecrit=shift;
switch($type) {
#the number of used db connection; for example
case(db) { $db_used_oid=".1.2.3.4.1.11";
($db_result,$db_exit)=&check_snmp($db_used_oid);
&time2exit($db_exit,$db_result); }
default { &check_snmp(); }
}
sub check_snmp{
$theoid=shift;
if ( !$thehost || !$theoid || !$thewarn || !$thecrit ) { &time2exit("UNKNOWN","Make sure to specify host,type,warning,critical values"); }
else {
($session,$error) = Net::SNMP->session(
-hostname => "$thehost",
-community => community_string,
-timeout => 10);
if (!$session){ &time2exit("UNKNOWN","$error"); }
else {
$result = $session->get_request($theoid);
if (!$result){ $exitstat="UNKNOWN"; $msg="an error occured"; }
$session->close;
%result = %$result;
foreach my $k (keys %result){
$snmp_result=$result{$k};}
}
}
if ($snmp_result > $thewarn){$exitstat="WARNING"; $msg=$snmp_result;}
if ($snmp_result > $thecrit){$exitstat="CRITICAL"; $msg=$snmp_result;}
if ($snmp_result !~ m/(\d)/g ){$exitstat="UNKNOWN"; $msg=$snmp_result;}
return ($snmp_result,$exitstat);
}#sub
sub time2exit{
$exitstat=shift;
$msg=shift;
switch($exitstat) {
case(UNKNOWN) { print "UNKNOWN - $msg\n";exit 3; }
case(WARNING) { print "WARNING - $msg\n";exit 1; }
case(CRITICAL){ print "CRITICAL - $msg\n";exit 2; }
default { print "OK - snmp stat is $snmp_result\n";exit 0; }
}
}
All I wanted to do was send an snmpget and return a numeric value. If the value was >$warn then throw a warning and if it was >$critical, then throw a critical. Pretty simple right?
Well, this one does it and can be extended easily because you can just keep adding more OIDs.
First of all, you need to make sure you install the Perl Net SNMP.
Then, you can use it like this:
command[check_mysnmp.db]=/opt/nrpe/libexec/check_mysnmp.pl localhost db 20 30
This tells the plug-in to connect to the localhost and to use the db option and set the warn threshold to >20 and critical to >30. In my environment, we have a maximum of 40 DB connections in the DB pool. We normally only see anywhere from 5 to 10 DB connections, therefore, I set the warning to 20 and the critical to 30. Once we hit 30, we definitely know there is a connection leak.
You can add as many OIDs as you want. Just keep adding more to the case statement in the plugin and change the OID to yours. Then, set the threshold to what you need.
So for example... Let's say, you have an OID .1.2.3.4.1.12 and this corresponds to the number of errors that has occurred. Let's say that you can tolerate up to 10 errors at any given time but if you start to see 50 to 100 errors, you want to know about it.
You add the following right after case(db):
case(apperr) { $apperr_oid=".1.2.3.4.1.12";
($apperr_result,$apperr_exit)=&check_snmp($apperr_oid);
&time2exit($apperr_exit,$apperr_result); }
Then, in your nrpe config, you would add:
command[check_mysnmp.apperr]=/opt/nrpe/libexec/check_mysnmp.pl localhost apperr 20 30
and here is the wonderful plugin....
#!/usr/bin/perl
#AUTHOR: GOU YANG
#PURPOSE: This is a nagios plugin to check snmp
#pass the host, the oid, the warn threshold and the critical threshold
#
# if we can't make an snmp connection =UNKNOWN (who knows what happened)
# if > warn threshold =WARNING (throw a warning)
# if > critical threshold =CRITICAL (throw a critical)
# if less than warn & critical =OK (must be ok)
#
# set $theoid = pick from the list of case below
# set $thewarn = value before throwing a warning
# set $thecrit = value before throwing a critical
use Switch;
use Net::SNMP;
$thehost=shift;
$type=shift;
$thewarn=shift;
$thecrit=shift;
switch($type) {
#the number of used db connection; for example
case(db) { $db_used_oid=".1.2.3.4.1.11";
($db_result,$db_exit)=&check_snmp($db_used_oid);
&time2exit($db_exit,$db_result); }
default { &check_snmp(); }
}
sub check_snmp{
$theoid=shift;
if ( !$thehost || !$theoid || !$thewarn || !$thecrit ) { &time2exit("UNKNOWN","Make sure to specify host,type,warning,critical values"); }
else {
($session,$error) = Net::SNMP->session(
-hostname => "$thehost",
-community => community_string,
-timeout => 10);
if (!$session){ &time2exit("UNKNOWN","$error"); }
else {
$result = $session->get_request($theoid);
if (!$result){ $exitstat="UNKNOWN"; $msg="an error occured"; }
$session->close;
%result = %$result;
foreach my $k (keys %result){
$snmp_result=$result{$k};}
}
}
if ($snmp_result > $thewarn){$exitstat="WARNING"; $msg=$snmp_result;}
if ($snmp_result > $thecrit){$exitstat="CRITICAL"; $msg=$snmp_result;}
if ($snmp_result !~ m/(\d)/g ){$exitstat="UNKNOWN"; $msg=$snmp_result;}
return ($snmp_result,$exitstat);
}#sub
sub time2exit{
$exitstat=shift;
$msg=shift;
switch($exitstat) {
case(UNKNOWN) { print "UNKNOWN - $msg\n";exit 3; }
case(WARNING) { print "WARNING - $msg\n";exit 1; }
case(CRITICAL){ print "CRITICAL - $msg\n";exit 2; }
default { print "OK - snmp stat is $snmp_result\n";exit 0; }
}
}
Thursday, September 13, 2012
gpg: decryption failed: No secret key
I installed gpg2 on my Mac. I encrypted my file and then tested and was able to decrypt my file just fine. So, I deleted the original file after I created my encrypted file. Two days later, I go to decrypt my file and what do I get? I get the below error message. I can't believe this. Why can't it find my private key?
Mac-mini:~ user$ gpg2 -d myfile.gpg
You need a passphrase to unlock the secret key for
user: "user <user@hotmail.com>"
2048-bit RSA key, ID 4E4D9FAB, created 2012-09-12 (main key ID D3C64D14)
gpg: problem with the agent: End of file
gpg: encrypted with 2048-bit RSA key, ID 4E4D9FAB, created 2012-09-12
"user <user@hotmail.com>"
gpg: public key decryption failed: Operation cancelled
gpg: decryption failed: No secret key
I did a little googling and realized that the problem is the agent.
So, I figure, I will just look for the gpg-agent process and kill it. Here are the steps
Mac-mini:~ user$ ps -eaf | grep gpg
501 3501 1 0 Wed04PM ?? 0:02.62 gpg-agent --daemon --use-standard-socket
501 7833 7603 0 7:12PM ttys000 0:00.00 grep gpg
and then...
Mac-mini:~ user$ sudo kill 3501
and then...
everything started working again!
Wednesday, September 12, 2012
How to setup ppolicy in OpenLDAP 2.3
STEP ONE - Prepare the environment
Install the openldap-servers-overlays RPM.
Edit slapd.conf and insert the following if it doesn't already exists...
include /etc/openldap/schema/ppolicy.schema
modulepath /usr/lib64/openldap
moduleload ppolicy.la
overlay ppolicy
ppolicy_default "cn=default,ou=ppolicy,dc=company,dc=com"
ppolicy_use_lockout
ppolicy_hash_cleartext
STEP TWO - Install the password check module
Next step is to install a password checker module, if you want to use a password checker module. It's easy to say no, but I recommend that you do. Anyway, you can get the source from their repository at:
http://tools.ltb-project.org/projects/ltb/files
This was the only password checker module I found when I was googling for one and it seems to work quite well.
Once you extract everything, you will want to edit the Makefile. Set the path to your openldap header files. You probably don't have it installed. If you do, great. If you don't, you can either install the source RPM or you can grab the source from openldap here:
http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=summary
On this web page, just find the openldap version you need and click on it. Since I am running OpenLDAP v2.3, I clicked on OPENLDAP_REL_ENG_2_3.
On the next page, I clicked on the snapshot link for "Update dates for release OPENLDAP_REL_ENG_2_3_43" since I am running v2.3.43
After you've extracted the source code, you need to execute "./configure" followed by "make depend"
That's it. You're done with the OpenLDAP package.
Back to the password checker. Now, I set LDAP_INC in the make file to the location where I extracted openldap source as follows:
LDAP_INC=-I/home/user/openldap-src/include \
-I/home/user/openldap-src/servers/slapd
That's it. Now, you are ready to compile the module. See what the output is supposed to look like below. By the way, I got an error the first time I ran make. It was because crack/cracklib was not installed. I ran 'yum install cracklib-devel cracklib crack' and that resolved it.
ltb-project-openldap-ppolicy-check-password-1.1 $ make
rm -f check_password.o check_password.so check_password.lo
rm -f -r .libs
gcc -g -O2 -Wall -fpic -DHAVE_CRACKLIB -DCRACKLIB_DICTPATH="\"/usr/share/cracklib/pw_dict\"" -DCONFIG_FILE="\"/etc/openldap/check_password.conf\"" -DDEBUG -c -I/home/user/openldap-src/include -I/home/user/openldap-src/servers/slapd check_password.c
gcc -shared -o check_password.so check_password.o -lcrack
ltb-project-openldap-ppolicy-check-password-1.1 $
You now should have a check_password.o and check_password.so file. Copy or move these two files in to your module path. In my case, I copied them in to /usr/lib64/openldap.
STEP THREE - Configure your server
Restart your openldap server.
Import the below in to your openldap server. The values I have are for testing purposes. You will need to modify it for your use.
dn: cn=users,ou=ppolicy,dc=company,dc=com
cn: users
objectclass: top
objectclass: device
objectclass: pwdPolicy
objectclass: pwdPolicyChecker
pwdallowuserchange: TRUE
pwdattribute: userPassword
pwdcheckmodule: check_password.so
pwdcheckquality: 2
pwdexpirewarning: 0
pwdfailurecountinterval: 0
pwdgraceauthnlimit: 0
pwdinhistory: 2
pwdlockout: TRUE
pwdlockoutduration: 600
pwdmaxage: 0
pwdmaxfailure: 4
pwdminage: 30
pwdminlength: 8
pwdmustchange: TRUE
pwdsafemodify: FALSE
Edit /etc/ldap.conf and insert or modify the following:
pam_password clear
pam_lookup_policy yes
Note: You need to NOT hash the password on the machine in order to allow openldap to be able to read the password. That way, the password history will be honored. If you set "pam_password md5" or anything other than clear, password history will not be honored. Don't worry about security though, just make sure you are using TLS. Also, don't worry about openldap storing the password in the clear because by default it doesn't. It should store it in SSHA like below. I took this screen shot using phpldapadmin "show internal attributes".
Create the configuration file for password checker at /etc/openldap/check_password.conf
The content of the conf file is fully explained at the LTB site:
http://ltb-project.org/wiki/documentation/openldap-ppolicy-check-password
Sunday, September 2, 2012
Hmong Yaj Secret Chicken Soup Recipe
I learned how to make this when I was just a little boy.
First, you get a pot of water, then you add your chicken (my favorite is the cornish game hen. you can find it in the frozen food area in most grocery stores).
Add a lot of lemon grass that you've cut in to halves (or large enough to sit in the pot but easy enough to remove later). The trick is to smash the lemon grass a few times to release the flavor but not enough so you can't easily remove it (you're not going to eat the lemon grass). Now, add some salt and let the water boil to cook the chicken.
As the blood is released, it will collect at the top of the water as a brown substance. If that bothers you, just skim it off.
Before you serve it, taste it. If it doesn't have enough salt, add some more. Hopefully, you did not add too much salt. Remove the lemon grass.
Finally, right before you serve it, add some crush black pepper.
So, here it is again...
1 x cornish game hen (or any chicken you want to use)
lots of lemon grass (use as much as you would like the lemon grass flavor)
some salt
some pepper
First, you get a pot of water, then you add your chicken (my favorite is the cornish game hen. you can find it in the frozen food area in most grocery stores).
Add a lot of lemon grass that you've cut in to halves (or large enough to sit in the pot but easy enough to remove later). The trick is to smash the lemon grass a few times to release the flavor but not enough so you can't easily remove it (you're not going to eat the lemon grass). Now, add some salt and let the water boil to cook the chicken.
As the blood is released, it will collect at the top of the water as a brown substance. If that bothers you, just skim it off.
Before you serve it, taste it. If it doesn't have enough salt, add some more. Hopefully, you did not add too much salt. Remove the lemon grass.
Finally, right before you serve it, add some crush black pepper.
So, here it is again...
1 x cornish game hen (or any chicken you want to use)
lots of lemon grass (use as much as you would like the lemon grass flavor)
some salt
some pepper
Installing Puppet-dashboard on Centos 5.7
I got puppet-dashboard up and going fairly quickly so I thought I share. The puppet site provides many different ways.
For me, I wanted to stick with RPMs.
I began by enabling the epel repo from fedora (since I was already using EPEL, otherwise I could have added the puppetlabs repo). (http://fedoraproject.org/wiki/EPEL) It's a pretty simple install if you want to use it and you don't already have it.
~# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
Now, begin by installing the third party dependencies like Apache, Ruby, RubyGems, MySQL, Ruby-MySQL.
~# yum install mysql-server httpd httpd-devel
(run this to get it going after the install mysql_secure_installation)
(but start it up first; i.e. 'service mysqld start')
~# yum install --enablerepo=epel ruby ruby-devel rubygems rubygem-rack rubygem-rails rubygem-rake ruby-mysql
Then, use the epel repo to perform the installation.~# yum --enablerepo=epel puppet-dashboard
After the installation, you have to configure the database.yml file.
~# updatedb
~# locate database.yml
/usr/share/puppet-dashboard/config/database.yml
~# vi /usr/share/puppet-dashboard/config/database.yml
I commented out everything except the following.
production:
database: dashboard
username: dashboard
password: somesecretpassword
encoding: utf8
adapter: mysql
Get a working settings.yml file
~# locate settings.yml
/usr/share/puppet-dashboard/config/settings.yml.example
~# mv /usr/share/puppet-dashboard/config/settings.yml.example /usr/share/puppet-dashboard/config/settings.yml
Create the DB
log in to your mysql server and run the following...
Create the tables
cd /usr/share/puppet-dashboard/config/
rake RAILS_ENV=production db:migrate
That's it! Puppet-dashboard is installed and configured but we still need to tell puppet server and puppet agent that we want reporting.
Add the following to [master] (puppet 2.6+)
reports = http, store
Add the following to [agent] (puppet 2.6+)
report = true
You can start using it by typeing the following (like the web page says)
sudo -u puppet-dashboard /usr/share/puppet-dashboard/script/server -e production
On a browser, go to http://$host:3000 and it should show up.
However, at one point or another, you'll want to install either passenger or thin. I went with passenger. I went to their website and followed their RPM installation instructions and it did not work. The link was dead so I ended up using gem to perform the installation. "gem install passenger"
Since, I'm using apache, I ran...
~# passenger-install-apache2-module
The script suggested to install the following since I didn't have it already installed.
~# yum install gcc-c++ curl-devel openssl-devel zlib-devel httpd-devel apr-devel apr-util-devel
For me, I wanted to stick with RPMs.
I began by enabling the epel repo from fedora (since I was already using EPEL, otherwise I could have added the puppetlabs repo). (http://fedoraproject.org/wiki/EPEL) It's a pretty simple install if you want to use it and you don't already have it.
~# rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
Now, begin by installing the third party dependencies like Apache, Ruby, RubyGems, MySQL, Ruby-MySQL.
~# yum install mysql-server httpd httpd-devel
(run this to get it going after the install mysql_secure_installation)
(but start it up first; i.e. 'service mysqld start')
~# yum install --enablerepo=epel ruby ruby-devel rubygems rubygem-rack rubygem-rails rubygem-rake ruby-mysql
Then, use the epel repo to perform the installation.~# yum --enablerepo=epel puppet-dashboard
After the installation, you have to configure the database.yml file.
~# updatedb
~# locate database.yml
/usr/share/puppet-dashboard/config/database.yml
~# vi /usr/share/puppet-dashboard/config/database.yml
I commented out everything except the following.
production:
database: dashboard
username: dashboard
password: somesecretpassword
encoding: utf8
adapter: mysql
Get a working settings.yml file
~# locate settings.yml
/usr/share/puppet-dashboard/config/settings.yml.example
~# mv /usr/share/puppet-dashboard/config/settings.yml.example /usr/share/puppet-dashboard/config/settings.yml
Create the DB
log in to your mysql server and run the following...
CREATE DATABASE dashboard CHARACTER SET utf8;
CREATE USER 'dashboard'@'localhost' IDENTIFIED BY 'somesecretpassword';
GRANT ALL PRIVILEGES ON dashboard.* TO 'dashboard'@'localhost';Create the tables
cd /usr/share/puppet-dashboard/config/
rake RAILS_ENV=production db:migrate
That's it! Puppet-dashboard is installed and configured but we still need to tell puppet server and puppet agent that we want reporting.
Add the following to [master] (puppet 2.6+)
reports = http, store
Add the following to [agent] (puppet 2.6+)
report = true
You can start using it by typeing the following (like the web page says)
sudo -u puppet-dashboard /usr/share/puppet-dashboard/script/server -e production
On a browser, go to http://$host:3000 and it should show up.
However, at one point or another, you'll want to install either passenger or thin. I went with passenger. I went to their website and followed their RPM installation instructions and it did not work. The link was dead so I ended up using gem to perform the installation. "gem install passenger"
Since, I'm using apache, I ran...
~# passenger-install-apache2-module
The script suggested to install the following since I didn't have it already installed.
~# yum install gcc-c++ curl-devel openssl-devel zlib-devel httpd-devel apr-devel apr-util-devel
Thursday, August 30, 2012
How to custom install Splunk Forwarder using Puppet
I've added "How to custom install Splunk Forwarder using Puppet - Part 2" to demonstrate how you could customize inputs.conf file for separate classes of servers. So, be sure to check that out if you had special inputs.conf for different classes/groups of servers. Anyway...
In this how to, I will demonstrate how I installed a second Splunk Forwarder instance using Puppet. (or, if you happen to just want Splunk to be installed in a different path)
By doing this, any new machines you add to your pool of Puppetized machines will get splunk installed and configured automatically.
Since the first Splunk was installed using yum, I could not use the puppet built-in to "ensure" the splunk package is installed. I needed a way to put Splunk in its own directory so it would be running side by side with the first Splunk (The first Splunk belongs to our hosting company).
This is a very basic manifest. It tells Puppet to run the rpm command unless 'rpm -qa | grep' returned a result. That way, Puppet will only install the package once.
First of all, you have to host your RPM somewhere where RPM can get to it. I already setup an internal repo so I just added the Splunk RPM to my repo. That way, I can just pull it using http.
exec { "install_splunkforward":
command => "/bin/rpm -ivh --prefix=/opt/custom/splunk/ http://example.com/myrepo/x86_64/splunkforwarder-4.2.5-113966-linux-2.6-x86_64.rpm",
unless => "/bin/rpm -qa | /bin/grep splunkforwarder-4.2.5-113966",
}
But what about configuring Splunk and starting it? That's easy! You sync all the necessary configurations in the correct order and then, execute splunk start.
Here is a list of necessary splunk files:
inputs.conf = contains a list of files you want splunk to monitor
outputs.conf = tells splunk where to send the files you are monitoring
cert4splunk.p12 = the certificate and private key to ensure splunk uses SSL
passwd = the splunk password file
splunk-launch.conf = splunk config file
web.conf = web settings
puppet-serial.txt = a file that puppet monitors for changes (if this file changes, puppet will sync this file and restart splunk. It's a way of restarting Splunk if you feel like it without making any changes to any of your regular configuration files)
The below is the order I have it set to.
file { "/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/splunk-launch.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => Exec["install_splunkforward"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/web.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/outputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/passwd":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/passwd",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local":
ensure => directory,
owner => splunk,
group => splunk,
mode => 755,
require => File["/opt/custom/splunk/splunkforwarder/etc/passwd"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/inputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local"],
}
file { "/etc/pki/tls/private/cert4splunk.p12":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/cert4splunk.p12",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/.puppet-serial.txt":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/puppet-serial.txt",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}
exec { "start_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk start --accept-license",
unless => "/opt/custom/splunk/splunkforwarder/bin/splunk status | /bin/grep 'splunkd is running'",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}
exec { "restart_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk restart --accept-license",
refreshonly => true,
subscribe => File["/opt/custom/splunk/splunkforwarder/.puppet-serial.txt"],
require => Exec["start_splunkforward"],
}
In this how to, I will demonstrate how I installed a second Splunk Forwarder instance using Puppet. (or, if you happen to just want Splunk to be installed in a different path)
By doing this, any new machines you add to your pool of Puppetized machines will get splunk installed and configured automatically.
Since the first Splunk was installed using yum, I could not use the puppet built-in to "ensure" the splunk package is installed. I needed a way to put Splunk in its own directory so it would be running side by side with the first Splunk (The first Splunk belongs to our hosting company).
This is a very basic manifest. It tells Puppet to run the rpm command unless 'rpm -qa | grep' returned a result. That way, Puppet will only install the package once.
First of all, you have to host your RPM somewhere where RPM can get to it. I already setup an internal repo so I just added the Splunk RPM to my repo. That way, I can just pull it using http.
exec { "install_splunkforward":
command => "/bin/rpm -ivh --prefix=/opt/custom/splunk/ http://example.com/myrepo/x86_64/splunkforwarder-4.2.5-113966-linux-2.6-x86_64.rpm",
unless => "/bin/rpm -qa | /bin/grep splunkforwarder-4.2.5-113966",
}
But what about configuring Splunk and starting it? That's easy! You sync all the necessary configurations in the correct order and then, execute splunk start.
Here is a list of necessary splunk files:
inputs.conf = contains a list of files you want splunk to monitor
outputs.conf = tells splunk where to send the files you are monitoring
cert4splunk.p12 = the certificate and private key to ensure splunk uses SSL
passwd = the splunk password file
splunk-launch.conf = splunk config file
web.conf = web settings
puppet-serial.txt = a file that puppet monitors for changes (if this file changes, puppet will sync this file and restart splunk. It's a way of restarting Splunk if you feel like it without making any changes to any of your regular configuration files)
The below is the order I have it set to.
file { "/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/splunk-launch.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => Exec["install_splunkforward"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/web.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/splunk-launch.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/outputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/web.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/passwd":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/passwd",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/system/local/outputs.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local":
ensure => directory,
owner => splunk,
group => splunk,
mode => 755,
require => File["/opt/custom/splunk/splunkforwarder/etc/passwd"],
}
file { "/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/inputs.conf",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local"],
}
file { "/etc/pki/tls/private/cert4splunk.p12":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/cert4splunk.p12",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/opt/custom/splunk/splunkforwarder/etc/apps/search/local/inputs.conf"],
}
file { "/opt/custom/splunk/splunkforwarder/.puppet-serial.txt":
ensure => present,
source => "puppet:///modules/prepapp/splunk-files/puppet-serial.txt",
group => "splunk",
owner => "splunk",
mode => "644",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}
exec { "start_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk start --accept-license",
unless => "/opt/custom/splunk/splunkforwarder/bin/splunk status | /bin/grep 'splunkd is running'",
require => File["/etc/pki/tls/private/cert4splunk.p12"],
}
exec { "restart_splunkforward":
command => "/opt/custom/splunk/splunkforwarder/bin/splunk restart --accept-license",
refreshonly => true,
subscribe => File["/opt/custom/splunk/splunkforwarder/.puppet-serial.txt"],
require => Exec["start_splunkforward"],
}
Sunday, August 26, 2012
Graphing JBoss Heap Memory
Update: This was done back in 2012. Today, there is Grafana. I would recommend that you plot your Java memory using Grafana. It is super easy to setup and use. However, I'll leave this post up for historical purposes.
-----------------------------------------------------------------------------------------------------------------
I am running Cacti and I looked around for a way to graph JBoss Heap Memory but I didn't find anything to plug in to Cacti. The closes I found was how to graph JBoss Memory. So, I built one myself.
The results for the Eden space looks like this: (of course, you can do the same thing for Survivor, Permanent and Old space)
First off, you can get Heap Memory by using twiddle.sh like:
/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false
This should produce something like:
<b>Total Memory Pools:</b> 5<blockquote><b>Pool: Code Cache</b> (Non-heap memory)<blockquote>Peak Usage : init:2555904, used:24166912, committed:24444928, max:50331648<br>Current Usage : init:2555904, used:24156416, committed:24444928, max:50331648</blockquote><br><b>Pool: PS Eden Space</b> (Heap memory)<blockquote>Peak Usage : init:1074266112, used:1384251392, committed:1384251392, max:1384251392<br>Current Usage : init:1074266112, used:760269616, committed:1345912832, max:1346764800</blockquote><br><b>Pool: PS Survivor Space</b> (Heap memory)<blockquote>Peak Usage : init:179044352, used:149028456, committed:350420992, max:350420992<br>Current Usage : init:179044352, used:0, committed:42991616, max:42991616</blockquote><br><b>Pool: PS Old Gen</b> (Heap memory)<blockquote>Peak Usage : init:2864709632, used:1832103216, committed:2864709632, max:2864709632<br>Current Usage : init:2864709632, used:1320418768, committed:2864709632, max:2864709632</blockquote><br><b>Pool: PS Perm Gen</b> (Non-heap memory)<blockquote>Peak Usage : init:23068672, used:228710608, committed:268435456, max:268435456<br>Current Usage : init:23068672, used:215421104, committed:216006656, max:268435456</blockquote><br></blockquote>
This is pretty useless to Cacti. So, what we need to do here is parse it. The cool thing is, the format is already setup for Cacti. We just need to extract what we need and remove the HTML format.
To extract it, I built a parser script from perl to remove the HTML. I pass the HTML format to the parser script. I remove Peak Usage and then I grep for what I want.
/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false | /usr/bin/perl /usr/local/bin/parser2.pl | /bin/grep -v "Peak Usage" | /bin/grep -A1 Eden | /bin/grep -v Eden
The output is:
Current Usage : init:1074266112, used:883869872, committed:1345912832, max:1346764800
(Looks pretty close to what we want)
I still need to remove Current Usage and the commas. So I add /bin/sed 's/Current Usage\ :\ //g' | /bin/sed 's/,//g'
The final results:
init:1074266112 used:883340128 committed:1345912832 max:1346764800
(Looks like Cacti can now use it.)
(see Example 9-1 of the following cacti doc
http://docs.cacti.net/manual:087:3a_advanced_topics.1_data_input_methods)
NOTE:
You probably don't want to run this on the cacti machine, especially if you have many JBoss instances to trend in Cacti because you will be calling too many twiddle.sh instances. Instead, you should probably place it on the JBoss machine if you don't have anywhere else to put it. Then, add the following to /etc/snmp/snmpd.conf... (I made the host $1 and Heap type $2)
extend Eden_mem /bin/bash /usr/local/bin/get_jboss_heap.sh localhost Eden
Restart snmpd and give it a test like this...
snmpget -v 1 -c com_string localhost 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1'
The output is:
NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1 = STRING: init:1074266112 used:141552664 committed:1303904256 max:1355939840
More details on extending snmpd can be found at:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html
The parser script looks like this (for jboss 4)
package HTMLStrip;
use base "HTML::Parser";
sub text {
my ($self, $text) = @_;
print $text;
}
my $var = new HTMLStrip;
while (<>) {
$_ =~ s/\<br\>/\<br\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;
$var->parse($_);
}
$var->eof;
For jboss 5, I found that I had to modify the syntax slightly:
$_ =~ s/\<br\/\>/\<br\/\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;
-----------------------------------------------------------------------------------------------------------------
I am running Cacti and I looked around for a way to graph JBoss Heap Memory but I didn't find anything to plug in to Cacti. The closes I found was how to graph JBoss Memory. So, I built one myself.
The results for the Eden space looks like this: (of course, you can do the same thing for Survivor, Permanent and Old space)
First off, you can get Heap Memory by using twiddle.sh like:
/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false
This should produce something like:
<b>Total Memory Pools:</b> 5<blockquote><b>Pool: Code Cache</b> (Non-heap memory)<blockquote>Peak Usage : init:2555904, used:24166912, committed:24444928, max:50331648<br>Current Usage : init:2555904, used:24156416, committed:24444928, max:50331648</blockquote><br><b>Pool: PS Eden Space</b> (Heap memory)<blockquote>Peak Usage : init:1074266112, used:1384251392, committed:1384251392, max:1384251392<br>Current Usage : init:1074266112, used:760269616, committed:1345912832, max:1346764800</blockquote><br><b>Pool: PS Survivor Space</b> (Heap memory)<blockquote>Peak Usage : init:179044352, used:149028456, committed:350420992, max:350420992<br>Current Usage : init:179044352, used:0, committed:42991616, max:42991616</blockquote><br><b>Pool: PS Old Gen</b> (Heap memory)<blockquote>Peak Usage : init:2864709632, used:1832103216, committed:2864709632, max:2864709632<br>Current Usage : init:2864709632, used:1320418768, committed:2864709632, max:2864709632</blockquote><br><b>Pool: PS Perm Gen</b> (Non-heap memory)<blockquote>Peak Usage : init:23068672, used:228710608, committed:268435456, max:268435456<br>Current Usage : init:23068672, used:215421104, committed:216006656, max:268435456</blockquote><br></blockquote>
This is pretty useless to Cacti. So, what we need to do here is parse it. The cool thing is, the format is already setup for Cacti. We just need to extract what we need and remove the HTML format.
To extract it, I built a parser script from perl to remove the HTML. I pass the HTML format to the parser script. I remove Peak Usage and then I grep for what I want.
/usr/local/bin/jboss-eap-5.1/jboss/bin/twiddle.sh -s jbossserver.domain.com -u user -p password invoke "jboss.system:type=ServerInfo" listMemoryPools false | /usr/bin/perl /usr/local/bin/parser2.pl | /bin/grep -v "Peak Usage" | /bin/grep -A1 Eden | /bin/grep -v Eden
The output is:
Current Usage : init:1074266112, used:883869872, committed:1345912832, max:1346764800
(Looks pretty close to what we want)
I still need to remove Current Usage and the commas. So I add /bin/sed 's/Current Usage\ :\ //g' | /bin/sed 's/,//g'
The final results:
init:1074266112 used:883340128 committed:1345912832 max:1346764800
(Looks like Cacti can now use it.)
(see Example 9-1 of the following cacti doc
http://docs.cacti.net/manual:087:3a_advanced_topics.1_data_input_methods)
NOTE:
You probably don't want to run this on the cacti machine, especially if you have many JBoss instances to trend in Cacti because you will be calling too many twiddle.sh instances. Instead, you should probably place it on the JBoss machine if you don't have anywhere else to put it. Then, add the following to /etc/snmp/snmpd.conf... (I made the host $1 and Heap type $2)
extend Eden_mem /bin/bash /usr/local/bin/get_jboss_heap.sh localhost Eden
Restart snmpd and give it a test like this...
snmpget -v 1 -c com_string localhost 'NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1'
The output is:
NET-SNMP-EXTEND-MIB::nsExtendOutLine."Eden_mem".1 = STRING: init:1074266112 used:141552664 committed:1303904256 max:1355939840
More details on extending snmpd can be found at:
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html
The parser script looks like this (for jboss 4)
package HTMLStrip;
use base "HTML::Parser";
sub text {
my ($self, $text) = @_;
print $text;
}
my $var = new HTMLStrip;
while (<>) {
$_ =~ s/\<br\>/\<br\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;
$var->parse($_);
}
$var->eof;
For jboss 5, I found that I had to modify the syntax slightly:
$_ =~ s/\<br\/\>/\<br\/\>\n/g;
$_ =~ s/\<blockquote\>/\<blockquote\>\n/g;
Tuesday, April 10, 2012
LDAP self-service password products
I setup OpenLDAP with phpldapadmin as the front-end. Everything works great (on the admin side). On the user side, I needed a tool for users. (perhaps phpldapuser? no, that doesn't exist)
I looked around for a while and found two open source solutions.
#1 The LDAP Tool Box(LTB) written in PHP- http://ltb-project.org/wiki/start
#2 Password Manager(PWM) written in JAVA - http://code.google.com/p/pwm/
I installed both and gave it a test run. Here are my thoughts.
LDAP Tool Box
LTB is super easy to install and configure. It's just a bunch of php pages. You dump the pages in to a directory you want, point apache to it and you're done. Configuring it is easy too, you just open up the config page and start typing in the values you want.
Password Manager
PWM is pretty easy as well. Since Tomcat wasn't already installed, I had to download Tomcat and do some quick configuration. PWM started up right away and dumped in to a lengthy configuration. I found out that if something wasn't available, then clicking "Advanced" will expose the feature I was looking for. After a while of going back and fourth, I finally had PWM up and running. I ran some tests and it seem to work.
Conclusion and Decision
Both products look like they both pretty much did the same thing. Although, PWM had some extra features, I didn't need it. I was specifically looking for something to allow users to change their password from a web UI and recover their password.
I decided to go with LTB because LTB allowed more flexibility and easier to work with. First of all, LTB, out of the box performed password hashing and it allowed LDAP field mapping. When I asked about password hashing with PWM, I was told to use the OpenLDAP ppolicy feature. That's fine but I wasn't interesting in configuring ppolicy just to use PWM (I was already using something else for the policy). PWM also expected that you configure the MAIL attribute (at least that's what I figure since I could not figure out how to map MAIL to something else). In LTB, it is easy to map fields. Last, PWM did not update ShadowLastChange. They said they fixed it but in the comments, they also said you had to go and configure some other setting.
Final thoughts... I really liked the clean interface of PWM and wanted to use it but I didn't want to deal with the complexity of the product. I already have 50 other products to worry about and self-service password was one that I wanted to deploy it and forget about it.
UPDATE: I ended up setting up ppolicy anyway. When setting up ppolicy, I wanted to use a password quality checker. It turns out that LTB provides such a module, so I wrote a blog on it. http://legendofgou.blogspot.com/2012/09/how-to-setup-ppolicy-in-openldap-23.html
So, I'd say the score is LTB 2 and PWM 0.
UPDATE: I downloaded the latest PWM to give it a test run. 10/2012
They cleaned up the UI and I was pretty excited. I started configuring the product only to hit a bunch of road blocks. i.e. after clicking "forgot password", I am asked to provide a user name. I typed in my test user name and I got an error 5019. I don't know why. Everything else worked. hmmm, may be one day, they will get it right. I will be sure to test again next year.
LTB 3 and PWM 0
I looked around for a while and found two open source solutions.
#1 The LDAP Tool Box(LTB) written in PHP- http://ltb-project.org/wiki/start
#2 Password Manager(PWM) written in JAVA - http://code.google.com/p/pwm/
I installed both and gave it a test run. Here are my thoughts.
LDAP Tool Box
LTB is super easy to install and configure. It's just a bunch of php pages. You dump the pages in to a directory you want, point apache to it and you're done. Configuring it is easy too, you just open up the config page and start typing in the values you want.
Password Manager
PWM is pretty easy as well. Since Tomcat wasn't already installed, I had to download Tomcat and do some quick configuration. PWM started up right away and dumped in to a lengthy configuration. I found out that if something wasn't available, then clicking "Advanced" will expose the feature I was looking for. After a while of going back and fourth, I finally had PWM up and running. I ran some tests and it seem to work.
Conclusion and Decision
Both products look like they both pretty much did the same thing. Although, PWM had some extra features, I didn't need it. I was specifically looking for something to allow users to change their password from a web UI and recover their password.
I decided to go with LTB because LTB allowed more flexibility and easier to work with. First of all, LTB, out of the box performed password hashing and it allowed LDAP field mapping. When I asked about password hashing with PWM, I was told to use the OpenLDAP ppolicy feature. That's fine but I wasn't interesting in configuring ppolicy just to use PWM (I was already using something else for the policy). PWM also expected that you configure the MAIL attribute (at least that's what I figure since I could not figure out how to map MAIL to something else). In LTB, it is easy to map fields. Last, PWM did not update ShadowLastChange. They said they fixed it but in the comments, they also said you had to go and configure some other setting.
Final thoughts... I really liked the clean interface of PWM and wanted to use it but I didn't want to deal with the complexity of the product. I already have 50 other products to worry about and self-service password was one that I wanted to deploy it and forget about it.
UPDATE: I ended up setting up ppolicy anyway. When setting up ppolicy, I wanted to use a password quality checker. It turns out that LTB provides such a module, so I wrote a blog on it. http://legendofgou.blogspot.com/2012/09/how-to-setup-ppolicy-in-openldap-23.html
So, I'd say the score is LTB 2 and PWM 0.
UPDATE: I downloaded the latest PWM to give it a test run. 10/2012
They cleaned up the UI and I was pretty excited. I started configuring the product only to hit a bunch of road blocks. i.e. after clicking "forgot password", I am asked to provide a user name. I typed in my test user name and I got an error 5019. I don't know why. Everything else worked. hmmm, may be one day, they will get it right. I will be sure to test again next year.
LTB 3 and PWM 0
Sunday, January 1, 2012
A note about stomach acid
For a long time, I always had terrible acid reflux aka heart burn or whatever doctors want to call it now-a-days. I used to religiously carry a stick of tums with me.
Then, I found out what was causing the stomach acid. Well, of course the drug companies don't want you to know and doctors are all about treating and not root cause analysis.
Ok, actually, I found out because I had no choice really... My acid problem became so bad that blood was coming out and anti-acids weren't helping. Anyway, I went to a health food store and purchased a bunch of vitamins. Two weeks went by and the acid problem was just getting worse. I almost broke down and went to the hospital but I knew that the doctors would only prescribe stronger medication and this was not what I wanted.
By this time, I was desperate. I went back to the health food store and told them about my acid problem. They recommended digestive enzymes. The idea is to take a digestive enzyme 15 to 30 minutes prior to eating any food. The digestive enzyme would help digest the food and the acid would not be produced. It worked great but now what?
Well, months had gone by and one day, I happen to take a vitamin called choline and voila, acid was completely gone.
The problem for me was that I had a deficiency in choline. Later on, I would find out that deficiencies in other things also caused acid. The one thing that is related to all this is STRESS. Stress burns our important vitamins and minerals and amino acids much faster than we can consume it back. While in this "stressed" state, problems like stomach acid occur.
A few other things I found that got rid of stomach acid for me:
- L-Lysine
- L-glutamin
- 5-HTP
- L-tyrosine
- D-Ribose
- Vitamin D
- Vitamin B-Complex
Remember that it only works if you really are low on it. You can get tested if you ask your doctor.
Then, I found out what was causing the stomach acid. Well, of course the drug companies don't want you to know and doctors are all about treating and not root cause analysis.
Ok, actually, I found out because I had no choice really... My acid problem became so bad that blood was coming out and anti-acids weren't helping. Anyway, I went to a health food store and purchased a bunch of vitamins. Two weeks went by and the acid problem was just getting worse. I almost broke down and went to the hospital but I knew that the doctors would only prescribe stronger medication and this was not what I wanted.
By this time, I was desperate. I went back to the health food store and told them about my acid problem. They recommended digestive enzymes. The idea is to take a digestive enzyme 15 to 30 minutes prior to eating any food. The digestive enzyme would help digest the food and the acid would not be produced. It worked great but now what?
Well, months had gone by and one day, I happen to take a vitamin called choline and voila, acid was completely gone.
The problem for me was that I had a deficiency in choline. Later on, I would find out that deficiencies in other things also caused acid. The one thing that is related to all this is STRESS. Stress burns our important vitamins and minerals and amino acids much faster than we can consume it back. While in this "stressed" state, problems like stomach acid occur.
A few other things I found that got rid of stomach acid for me:
- L-Lysine
- L-glutamin
- 5-HTP
- L-tyrosine
- D-Ribose
- Vitamin D
- Vitamin B-Complex
Remember that it only works if you really are low on it. You can get tested if you ask your doctor.
Subscribe to:
Posts (Atom)