There are many ways to remotely monitor your servers, but nagios is one of the most configurable and flexible ones. Once you have set up nagios, your localhost is being checked. But when there are real issues with your localhost, nagios probably isn’t going to send you any messages. That’s why you should set up remote monitoring: a nagios server that checks all your servers. Ideally, you set up multiple nagios servers or monitor your single nagios server as well. This article assumes a single nagios server.

So, remote monitoring. Nagios has a plugin named Nagios Remote Plugin Executor (NRPE for short). This plugin enables nagios to execute check scripts on remote host. What it checks, how often it checks it and when it sends out a warning can be easily configured (per server). In order for these remote checks to work, they need to be defined in two places:

• Nagios server: so it knows what to check and how to check it
• Remote host: so it knows what the nagios server is allowed to check

This article assumes you have nagios up and running for localhost. It also assumes you use Ubuntu or Debian (though it may very well work on other Linux distributions as well). The first thing to do is to make sure the NRPE plugin is installed on the nagios server:

sudo apt-get install nagios-nrpe-plugin

Once that is done, check if there’s a bunch of files (plugins) listed:

ls -al /usr/lib/nagios/plugins/

Next, you usually need to determine what services you would like to check and whether there is a plugin for it. You can write your own nagios plugins in whatever language you want. I’ve worked with some PHP nagios plugins, so it’s really not that hard. However, that’s something for another article and we’re going to work with some of the stock plugins now. I’m going to use the following plugins in my examples:

• Current Users (check_users)
• Current Load (check_load)
• Disk Space (check_all_disks)
• Zombie Processes (check_zombie_procs)
• Total Processes (check_total_procs)
• Swap (check_swap)

Nagios server

First thing you need to do, is to define the services on the nagios server. The nagios server needs to know what services to check for a remote host and how it needs to do that. I’ve added my services to /etc/nagios3/conf.d/services_nagios2.cfg (which is default on Ubuntu). The file consists of blocks that look like:

define service {
hostgroup_name local-servers
service_description SSH
check_command check_ssh
use generic-service
notification_interval 0
}

We’re going to make such a block for every remote check we need. I’m going to show one example here and attach the full list of definitions to this article (for reference or copy-paste). Here is the first remote service definition:

define service {
hostgroup_name generic-servers
service_description Current Users NRPE
check_command check_nrpe_1arg!check_users
use generic-service
notification_interval 0
}

I’m going to explain this block line by line, so you really understand what it all does. The ‘define service’ kind of speaks its purpose, so I’ll skip that.

hostgroup_name generic-servers

This assigns this service to a hostgroup (and defines it if it didn’t exist yet). You can later add a host to that hostgroup and it’ll automatically execute this check for that host (and all other hosts in that hostgroup).

service_description Current Users NRPE

This describes the service and is used for displaying this service on both the nagios web interface and in any nagios software you could use (like nagstamon or aNag).

check_command check_nrpe_1arg!check_users

This defines the actual command of the check. The first part, ‘check_nrpe_1arg’, says this is a check without any additional arguments. It says 1arg, because that one argument is ‘check_users’ (the exclamation mark is used as a separator for arguments). So this piece makes sure that the ‘check_users’ command is executed on the remote host.

use generic-service

The use keyword enables you to include templates. This time it’s the generic-service template. A template is actually a normal nagios configuration, in this case a service definition like the one we’re adding now. I use the default template that is installed with nagios (on Ubuntu). To see the service definition:

sudo less /etc/nagios3/conf.d/generic-service_nagios2.cfg

This service defines a lot of options that you now don’t have to have to define with every service definition you write. So with the use keyword, you can easily define basic definitions for groups of hosts. Now back to the last line:

notification_interval 0

This defines the number of time units before re-sending a notification in case of a problem with this service. A time unit is defined as interval_length (defined in /etc/nagios3/nagios.cfg), which defaults to 60 seconds. The number here is thus in minutes (x times 60 seconds). Set this to 0 to receive just one notification.

Now we’ve established what the definition does, you can add service definitions for all the plugins I listed and we can move on to the remote host. You may also skip doing this and add more checks later.

Remote host

Now on to the remote host we want to check. We start by installing the NRPE server (which includes the default set of plugins):

sudo apt-get install nagios-nrpe-server

Again, make sure the plugins are actually there:

ls -al /usr/lib/nagios/plugins/

Now it is important to make sure the nagios server is allowed to connect to the remote host. First of all, if you use a firewall, you need to open up port 5666 (TCP & UDP). Second, you need to define the allowed_hosts parameter in /etc/nagios/nrpe_local.cfg. Notice the ‘_local’, because there also is a nrpe.cfg. The nrpe.cfg contains the default configuration. You can override these configuration options in the nrpe_local.cfg file without getting merge issues when upgrading.

Make sure the nrpe_local.cfg line looks something like this:

######################################
# Do any local nrpe configuration here
######################################
allowed_hosts=127.0.0.1,192.0.2.20

I always leave 127.0.0.1 there. Although it is not strictly necessary, it could be useful for local testing. You can add as many IP addresses as you deem necessary there. I just added one IP, the one of the nagios server.

After setting this and restarting the nagios-nrpe-server, the nagios server can connect to the nagios server. Howerver, it cannot execute any checks yet. In order to be able to execute checks, the checks that were defined on the nagios server need to be defined here as well. Open up the file /etc/nagios/nrpe_local.cfg again and add the following lines (you can comment them out with #):

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_all_disks]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10%
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
command[check_swap]=/usr/lib/nagios/plugins/check_swap -w 50% -c 25%

By adding these lines, we have mapped the checks defined on the nagios server (behind check_nrpe_arg1!) to actual checks on the remote host. Let’s look at the first line:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10

This maps check_users to /usr/lib/nagios/plugins/check_users. You can even test the check by executing /usr/lib/nagios/plugins/check_users directly. If you do that, it gives a nice warning that you forgot two parameters:

Usage:
check_users -w <users> -c <users>

In the command definition, we’ve set these to 5 and 10. The -w is to set the warning threshold. The -c is to set the critical threshold. If five users are logged in at the same time, and the nagios servers checks it, it triggers a warning. If 10 users are logged in at the same time, it triggers a critical warning.

Now run the check again to see the actual result:

/usr/lib/nagios/plugins/check_users -w 5 -c 10

It should now say:

USERS OK – 1 users currently logged in |users=1;5;10;0

As you can see, every command has different threshold values. Notice the % with check_all_disks and check_swap. If you ever add a check, be sure to run in directly on the remote host to see if it returns the expected result. It’s never a waste to play with these checks a bit and fine-tune them.

Finally, restart the NRPE server so the new config gets loaded:

sudo service nagios-nrpe-server restart

Back to the nagios server

The last thing we need to do is to add a host to the hostgroup we have just added the check for. Still following me? We added the remote check to a hostgroup. If we now add hosts to that hostgroup, the hosts will automatically get the checks from that hostgroup.

On your nagios server, create a file in /etc/nagios3/conf.d. How you name it is up to you, but I use a scheme where I name the file after the host I define in it. So, say I want to add a host definition for alpha.example.net, I would use:

sudo vim /etc/nagios3/conf.d/alpha.example.net.cfg

Feel free to add a servers/ folder in conf.d/: nagios load all config files in that directory recursively. Now to the contents of the file. They should look like this:

define host {
host_name alpha
alias alpha.example.net
hostgroups general-servers
address 192.0.2.21
use generic-host
}

The lines host_name and alias are used to identify the host within nagios. Contrary to what you might expect, the host_name is the short version and the alias the long version of the host identification.

The hostgroups line is similar to what we’ve seen before. We have now added this host to just one hostgroup, the one we defined earlier. However, a host can belong to many more hostgroups. You can add it to more by comma-separating a list of hostgroups, like:

hostgroups general-servers,http-servers,critical-servers

The address refers to the IP address of the remote host This may also be a fully qualified domain name, so this is valid as well:

address alpha.example.net

The use statement has been explained before. Feel free to check the definition of generic-host and figure out what it does, it should lead to a better understanding of the host definition.

Now we’ve defined the host, reload (or restart, but reloading is faster) the nagios NRPE server:

sudo service nagios3 reload

…and you’re all set! If you now go to your nagios web interface, you should see the remote host and the services we’ve defined! You can now add all your hosts to nagios and have an overview of the full status of all of your servers!

Full list of service definitions

For reference, the full list of service definitions:

# NRPE Services
define service {
hostgroup_name generic-servers
service_description Current Users NRPE
check_command check_nrpe_1arg!check_users
use generic-service
notification_interval 0
}

define service {
hostgroup_name generic-servers
service_description Current Load NRPE
check_command check_nrpe_1arg!check_load
use generic-service
notification_interval 0
}

define service {
hostgroup_name generic-servers
service_description Disk Space NRPE
check_command check_nrpe_1arg!check_all_disks
use generic-service
notification_interval 0
}

define service {
hostgroup_name generic-servers
service_description Zombie Processes NRPE
check_command check_nrpe_1arg!check_zombie_procs
use generic-service
notification_interval 0
}

define service {
hostgroup_name generic-servers
service_description Total Processes NRPE
check_command check_nrpe_1arg!check_total_procs
use generic-service
notification_interval 0
}

define service {
hostgroup_name generic-servers
service_description Swap NRPE
check_command check_nrpe_1arg!check_swap
use generic-service
notification_interval 0
}

• Author
• Recent Posts

mpkossen

Maarten Kossen was the administrator of LowEndBox from 2013 to 2015, and brought many ideas and improvements to the website during his leadership. Today he is member of our community and LowEndTalk.

Latest posts by mpkossen (see all)

• How to Rapidly Install Java, OpenJDK & Oracle JDK on your VPS - December 14, 2015
• It’s been a great ride - December 14, 2015
• Cheap Windows VPS – $21/quarter 1GB KVM-based Windows VPS in 11 worldwide locations - November 30, 2015