LowEndBox - Cheap VPS, Hosting and Dedicated Server Deals

Server monitoring with Icinga 2 - Part 2: the node (Ubuntu host)

lowendtutorial

It’s been a long, long wait! But it’s finally here, the second part of the Icinga 2 tutorial that I’ve promised a while ago.

Last time, we set up an Icinga 2 server to which we are now going to add a host. Unlike with Nagios, you can add hosts “automatically” in Icinga 2 out of the box. That is, you have to do some things, but configuration is quite easy compared to getting started with NRPE. On top of that, it’s a lot more secure: all communication is secured by TLS with certificates, which Icinga 2 can set up for you automatically.

So, we’re going to add a host to our server. I’m assuming the host is running Ubuntu as well; any recent version will work.

Let’s get started!

The server node

In order to be able to add hosts securely, we have to go to the server and run the following command:

sudo icinga2 node wizard

This will start a wizard which will first ask you whether this is a satellite setup or not. Since this is the server or master you have to say ‘No’ here, to input an ‘n’:

Please specify if this is a satellite setup (‘n’ installs a master setup) [Y/n]: n

It then starts generating keys are certificates required for secured TLS communication. In addition to that, it adds these to the configuration, plus it ensures this server is listed as the master:

information/base: Writing private key to ‘/var/lib/icinga2/ca/ca.key’.
information/base: Writing X509 certificate to ‘/var/lib/icinga2/ca/ca.crt’.
information/cli: Initializing serial file in ‘/var/lib/icinga2/ca/serial.txt’.
information/cli: Generating new CSR in ‘/etc/icinga2/pki/icinga-server.csr’.
information/base: Writing private key to ‘/etc/icinga2/pki/icinga-server.key’.
information/base: Writing certificate signing request to ‘/etc/icinga2/pki/icinga-server.csr’.
information/base: Writing private key to ‘/etc/icinga2/pki/icinga-server.key’.
information/base: Writing certificate signing request to ‘/etc/icinga2/pki/icinga-server.csr’.
information/cli: Signing CSR with CA and writing certificate to ‘/etc/icinga2/pki/icinga-server.crt’.
information/cli: Copying CA certificate to ‘/etc/icinga2/pki/ca.crt’.
information/cli: Dumping config items to file ‘/etc/icinga2/zones.conf’.
information/cli: Created backup file ‘/etc/icinga2/zones.conf.orig’.

It then asks you for the host and port for the API. We have no reason to change these, so leave these empty:

Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

It then finalizes setting up this server as a master my editing some more configuration files:

information/cli: Enabling the APIlistener feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Created backup file ‘/etc/icinga2/features-available/api.conf.orig’.
information/cli: Updating constants.conf.
information/cli: Created backup file ‘/etc/icinga2/constants.conf.orig’.
information/cli: Updating constants file ‘/etc/icinga2/constants.conf’.
information/cli: Updating constants file ‘/etc/icinga2/constants.conf’.
Done.

Now restart your Icinga 2 daemon to finish the installation!

With that done, restart Icinga 2 in order to use the new settings:

sudo service icinga2 restart

And the master is good for now. Let’s move on to the host node!

The host node

On the host node, we’re first going to have to ensure the Icinga 2 repository is present:

add-apt-repository ppa:formorer/icinga

Press ENTER when it asks you to.

Note: If this command gives you an error, run ‘sudo apt-get install software-properties-common’ to get the ‘add-apt-repository’ command!

Once the repository has been added, update apt:

sudo apt-get update

And install Icinga 2:

sudo apt-get install icinga2

With that out of the way, we can initiate the same wizard as we did on the master:

sudo icinga2 node wizard

This time we answer ‘Yes’ when it asks us if this is a satellite setup by just hitting ENTER:

Please specify if this is a satellite setup (‘n’ installs a master setup) [Y/n]:

After which is starts a different wizard:

Starting the Node setup routine…
Please specifiy the common name (CN) [icinga-node]:
Please specifiy the local zone name [icinga-node]:

It asks you for the common name and the local zone name for this server. These default to the system’s hostname in fully qualified domain name format. These master uses the common name to connect to the server and the local zone name to identify it in configuration files. I just let these be.

It then goes on to ask you about your master node:

Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga-server
Do you want to establish a connection to the master from this node? [Y/n]:

Please enter the Common Name of your master node (in my case ‘icinga-server’). This is usually the hostname unless you’ve changed it. Then press ENTER when asked if you want to establish a connection to the master from this node. This triggers the next question:

Please fill out the master connection information:
Master endpoint host (Your master’s IP address or FQDN): icinga-server
Master endpoint port [5665]:
Add more master endpoints? [y/N]:

Enter the master’s IP address or Fully Qualified Domain Name (FQDN) here and accept the default port. Also press ENTER in order to not add more endpoints: we’re just working with one master right now.

Then it’s on to the connection for CSR auto-signing, the bit of magic that makes setting up a secure connection a bit easier for you:

Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [icinga-server]:
Port [5665]:

Accept the defaults here as well, because the master we have entered before is also our server for CSR auto-signing.

After this, Icinga 2 is going to save some configuration on the host node and start the setup of a secure connection. As part of this process the master is contacted:

information/base: Writing private key to ‘/etc/icinga2/pki/icinga-node.key’.
information/base: Writing X509 certificate to ‘/etc/icinga2/pki/icinga-node.crt’.
information/cli: Generating self-signed certifiate:
information/cli: Fetching public certificate from master (icinga-server, 5665):

information/cli: Writing trusted certificate to file ‘/etc/icinga2/pki/trusted-master.crt’.
information/cli: Stored trusted master certificate in ‘/etc/icinga2/pki/trusted-master.crt’.

The certificate that has been set up needs to be signed in order to prove that you’re actually in command of both servers and approve of this secure communication:

Please specify the request ticket generated on your Icinga 2 master.
(Hint: # icinga2 pki ticket –cn ‘icinga-node’):

This means you need to run the following command on the master:

sudo icinga2 pki ticket –cn ‘icinga-node’

And copy the output from that command, which looks like ‘ff84267fca3b0b29c4c88d94706c76f4247cac34’ to the host node.

information/cli: Processing self-signed certificate request. Ticket ‘ff84267fca3b0b29c4c88d94706c76f4247cac34’.

information/cli: Created backup file ‘/etc/icinga2/pki/icinga-node.crt.orig’.
information/cli: Writing signed certificate to file ‘/etc/icinga2/pki/icinga-node.crt’.
information/cli: Writing CA certificate to file ‘/etc/icinga2/pki/ca.crt’.

With all certificates signed and in place, we’re asked about the API again:

Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Just like at the master, we’re not going to touch these here.

The wizard now asks you whether to accept the configuration and commands from the master:

Accept config from master? [y/N]: y
Accept commands from master? [y/N]: y

Select ‘Yes’ for both. Unfortunately, the Icinga 2 documentation is a bit fuzzy on this part. There are several ways to setup up a client, for example with a local configuration or as an execution bridge. I’m aiming for the latter of the two here: the master is in control and sends commands, the host node just executes them and returns the results. This is closest to what NRPE does and should keep all the data on the master.

With this done, everything is being put in place to make this work:

information/cli: Disabling the Notification feature.
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Enabling the Apilistener feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Created backup file ‘/etc/icinga2/features-available/api.conf.orig’.
information/cli: Generating local zones.conf.
information/cli: Dumping config items to file ‘/etc/icinga2/zones.conf’.
information/cli: Created backup file ‘/etc/icinga2/zones.conf.orig’.
information/cli: Updating constants.conf.
information/cli: Created backup file ‘/etc/icinga2/constants.conf.orig’.
information/cli: Updating constants file ‘/etc/icinga2/constants.conf’.
information/cli: Updating constants file ‘/etc/icinga2/constants.conf’.
Done.

Now restart your Icinga 2 daemon to finish the installation!

After which you need to restart Icinga 2 on the host node:

sudo service icinga2 restart

And it’s back to the master.

Back to the server node (the master)

With the host node set up properly, only a few things remain on the master to be set up. First of all, we want to list the nodes on the master to see if our new host node is in there:

sudo icinga2 node list

The output should include the node you’ve just added.

Then, we update the configuration on the master so the host node is being included in checks, or in other words is being added to the master:

sudo icinga2 node update-config

The only thing that remains right now is to reload Icinga 2:

sudo service icinga2 reload

And we’re done! After a few minutes results should start popping up in the web interface!

mpkossen

26 Comments

  1. MP:

    This is awesome. Do you happen to have any instructions or a guide to make a windows machine a host node?? Thanks for the great tutorials.

    September 6, 2015 @ 12:40 am | Reply
  2. September 10, 2015 @ 1:58 pm | Reply
  3. Daniel:

    Great Tutorial. When i do “icinga2 node list”
    Its shows nothing on the server. And also nothing comes up
    in the web gui. Everything went thru without problems to this point.
    Any idea why its not listed?

    thanks

    October 16, 2015 @ 12:50 pm | Reply
    • Markus:

      @Daniel
      same problem here. Everything just worked fine, but i get an empty node list.
      Anyone with a solution?

      October 20, 2015 @ 10:26 am | Reply
      • Markus:

        Just did ‘icinga2 node update-config’ again and now it worked. Still dunno why it failed the first time.

        October 20, 2015 @ 10:29 am | Reply
        • siavash:

          I had same problem, after changing permission of icinga2 directory to 755 ( with -R ) the problem solved.

          January 2, 2016 @ 8:30 am | Reply
  4. Elbert:

    Thanks! This was really helpful in getting Icinga2 started. Are there instructions on how to add service checks on non-standard services that’s bundled with the setup from Parts 1 and 2?

    I would like to check ntp or mysql on a remote node. Will there be a Part 3?

    October 20, 2015 @ 3:54 pm | Reply
  5. Martin W:

    Thanks for this tutorial! Awesome, I find the docs on Icinga somewhat vague. The last two commands saved my day. Any chance you will also create a tutorial for load monitoring and displaying it using graphite/pnp4nagios or somethings else (don’t really know yet what does what and how to configure).

    October 29, 2015 @ 8:43 pm | Reply
  6. Mike:

    Hi. Your guides have been amazing thus far. I’ve had to modify a few things (running the latest Web 2 2.1.0 on Ubuntu Server 14.04.3), but it’s been pretty smooth.

    The part I’m stuck on is after the keys are generated on both sides, and I update check the node list on the server, I see my machine and the checks in the terminal, but then when I go to update the config, there are errors. It complains about zones not existing, and when I go into the actual commands themselves, they’re all importing satellite-commands and the check commands are all “check_dummy”.

    Any ideas? Desperate here.

    November 19, 2015 @ 12:34 am | Reply
    • Inge:

      I have the same problem. Error: Validation failed for object ‘host-name’ of type ‘Zone’; Attribute ‘parent’: Object ‘icinga-server’ of type ‘Zone’ does not exist. Any ideas ?

      November 23, 2015 @ 2:25 pm | Reply
      • bpbp:

        I had the same issue. I needed to update “zones.conf” and change:

        object Zone “master” {

        to

        object Zone “your.master.fqdn” {

        Looks like a bug in the node wizard for masters.

        November 27, 2015 @ 2:56 am | Reply
        • Inge:

          Solved! =)

          December 1, 2015 @ 8:16 am | Reply
        • Mike:

          Hi, I’m still a bit confused. I don’t have a FQDN. Would I just be using the IP address of my monitoring server?

          You’ve been so much help, thank you so much!

          January 21, 2016 @ 8:57 pm | Reply
          • Wulfy:

            most monitoring systems want a fqdn and wont accept just an ip, you can bypass this by using something like no-ip or dydns or adding an entry to the servers hosts file (i guess)

            January 21, 2016 @ 9:28 pm | Reply
  7. Starex:

    There are much tutorial on the web about this, but this documentations saves my live. It really works fine. Thx for the effort.

    December 10, 2015 @ 3:12 pm | Reply
  8. Gilbert Aispuro:

    Genius!

    December 29, 2015 @ 9:39 pm | Reply
  9. After struggeling for days with the initial setup this was the first really helpful and comprehensive tutorial I found. Setup worked like a breeze.
    For people running their hosts on CentOS like I do the installation is slightly different btw.

    Just run
    rpm –import http://packages.icinga.org/icinga.key

    wget http://packages.icinga.org/epel/ICINGA-snapshot.repo -O /etc/yum.repos.d/ICINGA-snapshot.repo

    yum makecache

    yum install icinga2 -y

    then follow the rest of this tutorial!

    January 26, 2016 @ 9:51 am | Reply
  10. IcingaNewbie:

    Thank you for posting this lucid and succinct guide. This is easily one of the best Icinga2 installation guides I have found online. The Icinga devs should use this guide as an example of what good documentation looks like.

    That being said I’m running into the same problem as Mike and Inge. When I execute the command to update the config file on the master ‘icinga2 node update-config’ I get the a list of errors:

    critical/config: Error: Validation failed for object ‘example.hostname.fqdn!apt’ of type ‘Service’; Attribute ‘zone’: Object ‘example.hostname.fqdn’ of type ‘Zone’ does not exist.

    I have tried user bpbp’s tip and verified /etc/icinga2/zones.conf has the correct host name pointing to the master host.

    … object Zone “your.master.fqdn” …

    but I still get the same error when I try and update the config. The documentation on Icinga’s site while very verbose is also poorly suited to help troubleshoot this particular problem. Any help would be greatly appreciated.

    February 6, 2016 @ 12:40 am | Reply
  11. akhil vasu:

    I had add remot host to the conf file but ist not load the web interface. why i had reload the icinga2 and also

    icinga2 node update-config
    information/cli: Updating node configuration for
    Node ‘ip-192-168-1-221.ec2.internal’ (last seen: Sat Feb 20 09:13:58 2016)
    * Host ‘ip-192-168-1-221.ec2.internal’
    * Service ‘apt’
    * Service ‘disk’
    * Service ‘disk /’
    * Service ‘http’
    * Service ‘icinga’
    * Service ‘load’
    * Service ‘ping4’
    * Service ‘ping6’
    * Service ‘procs’
    * Service ‘ssh’
    * Service ‘swap’
    * Service ‘users’
    please give a solution regarding the same.

    February 20, 2016 @ 9:20 am | Reply
    • akhil vasu,

      You need to use the Public DNS names not the Private DNS names. Log into AWS and get the listing that looks similar to:

      ec2-111-222-333-444.us-west-2.compute.amazonaws.com

      Hope this helps!

      April 11, 2016 @ 5:20 am | Reply
    • Ted Dargis:

      akhil vasu,

      You need to use the Public DNS not the Private DNS from your AWS console. It should look like:

      ec2-111-222-333-444.us-west-2.compute.amazonaws.com

      Hope this helps.

      April 11, 2016 @ 5:24 am | Reply
  12. Hi, would it be possible to integrate icinga on a ios platform?

    June 3, 2016 @ 11:16 am | Reply
  13. Mr Zog:

    No No No. HOW, bloody HOW do you copy the key to the satellite? You do NOT!
    You run ‘icinga2 node wizard’ on the satellite and paste in the key.

    March 14, 2017 @ 8:14 pm | Reply
  14. lauren:

    ok perfect, easy for a linux or windows system
    But if you want to add for example http://www.hw-group.com/software/Nagios/HWg-STE_nagios_plugin_en.html
    How do we do ?

    April 13, 2017 @ 2:05 pm | Reply
  15. Amanullah H:

    how i integrate the api with icinga2

    March 22, 2018 @ 7:40 am | Reply
  16. Saravanan:

    Icinga web console is running. I am able to connect to Master. But Icinga2 master is not listening on port 5665.

    root@ip-172-31-0-124:/etc/icinga2# ps -ef | grep icinga2
    nagios 2024 1 0 15:23 ? 00:00:00 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 –no-stack-rlimit daemon –close-stdio -e
    nagios 2057 2024 0 15:23 ? 00:00:00 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 –no-stack-rlimit daemon –close-stdio -e
    root 2079 1735 0 15:28 pts/0 00:00:00 grep –color=auto icinga2
    root@ip-172-31-0-124:/etc/icinga2#

    root@ip-172-31-0-124:/etc/icinga2# netstat -ntp | grep 5665
    root@ip-172-31-0-124:/etc/icinga2#

    While adding client to master i receive below error:
    critical/TcpSocket: Invalid socket: 10060, “A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.”

    critical/pki: Cannot connect to host ‘13.126.37.64’ on port ‘5665’

    critical/cli: Failed to fetch certificate from host.
    Please assist.

    November 9, 2018 @ 10:31 am | Reply

Leave a Reply to Markus Cancel reply

Some notes on commenting on LowEndBox:

  • Do not use LowEndBox for support issues. Go to your hosting provider and issue a ticket there. Coming here saying "my VPS is down, what do I do?!" will only have your comments removed.
  • Akismet is used for spam detection. Some comments may be held temporarily for manual approval.
  • Use <pre>...</pre> to quote the output from your terminal/console, or consider using a pastebin service.

Your email address will not be published. Required fields are marked *