CONTENTS

RELATED RESOURCES

• Page:
  SSH troubleshooting (Knowledge Base)
• Page:
  SSH based Monitoring (Documentation)

This article describes how to monitor hosts with GroundWork Monitor 8 by using SSH to execute monitoring plugins and scripts on remote servers. 

SSH is a secure communication protocol that can be used to login to and execute commands on remote servers. GroundWork can monitor metrics and services on remote servers by using an SSH plugin, check_by_ssh, for checking such services as disk space, load, and memory.

Using the SSH plugin to execute commands on remote hosts is fairly simple in practice, although there are some one-time setup tasks that must be performed first. The remote server requires a running SSHD, which is the daemon program for SSH that listens for a connection. This should normally be installed and running by default. The remote server also requires the public key authentication to be enabled, an ordinary user login (typically nagios, but you can change that  to any other username) with a home directory. In addition, SSH monitoring requires a set of Nagios plugins resident on the remote server. Each variety of servers must get the specific plugins that are compatible, and have the dependencies satisfied so that they will run on that server. You need to obtain and install plugin packages, and some are included on the Downloads page. These packages are for generic Linux hosts and most Linux variants will work with them.  All the required libraries and Perl modules are included. You could also choose to copy plugins from a working target system with matching OS and architecture, or download plugins from alternate sites per the site instructions.

Set up login accounts

Define remote login account in GroundWork

All of the default SSH-based command definitions in GroundWork use the username stored in the user17 nagios macro definition for remote login purposes. The default value for this macro assignment is nagios, although any username can be used.

If you wish to define a different username, perform the following steps:

1. Log in to GroundWork as an Administrator (default is user admin).
2. Select Configuration > Nagios Monitoring > Control > Nagios resource macros.
3. From the list of macros, select the USER17 entry, which will cause the main window to be reloaded with the macro value in an edit field at the top of the screen.
4. Change the USER17 value to a desired username value, click Update.
5. After a commit operation (Configuration > Nagios Monitoring > Control > Commit), the new username will be used for all subsequent SSH service definitions that reference the user17 macro.

Create login account on remote server

The SSH username must have a corresponding user account on each machine that will be used for monitoring purposes. If the user account does not already exist, it must be created before that system can be accessed by the SSH plugin. User account management routines are platform-specific, so you should refer to your system documentation for information on how to create a new user. The username assigned to the account must be identical to the username specified in the user17 macro (the default is nagios), and must also have a home directory that can store the SSH configuration directory and public key.

1. Login to the remote system with an administrative account, such as root, (or use sudo).

2. Create the nagios user with the following command, which also makes a home directory (e.g., username of nagios):
   

   root@remoteserver:~# useradd -m nagios

3. Verify that the user's home directory is owned by the new user account, and has the user permissions to read, write, and enter the directory. For example:
   

   root@remoteserver:~# ls -lad /home/nagios
   drwxr-xr-x 2 nagios nagios 4096 May 11 20:39 /home/nagios

   Some level of access for the account's primary group are often allowed, and this is typically a matter of local administrative policy. However, some SSH servers are known to prohibit logins for accounts with insufficiently secure home directories (refer to your system's SSHD documentation for details).

4. You may want to assign a password for the user account to allow easier access, although a password is not used by the SSH public-key authentication mechanism. For example, the command below is used to assign a password to the default nagios username, which will then prompt for a new user password.
   

   passwd nagios

   It's better not to use passwords at all. Consider using only SSH keys for access to system running SSHD.

5. Now we'll create the SSH directory and set its permissions. Change to the nagios home directory:
   

   root@remoteserver:~# cd /home/nagios

   Create the .ssh directory:

   root@remoteserver:/home/nagios~# mkdir .ssh

   Change access permissions (owner can read, write and execute):

   root@remoteserver:/home/nagios~# chmod 700 .ssh

   Change file ownership and group (e.g., nagios.nagios):

   root@remoteserver:/home/nagios~# chown nagios.nagios .ssh

6. Once the user account has been created and enabled, verify that the user is able to login to the system with the assigned password, and that the user is able to create and edit files in their home directory.

Configure public key authentication

Enable public key authentication on remote server

SSH server configuration mechanisms are system-specific, so you may need to review your system documentation for more information. Most Linux systems use the open source OpenSSH package, which typically uses a configuration file named /etc/ssh/sshd_config, although its location will often vary by distribution. Most Linux distributions with OpenSSH also have public key authentication enabled by default, but if you need to specifically enable it you can do so by changing or adding a PubkeyAuthentication configuration directive with value of yes.

1. Open a terminal session to the remote server.

2. Become the root user:

   user@remoteserver:~$ sudo su -

3. Edit the SSHD configuration file:

   root@remoteserver:~$ vi /etc/ssh/sshd_config

4. Locate the entry for PubkeyAuthentication (under # Authentication) and make sure it's set to yes and not commented out, then save the file:
   

   PubkeyAuthentication yes

5. Restart the daemon if the directive was modified, for example:

   root@remoteserver:~# service sshd restart

Copy keys from GroundWork to remote server

Copy the key on the GroundWork server

1. Open a terminal session to the GroundWork server.

2. Become the user used to install and run GroundWork, and change to the gw8 directory:
   

   user@groundworkserver:~$ cd gw8

   
   

3. Using the docker cp command, copy the id_rsa.pub public key file to the local directory:

   user@groundworkserver:~/gw8$ docker cp dockergw8_nagios_1:/home/nagios/.ssh/id_rsa.pub .

4. Listing the contents of the directory, you should see the id_rsa.pub file:

   user@groundworkserver:~/gw8$ ls
   id_rsa.pub

5. Copy the contents of the file which will be used in the next section, for example, to show the contents on the screen for copy-paste:

   user@groundworkserver:~/gw8$ cat id_rsa.pub

Paste the key on the remote server

1. Open (or return to) a terminal session to the remote server.

2. Become the root user, then the nagios user, and change to the .ssh directory:

   user@remoteserver:~$ sudo su – 
   root@remoteserver:~# su – nagios
   nagios@remoteserver:~$ cd .ssh

3. Edit the authorized_keys file and paste in the previously copied key from above. You can append it to any existing content (don't overwrite unless you are sure you know it's necessary). Then save the file:

   nagios@remoteserver:~/.ssh$ vi authorized_keys

4. Adjust the permissions on the authorized_keys file, as most modern ssh servers require:

   nagios@remoteserver:~/.ssh$ chmod 600 authorized_keys

Test SSH to the remote server from the GroundWork server

Host key verification isn't done from the Nagios container on the GroundWork server, so you don't have to ssh to servers individually and accept the keys. If you want to check that the system is set up correctly to run the plugins, you can test it like this:

1. Open (or return to) a terminal session to the GroundWork server.

2. Change to the gw8 directory:

   user@groundworkserver:~$ cd gw8

3. Using the docker exec command launch a bash session on the Nagios container, you should receive a command prompt as root user on the Nagios container:

   user@groundworkserver:~/gw8$ docker-compose exec nagios bash
   root@a7ce873cfc8c:/src#

4. Become the nagios user on the Nagios container:

   root@a7ce873cfc8c:/src# su - nagios

5. Enter the command to SSH to the remote server. You should then be connected to the remote server from your GroundWork server. To close the connection type exit. You should not be prompted for a password.
   

   $ ssh remoteserver
   nagios@remoteserver:~$ exit

   If you set up a different user with the ssh key, just ssh as that user, for example:

   $ ssh username@remoteserver
   username@remoteserver:~$ exit

Install plugins on the remote server

There are some plugin packages that GroundWork supplies for convenience, which are described here. Please note that there are many other packages of Nagios plugins in existence, and you can use them just as well, if not better, than these. Also, it's possible to use the GDMA plugins if GDMA is installed. See the Using GDMA Plugins with SSH section below for more information. 

1. Open a browser to the Downloads page, and under the section Linux Plugin Packages for SSH Monitoring, download the two plugin files (for 32 or 64).
2. Open a terminal session to your desktop (or where ever the plugin files were downloaded), and copy the downloaded files to the "/ " directory on the remote server.

3. From a terminal session to the remote server, become the root user:

   user@remoteserver:~$ sudo su -
   root@remoteserver:~# 

4. Change file permissions, for example:

   root@remoteserver:~# chmod +x installplugs64.sh

5. Run the script, then verify the files are in the correct directory:
   

   root@remoteserver:~# ./installplugs64.sh nagios
   root@remoteserver:~# cd /home/nagios/libexec
   ls

6. Once all the files have been put into the correct directory on the remote host, you can verify they will work as expected by logging into the GroundWork server under the configured SSH account, and then providing one of the remote commands as a parameter to an ssh command. This will cause the SSH client to login to the remote host with the user's credentials, execute the specified command, display any output that was returned and then close the SSH session. For example, the following command will look at the amount of free swap space in use on the destination host, and generate a warning if there are less than 100 bytes of free swap space, or generate a critical error if there are less than 50 bytes:

   user@groundworkserver:~$ cd gw8
   user@groundworkserver:~/gw8$ docker-compose exec nagios bash
   root@5a9ccde65832:/src# su - nagios
   $ ssh 10.4.50.169 libexec/check_swap -w 100 -c 50
   SWAP OK - 100% free (1349 MB out of 1349 MB) |swap=1349MB;0;0;0;1349

   If this command completes successfully, the Nagios SSH plugin commands should also be able to work as expected.

   Perl interpreter location

   The Perl scripts in the libexec directory are configured to use a Perl interpreter that is expected to be located in /usr/local/groundwork/common/bin, however that directory will not exist on the remote machine by default. You may need to update the Perl scripts to use a different interpreter, or create a symbolic link from an existing Perl interpreter to that directory so that one will be located.

Import profiles and create a new host

Lastly, in GroundWork Monitor, create a new host to be monitored using GroundWork SSH profiles.

Import the SSH profile

First, import the SSH profile to make it available to hosts you want to monitor with SSH. 

1. In GroundWork, select Configuration > Nagios Monitoring > Profiles.
2. Click Profile importer > Import > SSH.
3. Check the box for service-profile-ssh-unix.xml, and click Import.

Create a new host

1. Select Configuration > Nagios Monitoring > Hosts > Host wizard.
2. Enter a Host name, Alias, and the IP address.
3. For Host profile, select host-profile-ssh-unix.
4. Commit the configuration (Configuration > Nagios Monitoring > Control > Commit).
5. That’s it. In the Status Summary (Dashboards > Status), you should start to see functioning GroundWork SSH profile services for the remote host and information coming in along with performance data.

Using GDMA plugins with SSH

In GroundWork 8, you might want to add your own plugins as you did in prior versions. This isn't so easy if you use other packages of plugins, or if the plugins you want to add are ones that are best run actively from the GroundWork server. You can copy them into the nagios container, but resolving the dependencies there will be problematic, since the system volumes that house dependencies are volatile and reset when the nagios container restarts. GDMA plugins can be run via SSH, however, and the dependencies are very much the same as in prior versions (including the perl interpreter, etc.). Also, GDMA allows you to add plugins and dependencies as files to the GroundWork server, which can then be downloaded to all running GDMA installs, so you don't have to touch them all more than once. 

If you have GDMA installed on the remote host(s), you can enable SSH execution of plugins there by the following procedure:

1. Add the Nagios user and keys as described above.

2. Instead of adding plugins from a package, instead add the nagios user to the gdma group on the remote system: 

   root@remoteserver:~/# usermod -g gdma nagios

3. Symlink the libexec directory from gdma to the /home/nagios directory: 

   root@remoteserver:~/# su - nagios
   nagios@remoteserver:~/home/nagios$ ln -s /usr/local/groundwork/nagios/libexec libexec

4. Test the execution of the plugins as above, and proceed to use the profiles, commands, and services as needed. 

   GDMA user in place of Nagios user

   You can alternatively use the gdma user in place of the nagios user. Just add the key to the gdma user ~gdma/.ssh/authorized_keys file, symlink the libexec directory under the gdma home, and change the macro USER17 to gdma in the Nagios resource macros screen.