Secure Shell (SSH) provides encrypted terminal access to remote hosts, and also provides the ability for users to execute commands on the remote hosts. In simple terms, this means that a user on one host can use SSH to execute commands on another host in the same basic manner as if those commands were being executed locally, with the extra benefit of having the commands and response data being exchanged across a secure communications channel.This article discusses some best practices and general uses of SSH for monitoring. To see how to set this up in GroundWork 8, jump down to Configuring SSH.
Nagios provides an SSH plugin that can be used to build a variety of service profiles in GroundWork Monitor 8. In this setup, the administrator defines one or more profiles that describe the commands that need to be executed on the remote hosts, and also defines a global SSH username for remote login purposes. From there, Nagios is able to call the SSH plugin with the necessary parameters as needed, with the plugin establishing a secure connection to the specified host, executing the specified command, and then returning the program output to Nagios for processing.
This model is especially important when you need to monitor something that is only available on the local command line. Locally installed management utilities can often be wrapped by plugins that can be executed over an SSH session.
Apart from specific services that cannot be easily monitored, many administrators also use the SSH plugin to monitor system-level resources by simply copying the entire set of Nagios local commands to a remote host and then using the SSH plugin to login to the remote server and execute those commands. Although this is not technically required (there are several other means of obtaining most of the same data), this route can simplify data collection if an administrator has to use the SSH plugin for other tasks on those systems anyway. SSH is also a commonly allowed protocol (being encrypted helps satisfy security requirements), so this method is often used for convenience.
For these reasons, GroundWork Monitor 8 includes the plugins and profiles needed to execute "check_by_ssh" services.
In GroundWork Monitor 8, there is less of an opportunity to add custom plugins to the GroundWork internal Nagios container, and so custom plugins can be most easily added, run, and maintained separately via this method, via NRPE, or preferably via GDMA. In fact, installing GDMA on the GroundWork server is an easy way to load all the standard plugins and their dependencies, and gives you an easy way to add custom plugins to GroundWork Monitor 8. Combining this with check_by_ssh will let you run your custom plugins with minimal adjustments.
Once the steps in the configuration reference below have been successfully completed, the Nagios SSH plugin should be able to seamlessly login to any target host that has the correct username and public key file. From this point forward, you will only need to be concerned with maintaining the key files for access - the passwords do not matter. One good practice is to disable password authentication on the individual systems, since they will not be needed by the plugin, and are not needed by anyone who has a key. Additional measures for further tightening access to the SSH plugin are discussed here http://www.linuxjournal.com/article/8759, if higher levels of security are needed or desired.
When Nagios issues a command on a remote server, it tries to ensure that the correct command is being called by looking for in a specific subdirectory under the primary user account's home directory. In order to ensure that the default set of SSH service definitions function as expected, it is necessary to ensure that these commands are copied into the appropriate subdirectory on the remote hosts. By default, this subdirectory is specified in the Nagios user22 macro definition, with the value of
libexec (Configuration > Nagios Monitoring > Control > Nagios resource macros).
The GroundWork Distributed Monitoring Agent (GDMA) has an entire set of plugins and dependencies included, and is consistently maintained. These can be used via SSH as well as (or instead of) by the agents poller. GroundWork also provides (much older) downloadable Linux Plugin Packages for SSH Monitoring for both 32 and 64 bit architectures. You may use these packages of compiled plugins for SSH-based monitoring on hosts that you want to monitor with GroundWork. These packages are for generic Linux hosts, and are intended for convenience in monitoring legacy Linux systems. Most Linux variants will work. All the required libraries and Perl modules are included.
If the operating system and/or processor architecture of the target host is not supported by the GDMA or the compiled packages, you may need to download alternate plugins from internet sources, and install them on the target host system. Make sure to put them into the
libexec directory under the SSH user account's home directory, and make sure that account has sufficient permission to execute the programs.
This article describes how to monitor hosts with GroundWork Monitor 8 by using SSH to execute monitoring plugins and scripts on remote servers.
SSH is a secure communication protocol that can be used to login to and execute commands on remote servers. GroundWork can monitor metrics and services on remote servers by using an SSH plugin, check_by_ssh, for checking such services as disk space, load, and memory.
Using the SSH plugin to execute commands on remote hosts is fairly simple in practice, although there are some one-time setup tasks that must be performed first. The remote server requires a running SSHD, which is the daemon program for SSH that listens for a connection. This should normally be installed and running by default. The remote server also requires the public key authentication to be enabled, an ordinary user login (typically nagios, but you can change that to any other username) with a home directory. In addition, SSH monitoring requires a set of Nagios plugins resident on the remote server. Each variety of servers must get the specific plugins that are compatible, and have the dependencies satisfied so that they will run on that server. You need to obtain and install plugin packages, and some are included on the Downloads page. These packages are for generic Linux hosts and most Linux variants will work with them. All the required libraries and Perl modules are included. You could also choose to copy plugins from a working target system with matching OS and architecture, or download plugins from alternate sites per the site instructions.
Define remote login account in GroundWork
All of the default SSH-based command definitions in GroundWork use the username stored in the user17 nagios macro definition for remote login purposes. The default value for this macro assignment is nagios, although any username can be used.
If you wish to define a different username, perform the following steps:
- Log in to GroundWork as an Administrator (default is user admin).
- Select Configuration > Nagios Monitoring > Control > Nagios resource macros.
- From the list of macros, select the USER17 entry, which will cause the main window to be reloaded with the macro value in an edit field at the top of the screen.
- Change the USER17 value to a desired username value, click Update.
- After a commit operation (Configuration > Nagios Monitoring > Control > Commit), the new username will be used for all subsequent SSH service definitions that reference the user17 macro.
Create login account on remote server
The SSH username must have a corresponding user account on each machine that will be used for monitoring purposes. If the user account does not already exist, it must be created before that system can be accessed by the SSH plugin. User account management routines are platform-specific, so you should refer to your system documentation for information on how to create a new user. The username assigned to the account must be identical to the username specified in the user17 macro (the default is nagios), and must also have a home directory that can store the SSH configuration directory and public key.
- Login to the remote system with an administrative account, such as root, (or use sudo).
Create the nagios user with the following command, which also makes a homedirectory (e.g., username of nagios):
useradd -m nagiosCODE
Verify that the user's home directory is owned by the new user account, and has the user permissions to read, write, and enter the directory. The command below will return similar to drwxr-xr-x 2 nagios nagios 4096 May 11 20:39 /home/nagios:
ls -lad /home/nagiosCODE
Some level of access for the account's primary group are often allowed, and this is typically a matter of local administrative policy. However, some SSH servers are known to prohibit logins for accounts with insufficiently secure home directories (refer to your system's SSHD documentation for details).
You may want to assign a password for the user account to allow easier access, although a password is not used by the SSH public-key authentication mechanism. For example, the command below is used to assign a password to the default nagios username, which will then prompt for a new user password.
It's better not to use passwords at all. Consider using only SSH keys for access to system running SSHD.
Now we'll create the SSH directory and set its permissions. Change to the nagios home directory:
Create the .ssh directory:
Change access permissions (owner can read, write and execute):
chmod 700 .sshCODE
Change file ownership and group (e.g., nagios.nagios):
chown nagios.nagios .sshCODE
- Once the user account has been created and enabled, verify that the user is able to login to the system with the assigned password, and that the user is able to create and edit files in their home directory.
Public Key Authentication
Enable public key authentication on remote server
SSH server configuration mechanisms are system-specific, so you may need to review your system documentation for more information. Most Linux systems use the open source OpenSSH package, which typically uses a configuration file named /etc/ssh/sshd_config, although its location will often vary by distribution. Most Linux distributions with OpenSSH also have public key authentication enabled by default, but if you need to specifically enable it you can do so by changing or adding a PubkeyAuthentication configuration directive with value of yes.
Open a terminal session to the remote server.
Become the root user:
sudo su -CODE
Edit the SSHD configuration file:
Locate the entry for PubkeyAuthentication (under # Authentication) and make sure it's set to yes and not commented out, then save the file:
Restart the daemon if the directive was modified, for example:
service sshd restartCODE
Copy Keys from GroundWork to Remote Server
The public key data can be copied into the authorized_keys file using any means available, but you must be careful to avoid adding any extra characters such as line feeds or carriage returns. The following is a very simple manual example, and there are many automated systems that handle key distribution which can be used instead.
Copy the key on the GroundWork server
- Open a terminal session to the GroundWork server.
Become the user used to install and run GroundWork, and change to the gw8 directory:
Using the docker cp command, copy the id_rsa.pub public key file to the local directory:
docker cp dockergw8_nagios_1:/home/nagios/.ssh/id_rsa.pub .CODE
Listing the contents of the directory, you should see the id_rsa.pub file:
Copy the contents of the file which will be used in the next section, for example, to show the contents on the screen for copy-paste:
Paste the key on the remote server
- Open (or return to) a terminal session to the remote server.
Become the root user, then the nagios user, and change to the .ssh directory:
sudo su – su – nagios cd .sshCODE
Edit the authorized_keys file and paste in the previously copied key from above. You can append it to any existing content (don't overwrite unless you are sure you know it's necessary). Then save the file:
Adjust the permissions on the authorized_keys file, as most modern ssh servers require:
chmod 600 authorized_keysCODE
Test SSH to the remote server from the GroundWork server
This process accesses the root shell in the nagios container. Please use Docker commands carefully.
Host key verification isn't done from the Nagios container on the GroundWork server, so you don't have to ssh to servers individually and accept the keys. If you want to check that the system is set up correctly to run the plugins, you can test it like this:
Open (or return to) a terminal session to the GroundWork server.
Change to the gw8 directory:
Using the docker exec command launch a bash session on the nagios container, you should receive a command prompt as root user on the nagios container (e.g., root@a7ce873cfc8c:/src#):
Become the nagios user on the nagios container:
su - nagiosCODE
Enter the command to SSH to the remote server. You should then be connected to the remote server from your GroundWork server. To close the connection type exit. You should not be prompted for a password.
If you set up a different user with the ssh key, just ssh as that user, for example:
Once the steps above have been successfully completed, the Nagios SSH plugin should be able to seamlessly login to any target host that has the correct username and public key file.
Install Plugins on Remote Server
There are some plugin packages that GroundWork supplies for convenience, which are described here. Please note that there are many other packages of Nagios plugins in existence, and you can use them just as well, if not better, than these. Also, it's possible to use the GDMA plugins if GDMA is installed. See the section below Using GDMA Plugins with SSH for more information.
- Open a browser to the Downloads page, and under the section Linux Plugin Packages for SSH Monitoring, download the two plugin files (for 32-bit or 64-bit).
- Open a terminal session to your desktop (or where ever the plugin files were downloaded), and copy the downloaded files to the "/ " directory on the remote server.
From a terminal session to the remote server, become the root user:
sudo su -CODE
Change file permissions, for example:
chmod +x installplugs64.shCODE
Run the script, then verify the files are in the correct directory:
./installplugs64.sh nagios cd /home/nagios/libexec lsCODE
Once all the files have been put into the correct directory on the remote host, you can verify they will work as expected by logging into the GroundWork server under the configured SSH account, and then providing one of the remote commands as a parameter to an ssh command. This will cause the SSH client to login to the remote host with the user's credentials, execute the specified command, display any output that was returned and then close the SSH session. For example, the following command will look at the amount of free swap space in use on the destination host, and generate a warning if there are less than 100 bytes of free swap space, or generate a critical error if there are less than 50 bytes:
If this command completes successfully, the Nagios SSH plugin commands should also be able to work as expected.
The Perl scripts in the libexec directory are configured to use a Perl interpreter that is expected to be located in /usr/local/groundwork/common/bin, however that directory will not exist on the remote machine by default. You may need to update the Perl scripts to use a different interpreter, or create a symbolic link from an existing Perl interpreter to that directory so that one will be located.
Import Profiles and Create a New Host
Lastly, in GroundWork Monitor, create a new host to be monitored using GroundWork SSH profiles.
Import the SSH profile
First, import the SSH profile to make it available to hosts you want to monitor with SSH.
- In GroundWork, select Configuration > Nagios Monitoring > Profiles.
- Click Profile importer > Import > SSH.
- Check the box for service-profile-ssh-unix.xml, and click Import.
Create a new host
- Select Configuration > Nagios Monitoring > Hosts > Host wizard.
- Enter a Host name, Alias, and the IP address.
- For Host profile, select host-profile-ssh-unix.
- Commit the configuration (Configuration > Nagios Monitoring > Control > Commit).
- That’s it. In the Status dashboard (Dashboards > Status), you should start to see functioning GroundWork SSH profile services for the remote host and information coming in along with performance data.
Using GDMA Plugins with SSH
In GroundWork 8, you might want to add your own plugins as you did in prior versions. This isn't so easy if you use other packages of plugins, or if the plugins you want to add are ones that are best run actively from the GroundWork server. You can copy them into the nagios container, but resolving the dependencies there will be problematic, since the system volumes that house dependencies are volatile and reset when the nagios container restarts. GDMA plugins can be run via SSH, however, and the dependencies are very much the same as in prior versions (including the perl interpreter, etc.). Also, GDMA allows you to add plugins and dependencies as files to the GroundWork server, which can then be downloaded to all running GDMA installs, so you don't have to touch them all more than once.
If you have GDMA installed on the remote host(s), you can enable SSH execution of plugins there by the following procedure:
Add the nagios user and keys as described above.
Instead of adding plugins from a package, instead add the nagios user to the gdma group on the remote system:
usermod -g gdma nagiosCODE
Symlink the libexec directory from gdma to the /home/nagios directory:
Test the execution of the plugins as above, and proceed to use the profiles, commands, and services as needed.
You can alternatively use the gdma user in place of the nagios user. Just add the key to the gdma user ~gdma/.ssh/authorized_keys file, symlink the libexec directory under the gdma home, and change the macro USER17 to gdma in the Nagios resource macros screen.