SSH Monitoring
About SSH
fas fa-forward
SKIP TO TUTORIAL
Secure Shell (SSH) provides encrypted terminal access to remote hosts, and also provides the ability for users to execute commands on the remote hosts. In simple terms, this means a user on one host can use SSH to execute commands on another host in the same basic manner as if those commands were being executed locally, with the extra benefit of having the commands and response data being exchanged across a secure communications channel. This page discusses some best practices and general uses for monitoring hosts using SSH. To see how to set this up in GroundWork Monitor 8, click the link above to jump down to the tutorial.
Nagios provides an SSH plugin that can be used to build a variety of service profiles in GroundWork. In this setup, the administrator defines one or more profiles that describe the commands that need to be executed on the remote hosts, and also defines a global SSH username for remote login purposes. From there, Nagios is able to call the SSH plugin with the necessary parameters as needed, with the plugin establishing a secure connection to the specified host, executing the specified command, and then returning the program output to Nagios for processing.
This model is especially important when you need to monitor something that is only available on the local command line. Locally installed management utilities can often be wrapped by plugins that can be executed over an SSH session.
Apart from specific services that cannot be easily monitored, many administrators also use the SSH plugin to monitor system-level resources by simply copying the entire set of Nagios local commands to a remote host and then using the SSH plugin to login to the remote server and execute those commands. Although this is not technically required (there are several other means of obtaining most of the same data), this route can simplify data collection if an administrator has to use the SSH plugin for other tasks on those systems anyway. SSH is also a commonly allowed protocol (being encrypted helps satisfy security requirements), so this method is often used for convenience.
For these reasons, GroundWork includes the plugins and profiles needed to execute "check_by_ssh" services.
In GroundWork 8, there is less of an opportunity to add custom plugins to the GroundWork internal Nagios container, and so custom plugins can be most easily added, run, and maintained separately via this method, via NRPE, or preferably via GDMA. In fact, installing GDMA on the GroundWork server is an easy way to load all the standard plugins and their dependencies, and gives you an easy way to add custom plugins to GroundWork 8. Combining this with check_by_ssh will let you run your custom plugins with minimal adjustments.
Once the steps in the configuration reference below have been successfully completed, the Nagios SSH plugin should be able to seamlessly login to any target host that has the correct username and public key file. From this point forward, you will only need to be concerned with maintaining the key files for access - the passwords do not matter. One good practice is to disable password authentication on the individual systems, since they will not be needed by the plugin, and are not needed by anyone who has a key. Additional measures for further tightening access to the SSH plugin are discussed here http://www.linuxjournal.com/article/8759, if higher levels of security are needed or desired.
When Nagios issues a command on a remote server, it tries to ensure the correct command is being called by looking for in a specific subdirectory under the primary user account's home directory. In order to ensure the default set of SSH service definitions function as expected, it is necessary to ensure these commands are copied into the appropriate subdirectory on the remote hosts. By default, this subdirectory is specified in the Nagios user22 macro definition, with the value of libexec
(Configuration > Nagios Monitoring > Control > Nagios resource macros).
The GroundWork Distributed Monitoring Agent (GDMA) has an entire set of plugins and dependencies included, and is consistently maintained. These can be used via SSH as well as (or instead of) by the agents poller. GroundWork also provides (much older) downloadable Linux Plugin Packages for SSH Monitoring for both 32 and 64 bit architectures. You may use these packages of compiled plugins for SSH-based monitoring on hosts you want to monitor with GroundWork. These packages are for generic Linux hosts, and are intended for convenience in monitoring legacy Linux systems. Most Linux variants will work. All the required libraries and Perl modules are included.
If the operating system and/or processor architecture of the target host is not supported by the GDMA or the compiled packages, you may need to download alternate plugins from internet sources, and install them on the target host system. Make sure to put them into the libexec
directory under the SSH user account's home directory, and make sure that account has sufficient permission to execute the programs.
The tutorial below describes how to monitor hosts with GroundWork by using SSH to execute monitoring plugins and scripts on remote servers. SSH is a secure communication protocol that can be used to login to and execute commands on remote servers. GroundWork can monitor metrics and services on remote servers by using an SSH plugin, check_by_ssh, for checking such services as disk space, load, and memory.
Using the SSH plugin to execute commands on remote hosts is fairly simple in practice, although there are some one-time setup tasks that must be performed first. The remote server requires a running SSHD which is the daemon program for SSH that listens for a connection and should normally be installed and running by default. The remote server also requires the public key authentication to be enabled, an ordinary user login (typically nagios, but you can change it to any other username) with a home directory. In addition, SSH monitoring requires a set of Nagios plugins resident on the remote server. Each variety of servers must get the specific plugins that are compatible, and have the dependencies satisfied so they will run on that server. You need to obtain and install plugin packages, and some are included on the Downloads page. These packages are for generic Linux hosts and most Linux variants will work with them. All the required libraries and Perl modules are included. You could also choose to copy plugins from a working target system with matching OS and architecture, or download plugins from alternate sites per the site instructions.
SSH Tutorial
This tutorial shows how to monitor hosts with GroundWork using SSH to execute monitoring plugins and scripts on remote servers.
fas fa-chalkboard-teacher WHAT YOU WILL LEARN
GroundWork server SSH configuration
Remote server SSH configuration
fas fa-sign-in-alt GETTING STARTED
Log in to GroundWork Monitor as a user with an Administrator role (e.g., admin/admin) and open terminal sessions for your GroundWork server and Remote server(s).
fas fa-clock 20 30 60 90 MIN
Define remote login account (on GroundWork server)
All of the default SSH-based command definitions in GroundWork use the default username nagios stored in the USER17 macro definition for remote login purposes. Any username can be used. If you wish to define a different username, perform the following steps, otherwise continue to creating a remote login account below.
Log in to GroundWork Monitor as a user with the Admin role, e.g., admin/admin.
Select Configuration > Nagios Monitoring > Control > Nagios resource macros.
From the list of macros, select the USER17 entry, which will cause the main window to be reloaded with the macro value in an edit field at the top of the screen.
Change the USER17 value to a desired username value, click Update.
After a commit operation (Configuration > Nagios Monitoring > Control > Commit), the new username will be used for all subsequent SSH service definitions that reference the USER17 macro.
Create remote login account (on Remote server)
The SSH username, described above, must have a corresponding user account on each remote server that will be used for monitoring purposes. If the user account does not already exist, it must be created before that system can be accessed by the SSH plugin. User account management routines are platform-specific, you may want to refer to your system documentation for information on how to create a new user. The username assigned to the account must be identical to the username specified in the USER17 macro (the default is nagios), and must also have a home directory to store the .ssh configuration directory and public key.
Login to the Remote server with an administrative account, such as root, (or use sudo).
Create the nagios user with the following command, which also makes a home directory:
useradd -m nagios
CODEVerify the user's home directory is owned by the new user account, and has the user permissions to read, write, and enter the directory. The command below will return similar to drwxr-xr-x 3 nagios nagios 4096 Nov 18 19:57 /home/nagios:
ls -lad /home/nagios
CODESome level of access for the account's primary group are often allowed, and this is typically a matter of local administrative policy. However, some SSH servers are known to prohibit logins for accounts with insufficiently secure home directories (refer to your system's SSHD documentation for details).
You may want to assign a password for the user account to allow easier access, although a password is not used by the SSH public key authentication mechanism. For example, the command below is used to assign a password to the default nagios username, which will then prompt for a new user password.
passwd nagios
It's better not to use passwords at all. Consider using only SSH keys for access to system running SSHD.
Now we'll create the SSH directory and set its permissions.
Change to the nagios home directory:
cd /home/nagios
CODECreate the .ssh directory:
mkdir .ssh
CODEChange access permissions (owner can read, write and execute):
chmod 700 .ssh
CODEChange file ownership and group (e.g., nagios.nagios):
chown nagios.nagios .ssh
CODE
Once the user account has been created and enabled, verify the user is able to login to the system (with the assigned password), and the user is able to create and edit files in their home directory.
Enable public key authentication (on Remote server)
SSH server configuration mechanisms are system specific, you may need to review your system documentation for more information. Most Linux systems use the open source OpenSSH package, which typically uses a configuration file named /etc/ssh/sshd_config, although its location will often vary by distribution. Most Linux distributions with OpenSSH also have public key authentication enabled by default, however if you need to specifically enable it you can do so by changing or adding a PubkeyAuthentication configuration directive with value of yes.
Open a terminal session to the Remote server.
Become the root user:
sudo su -
CODEEdit the SSHD configuration file:
vi /etc/ssh/sshd_config
CODELocate the entry for PubkeyAuthentication (under # Authentication:) and make sure it's set to yes and not commented out, then save the file:
PubkeyAuthentication yes
Restart the daemon if the directive was modified, for example:
service sshd restart
CODE
The public key data in the id_rsa.pub file on the GroundWork server can be copied into the authorized_keys file on the Remote server using any means available. You need to be careful to avoid adding any extra characters such as line feeds or carriage returns. The following is a very simple manual example, and there are many automated systems that handle key distribution which can be used instead.
Copy public key data (on GroundWork server)
Open a terminal session to the GroundWork server.
Become the user used to install and run GroundWork (e.g., gwos), and change to the gw8 directory:
sudo su - su - gwos cd gw8
CODEUsing the docker cp command, copy the id_rsa.pub public key file to the local directory:
docker cp dockergw8_nagios_1:/home/nagios/.ssh/id_rsa.pub .
CODEListing the contents of the gw8 directory, you should see the id_rsa.pub file:
ls
CODECopy the contents of the file which will be used in the next section, for example, to show the contents on the screen for copy/paste you can use the following command:
cat id_rsa.pub
CODE
Paste public key data (to Remote server)
Open (or return to) a terminal session to the Remote server.
Become the root user, then the nagios user, and change to the .ssh directory:
sudo su – su – nagios cd .ssh
CODEEdit the authorized_keys file and paste in the previously copied public key from above. You can append it to any existing content (don't overwrite unless you are sure you know it's necessary).
vi authorized_keys
CODEThen save the file.
Adjust the permissions on the authorized_keys file, as most modern ssh servers require:
chmod 600 authorized_keys
CODE
Test SSH login to the Remote server (on GroundWork server)
This process accesses the root shell in the nagios container. Please use Docker commands carefully.
Host key verification is not done from the Nagios container on the GroundWork server, so you do not have to SSH to servers individually and accept the keys. If you want to check the system is set up correctly to run the plugins, you can test as follows:
Open (or return to) a terminal session to the GroundWork server.
Change to the gw8 directory:
cd gw8
CODEUsing the docker exec command launch a bash session on the nagios container, you should receive a command prompt as root user on the nagios container (e.g., root@a7ce873cfc8c:/src#):
docker-compose exec nagios bash
Become the nagios user on the nagios container:
su - nagios
CODEEnter the command to SSH to the Remote server. You should then be connected to the Remote server from your GroundWork server. To close the connection type exit. You should not be prompted for a password.
ssh <remoteserver> exit
If you set up a different user with the SSH key, just SSH as that user, for example:
ssh username@remoteserver exit
Once the steps above have been successfully completed, the Nagios SSH plugin should be able to seamlessly login to any target host that has the correct username and public key file.
Install plugins (on Remote server)
There are some plugin packages GroundWork supplies for convenience. Please note there are many other packages of Nagios plugins in existence, and you can use them just as well, if not better. Also, it's possible to use the GroundWork Monitor Distributed Agent (GDMA) plugins if GDMA is installed. See the section below Using GDMA Plugins with SSH for more information.
Open a browser to the GroundWork Downloads page, navigate to the section Linux Plugin Packages for SSH Monitoring, and download the installer script and plugins package for your architecture (you will need both files for either 32 or 64 bit).
Open a terminal session to your desktop, or where ever the plugin files were downloaded, and copy the downloaded files (each) to the /tmp/ directory on the Remote server.
From a terminal session to the Remote server, become the root user:
sudo su -
CODEChange to the tmp directory, the location the plugin files were copied to:
cd /tmp
CODEChange file permissions, for example:
chmod +x installplugs64.sh
CODERun the script, then verify the files are in the correct directory:
./installplugs64.sh nagios cd /home/nagios/libexec ls
CODEOnce all the files have been put into the correct directory on the Remote server, you can verify they will work as expected by logging into the GroundWork server under the configured SSH account, and then providing one of the remote commands as a parameter to an SSH command. This will cause the SSH client to login to the remote host with the user's credentials, execute the specified command, display any output returned and then close the SSH session.
For example, the command check_by_ssh_swap will look at the amount of free swap space in use on the Remote host, and generate a warning if there are less than 100 bytes of free swap space, or generate a critical error if there are less than 50 bytes:cd gw8 docker-compose exec nagios bash su - nagios
If this command completes successfully, the Nagios SSH plugin commands should also be able to work as expected.
The Perl scripts in the libexec directory are configured to use a Perl interpreter that is expected to be located in /usr/local/groundwork/common/bin, however that directory will not exist on the Remote server by default. You may need to update the Perl scripts to use a different interpreter, or create a symbolic link from an existing Perl interpreter to that directory so that one will be located.
Import SSH profile (from GroundWork UI)
Lastly, in the GroundWork Monitor UI, create a new host to be monitored using SSH profiles. First, import the SSH profiles to make available to hosts you want to monitor with SSH.
In GroundWork, select Configuration > Nagios Monitoring > Profiles.
Click Profile importer > Import > SSH.
Check the box for service-profile-ssh-unix.xml, and click Import.
Then create a new host by selecting Configuration > Nagios Monitoring > Hosts > Host wizard.
Enter a Host name, Alias, and the IP address.
For Host profile, select host-profile-ssh-unix., click Next >> four times.
Commit the configuration by selecting Configuration > Nagios Monitoring > Control > Commit.
That’s it. In the Status dashboard (Dashboards > Status), you should start to see functioning GroundWork SSH profile services for the Remote host and information coming in along with performance data.
Using GDMA Plugins with SSH
In GroundWork Monitor 8, you might want to add your own plugins as you did in prior versions. This isn't so easy if you use other packages of plugins, or if the plugins you want to add are ones that are best run actively from the GroundWork server. You can copy them into the nagios container, but resolving the dependencies there will be problematic, since the system volumes that house dependencies are volatile and reset when the nagios container restarts. GDMA plugins can be run via SSH, however, and the dependencies are very much the same as in prior versions (including the perl interpreter, etc.). Also, GDMA allows you to add plugins and dependencies as files to the GroundWork server, which can then be downloaded to all running GDMA installs, so you don't have to touch them all more than once.
If you have GDMA installed on the Remote host(s), you can enable SSH execution of plugins there by the following procedure:
Add the nagios user and keys as described above.
Instead of adding plugins from a package, instead add the nagios user to the gdma group on the Remote system:
usermod -g gdma nagios
CODESymlink the libexec directory from gdma to the /home/nagios directory:
su - nagios ln -s /usr/local/groundwork/nagios/libexec libexec
Test the execution of the plugins as above, and proceed to use the profiles, commands, and services as needed.
You can alternatively use the gdma user in place of the nagios user. Just add the key to the gdma user ~gdma/.ssh/authorized_keys file, symlink the libexec directory under the gdma home, and change the macro USER17 to gdma in the Nagios resource macros screen.
Related Resources
There is no content with the specified labels