About Using HTTPS

Many customers prefer the security of HTTPS when GDMA clients contact the GroundWork server to auto-register the client, and to download configuration files and plugins from the server to the client, as well as for the posting of results of service checks. GroundWork 8.x makes HTTPS access to the GroundWork server a standard part of its operation, so it is important to understand how GDMA clients must be configured to interact with the server.Starting with GDMA 2.3.2, a GDMA client configured to use HTTPS for these purposes insisted on having a valid server certificate file installed on the client, so the client can ensure it has a trustworthy connection to the server. This is the basic protection required to prevent a man-in-the-middle (MITM) attack. Previous releases of GDMA only supported the encryption capabilities of SSL, and did not use certificates to otherwise protect the data on the wire. So any site using an older release that wishes to enforce this extra level of protection should upgrade to the the latest release, which is currently GDMA 2.7.1.

Starting with GDMA 2.7.1, the handling of TLS certificates has changed. The previous model of necessarily installing a server certificate on the GDMA client has changed to a model that more closely resembles how an ordinary web browser operates. In that model, whether or not you need to install certificates on the GDMA client depends on the details of how you provision the TLS certificates that are installed on the server. Customers who are able to provision their GroundWork server with TLS certificates obtained from a common public Certificate Authority will not need to install any certificates on the GDMA client. Instead, the client will analyze the certs sent to it by the server, and validate them against a standard set of root-CA certs built into the client. Customers in other situations will need to take steps similar to but now different from the client-side cert installation that was used in older GDMA releases. For details, see the Certificates with Options section below.

In addition to certificate handling, another consideration applies. Modern HTTPS uses the TLS 1.2 protocol or even TLS 1.3 for Transport Layer Security, and earlier versions of TLS (and all versions of its predecessor SSL) are now deprecated. GroundWork 8.x is configured by default to use TLS 1.2 when interacting with clients via HTTPS. It is therefore important that the GDMA client be able to talk this protocol, which means the use of GDMA 2.5.0 or later (with some exceptions, not described here) on the clients.

Dealing with Certificates

GroundWork 8.x supports HTTPS on the server side out of the box, so while no configuration is strictly necessary, you will probably want to add your own, valid certificates to the GroundWork 8 server. The self-signed certs it includes will eventually expire. They are good for a year from installation, and self-signed certs are not terribly secure in the first place, so managing your own server-side certificates is the preferred solution. See Adding Certificates to HTTPS for instructions on managing certificates, and Generating Let's Encrypt SSL Certificates for an example of generating a valid CA certificate for free. 

For purposes of supporting GDMA over HTTPS, the critical files are the server private key file (which we will call privkey.pem), the server certificate (which we will call server.crt), and possibly an intermediate certificate (intermediate.pem). Your actual server names may vary, of course. These are just examples. 

For GDMA purposes, you might need a certificate related to your GroundWork server, as described in Certificates with Options below, and (if desired) a Certificate Revocation List (CRL) file. The CRL file is allowed and used if present, but is not required by the GDMA client software.  You may put one in play even if you know of no certificates that should be revoked. The GDMA software uses the CRL so if you ever actually want to revoke certain certificates you have the built-in capability to do so.

Managing TLS certificates with OpenSSL is a broad topic. Here, we will only scratch the surface. If you want more details, dive into the OpenSSL documentation as well as other sources on the Internet.

Specifics of the server-side configuration for supporting GDMA are given in the GroundWork server setup for HTTPS section below.

Forewarned should be forearmed

Managing TLS certificates in your infrastructure is a long-term process. Realizing that and acting on that realization beforehand will save you considerable grief later on.

  • Certificates and certificate revocation lists will eventually be replaced. Partly because they are distributed across a large number of machines, it pays to have well-understood and practiced procedures already established when you need to do this in a hurry.
  • You must carefully specify certificate and revocation-list lifetimes, be aware when they are about to expire, and be proactive about replacing them on all your machines before that happens.
  • Particularly because those lifetimes are often specified in very long time periods, such as several years, people are likely to forget about upcoming expiration times by the time they finally do occur. Establish some way to keep track of such infrequent maintenance activities in a way that it won't be a sudden surprise when the expiration date arrives. Or establish and test automated procedures for upgrading such files, and replace them more frequently. 
  • Certificate management is made a lot easier if you establish certain conventions from the beginning, such as numbering your certificate and revocation-list files so you can tell which is which when new ones appear in your infrastructure. The commands we show in this document adopt a simple standard for such numbering. You can adopt whatever policies and conventions you want for your own site, but you should definitely think about such issues in advance of deploying HTTPS in your infrastructure.

Certificates with Options

This section applies to GDMA 2.7.1 or later and provides installation notes which are critical for security.

Any installation of GroundWork Monitor that uses HTTPS, which is highly recommended, has to deal with TLS certificates and maintenance. The recommended approach is to obtain TLS certs from a public or private Certificate Authority, depending on your particular situation.

This release of GDMA simplifies TLS certificate handling on the client side, by whenever possible making automatic the process of using certs provided by the server, much as common browsers do. If your server uses certs which are provided by one of the standard public Certificate Authorities, you should no longer need to manually install TLS certs on the GDMA client. That said, in this release, we still rely on a manually-installed Certificate Revocation List to understand that particular certs should be treated as having been revoked.

To use a TLS certificate from a common public Certificate Authority, set up the GroundWork Monitor server and the GDMA client as outlined below.

  • If you obtained your certs from an uncommon public CA, its root CA public certificate might not already be known to the GDMA client. That case can be handled as though it were a private CA, requiring installation of the root CA public certificate on the GDMA client. See the next item instead of this one.
  • On the GroundWork Monitor server:
    • Install the server private key, the server leaf public certificate, and the set of intermediate public certificates, in that order on the command you use to perform the cert installation. In that process, you must NOT include the public-CA root CA public certificate.  See Adding Certificates to HTTPS for the essential commands.
  • On the GDMA client:
    • To keep your setup secure, you must explicitly disable allowing downloaded self-signed certs to be used, on the client itself. That is done either by having the GDMA installer do so by careful selection of the option to do so, or by manual adjustment in the gdma_auto.conf file on the GDMA client after installation.
    • No manual install of any public certs (leaf, intermediate, or root) is needed on the GDMA client.

To use a TLS certificate from a private Certificate Authority, setup is needed on both the server and the client. In brief, for a GroundWork 8.x based system, the process is:

  • On the GroundWork Monitor server:
    • Install the server private key, the server leaf public certificate, and the set of intermediate public certificates, in that order on the command you use to perform the cert installation. In that process, you must NOT include the private-CA root CA public certificate.  See Adding Certificates to HTTPS for the essential commands.

  • On the GDMA client:

    • To keep your setup secure, you must explicitly disable allowing downloaded self-signed certs to be used, on the client itself. That is done either by having the GDMA installer do so by careful selection of the option to do so, or by manual adjustment in the gdma_auto.conf file on the GDMA client after installation.

    • Install only the private-CA root CA public certificate, in the groundwork/gdma/certs/ directory, in a file with a .pem filename extension.  Most of the slight inconvenience of installing the cert on the client, along with the following steps, can be covered by extending whatever automation processes you use to install the GDMA client in the first place.

    • Make sure the private-CA root CA public certificate file you just installed is owned by the same user you have installed the GDMA client to run as, and make sure it is only writable by the owner of the file.

    • Run c_rehash on the directory containing the private-CA root CA public certificate.

      • On a Windows system, the commands would be like:

        cd {path-to}\gdma\certs
        ..\..\common\bin\c_rehash

      • On a Linux or AIX system, the commands would be:

        cd /usr/local/groundwork/gdma/certs
        ../../common/bin/c_rehash ./

      • On a Solaris system, the commands would be:

        cd /opt/groundwork/gdma/certs
        ../../common/bin/c_rehash ./

If you use a self-signed TLS certificate on your GroundWork Monitor server, you must choose which type of setup to establish on the GDMA client.

  • The following choice is the safe way to handle a server's self-signed certificate.  On the GDMA client:

    • To keep your setup secure, you must explicitly disable allowing downloaded self-signed certs to be used, on the client itself.  That is done either by having the GDMA installer do so by careful selection of the option to do so, or by manual adjustment in the gdma_auto.conf file on the GDMA client after installation.

    • Install the self-signed public cert manually in the groundwork/gdma/certs/ directory on each GDMA client, in a file with a .pem filename extension. Most of the slight inconvenience of installing the cert on the client, along with the following steps, can be covered by extending whatever automation processes you use to install the GDMA client in the first place.

    • Make sure the self-signed public certificate file you just installed is owned by the same user you have installed the GDMA client to run as, and make sure it is only writable by the owner of the file.

    • Run c_rehash on that directory, using the platform-specific commands shown in the previous item.

  • Alternatively, you may allow the GDMA client to trust a self-signed cert when it is downloaded directly from the server it contacts, using options available at install time and reflected in the client's groundwork/gdma/config/gdma_auto.conf file. This behavior is defaulted as enabled, but note that leaving it set that way is inherently insecure, and definitely not recommended. An attacker may substitute their own certificates and get them installed on the GDMA client, which will thereafter treat them as trusted. This mode of operation is provided mainly as a convenience for initial proof-of-concept installations, before you shift to some other mechanism for providing certs (manually-installed self-signed certs, certs provided by a private Certificate Authority within your infrastructure, or certs provided by a well-known public Certificate Authority).

    • Allowing self-signed automatically-downloaded certificates to be used by the GDMA client comes with associated threats. To make those threats somewhat more visible, when auto-downloaded self-signed certs are in play, certain service-check results may be prefixed by the word "INSECURE:". This happens for the configured Poller_Service (typically, either gdma_poller for a standalone GDMA client, wchild_poller for a Windows GDMA client operating in multi-host mode, or unix_child_poller for a non-Windows GDMA client operating in multi-host mode). This is your reminder that you ought to improve the situation by obtaining and installing proper certs from wherever your organization approves. You may choose to disable this prefixing, but if you do so, that will be an act of declaring your own conscious choice to take on any associated risks of making the limited security model invisible.

GroundWork Server as Target Server

Previous instructions on setting up GDMA clients sometimes recommended use of the virtual gdma-autohost name for the GroundWork server in the Target_Server parameter on the GDMA client, this being the default Target Server hostname provided by the GDMA installer. Use of the virtual hostname, which would need to be resolved by your DNS to point to the GroundWork server, allows a level of indirection in how that name is interpreted.

With the use of SSL certificates, that convention is no longer possible because the hostname in the Target_Server parameter must exactly match the hostname used in the SSL certificate, and the hostname in the SSL certificate must match the hostname of the GroundWork server in order for the certificate to be seen as valid for accessing that server. (Use of the gdma-autohost name as the value of the GDMA_Auto_Host parameter can continue as before, because this use of the name is not tied to the use of SSL certificates. It relates to content passed over the connection to the server, not the establishment of the connection.)

Because matching the name is so critical, setting the Target_Server value to the appropriate server name necessary, for both existing GDMA and new GDMA clients. If you're not sure of the  exact hostname contained within a certificate, say one named server_01.pem, you can check it with the following command where the "displayedCN" field will contain the hostname:

cd /usr/local/groundwork/gdma/certs
/usr/local/groundwork/common/bin/openssl x509 -noout -text -in server_01.pem \
    | fgrep Subject:

GDMA Client Restrictions

When setting a target server, be aware that the server itself will accept connections only from allowed networks. If you aren't getting the results for your GDMA checks (they stay pending and eventually fail freshness checks), and you see an http 403 (Forbidden) error in the poller log, you can check if your client is allowed by check the value http_listener_allowed_clients in the /usr/local/nagios/etc/bronx.cfg file in the Nagios container. This parameter defaults to private IP address ranges (10.0.0.0/8,172.16.0.0/12,192.168.0.0/16). To edit this file:

  1. Change to the gw8 directory:

    cd gw8
    CODE

  2. Edit the file:

    docker-compose exec nagios vi /usr/local/nagios/etc/bronx.cfg
    CODE
  3. Adjust the parameter as necessary to allow your clients access, and save the file.

  4. Restart the nagios container: 

    docker-compose kill nagios
    CODE 
    docker-compose up -d nagios
    CODE

Transitioning to HTTPS

Customers who have many GDMA clients will naturally want to know how to transition all of them from HTTP/NSCA to HTTPS, and whether a big-bang approach is required. The transition plan depends on your current setup. Generally, however, you can use HTTP-only GDMA clients on GroundWork 8 if you need to. As GroundWork 8 ships with HTTPS only, you will need to either override this setting (not recommended), or tell GDMA not to pull the configuration files from GW8 until you finish upgrading. Feel free to submit a ticket in GroundWork Support if you would like our advice. 

You should upgrade and update all your GDMA installations to use certificates in a reasonable time frame. In broad strokes:

HTTP (GDMA 2.3.1 or earlier) to HTTPS (GDMA 2.3.2 or later)

  • Upgrade the GDMA release on individual clients, specifying the use of HTTPS, downloading (or installing) the server certificate and (optionally) a certificate revocation list.

  • Modify the gdma_auto.conf file on each and every updated GDMA client so that the Target_Server parameter specifies the https:// protocol and the correct matching GroundWork 8.x server hostname.

You don't need a big-bang approach (converting all of your GDMA clients instantly). Existing GDMA clients will continue to operate and report production results throughout the conversion process. Still, once you've started the conversion, it's best to run it through to completion as soon as possible. If you need to transition away from GroundWork 7.x to a new GroundWork 8 server running in parallel, you can simply add the second server to the target server list and have the GDMAs report to both, until the transition is complete. 

Watch out for conflicting Target_Server definitions

Some customers set a variety of client-configuration parameters (other than the commands to run) in host externals. If your setup does this for the Target_Server parameter, you must ensure that the protocol (http:// or https://) in this secondary definition is always in lockstep with the definition in the client's own gdma/config/gdma_auto.conf file, at every stage of the transition.

Managing SSL Certificates

 You will likely replace certificates from time to time. When this happens, it can be considerably confusing if you always use the same file names for the old and new certificate files and certificate revocation list (CRL) files. To counter that problem, in the discussion below we adopt a strategy of numbering the certificate and CRL files, so it will be clear what is happening when you upgrade from one certificate to another. We also use certain temporary names during invalidation procedures, both to clearly label the nature of old and new files, and to collapse multiple cases into a single, easy-to-follow process.

Installing HTTPS on GDMA Clients

Supporting GDMA communication over HTTPS requires both server-side and client-side configuration.

Configuring HTTPS Option on GroundWork 8 Servers

Please see Adding Certificates to HTTPS for a complete discussion of adding certificates, adjusting server names, and alternate names and certificates. Once you have your certificates loaded and the names are matching, you have the option of configuring a certificate revocation list. The following instructions will show you how to do this, but have not been updated for GroundWork 8 as of this writing. Therefore you will need to use an alternate procedure, either with a GroundWork 7 server or one of your own devising. 

Obtaining CRLs for non-self-signed certificates is more complex

The process below works fine if you are using self-signed certificates. If you get your certificates from some other source, the procedure will be different; you will likely need to obtain the CRL, if you desire to use it, from the upstream certificate provider. And then you may need to convert it into the .pem format which the GroundWork software can deal with. Details regarding the procurement and conversion of a CRL under these conditions are beyond the scope of this article.

To generate a CRL, you will execute the following commands as the nagios user.

  1. Before you run the openssl command below, first make sure you have a /usr/local/groundwork/common/openssl/index.txt file on your system. If not, create it this way:

    touch /usr/local/groundwork/common/openssl/index.txt
    CODE

    • The index.txt file will store details of previously revoked certificates, so they can be included in future CRL files without your needing to specify them again.

  2. Similarly, before you run the openssl command below, first make sure you have a /usr/local/groundwork/common/openssl/crlnumber file on your system. If so, leave it alone. If not, create it this way:

    echo 01 > /usr/local/groundwork/common/openssl/crlnumber
    CODE

    • The crlnumber file keeps track of the sequence numbers of revoked certificates; the openssl command below will henceforth update it as certificates are revoked. We will also use the number in this file to distinguish the CRL files that we generate over time.

  3. In the following openssl command, the -crldays option value should be specified to reflect how often you will update the certificate revocation list on all of your GDMA clients. Here in this example, we set it to a time period of three years, that being the same time period we are using in our examples for generating SSL certificates. (If your CRL expires before you revoke any certificates and have to regenerate a new revocation list for that reason, just generate a new one and distribute it to all your clients which were using the expired list. The CRL is simply a signed copy of your master list of revoked certificates, combined with a validity date and placed into a standard format. You can create a new copy whenever you want.)

    cd /usr/local/groundwork/apache2/conf
/usr/local/groundwork/common/bin/openssl ca -gencrl \
    -keyfile server.key -cert server.crt \
    -out crl_`cat ../../common/openssl/crlnumber`.pem -crldays 1095

    It's not actually necessary to store the output revocation-list file (e.g., crl_01.pem) in the same directory as the key and certificate files, but having it there makes it a lot easier to find later on (when you will copy certificate and revocation-list files to your client machines).

  4. To make future file copying more convenient, make a copy of your certificate file under the name by which it will be known on the client machines. Embedding a trivial serial number in the copied filename will make it much easier to distinguish which file is which later on, so change the command shown here to choose the next number in sequence at your site. The first such copy command might look like this if using a self-signed certificate:

    cd /usr/local/groundwork/apache2/conf
cp -p server.crt server_01.pem

    If using a chained certificate that contains a root and server certificate, it may look like this:

    cd /usr/local/groundwork/apache2/conf
cat server-ca-root.crt server.crt > server_01.pem

Upgrading Existing Clients from HTTP to HTTPS

Installing HTTPS on existing GDMA clients takes the following steps:

  1. Transfer the server certificate file (e.g., server_01.pem) and (optionally) the certificate revocation list file (e.g., crl_01.pem) from the GroundWork server to the GDMA client. These files must have a .pem filename extension on the client, so you can make a copy of the certificate file if needed. 

  2. Put the file(s) into the gdma/certs directory (e.g., on Linux GDMA, this would be in /usr/local/groundwork/gdma/certs/).

  3. Make sure that each of the files in this directory is owned by either the GDMA user account that owns all the rest of the GDMA client files, or (on UNIX-like systems) by root.

  4. Make sure that each of the files has no write permissions to anyone other than the file owner.

  5. Make sure that none of the files you are installing is completely empty.

  6. Create required symlinks, by running the c_rehash program. On the Windows platform, the directory argument is defaulted, so you don't have to mention it explicitly. On all other platforms, you must mention the path to the gdma/certs directory explicitly, but it can be given as a simple relative pathname, as we do in the following sample commands.

    • On a Windows system, the commands would be like:
      cd {path-to}\gdma\certs
..\..\common\bin\c_rehash
    • On a Linux or AIX system, the commands would be:
      cd /usr/local/groundwork/gdma/certs
../../common/bin/c_rehash ./
    • On a Solaris system, the commands would be:
      cd /opt/groundwork/gdma/certs
../../common/bin/c_rehash ./

  7. Verify that the gdma_auto.conf file specifies you want to use HTTPS, by using that protocol in the Target_Server definition. For example:

    Target_Server = "https://gwserver.mydomain.com"

  8. In that Target_Server value, be sure you use exactly the same server name, unqualified or qualified, as you used when you created the certificate for that server.

  9. Restart GDMA on the client.

Installing New Clients to Use HTTPS

If you are just starting to use GDMA and have no clients as yet, follow the instructions to set up your server name and optionally add a CA certificate before you install any GDMA clients.

Once your server runs with HTTPS, installing new GDMA clients to use HTTPS is easy.

  1. When the installer asks for the Target Server hostname, specify the name of the GroundWork server exactly as it is included in your SSL certificate.

  2. When the installer asks for the protocol to use for communication with the GroundWork server, select HTTPS.

  3. When the installer asks whether you wish to start the GDMA service after the installation, select No.

  4. Follow all the steps listed in the preceding section for installing certificate files on the GDMA client.

  5. Start (or restart, on Windows) the GDMA client.

Note, command-line options are available for unattended-mode installs, for selecting the installation mode, target server name, communication protocol, and (on non-Windows platforms) whether to start the service at the end of the install, as well as other parameters you may wish to set. Run the installer with the --help option to see these options listed.

Downloading Certificates from GroundWork 8 Server

If security is less of a concern and you can tolerate self-signed certificates, you may be able to use the certificate downloading and self-signed certificates option. These options are not fully documented as of this writing, and are not functional on Windows GDMA version 2.7.0. Further documentation is pending. 

Replacing Non-Downloaded Certificates

If you use the manual method of GDMA installation with certificates described above, you will have to re-do this at expiration time. There's a way to make this easier however. It is possible to have two server names and two certificates active in GroundWork 8 at the same time. If you need to transition working GDMA clients with expiring certificates incrementally, you can simply load the alternate name and certificate, and use it as the new target server. Once all the clients are transitioned, you can update the primary certificate. The next time, you can switch again. 

As GroundWork adds the functionality to more fully support and verify downloaded certificates, this case will be come moot. You can simply adjust the GDMA clients to download the certificates each time. 

Troubleshooting an HTTPS Connection

Very little information is provided by the HTTPS-support software when the certificate and certificate revocation list do not work as intended. If you enable logging in the GDMA poller, via the Enable_Local_Logging option in the gdma_auto.conf file, a small amount of information may be gleaned. Here is an example, with the lines wrapped for easier viewing here:

[Thu Apr  4 16:55:52 2013] Failed to get https://gwserver/gdma/gwmon_linuxgdma.mydomain.com.cfg --
    500 Can't connect to gwserver:443 (certificate verify failed)
[Thu Apr  4 16:55:52 2013] Failed to get https://gwserver/gdma/gwmon_linuxgdma.cfg --
    500 Can't connect to gwserver:443 (certificate verify failed)
[Thu Apr  4 16:55:52 2013] Failed to fetch the host config file.
[Thu Apr  4 16:55:52 2013] ERROR:  Auto-registration request was not processed;
    HTTP/S response code = 500.
[Thu Apr  4 16:55:52 2013] XML response was:
Can't connect to work.groundwork.groundworkopensource.com:443 (certificate verify failed)

LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error error:
    14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed at
    /usr/local/groundwork/perl/lib/site_perl/5.8.9/LWP/Protocol/http.pm line 51.

The phrase certificate verify failed, repeated several times as shown above indicates that this is the issue. So it's important to follow the certificate-install instructions carefully on every GDMA client. The main things to check are:

  • Is the certificate file (e.g., server_01.pem) in place on the client, in the gdma/certs directory, with restricted ownership and permissions?

  • If you are supporting CRL files on your clients, is the certificate revocation list (e.g., crl_01.pem) also in place on the client, in the gdma/certs directory, with restricted ownership and permissions?

  • Is the certificate file up-to-date with the current copy on the server?

  • If you are supporting CRL files on your clients, is the certificate revocation list file up-to-date with the current copy on the server?

  • Does the certificate reflect the actual hostname used by the server?

  • Does the Target_Server option in the client gdma_auto.conf file reflect the exact same form of the hostname as is present in the certificate?

  • Have the appropriate hash symlinks or copies been made in the gdma/certs directory, using the c_rehash program?

If all of those things check out, perhaps you really do have a Man-In-The-Middle attack going on, and the client should not be validating the certificate under that condition! (Proving such a situation is happening is beyond the scope of this document. Consult your local network folks.)

Related Resources