What is GDMA?

The GroundWork Distributed Monitoring Agent (GDMA) is a monitoring agent that runs on Linux, MS Windows, Solaris, and AIX.

Using plugins that follow the same standard as Nagios, GDMA can execute service checks on the host where it is installed and/or other hosts, making it a flexible way to do distributed monitoring. It is lightweight, and highly configurable. It supports HTTPS and/or NSCA encrypted communication with a GroundWork server, and forwards the service check results to a Bronx event broker in a Nagios instance, typically running in a GroundWork Monitor server. GDMA code is open source. 

While manual configuration is supported, you can manage GDMA directly from the GroundWork server using the Nagios Monitoring configuration application, Monarch. It's also possible to automate much of the configuration of the agent using two main automation methods, Auto Registration and Auto Setup. The Auto Setup method of configuration works with the Monarch configuration database to detect running software, file and content, as well as many other particular aspects of the platform to be monitored. The results of the discovery are then used to assign service checks to the system, making it possible to accommodate many, many different types of deployments with a single automation instructions file.

Why Use GDMA?

  • Distributed: Monitoring systems can be overwhelmed if asked to do too many things from a single instance. GDMA allows you to offload the GroundWork Monitor server of nearly all the active checks you would otherwise execute with Nagios, and can reduce the need to deploy GroundWork Child Servers. In addition, allowing the monitored system to execute its own checks, you gain the ability to access resources present only on the machine without opening ports or running services like SNMP or even SSH. 

  • Unidirectional: GDMA only needs to reach out to the GroundWork server to send results and (optionally) pick up configuration changes and new plugins. You never need connectivity outbound from the GroundWork server to the GDMA host, making it ideal for cross-firewall communication.

  • Multiplatform: GDMA uses a largely unified code base on Windows and Linux/Unix based systems. This means administration is very similar between platforms, and therefore simpler. Its features for tuning, multi-host mode, and reliable spooling of results are the same across all platforms, and its lightweight nature is preserved wherever it is run.  

  • Flexible: You can run any plugin you like as a service check in Nagios, and GDMA also lets you be this flexible. In fact, you can enable plugin downloads in GDMA and let selected agents update their plugin sets automatically, dependencies included. Simple, file-based configurations allow integration with configuration management systems. Multi-host mode allows you to deploy just one GDMA and monitor all the systems in a moderate-sized remote location. 

  • Secure: Support for restricted (non-privileged) user accounts, encrypted communication, and no open inbound ports make GDMA hard to compromise without on-system privileged access. SSL Certificate download, validation, and revocation options keep the encryption secure.

GDMA Automation Methods

Which method should I use? The two automation methods for GDMA are Auto Registration and Auto Setup.

  • If you want to simply register a host with standard GDMA profiles, you will want to use GDMA Auto Registration.
  • If you want auto detection of services based on conditions found on a host such as open ports, running services, or running processes, or additionally to detect file systems, directories, files or text in files, you will want to use GDMA Auto Setup.

Auto Registration

Auto Registration allows a GDMA client to send in a small amount of basic information about itself, and have that client machine be registered in Monarch and have initial externals built so the client can begin monitoring. Nagios will begin accepting monitoring results from the GDMA client the next time a Commit is run.

The basic information sent in for Automated Agent Registration is essentially the hostname, IP address, and the name of a host profile and optional service profile to apply. The profiles are a good start, and may suffice if you have a fleet of identical machines, but there is no significant per-host customization reflecting the exact set of services being run on the GDMA client.

Auto Registration actions

  • The GDMA client wakes up, and looks for a local config file.
  • If it cannot find a local config file, it reaches over to the GroundWork target server and looks there.
  • If the GroundWork target server does not have a config file ready to go, the GDMA client runs Auto Registration, sending in a packet (described below).
  • The GroundWork target server is supposed to respond to the packet by:
    • registering the GDMA client in the configuration database Monarch;
    • applying the host profile and service profile, if requested by the GDMA client and if the named profiles are available in Monarch;
    • building externals for the GDMA client, placing them in a standard location on the GroundWork target server known to the GDMA client;
    • returning a success/failure indicator to the GDMA client, along with the hostname by which the GDMA client is now registered.
  • If the registration succeeded, the GDMA client will:
    • adopt the hostname sent back by the GroundWork target server and use it for all further interaction with the GroundWork target server;
    • reach over to the GroundWork target server to pick up the newly-created externals file;
    • begin monitoring, using the downloaded externals.

That sequence of actions is repeated on subsequent client processing cycles as long as the GDMA client does not have a config-file in hand. A back-off algorithm controls the interval between retries, and if the client never successfully registers, it will eventually stop trying. The auto-registration packet contains the following scalar strings, sent to the GroundWork target server as standard URI parameters in a HTTP/S POST request:

  • auto-registration username
  • auto-registration password
  • agent type
  • hostname
  • host IP address
  • host MAC address
  • operating system
  • host profile name
  • service profile name

The server-side auto-registration actions are scripted. The standard behavior is described just above. Optionally, a customer can implement custom logic to validate and possibly modify the data sent in by the GDMA client before that data is used to register the client in Monarch. Such local customization is intended to serve a few common purposes, but does not in general result in a fully fleshed-out notion of what the client should be monitoring aside from the services noted in the host and service profiles that end up being applied. Monarch itself is largely a passive player in auto-registration. It provides the place to store the configuration data and the code to apply host and service profiles and to build externals. The auto-registration scripting calls the Monarch facilities that it needs.

Auto Setup

GDMA Auto Setup is a generalization of Auto Registration. GDMA Auto Setup adds the capability for the GDMA client to run a guided discovery of resources available on the client, making it possible to automatically adapt to the particulars of that client when monitoring is configured on the GroundWork server.

Auto Setup actions

GDMA Auto Setup actions are similar to those for Auto Registration. Auto Setup is enabled via a boolean Enable_Auto_Setup option in the gdma_auto.conf file. This option is settable by the GDMA installer, either manually via a UI-based install or via command-line option on an unattended-mode install. You can also edit the config file after an install. For that purpose, it makes sense to tell the installer not to start the GDMA client as soon as the install is done. That way, you have a chance to modify the gdma_auto.conf file before GDMA starts up the first time. Here's the sequence:

  • The GDMA client wakes up, and looks for a local config file (previously downloaded from the GroundWork target server during an earlier run).
  • If it cannot find a local config file, it reaches over to the GroundWork target server and looks there.
  • If the GroundWork target server does not have a config file ready to go, the GDMA client checks the gdma_auto.conf options for Enable_Auto_Setup. If that is enabled, the client will initiate auto-setup processing. Otherwise, it will fall back to attempting Auto-Registration, as noted above. The rest of this sequence assumes that Enable_Auto_Setup is enabled.
  • The GDMA client will reach over to the GroundWork target server, looking for an auto-discovery trigger file and then an auto-discovery instructions file for this host.
  • If the GDMA client finds both files for this host on the GroundWork target server, it will download those files and run auto-discovery actions as described therein.
  • The auto-discovery processing will result in a set of auto-discovery results. Those results will be sent to the GroundWork target server as part of an auto-discovery results packet.
  • The GroundWork target server is supposed to respond to the auto-discovery packet by:
    • analyzing the packet to ensure that it is valid;
    • configuring the GDMA client in Monarch;
    • removing services from a previously configured host which are no longer present in the auto-discovery results;
    • applying a host profile and service profile(s), as listed in the results;
    • customizing services, and creating service instances as necessary, to reflect details of the auto-discovery results;
    • building externals for the GDMA client, placing them in a standard location on the GroundWork target server known to the GDMA client;
    • removing the trigger file on the GroundWork target server so the GDMA client won't re-run discovery until a new trigger file shows up;
    • returning a success/failure indicator to the GDMA client, along with the hostname by which the GDMA client is now registered.
  • If the configuration succeeded, the GDMA client will:
    • adopt the hostname sent back by the GroundWork target server and use it for all further interaction with the GroundWork target server;
    • reach over to the GroundWork target server to pick up the newly-created externals file;
    • begin monitoring, using the downloaded externals.

Key to all of this processing is the auto-discovery instructions file, which must be defined to reflect the kinds of resources that might be present and need monitoring. 

In brief, it tells the GDMA client what resources to look for and how to look for them, what host and service profiles to apply if they are found, and how those profiles should be customized based on the details of the discovery matching on that particular host.

There will also be a means to trigger an auto-setup cycle later on, after the GDMA client is in operation.

Auto Setup examples

Let's start with some simple examples of how you would set a GDMA client to run certain auto-discovery actions. As you can see from the comments, this formatted file allows you to set the host profile based on the detected OS, apply a service profile based on the existence of a running process that matches a regex, or the existence of a file (influxdb.conf, in this example), and a port being open.

format_version = "1.0"

    # Apply the "gdma-linux-host" host profile if we're running on a Linux box.
    <host "Linux">
        type = os_type
        pattern = "linux"
        host_profile = "gdma-linux-host"
    </host>

    # Apply the "apache-web-server" service profile if Apache Web Server is currently running.
    <service "Apache">
        type = full_process_command
        pattern = "/(httpd\.bin)(?:\s+-f\s+(\S+)|$)"
        service = "apache-web-server"
    </service>

    # Apply the "influxdb-server" service profile if InfluxDB is installed,
    # whether or not it is currently running.
    <service "InfluxDB Server">
        type = file_name
        resource = "/etc/influxdb/influxdb.conf"
        pattern = "/influxdb.conf$"
        service_profile = "influxdb-server"
    </service>

    # Apply the "sendmail" service profile if the standard sendmail port is currently open
    # on any network interface of the GDMA client machine (i.e., if sendmail is running).
    <service "sendmail">
        type = open_local_port
        resource = "0.0.0.0/32"
        pattern = "25"
        service_profile = "sendmail"
    </service>

Related Resources