This page provides an introduction to GDMA Auto Setup and how it is put into play.

Purpose

GDMA Auto-Setup allows capabilities on the GDMA client to be automatically reflected in what gets monitored, with reduced manual work on the server. Of course, you do still need to decide and define what resources are important to you and deserve monitoring, and provide the specific commands to monitor them. But thereafter, the details of applying those definitions to individual GDMA clients and services can be handled automatically, depending on when you choose to tell the GDMA clients to look around at their own contexts and send in discovery data to configure the monitoring.

Auto-Setup is good not just for initial client configuration, but also for re-discovery and reconfiguration as provisioning is changed on the client.

A word of advice... The GDMA Auto-Setup documentation is long and detailed, but you shouldn't be frightened of it.  At the heart of it are sensor definitions, and in practice most of them turn out to be individually fairly simple.  You can look at examples, start with just a few of the easiest cases that apply to your situation, and work up from there.

A Bird's-eye View of How It Works

GDMA Auto-Setup has components on both the GDMA client and the GroundWork server.

GroundWork Monitor itself starts out knowing nothing about the particular GDMA client resources that you want to be monitored. So it is up to you to construct a kind of generic "instructions" file that will often apply to multiple GDMA clients, that describes what you care about and how that should be reflected in Monarch (and ultimately, in the configuration for Nagios). The instructions consist of a bunch of sensor definitions. Each sensor looks around the system for matching resources, and if it finds them, it generates configuration data to send to the server. This process is called "auto-discovery".

So you get the idea quickly as to how much setup is still needed, here are some sample sensor definitions, for Windows GDMA:

<host "Windows">
    type = os_type
    pattern = "windows"
    host_profile = "gdma-windows-host"
</host>

<service "Service Host">
    type = full_process_command
    resource = " "NT AUTHORITY\\NETWORK SERVICE" "
    cardinality = multiple
    # Match a few of the svchost process command lines, those whose principal
    # argument starts with an uppercase character (for arbitrary demo purposes).
    # Meanwhile, match the command path itself case-insensitively, and capture
    # the key command argument as part of the pattern-match results.
    pattern = "(?i:C:\\\\windows\\\\system32\\\\svchost.exe)\\s+-k\\s+([A-Z][A-Za-z]*)"
    service = "windows-svchost"
    instance_suffix = "_$SANITIZED1$"
    instance_ext_args = "$MATCHED1$!40!60"
</service>

When those discovery instructions are processed on a Windows machine, they will cause the gdma-windows-host host profile to be applied to the host configuration, and likely multiple individualized instances of the windows-svchost service to be applied as well.

In addition to the instructions, you must also create a trigger file. It tells both the GDMA client and the server-side code how to treat the instructions and the discovery results. We'll have more to say on this later. A typical full trigger file looks like this:

last_step = "do_configuration"
if_duplicate = "optimize"
soft_error_reporting = "ignore"

Sensors can look for open ports, mounted filesystems, existence of files, and a number of other signals to see what must be configured. Some customers find it useful to look at running-process command lines. Details of resources can be pattern-matched, and the matched strings can be used to customize the configuration. For instance, you might have several copies of a Java process, each running on a different port specified on the command line. That data can be used to configure multiple instances of the same base service on that host. Corresponding GDMA externals get customized and built automatically, reflecting the particular Java processes to be monitored.

Customer Interactions and Responsibilities

You are responsible for defining whatever instructions files you need. Typically, you will classify your GDMA clients into just a few categories, and build an instructions file for each category. Since auto-discovery will find the particular resources that are present on each separate GDMA client, it should not be necessary to customize the instructions for each individual client machine. Sensor definitions in the instructions for resources not present on a given machine won't match, and will just be ignored.

You must also define a few master trigger files. Typically, you will only need a couple of them, one to trigger dry-run discoveries, and one to trigger live-action discoveries.

You will store your master instructions and trigger files in some location that you fully control. When you wish to have a particular GDMA client run a pass of auto-discovery, you will run one or two commands to copy specific master instructions and trigger files into a location where that GDMA client will look for them. When it finds them there on its next polling cycle, it will run auto-discovery and generally send the results back to the server. The server will typically reconfigure Monarch, and in a live-action run, it will commit those changes to the database and re-build the externals for pickup by the GDMA client. The client will then begin monitoring with the new setup. The next time you run a Commit operation, Nagios will recognize all the configuration changes and begin monitoring with them in play. This leaves you in complete control, so you can synchronize deploying or reprovisioning your GDMA clients and having the new or modified setups folded into the production monitoring.

Developing Sensor Definitions

What Goes Into a Sensor?

Here is a conceptual map of the questions you must ask when creating a sensor definition.  Each question is answered by either including a directive that reflects the answer you have decided upon, or just accepting the default value for that directive.  For complex directive values, you will want to learn about the rules for handling embedded backslash (\) and double-quote (") characters.  This is just an overview of the sensor directives; details, when you need them, can be found in the Quick Reference for Auto Setup and Configuration Reference for Auto Setup pages.

Top-Level QuestionSecondary QuestionSensor Directive/Directive Value

How do I identify the capabilities I want to monitor?

Identifying capabilities comes down to matching certain data on the GDMA client.  If matching succeeds, the named configuration object in the next section will become part of the desired configuration.  Sometimes, it helps to capture some details about the particular capabilities you have found, so those details can be used as part of the object configuration.

What is the best kind of resource to probe to see if those capabilities are present on this machine?

type

One of the supported types:

os_type
os_version
os_bitwidth
machine_architecture
file_name
symlink_name
mounted_filesystem
file_content
running_system_service
full_process_command
open_local_port
open_named_socket

Can I limit or filter the resources under consideration at a broad level, before attempting to match them precisely?

resource

Depends on the sensor type;

for file_name, symlink_name, directory_name:

    fileglobs

for mounted_filesystem:

    fllesystem types

for file_content:

    fileglobs

for full_process_command:

    userlist

for open_local_port:

    IP address blocks

How many of the resources that will match my search pattern do I want to monitor individually?

cardinality

single, first, or multiple; first is used when more than one resource may match the pattern but you only want to configure based on one of them

Out of all the resources left after resource filtering, how do I select exactly the ones I car about? And how do I pick out strings that uniquely identify those resources and can be used to configure them?

pattern

Perl regular-expression pattern, possibly specifying captured strings.

Do I need to transform some characters in captured srings before using them for configuration?  (This includes capitalization, changing punctuation, deleting some characters, squashing consecutive duplicate characters, and arbitrary character replacement.)

transliteration

Perl "tr" operator character mapping.  Multiple copies of this directive may be included in the sensor, as many as needed.

Do I need to exclude certain characters from the captured and transliterated strings, either for security purposes or just to make them more suitable for use in configuration?

sanitization

Perl "tr" operator character-deletion searchlist.

How do I configure monitoring of the resources I have identified?

Often, when you configure said monitoring, you need to make the values used in that configuration adapt to the specific characteristics of those resources.  When that is necessary, it is done by using the $MATCHED#$ and $SANITIZED#$ macros which result from pattern matching and subsequent transliteration and sanitization of the captured strings, by referencing those macros in the values of the directives listed here.

What object am I trying to configure?

The sensor directives in this group are mutually exclusive.  One of them must be included in every sensor definition.  In all cases, the value represents a name in the Nagios configuration tool.

host_profile

Name of an existing host profile.

service_profile

Name of an existing service profile.

service

Name of an existing generic service.

What Nagios command, if any, should override the command(s) associated with the object I am configuring?

check_command

Name of an existing command (just the Nagios command name, not the path to whatever program it runs or the full command line).

What strings should be used as $ARG#$ macro values for the base-service command(s) associated with the object I am configuring?

command_arguments

What strings should be used as $ARG#$ macro values in service externals associated with the base service(s) in the object I am configuring?

externals_arguments

What string should be used as the instance suffix, when services are created using one or more service instances?

instance_suffix

A string, typically starting with an underscore, whose value uniquely identifies each service instance.  Typically limited to alphanumerics, underscore, and possibly hyphen characters.

What strings should be used as $ARG#$ macro values for the service-instance command(s) associated with the object I am configuring?

instance_cmd_args

What strings should be used as $ARG#$ macro values in service externals associated with the services I am creating, when service instances are used?

instance_ext_args

Do I want this sensor definition to be operational?

You have a choice as to whether each sensor definition in the instructions file will be active during the discovery phase.  Sensors are enabled by default; disabling a sensor using the following directive is easier than commenting out the whole thing.

Should this sensor be in play during discovery?

enabled

yes, no, or certain equivalent terms

Sensor Construction Rules

Here are some common-sense observations and rules for developing sensor definitions.

  1. When possible, you should used fixed or static sensors (e.g., type = file_name or type = file_content) instead of dynamic sensors (e.g., type = full_process_command or type = open_local_port) to identify capabilities to be monitored.  That's because if a given dynamic resource happens to be down at the moment that discovery happens, a dynamic sensor probably won't see it, and it won't be included in the generated configuration.  See Quick Reference for Auto Setup and Configuration Reference for Auto Setup for more information on sensor types.
  2. Do pay attention to security and pattern-match sanitization.  For instance, typically you will want to filter out all punctuation that may be used as shell metacharacters, to prevent any command-injection attacks.
  3. You don't need to use every possible directive in every sensor definition.  For instance, usually the command will be specified in the generic service in the Nagios configuration tool, and there will be no reason to override that choice during discovery.  And various directives either may not be needed or can just be defaulted.
  4. An instance_suffix will typically only be specified either when cardinality = multiple, or when you want to prepare up front for the day when that might be so.

Sensor Development Workflow

  1. Establish a master repository outside of the GroundWork product itself, where you will maintain all your instructions and trigger files.
  2. Make sure that repository is backed up on a regular basis, as it will contain precious cargo.
  3. Create a few standard trigger files in your master repository, and you'll probably never need to edit them ever again. See Common Trigger Files.
  4. Create some shell aliases to make invoking the autosetup tool convenient in your environment.  You will be using that tool a lot, so this tiny investment up front will quickly pay off.
  5. You can safely test out sensor definitions being newly defined or updated, using dry runs of discovery.  See the common trigger files.
  6. You generally don't want to test sensor development on a GDMA machine which has been installed and is already running using GDMA Auto-Registration.  That would cause said machine to be configured using whatever host profile and possibly service profile you specified in the client configuration, which might conflict with the configuration you will be generating using Auto-Setup sensors.
  7. in normal operation, each pass of discovery is only initiated when the GDMA client finds a new trigger file which is more recent than the most recent instructions file it can find.  But also, the GDMA poller only checks for such files at the beginning of each polling cycle, typically once every 10 minutes.  That will be annoyingly slow when you are modifying sensor definitions and want immediate feedback as to how well they work.  So on the GDMA client you are using for such testing, you will want to temporarily modify the Poller_Proc_Interval value in the gdma/config/gdma_auto.conf file to something shorter, like 60 (seconds).  Alternatively, you can do your sensor development directly on the GDMA client, using the gdma/bin/discover tool to initiate each new pass of discovery.

System Interactions

File timestamps are used to tell whether a new pass of discovery is to be run, and whether a trigger file applies to the instructions file currently available to the GDMA client. Therefore, when you want to run auto-discovery, it is necessary to have the instructions file planted first and then a new copy of the trigger file planted on the server. We will describe that process later on in more detail.

The last_step directive in the trigger file controls how far the discovery processing runs before it stops. The most commonly used early-stop stage in this regard is:

last_step = "test_configuration"

That will perform a dry run of all discovery on the client and all configuration changes on the server. At the last moment, just before the configuration changes are committed for production use, they are rolled back instead. This allows you to fully test your instructions before putting them into production.

In general, we try to make the results of discovery visible on the server, so in production you don't need to reach over to an individual client to see what happened there. For diagnosis in rare cases, it may be necessary to initiate auto-discovery directly on the GDMA client, using the gdma/bin/discover tool provided there.  As noted above, running the discover tool on the client can also be useful when developing sensor definitions, to speed things up.

The server-side autosetup command is the main touchpoint for your control of Auto-Setup. It is used to install instructions and trigger files where the GDMA clients can see them, and to perform other maintenance. This all-purpose tool provides easy access to both the client-side discovery results and subsequent server-side analysis. Detailed descriptions can be found in Managing Auto Setup and Tools and Files for Auto Setup.

Related Resources