Examples of Auto Setup
Provides examples of using GDMA Auto Setup in practice. See also Practical Advice for Auto Setup.
An End-to-end Complete Example
Let's use the resources known to be on every Linux GDMA machine to create an example that we can examine in all its aspects.
Every Linux GDMA machine runs two Perl daemons, named gdma_poll.pl
(the poller) and gdma_spool_processor.pl
(the spooler). We will undertake to monitor those two processes in some way. To do so, we first create a gdma-daemon
base service on the GroundWork server, and associate the following service external with it, literally as shown (with embedded macro references that will expand to appropriate values when externals are built):
Check_$BASESERVICEDESC$[$INSTANCE$]_Enable="ON" Check_$BASESERVICEDESC$[$INSTANCE$]_Service="$SERVICEDESC$" Check_$BASESERVICEDESC$[$INSTANCE$]_Command="check_gdma -t $INSTANCESUFFIX$ -w $ARG2$ -c $ARG3$" Check_$BASESERVICEDESC$[$INSTANCE$]_Check_Interval="1"
check_gdma
program you see there is presumably a custom plugin that we will have set up to be downloaded to Linux GDMA machines. The exact nature of that plugin is not specified here; we are just using it and its arguments as a prop to correspond to the instance externals arguments we will generate in our discovery results. To be clear, there is only the one external associated with the base gdma-daemon
service. As many customized copies of that one external will be generated as needed when externals are built, corresponding to the particular service instances that get generated in our Auto-Setup processing.
For purposes of exposition here, we assume that the instructions and trigger files reside on the GDMA client, and we will run the discover
tool there to perform our testing.
Our sample instructions file looks like this:
# Sample instructions file for discovering the principal # GDMA daemon processes. format_version = "1.0" # Generic GDMA host profile, for whatever platform we're on. <host "GDMA host"> type = os_type pattern = "(.*)" host_profile = "gdma-$SANITIZED1$-host" </host> <service "GDMA Poller"> type = full_process_command cardinality = "first" # $MATCHED1$ will be "poll" for this sensor. pattern = "/.perl.bin\s+/[a-z/]+?/gdma_(poll).pl" service = "gdma-daemon" instance_suffix = "_$MATCHED1$" instance_ext_args = "$MATCHED1$!20!10" </service> <service "GDMA Spooler"> type = full_process_command cardinality = "first" # $MATCHED1$ will be "spool" for this sensor. pattern = "/.perl.bin\s+/[a-z/]+?/gdma_(spool)_processor.pl" service = "gdma-daemon" instance_suffix = "_$MATCHED1$" instance_ext_args = "$MATCHED1$!20!10" </service>
# dry-run trigger file, # for tentative testing if_duplicate = "optimize" soft_error_reporting = "ignore" change_policy = "non_destructive" last_step = "test_configuration"
last_step
value is set to allow us to send results to the server, which we will do as part of our testing.
While logged in as the gdma
user, we run a pass of discovery like this:
cd /usr/local/groundwork/gdma/bin ./discover /tmp/sample_trigger /tmp/sample_instructions
discover
command yields the following output:================================================================ Discovery instructions, as read and parsed ================================================================ $instructions = { 'format_version' => '1.0', 'host' => { 'GDMA host' => { 'cardinality' => 'single', 'host_profile' => 'gdma-$SANITIZED1$-host', 'pattern' => '(.*)', 'type' => 'os_type' } }, 'service' => { 'GDMA Poller' => { 'cardinality' => 'first', 'instance_ext_args' => '$MATCHED1$!20!10', 'instance_suffix' => '_$MATCHED1$', 'pattern' => '/.perl.bin\s+/[a-z/]+?/gdma_(poll).pl', 'service' => 'gdma-daemon', 'type' => 'full_process_command' }, 'GDMA Spooler' => { 'cardinality' => 'first', 'instance_ext_args' => '$MATCHED1$!20!10', 'instance_suffix' => '_$MATCHED1$', 'pattern' => '/.perl.bin\s+/[a-z/]+?/gdma_(spool)_processor.pl', 'service' => 'gdma-daemon', 'type' => 'full_process_command' } } }; --------------------------------------- OS type, as determined during discovery --------------------------------------- linux -------------------------------------------------------------- Full process commands for all users, as found during discovery -------------------------------------------------------------- <b>... lots of process command lines, not shown here ...</b> /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_poll.pl /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_spool_processor.pl <b>... lots of process command lines, not shown here ...</b> ================================================================ Mostly-unanalyzed discovery results ================================================================ packet_version = 1.0 succeeded = true last_step = test_configuration if_duplicate = optimize soft_error_reporting = ignore change_policy = non_destructive registration_agent = GDMA Auto-Setup hostnames = myhost.mydomain.com myhost ip_addresses = 10.7.3.109 172.24.89.52 fe80::111:ea85:cafe:1729 mac_addresses = 00:19:c2:87:a3:f9 os_type = linux discovered_host_profiles: gdma-linux-host discovered_services: gdma-daemon ---------------------------------------------------------------- Sensor results ---------------------------------------------------------------- Sensor: sensor_name = GDMA host type = os_type cardinality = single enabled = true matched = true host_profile_name = gdma-linux-host Sensor Instance: qualified_resource = linux Raw matched strings, quoted for visibility: $MATCHED1$ = 'linux' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'linux' Sensor: sensor_name = GDMA Poller type = full_process_command cardinality = first enabled = true matched = true service_name = gdma-daemon Sensor Instance: qualified_resource = /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_poll.pl Raw matched strings, quoted for visibility: $MATCHED1$ = 'poll' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'poll' instance_suffix = _poll instance_ext_args = poll!20!10 Sensor: sensor_name = GDMA Spooler type = full_process_command cardinality = first enabled = true matched = true service_name = gdma-daemon Sensor Instance: qualified_resource = /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_spool_processor.pl Raw matched strings, quoted for visibility: $MATCHED1$ = 'spool' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'spool' instance_suffix = _spool instance_ext_args = spool!20!10
discover
tool saw at different stages of its execution. And that helps tremendously with developing and testing sensor definitions. Let's walk through each of those sections. It may help if you use two browser windows to display two copies of this page side-by-side, so you can read these explanations in one window while you examine the output above in a second window.
Section | Description |
---|---|
| This section is a summary of how |
| The presence of these sections, and others like them, depends on which sensor types you have enabled in your instructions. Each distinct kind of probe will generate a separate section listing all the resources under consideration for matching, so you can understand what your sensor |
| This section summarizes the discovery results and the trigger data that was used to generate them. (The trigger data is passed through because it affects further processing.) In particular, here you will find a short list of the profiles and services that are in play because of matched sensors. |
| This section contains the meat of the sensor results (apologies to vegetarians). The results of processing each individual sensor are presented, so you can see exactly what transpired. |
If there had been errors in our instructions, the discover
tool would have let us know about that. In this case, we didn't, so we won't cover that here. Let's take the next step and run the same discovery, but this time we will capture the discovery results into a local file and send them to the GroundWork server. Before we do that, let's run the autosetup tool on the GroundWork server while logged in as the nagios
user, to see what it shows.
% cd /usr/local/groundwork/gdma/bin % ./autosetup status -t -a ERROR: There are no hosts with files to list status for.
./discover /tmp/sample_trigger /tmp/sample_instructions /tmp/results 1
======================================= Sending discovery results to the server ======================================= NOTICE: Auto-Setup request was processed. server response is: {"status":"success","message":"Determined hostname is 'myhost.mydomain.com'. Your discovery results were successfully tested against the database."} $result_data = { "message" => "Determined hostname is 'myhost.mydomain.com'. Your discovery results were successfully tested against the database.", "status" => "success" }; NOTICE: Auto-Setup call succeeded, with these server-side messages: Determined hostname is 'myhost.mydomain.com'. Your discovery results were successfully tested against the database.
cat /tmp/results
% ./autosetup status -t -a instructions trigger results analysis hostname =============== =============== =============== =============== =================== --- --- Jan 6 00:50:45 Jan 6 00:50:45 myhost.mydomain.com
% ./autosetup print results myhost Reporting results for alternative host: myhost.mydomain.com Showing discovery results for host "myhost.mydomain.com" as saved on Wed Jan 6 00:50:45 2021. ================================================================ Mostly-unanalyzed discovery results ================================================================ packet_version = 1.0 succeeded = true last_step = test_configuration if_duplicate = optimize soft_error_reporting = ignore change_policy = non_destructive registration_agent = GDMA Auto-Setup hostnames = myhost.mydomain.com myhost ip_addresses = 10.7.3.109 172.24.89.52 fe80::111:ea85:cafe:1729 mac_addresses = 00:19:c2:87:a3:f9 os_type = linux discovered_host_profiles: gdma-linux-host discovered_services: gdma-daemon ---------------------------------------------------------------- Sensor results ---------------------------------------------------------------- Sensor: sensor_name = GDMA host type = os_type cardinality = single enabled = true matched = true host_profile_name = gdma-linux-host Sensor Instance: qualified_resource = linux Raw matched strings, quoted for visibility: $MATCHED1$ = 'linux' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'linux' Sensor: sensor_name = GDMA Poller type = full_process_command cardinality = first enabled = true matched = true service_name = gdma-daemon Sensor Instance: qualified_resource = /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_poll.pl Raw matched strings, quoted for visibility: $MATCHED1$ = 'poll' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'poll' instance_suffix = _poll instance_ext_args = poll!20!10 Sensor: sensor_name = GDMA Spooler type = full_process_command cardinality = first enabled = true matched = true service_name = gdma-daemon Sensor Instance: qualified_resource = /usr/local/groundwork/perl/bin/.perl.bin /usr/local/groundwork/gdma/bin/gdma_spool_processor.pl Raw matched strings, quoted for visibility: $MATCHED1$ = 'spool' Sanitized match strings, quoted for visibility: $SANITIZED1$ = 'spool' instance_suffix = _spool instance_ext_args = spool!20!10
discover
tool showed on the GDMA client. One thing to notice is that we didn't have to type the fully-qualified hostname of the GDMA client; the autosetup
tool was able to scan through the results files it had in hand and find the right one for us. Note also that you don't see a full listing of all the resources that were potential sensor matches; for that, you'll need to run the client-side tool.
Next, let's look at the analysis of the discovery results:
% ./autosetup print analysis myhost Reporting analysis for alternative host: myhost.mydomain.com Showing discovery analysis for host "myhost.mydomain.com" as saved on Wed Jan 6 00:50:45 2021. NOTICE: The discovery results passed basic validation (without comparison to the current content of the database). ------------------------------------------------------------------------------------------ Consolidated discovery results: { "change_policy" : "non_destructive", "chosen_address" : "10.7.3.109", "chosen_alias" : "myhost.mydomain.com", "chosen_hostname" : "myhost.mydomain.com", "host_profiles" : { "gdma-linux-host" : { "instances" : [ {} ] } }, "hostnames" : [ "myhost.mydomain.com", "myhost" ], "if_duplicate" : "optimize", "ip_addresses" : [ "10.7.3.109", "172.24.89.52", "fe80::111:ea85:cafe:1729" ], "last_step" : "test_configuration", "mac_addresses" : [ "00:19:c2:87:a3:f9" ], "os_type" : "linux", "registration_agent" : "GDMA Auto-Setup", "service_profiles" : {}, "services" : { "gdma-daemon" : { "instances" : [ { "instance_ext_args" : "poll!20!10", "instance_suffix" : "_poll" }, { "instance_ext_args" : "spool!20!10", "instance_suffix" : "_spool" } ] } }, "soft_error_reporting" : "ignore" } ------------------------------------------------------------------------------------------ Database changes: * adding service gdma-daemon to host myhost.mydomain.com * adding inheritance overrides to service gdma-daemon on host myhost.mydomain.com * adding instance gdma-daemon_poll to service gdma-daemon on host myhost.mydomain.com * adding instance gdma-daemon_spool to service gdma-daemon on host myhost.mydomain.com Rolling back database changes, because this was just a test run.
<service>
sensors in our instructions got combined to create separate service instances of the base gdma-daemon
service. Also at the end, we see a record of the individual database changes that would result from this discovery. They were put in place to verify that they would work, then backed out because this was just a dry run (last_step = "test_configuration"
in our trigger file, carried through as part of the discovery results).
The next step is to copy your working instructions file from the GDMA client to the GroundWork server, and stash it in your master repository. You might do the same with the trigger file if that were not a stock-standard trigger file already in your repository. We won't show those steps here, but for purposes of exposition, we have established these files on our server:
-rw-r--r--. 1 nagios nagios 173 2021-01-06 00:28:20 /home/autosetup/dry_run_trigger -rw-------. 1 nagios nagios 900 2021-01-06 00:33:03 /home/autosetup/gdma_daemon_instructions
% cd /usr/local/groundwork/gdma/bin % ./autosetup status -t -a instructions trigger results analysis hostname =============== =============== =============== =============== =================== --- --- Jan 6 00:50:45 Jan 6 00:50:45 myhost.mydomain.com % ./autosetup install -p /home/autosetup/gdma_daemon_instructions myhost % ./autosetup install -p /home/autosetup/dry_run_trigger myhost % ./autosetup status -t -a instructions trigger results analysis hostname =============== =============== =============== =============== =================== Jan 6 14:52:50 Jan 6 14:52:51 --- --- myhost --- --- Jan 6 00:50:45 Jan 6 00:50:45 myhost.mydomain.com
[Wed Jan 06 15:06:06 2021] NOTICE: Failed to get trigger file http://myserver/gdma_trigger/myhost.mydomain.com_trigger -- 404 Not Found [Wed Jan 06 15:06:06 2021] NOTICE: fetch_auto_setup_files: Pull failure for http://myserver/gdma_trigger/myhost.mydomain.com_trigger [Wed Jan 06 15:06:07 2021] NOTICE: File timestamp analysis shows that a new pass of discovery should be run. [Wed Jan 06 15:06:11 2021] NOTICE: A dry-run pass of discovery has been run, and the results have been saved. [Wed Jan 06 15:06:11 2021] ERROR: Cannot open the old results file "/usr/local/groundwork/gdma/tmp/myhost_last_dry_results" (No such file or directory). [Wed Jan 06 15:06:11 2021] NOTicE: These discovery results are different from those of the last dry-run pass of discovery. [Wed Jan 06 15:06:11 2021] NOTICE: The new discovery results are being sent to the server, because they are not completely the same as what has gone before. The difference might be as trivial as some change in the trigger options, not the results of the sensor processing. [Wed Jan 06 15:06:13 2021] NOTICE: Auto-Setup request was processed. [Wed Jan 06 15:06:13 2021] NOTICE: Auto-Setup call succeeded, with these server-side messages: Determined hostname is 'myhost.mydomain.com'. Your discovery results were successfully tested against the database. [Wed Jan 06 15:06:13 2021] NOTICE: Failed to get config file http://myserver/gdma/gwmon_myhost.mydomain.com.cfg -- 404 Not Found [Wed Jan 06 15:06:13 2021] NOTICE: Failed to get config file http://myserver/gdma/gwmon_myhost.cfg -- 404 Not Found [Wed Jan 06 15:06:13 2021] ERROR: Failed to fetch the host config file [Wed Jan 06 15:06:13 2021] ERROR: reload_config: Cannot open configuration file /usr/local/groundwork/gdma/config/gwmon_myhost.cfg (No such file or directory). [Wed Jan 06 15:06:13 2021] ERROR: Failed to reload config.
Back on the server, we can see the discovery results have come in, and been analyzed:
% ./autosetup status -t -a instructions trigger results analysis hostname =============== =============== =============== =============== =================== Jan 6 14:52:50 Jan 6 14:52:51 --- --- myhost --- --- Jan 6 15:06:13 Jan 6 15:06:13 myhost.mydomain.com
myhost
, and the GDMA client was able to find those files on the server. Note that the client returned the discovery results, and then analysis was done, using the hostname myhost.mydomain.com
instead (notice that those timestamps have been updated). This is perfectly legitimate, but it is something to watch for when you use hostnames on the autosetup
command line. You may wish to always use the fully-qualified hostname to avoid this kind of confusion.
Here are some things we hope you learned from that example:
- You can develop and test your instructions and trigger files while working from either the GDMA client or the GroundWork server. That said, if you encounter difficulties, it may be helpful to initiate discovery from the client machine, since the output you can see there will include lists of all the resources that your sensor patterns will be matched against. That level of detail can be invaluable when developing your sensor
pattern
directives. - Perl regular expressions used in
pattern
directives can be simple or complex, depending on what you are trying to match. In this example, we used fairly complex patterns, just to show that such things are possible. If you are not familiar with Perl regex's, you are strongly advised to get yourself a copy of the book Programming Perl, 4th edition. Chapter 5 is all about pattern matching. Learning this material will serve you well in many contexts going forward, because Perl-Compatible Regular Expressions are used in many places in practical programming. Start simple, and look up things as you need them; there's no reason to be frightened by all the advanced capabilities that are available. - The output from the
discovery
andautosetup
tools is designed to show you all the relevant detail in compact form. While we don't document every last item in that output, it should be fairly self-explanatory. Use it to guide your work. - You can do dry-run testing and see exactly what database changes would result from a pass of discovery, to verify that your instructions will do what you want them to. Dry-run tests are safe even on a production system, as the database changes will be automatically rolled back.
Samples
Sample Sensor Definitions
- basic host-OS discovery and application of some standard host profile
- some service-discovery sensors we can imagine will be most common
Sample Trigger Files
- standard dry-run and live-action trigger files, which will probably be almost exclusively what you will use all the time (refer to the files documented in the Quick Reference page)
Sample Use of the autosetup tool
See The autosetup Tool.
Other Examples
- links to short embedded examples on the sibling pages; fill in once the other pages are done
- whatever example we have for auto-discovery of Windows disks, as Joey has validated and improved it
- whatever examples we might have from customers
- whatever other examples we can think of, including some made-up examples of application-monitoring situations, not just collecting system-level measurements
DBus
The DBus is a standard Linux service. Normally it's not something you would want to monitor, so we have disabled that sensor in the following example. But it could be enabled and used for dry-run testing, to ensure that the GDMA client can interoperate with the GroundWork server as regards downloading instructions and trigger files, and submitting discovery results.
format_version = "1.0" # Apply the "gdma-linux-host" host profile if we're running on a Linux box. <host "Linux"> type = os_type pattern = "linux" host_profile = "gdma-linux-host" </host> # Apply the "dbus-service" service if the DBus service is running. <service "DBus"> type = open_named_socket pattern = "^/var/run/dbus/(system_bus_socket)$" service = "dbus-service" enabled = false </service>
Python 2 Processes
The notion that you might want to monitor all Python 2 processes on your machine is perhaps fanciful, but the notion does provide an opportunity to demonstrate useful application of sensor directives. The interpretation of the instance_ext_args
in the Python2
sensor would obviously depend on the construction of the service external for that service, not shown here.
format_version = "1.0" # Generic GDMA host profile, for whatever platform we're on. <host "GDMA host"> type = os_type pattern = "(.*)" host_profile = "gdma-$SANITIZED1$-host" </host> # Processes running as Python 2 scripts, such as firewalld or tuned. # The command lines we are matching against look like this: # /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid # /usr/bin/python2 -Es /usr/sbin/tuned -l -P <service "Python2"> type = full_process_command resource = "root" cardinality = multiple pattern = "^\S+/(python2)\s+(?:-Es\s+)?\S*/([^/\s]+)(?:\s|$)" service = "python2-program" instance_suffix = "_$SANITIZED2$" instance_ext_args = "foo!$MATCHED1$!bar!$SANITIZED2$!abc" </service>
Discrimination Mostly by OS Release
This example shows that you can combine multiple sensors of similar type in the same instructions file and use that one file for discovery on multiple disparate hosts, in a situation where you can reasonably expect that only one of the sensors will match.
format_version = "1.0" <host "CentOS 7.4"> type = os_version pattern = ^7\.4\. host_profile = "centos7-host" </host> <host "CentOS 8.2"> type = os_version pattern = ^8\.2\.$ host_profile = "centos8-host" </host> <host "Ubuntu 18.04"> type = os_version pattern = ^18\.04$ host_profile = "ubuntu-host" </host> <host "Raspberry Pi"> type = machine_architecture pattern = ^arm$ host_profile = "raspberry-pi-host" </host>
Checking for Specific Processes
An application like a web server will often run many copies of a particular binary, to handle the load. But all we need is to recognize one copy, for configuration purposes.
format_version = "1.0" # Generic GDMA host profile, for whatever platform we're on. <host "GDMA host"> type = os_type pattern = "(.*)" host_profile = "gdma-$SANITIZED1$-host" </host> # Apply the "apache-web-server" service if, not surprisingly, # Apache Web Server is currently running. <service "Apache 2.4"> type = full_process_command cardinality = first # Matches a command line like "/usr/sbin/apache2 -k start". pattern = "/apache2\s+-k\s+\S+" service = "apache-web-server" </service>
Windows Daemons
Simple daemons may take simple sensors to recognize.
format_version = "1.0" <host "Windows"> type = os_type pattern = "windows" host_profile = "gdma-windows-host" </host> <service "DNS Client"> type = running_system_service pattern = "^DNS Client$" service = "windows-dns-client" </service> <service "Search Indexer"> type = full_process_command pattern = "^searchindexer.exe$" service = "windows-search-indexer" </service>
Request for Feedback
GroundWork invites customers to contact us and contribute examples you have found useful and might be generally interesting, or tell us about examples you would like to see included. We are especially interested in examples that make good use of complex pattern
s, or that demonstrate use of the less-common sensor directives.
Related Resources
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
-
Page: