Auto Registration Troubleshooting
Common Issues
If auto-registration is completing but monitoring data from GDMA clients is obviously not being processed by the server, perhaps the problem is that the server does not have the Bronx listener port (by default, port 5667) open to the clients. See the
listener_port
value in theconfig/bronx.cfg
file on the server, and theSpooler_NSCA_Port
value in thegdma/config/gdma_auto.conf
file on the GDMA client. These values must match, and you must not have a firewall (or packet filtering such as iptables on the server) blocking this port.
The mechanism for allowing access to a given server port will depend on the particular Linux distribution you are running on the server. On RHEL 6.3, for instance, you would need a line such as the following in the/etc/sysconfig/iptables
file, and then have iptables restarted to pick up this configuration:-A INPUT -m state --state NEW -m tcp -p tcp --dport 5667 -j ACCEPT
If you think that auto-registration is not doing what it ought to, first look in the server-side log file below. This pathname is configured in the
config/register_agent.properties
file on the server, and listed also in the/etc/logrotate.d/groundwork
file to enable regular log rotation./usr/local/groundwork/monarch/var/gdma/register_agent.log
- Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file if logging is enabled on the client by enabling it in the
gdma/config/gdma_auto.conf
file. If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file. If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files.Enable_Local_Logging = "on"
- If an auto-registration request doesn't yield any positive results, the poller won't keep banging on the server every iteration. It waits at least an hour before trying again. This interval is currently hardcoded. Due to slight delays in processing a single auto-registration request, in practice this means it won't try again for 70 minutes.
- If you want to speed things up for testing, especially to avoid the initial 10-minute wait before the first auto-registration attempt after the poller starts and finds it has no local copy of the host configuration file, you will probably want to set
Poller_Proc_Interval
to60
. If you change this value for testing, be sure to change it back (typically to600
) for production use! If auto-registration fails on a given attempt and you don't want to wait the extra hour for the following attempt, just bounce GDMA on the client machine. The Docker Commands page describes the various platform-specific commands used to start and stop the GDMA client. - Time sync: Many installations do not properly synchronize time with NTP. This is particularly an issue with Virtual environments, where time slicing may introduce latency and drift to otherwise stable time signatures. For this reason, GDMA has been made tolerant of time drift. In the setup instructions, the file
/usr/local/groundwork/config/bronx.cfg
was modified to allow up to 900 seconds latency and 90 seconds imminence, or future timestamps, on incoming messages. If the time synchronization is sufficiently off to be outside this 16.5 minute envelope, the GroundWork server will reject the incoming message. To correct this situation, you must either:- Synchronize the time on all monitored systems and the GroundWork server, or
Widen the envelope by changing the parameters in
bronx.cfg
. It is highly recommended that the time be synchronized. You will need to restart Nagios after you make a change.The GroundWork Monitor version of send_nsca supplied with GDMA allows for the sending of old time stamps, which are the time stamps associated with when the check was run, not when the spooler has sent the data, which is potentially much later. This allows the system to accurately process performance data for checks that have been spooled, potentially for long periods.
Encryption mismatch: A common security requirement GroundWork Monitor supports is the encryption of data sent over the NSCA channel. By default, however, GDMA is installed with encryption turned off, which is also the default on the GroundWork server. Thus it is possible to have an encryption mismatch, for example, if you enable encryption on the GroundWork server (in
bronx.cfg
), and fail to enable it on the GDMA insend_nsca.cfg
. Always make sure these files are consistent between GDMA (send_nsca.cfg
) and GroundWork (bronx.cfg
).
One way to manage this situation is to set up GDMA to use an alternatesend_nsca.cfg
file. This file can be downloaded to the GDMA as if it were a plugin, and specified for use via the host external using the GDMA parameter calledSpooler_NSCA_Config
. For example:
If you create a file called:send_nsca_gdma_encrypted.cfg
and you set it up for download as a plugin for Linux, it will be placed on the Linux GDMA in:
/usr/local/groundwork/gdma/tmp/send_nsca_gdma_encrypted.cfg
That means you could set up the GDMA to start using this file at the same time you switch the GroundWork server to use encryption, just by changing:
Spooler_NSCA_Config="/usr/local/groundwork/gdma/tmp/send_nsca_gdma_encrypted.cfg"
Generating Output
Generating Logs
In general, GDMA will not output much in the way of log information unless you specify that it should do so. If you want logs, you can use the Enable_Local_Logging
parameter to output log data to the file specified in the Logdir parameter, which is the log subdirectory by default. You should not leave this enabled permanently, as there is no facility to remove old logs.
Client logs
The gdma_auto.conf
file is located in the /usr/local/groundwork/gdma/config
directory.
- Auto-registration error messages listed in the server log are generally also returned to the GDMA client, and will be recorded in the client's poller log file, if logging is enabled (e.g.,
Enable_Local_Logging = "on"
) on the client to output log data to the file specified in theLogdir
parameter, which is the log subdirectory by default. If the client is having trouble contacting the server in the first place, that may not be evident without enabling logging on the client and looking in the poller log file. If you do enable logging on the client, it is highly recommended that you disable it once you have diagnosed and fixed the problem, because the current version of GDMA never rotates its log files. - Example with logging parameter "on": linux gdma client gdma_auto.conf example.txt
- Example: linux gdma client log directory example.txt
Nagios log
After a Commit operation, the auto registration process begins and a Nagios log message will announce the new GDMA. As soon as the auto-registration cron job runs, the message is matched to the profile, the host definition is created with that profile, and the Monarch group gets the host added.
- Nagios Log - The
nagios.log
file is located in the/usr/local/groundwork/nagios/var
directory.
- Example: gdma server nagios.log example.txt
Agent log
If you think auto registration is not operating as it should you can look in the register_agent.log file.
- Register Agent Log - The
register_agent.log
file is in the/usr/local/groundwork/monarch/var/gdma
directory. - Example: gdma server register_agent.log example.txt
Running GDMA Poller and Spooler By Hand
You can also generate useful output (the same as what gets placed in the log) by running the GDMA Poller and/or Spooler by hand.
On Windows
Poller output - Open a command shell, and type:
cd \Program Files (x86)\groundwork\gdma\bin
CODEgdma_poll.exe -i -d2 -x
CODE- As the Windows GDMA can be installed in alternate locations, you may need to enter a different path. For example in 64-bit Windows, GDMA is installed in
\Program Files (x86)\groundwork\gdma
by default. - This will give you output similar to the following: window gdma client poller output example.txt
- There are several diagnostic steps GDMA goes through each time it runs. Errors in the retrieval of the config file, or with a syntax error in that file, will be presented near the top of this output. Any problems running the checks will appear in the last sections.
If all of that is working ok, then you might try running the spooler:
gdma_spool_processor.exe -i -d2 -x
CODE- This will give you output similar to the following: windows gdma client spooler output example.txt
- The spooler merely takes output from the checks that have been run and sends it to the Target server or servers. Of interest here is if it succeeded in contacting the Target, which will be indicated as Processing live targets. Dead targets are Target servers that can't be reached for some reason, and indicate a down primary or standby system, or network issues in contacting the Target. The spooler will keep trying until it reaches a dead target, or times out according to the
Spooler_Retention_Time
andSpooler_Max_Retries
parameters. - For UNIX, the command is similar, but you must run as the gdma user, with the full execution environment of that user, and specify the GroundWork-supplied perl interpreter. The output is essentially the same for UNIX as for Windows.
On Linux and AIX
Poller output - Open a terminal session, and enter:
su - gdma
CODEcd /usr/local/groundwork/gdma/bin
CODE/usr/local/groundwork/perl/bin/perl gdma_poll.pl -i -d2 -x
CODE- Example: linux gdma client poller output example.txt
Spooler output - Open a terminal session, and enter:
su - gdma
CODEcd /usr/local/groundwork/gdma/bin
CODE/usr/local/groundwork/perl/bin/perl gdma_spool_processor.pl -i -d2 -x
CODE- Example: linux gdma client spooler output example.txt
On Solaris
Poller output - Open a terminal session, and enter:
su - gdma
CODEcd /opt/groundwork/gdma/bin
CODE/opt/groundwork/perl/bin/perl gdma_poll.pl -i -d2 -x
CODESpooler output - Open a terminal session, and enter:
su - gdma
CODEcd /opt/groundwork/gdma/bin
CODE/opt/groundwork/perl/bin/perl gdma_spool_processor.pl -i -d2 -x
CODE
Related Resources
-
Page:
-
Page:
-
Page:
-
Page:
-
Page: