Overview

From a technical standpoint, Syslog Monitoring has several visible components that need to be installed and configured:

  • A tsyslogd container that runs the Syslog Monitoring.
  • Configuration within the gw8/docker-compose.override.yml file to establish the tsyslogd container within your GroundWork Monitor 8 system.
  • Two configuration files stored on the GroundWork Monitor host, that control the run-time operation of the tsyslogd container.
  • Setup for OS-provided rsyslogd daemons to have them forward selected portions of the syslog message stream to the tsyslogd container.
  • Setup for firewall permissions to receive syslog messages from selected remote systems.
  • Setup for whatever alerts you wish to generate for syslog messages that represent certain states of your infrastructure.

That said, when considering the use of Syslog Monitoring, it is probably best to skip the technical details to begin with, and start instead by collecting and examining the syslog messages that you are considering for Syslog Monitoring.

  • Can you force your devices and applications to generate test messages against which you can generate regular expressions to match those messages?
  • Can you collect an adequately comprehensive set of messages so you believe you understand the particular messages you wish to recognize?
  • Can you assign a sensible state (okay, warning, critical, unknown, or just ignore) to each of the messages you might send to the monitoring system?

Only once you have that data in hand, for at least one device or application, is it appropriate to follow the rest of the instructions here.  It is not necessary to have a total and complete set of messages before you implement this feature.  You can start small, deflecting all previously-unseen messages to a warning state so they become visible, then reclassifying those messages to other states (including ignoring some messages) as you gain experience with the messages your devices and applications generate.

Prerequisites

To successfully install the syslog-monitoring feature, you will need to prepare the following beforehand:

  • You must have downloaded the config-file tarball from the Syslog Monitoring Configuration Files section of our Downloads page.
  • To obtain a value for the agent_id option in the telegraf-syslog.conf file, as described below, you must have the uuidgen or uuid program installed on your GroundWork server.  Depending on the Linux distribution, these commands will be included in the util-linux, uuidgen, or uuid package.
  • You must have a limited-privileges user configured in the Administration > Users screen within GroundWork Monitor.  See Configuring Telegraf for more information on this user.
  • You must understand whether your GroundWork server is running SELinux.  See SELinux constraints on port numbers for more information on this topic.

Selecting port numbers for the tsyslogd container

Choosing which port to use for the tsyslogd container raises a number of issues.  GroundWork does not know of the topology or complexity of the existing syslog-message handling within your infrastructure.  Our intent is to disturb that setup as little as possible, and only accept feeds of selected messages that may be of interest to the monitoring system.

The following port numbers have been historically used for various types of syslog-message handling.

  • 514/udp:  Used for submission of classic BSD-syslog messages over UDP; see RFC3164: The BSD syslog Protocol.  Also used for RFC5424-formatted messages sent over UDP, as described in RFC5426: Transmission of Syslog Messages over UDP.  You will find this listed as the syslog entry in the /etc/services file.
  • 514/tcp:  This port is sometimes abused for receiving syslog messages over TCP, though it is not officially sanctioned for that use.  You will find this port listed instead as the passwordless shell service in the /etc/services file, which may partly why this usage is somewhat common — in today's security-conscious world, nobody wants to open up their systems to that kind of command execution from remote systems.  Since usage of this port for syslog messages is not officially documented, the format of messages sent to this port is not defined anywhere.  Nonetheless, it can serve as a useful rsyslogd-daemon endpoint for debugging, as noted in the Syslog Monitoring Testing and Troubleshooting page.
  • 601/tcp:  Used for the BEEP protocol applied to syslog, as defined in RFC3195: Reliable Delivery for syslog.  You will find this listed as the syslog-conn entry in the /etc/services file.
  • 6514/tcp:  Used for RFC5424-formatted messages sent using TLS, as described in RFC5425: Transport Layer Security (TLS) Transport Mapping for Syslog.  You will find this listed as the syslog-tls entry in the /etc/services file.
  • 10514/tcp:  Historically used by rsyslog setups for RFC5424-formatted messages sent using TLS.  The rsyslog documentation was changed in 2019 to refer to the standard port 6514 for this purpose instead, but you may find the old usage still grandfathered in, such as within SELinux profiles.
  • 20514/tcp:  Typically used by rsyslog setups for the RELP protocol; see Using TLS with RELP for more information.

Having all those choices makes for something of a confusing mess.  Inasmuch as we don't want to get into the extra details of TLS connections here, we choose a port for our example setup that does not conflict with any of those.  Using a separate port means there should be no confusion with our choice of protocol and data format with what is generally present on one of the standard ports just listed.  In our example, we choose:

  • 26514/tcp:  Used for RFC5424-formatted messages sent via TCP, but without TLS in play.

Port 26514/tcp is currently unassigned by IANA, so there should be no confusion with any established use.

For simplicity in setup and understanding, we use port 26514/tcp both outside and inside the tsyslogd container, as shown in this line in the setup below.

    ports:
      - "26514:26514/tcp"

Note that there are likely to be special considerations if your GroundWork server runs SELinux.  See Advanced setup below and the Syslog Monitoring Testing and Troubleshooting page for more information.

Configuring Docker-Compose to run the tsyslogd container

The following text must appear under the services: category in your gw8/docker-compose.override.yml config file on the GroundWork host:

  # Telegraf acting in a capacity as a specialized syslog daemon.
  tsyslogd:
    ## We use the following image at least until our classify plugin and improvements
    ## to our groundwork plugin make it into the upstream Telegraf distribution.
    image: groundworkdevelopment/telegraf-docker:master
    ## We use "config-syslog/" instead of just "config/" as the location of the volume on
    ## the GroundWork host so as to distinguish that directory from any similar location
    ## used for purposes other than this container.  We place two files in this directory:
    ## "telegraf-syslog.conf", and "classify.toml".  (We intentionally don't just use the
    ## simpler name "telegraf.conf", to avoid any confusion with a copy of telegraf.conf
    ## used for other purposes, such as SNMP trap handling, even if that other copy of
    ## telegraf.conf sits in some other directory.)
    volumes:
      - ${PWD}/config-syslog/:/etc/telegraf/
    depends_on:
      - groundwork
    ## The container startup configuration is somewhat complex to accommodate Foundation
    ## not yet being up right when this container starts up.
    entrypoint: ["bash", "-c"]
    command: ["while test $$(wget -qO - groundwork:8080/index.html | grep 'Groundwork Server' | wc -l) != 2;
               do echo 'awaiting groundwork health'; sleep 10; done;
               PATH=/opt/telegraf:$$PATH
               exec telegraf --config /etc/telegraf/telegraf-syslog.conf"]
    ## Externally-available port that local and remote rsyslogd daemons can forward
    ## data to, but only using RFC5424-format messages.  That constraint ensures that
    ## all the data fields needed by the Telegraf syslog input plugin are available,
    ## and uses the rsyslogd program for parsing older RFC3164-format messages, which
    ## we find that the syslog input plugin has difficulty parsing correctly.
    ## See the GroundWork documentation on the selection of outside:inside port numbers
    ## here.  We generally use port 26514 on the outside of the container for rsyslogd
    ## daemons to forward messages to, so as not to conflict with standard existing
    ## syslog-related ports in that context.  The inside-the-container port we specify
    ## here must match the "server" port specified for the syslog input plugin.
    ports:
      - "26514:26514/tcp"

That setup alone will not be enough to get you a running container.  You must continue on and configure at least the Telegraf portion of the container, as discussed in the next section, before the container will stay up and running once started.

Configuring run-time operation of the tsyslogd container

Two configuration files provide control of how the tsyslogd container operates.  One of them configures Telegraf itself, configuring global options, the selection of plugins to run, and the config options for most of the plugins.  The other file controls the detailed operation of the classify plugin.  It is this second file that a customer will modify to adapt to the kinds of syslog messages seen in their environment.  These files are not currently provided as part of the GW8 install; instead, they are available from the Syslog Monitoring Configuration Files section of our Downloads page.  Once downloaded, unroll the tarball (here, presuming you parked it in the /tmp/ directory):

cd ~/gw8
tar xvfz /tmp/config-syslog.8.3.1.tar.gz

That will create a ~/gw8/config-syslog/ subdirectory containing the two config files.  Make sure you put this subdirectory under control of whatever backup procedures you have in place, as it will contain critical setup data for your system.

As you put this capability into play, you may wish to gradually extend the kinds of messages it recognizes.  And in the process of such deployment, mistakes will be made.  To shake things out and prove to your satisfaction that syslog messages are being handled as you intend, see the Syslog Monitoring Testing and Troubleshooting page.

Configuring Telegraf

The telegraf-syslog.conf file must be placed in the gw8/config-syslog/ directory.  The fields you will need to modify are these, all in the OUTPUT PLUGINS section for the [[outputs.groundwork]] plugin.

  ## Agent uuid for GroundWork API Server. 
  agent_id = ""

  ## Username and password to access GroundWork API.
  username = ""
  password = ""

To obtain a value for the agent_id option, you must have the uuidgen or uuid program installed, as described in the Prerequisites section above.  Run the uuidgen or uuid command without arguments, and use the output of that command as the agent_id.  You will end up with something like:

agent_id = "77118f93-692d-48c2-bed7-a491dfdc1c64"

The username and password options must be filled in with values configured in the Administration > Users screen within GroundWork Monitor.  This should be an unprivileged user created for this particular purpose.

Once you have configured that much, the tsyslogd container can be started and should remain running.  However, further adaptation to your local conditions is still required.  For that part, note that the telegraf-syslog.conf file as you downloaded it comes preconfigured to refer to the classify.toml file, which is the subject of the next section.

# Classify Telegraf data points according to user-specified rules.
[[processors.classify]]
  order = 200
  ## The detailed configuration data for the classify plugin lives
  ## in a separate file, whose path is given here.
  classify_config_file = '/etc/telegraf/classify.toml'

Configuring the classify plugin processing

Classification of syslog messages works in two stages, as described in the About Syslog Monitoring page.  The first stage uses a part of the incoming data (the name of the host that generated the message) to group this message into a bucket of messages from hosts with similar functionality, so the same regular expressions can be used to match messages from all of them without tedious duplication in the config file.  The second stage analyzes the syslog message by attempting to match it against a series of regular expressions that you provide, to determine which category of status should be assigned to this message.  More detail is provided in the Classify Processor Plugin page.

The classify.toml fle is where you will configure all the regular expressions used to match the syslog messages you expect to see.  This file must be placed in the gw8/config-syslog/ directory on the GroundWork Monitor host system.  Once you have it there, you will want to configure these options:

  • selector_mapping (a mapping from hostnames to regex group names)
  • default_regex_group
  • mapped_selector_regexes.MY_REGEX_GROUP (one variant for each distinct MY_REGEX_GROUP name you have listed in the selector_mapping and default_regex_group options).  Each variant will include the category names to be used as the classification result if that regex group is chosen, along with the regular expressions that should be matched against syslog messages to see if they fall into those categories.  Matching will occur in the order that you define regexes within the regex group.  Each category within a given regex group can only be mentioned once, but you are free to re-order the categories under a regex group as needs dictate.  Attempts to match regular expressions within each category will be made in the order in which those regexes are listed in the setup for that category.
  • default_category
  • drop_categories (which categories should be ignored, with messages so classified not being passed to GroundWork Monitor)
  • aggregation options, as desired.  A future version of this documentation will describe how you might want to collect and display consolidated statistics on syslog message handling within the tsyslogd container.

A very detailed description of all of those options, including their meanings, their effects on the classifier, and their acceptable formats in the config file, is presented in the Configuration documentation for the Telegraf classify plugin.  In practice, when using the classify plugin in the context of handling syslog messages and sending them to GroundWork Monitor, there are only four category names that make sense to define for data that should reach GroundWork:

SERVICE_OK
SERVICE_WARNING
SERVICE_UNSCHEDULED_CRITICAL
SERVICE_UNKNOWN

Any messages that don't fit one of those categories should be configured to be dropped.

Configuring the rsyslog service to receive syslog messages

Your OS distribution will presumably have come configured out-of-the-box so most system services can send syslog messages to the rsyslogd daemon and have them received there.  However, the transport used for such local messages has evolved over time, and that setup alone might not be adequate for our purposes here.  There are two situations of particular note that require you to take some explicit action to configure the rsyslog service to accept incoming messages.

  • If you need to debug any of your rsyslog service or tsyslogd container setup, you will want some simple way to send in crafted test messages.  See the Syslog Monitoring Testing and Troubleshooting page for general discussion of that process.  Here, we note that it is simplest to have the rsyslogd daemon open a TCP port that can receive messages sent by the logger command.
  • If the rsyslog service on your GroundWork server will act as a relay for syslog messages from remote devices and hosts, it will need to have some public port open for that purpose.  You will probably need to discuss that with your network administrators, because firewall settings (not discussed here) will also be involved.  The notes on port usage given for Selecting port numbers for the tsyslogd container also apply to selecting the right port for the rsyslog service (as opposed to the tsyslogd container).  Obviously, the service and the container must use different port numbers on the GroundWork host, so they don't collide.

The basic directives for opening up a network port for rsyslog to accept incoming messages are simple, and just involve uncommenting some lines in the as-delivered /etc/rsyslog.conf file.  On a CentOS7 system, for instance, these are the relevant directives:

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

Uncomment one or both of those pairs, and set the port number differently if you need to for your situation.  Pay attention to SELinux constraints on port numbers if you are running SELinux on your machine.  On CentOS8, the equivalent lines look slightly different:

# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
#module(load="imudp") # needs to be done just once
#input(type="imudp" port="514")

# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
#module(load="imtcp") # needs to be done just once
#input(type="imtcp" port="514")

We said "basic directives" above because the rsyslog documentation describes a lot of options you may or may not wish to apply in your situation.  Discussion of those options is beyond the scope of what we are describing here, but you should at least look through that material (for the version of rsyslog that you are running) to satisfy yourself that your system will operate as intended.

Once you have made appropriate changes, bounce the service to pick up changes:

service rsyslog restart

Forwarding messages from rsyslogd to the tsyslogd container

The idea behind Syslog Monitoring in GW8 is that we do not wish to interfere with OS-provided syslog-message handling.  The usual system files, such as /var/log/kern.log and /var/log/messages, should continue to log particular kinds of syslog messages as would happen in an unmodified system.  Forwarding of messags to the tsyslogd container is therefore done as a strict extension to the existing setup, not replacing it.  To that end, you will create a new /etc/rsyslog.d/90-groundwork.conf file so as not to disturb the existing configuration.  Once changes are made there, you can bounce the rsyslogd process with a service rsyslog restart command, and the extra configuration will be included in the operational setup.

The setup shown here for forwarding messages from remote servers does so by directly targeting the tsyslogd container's externally visible port, bypassing the rsyslogd daemon on the GroundWork Monitor host.  This seems like the simplest and most efficient means to process such messages.  An alternative setup would route such messages through the GroundWork Monitor host's rsyslogd process.  You can work out the details for doing that if you are so inclined.

The rsyslog feature provided by the Linux OS has detailed documentaton beyond just the simple man rsyslogd and man rsyslog.conf manual pages.  A separate OS-provided documentation package can be installed on your machine to provide HTML pages with in-depth information on the release you have installed, or you can view that information online for the latest release, at https://www.rsyslog.com.

Messages from rsyslogd on the GroundWork Monitor server

If you wish to send syslog messages from the GroundWork Monitor server's own rsyslogd daemon to the tsyslogd container, here is the setup you will need in the /etc/rsyslog.d/90-groundwork.conf file on the GroundWork Monitor host.  Modify the facility.priority setting (shown here as user.err) as appropriate for your own local conditions.

# Local forwarding rules for rsyslog, sending chosen messages strictly
# in RFC5424 format to the tsyslogd container port on this GroundWork
# Monitor server.
#
# For more information, see rsyslog.conf(5) and /etc/rsyslog.conf

# This rule forwards selected messages to the GroundWork tsyslogd container.
# Adjust the selector field as desired, to choose which kinds of messages
# get forwarded.  The "(o)" string specifies octet-counting framing, which
# is needed for clean compatibility with the Telegraf syslog input plugin.
# The ";RSYSLOG_SyslogProtocol23Format" template specifies formatting of
# messages according to the RFC5424 standard, which we want so the nature
# of the data fields is well-controlled.
user.err    @@(o)127.0.0.1:26514;RSYSLOG_SyslogProtocol23Format

Bounce the rsyslog service after first establishing this setup, and again in the future after making changes:

service rsyslog restart

Messages from rsyslogd on remote servers

For messages that originate off the GroundWork server, you must make some decisions about the syslog-message routing topology that makes sense at your site.  The easiest solution from a debugging point of view is not to send message directly to port 26514 on the GroundWork server, but instead to route them to the rsyslog daemon on the GroundWork server using its native facilities for receiving messages.  The rsyslogd daemon can then act as a relay to the tsyslogd container for selected messages.  This provides a huge advantage:  if you have any connectivity issues, you can factor out the tsyslogd container as part of the remote-to-GW-server path, and debug the situation using ordinary rsyslog configuration for writing messages to files on the GW server.  You can use the standard device documentation and rsyslog documentation to decipher how to get the messages properly forwarded to the GW-server rsyslog daemon.  A second advantage is that using just the rsyslogd daemon on the GW server to relay syslog messages to the tsyslogd container guarantees that they will all arrive at the container in exactly the same format as regards framing and message structure (RFC5424).  Otherwise, you will have more work to do for each individual device to guarantee that those parameters are set correctly on every sending device.  It's better to work out all of that detail in the device-to-relay link, again factoring out the tsyslogd container as part of what needs to be looked at while setting up sending in remote messages.

In the following diagram, we show the physical flow of messages from original sources and thence through the system, when using rsyslog on the GroundWork server as a relay for messages from other hosts and devices.  Contrast this with the diagram in How Syslog Monitoring Works that shows only the logical flow, which reflects a setup where remote machines forward directly to the tsyslogd container.

If you do wish to send syslog messages from other machines directly to the tsyslogd container (note again, this is not generally recommended), here is the setup you will need in the /etc/rsyslog.d/90-groundwork.conf file on those machines.  Obviously, you will need to replace MY-GROUNDWORK-SERVER with the hostname or IP address of the GroundWork Monitor host machine.  Modify the facility.priority setting (shown here as user.err) as appropriate for your own local conditions.

# Remote forwarding rules for rsyslog, sending chosen messages strictly
# in RFC5424 format to the GroundWork Monitor tsyslogd container port.
#
# For more information, see rsyslog.conf(5) and /etc/rsyslog.conf

# This rule forwards selected messages to the GroundWork tsyslogd container.
# Adjust the selector field as desired, to choose which kinds of messages
# get forwarded.  The "(o)" string specifies octet-counting framing, which
# is needed for clean compatibility with the Telegraf syslog input plugin.
# The ";RSYSLOG_SyslogProtocol23Format" template specifies formatting of
# messages according to the RFC5424 standard, which we want so the nature
# of the data fields is well-controlled.
user.err    @@(o)MY-GROUNDWORK-SERVER:26514;RSYSLOG_SyslogProtocol23Format

Bounce the rsyslog service after first establishing this setup, and again in the future after making changes:

service rsyslog restart

Establishing firewall rules for accepting remote syslog messages on the GroundWork server

For a remote rsyslogd daemon to send messages to the GroundWork server, you will probably need to set up firewall rules to accept such traffic.  Details of that sent depend on the firewall you have in play, and what port you have selected for receipt of remote messages (either by rsyslogd on the GroundWork server, which would be the recommended configuration, or by the tsyslogd container).

Advanced setup

What we have presented above is a basic setup, so we could concentrate on the details specific to the tsyslogd container.  However, that configuration may not suffice for your own infrastructure, for a variety of reasons, such as:

Most of that sort of thing is outside the bounds of our documentaton here.  You first need to know that the selection of message-forwarding topology and port-number usage is up to you to decide, and the example port 26514 used above can be altered if some other port number is better suited for your infrastructure.  That said, it is best to insist on octet-counting framing and on RFC5424 message structure as the messages are received by the tsyslogd container, as otherwise you are in for some long debugging sessions.

SELinux constraints on port numbers

Special consideration applies when your GroundWork server runs SELinux.  It turns out that in that context, the rsyslogd daemon will be subject to constraints on which port numbers it can open for listening for incoming messages and for connecting to syslog relays and collectors while forwarding messages.  It can be infuratingly difficult to understand why you are experiencing a malfunction when this is in play, so we provide some guidance here.

You can check whether SELinux is imposing port-access contraints by seeing if SELinux is operating:

% getenforce
Enforcing

That's what you will see when SELinux is controlling the ports accessible by the rsyslogd daemon.  You can see which particular ports that daemon can access using the following command:

% semanage port -l | fgrep syslog
syslog_tls_port_t              tcp      6514, 10514
syslog_tls_port_t              udp      6514, 10514
syslogd_port_t                 tcp      601, 20514
syslogd_port_t                 udp      514, 601, 20514

That sample output shows the ports associated with syslog handling in a standard CentOS7 SELinux security profile.  You will notice that port 26514 is not among those ports, so in this configuration, the rsyslogd daemon will be unable to use that port to forward messages to the tsyslogd container.  Your choices to solve this are:

  • Disable SELinux.  We mention this for completeness, but you probably have it running for good reason, so this choice is not recommended.
  • Use one of the ports you see listed in the semanage output on your machine.  Those ports won't cause any conflict with SELinux, but they may cause conflict and confusion with ports you are otherwise using in your infrastructure for syslog message transport, or you may end up using such a port number in a way that does not align with the standard protocol and message format that is associated with that port.  That could cause a lot of administrative confusion down the line.

  • Tell SELinux to allow the rsyslogd daemon access to a port of your choosing.  For example, to add access to the same port 26514 as we have shown above, you can say:

    semanage port -a -t syslogd_port_t -p tcp 26514

    It will take perhaps 15 seconds for that change to percolate through the system and have the rsyslogd daemon know about the change.  To see that your addition has been successful:

    % semanage port -l | fgrep syslog
syslog_tls_port_t              tcp      6514, 10514
syslog_tls_port_t              udp      6514, 10514
syslogd_port_t                 tcp      26514, 601, 20514
syslogd_port_t                 udp      514, 601, 20514

    Alternatively, look just for all local changes to the standard compiled SELinux policies:

    % semanage port -l -C
SELinux Port Type              Proto    Port Number

syslogd_port_t                 tcp      26514

    To thereafter remove that port from special consideration:

    semanage port -d -p tcp 26514

    See the semanage-port man page for full details on these commands.

Note that there is some unexplained leeway in the port usage by the rsyslogd daemon in the SELinux context, even without any local adjustments to the SELinux policy.  In particular, suppose you have the daemon listen on port 514/tcp using these lines in the /etc/rsyslog.conf file, as they would appear on a CentOS7 system (equivalent directives on CentOS8 use a different syntax):

$ModLoad imtcp
$InputTCPServerRun 514

Then the daemon can receive syslog messages on that TCP port, even though it is not listed as being available in the output of a "semanage port -l | fgrep syslog" command.  We take advantage of this loophole in our configuration of rsyslog as shown, for debugging purposes described in Syslog Monitoring Testing and Troubleshooting.

Setting up alerts for syslog messages

Messages received by the tsyslogd container that are classified and not dropped will end up being reported to GroundWork Monitor via the syslog service on the hostname included in the syslog message.  That, of course, should be the name of the host that sent the message.  The status of that service will be set as determined by a combination of the regular-expressions you define and the messages that actually come through.  That status can be used to drive notifications in the usual manner, using the features of GroundWork Messenger.

Related Resources

About Syslog Monitoring

Classify Processor Plugin

Configuration for the classify processor plugin

Syslog Monitoring Testing and Troubleshooting

The rocket-fast Syslog Server - Documentation on the rsyslogd daemon and its configuration

RSyslog Documentation

RFC3164 - The BSD syslog Protocol, which documents the now-deprecated older syslog-message format and its uncontrolled variations

RFC5424 - The Syslog Protocol

RFC5425 - Transport Layer Security (TLS) Transport Mapping for Syslog

GroundWork Messenger