From a technical standpoint, Syslog Monitoring has several visible components that need to be installed and configured:
tsyslogdcontainer that runs the Syslog Monitoring.
- Configuration within the
gw8/docker-compose.override.ymlfile to establish the
tsyslogdcontainer within your GroundWork Monitor 8 system.
- Two configuration files stored on the GroundWork Monitor host, that control the run-time operation of the
- Setup for OS-provided
rsyslogddaemons to have them forward selected portions of the syslog message stream to the
- Setup for firewall permissions to receive syslog messages from selected remote systems.
- Setup for whatever alerts you wish to generate for syslog messages that represent certain states of your infrastructure.
That said, when considering the use of Syslog Monitoring, it is probably best to skip the technical details to begin with, and start instead by collecting and examining the syslog messages that you are considering for Syslog Monitoring.
- Can you force your devices and applications to generate test messages against which you can generate regular expressions to match those messages?
- Can you collect an adequately comprehensive set of messages so you believe you understand the particular messages you wish to recognize?
- Can you assign a sensible state (okay, warning, critical, unknown, or just ignore) to each of the messages you might send to the monitoring system?
Only once you have that data in hand, for at least one device or application, is it appropriate to follow the rest of the instructions here. It is not necessary to have a total and complete set of messages before you implement this feature. You can start small, deflecting all previously-unseen messages to a warning state so they become visible, then reclassifying those messages to other states (including ignoring some messages) as you gain
experience with the messages your devices and applications generate.
To successfully install the syslog-monitoring feature, you will need to prepare the following beforehand:
- You must have downloaded the config-file tarball from the Syslog Monitoring Configuration Files section of our Downloads page.
- To obtain a value for the
agent_idoption in the
telegraf-syslog.conffile, as described below, you must have the
uuidprogram installed on your GroundWork server. Depending on the Linux distribution, these commands will be included in the
- You must have a limited-privileges user configured in the Administration > Users screen within GroundWork Monitor. See Configuring Telegraf for more information on this user.
- You must understand whether your GroundWork server is running SELinux. See SELinux constraints on port numbers for more information on this topic.
Selecting port numbers for the tsyslogd container
Choosing which port to use for the
tsyslogd container raises a number of issues. GroundWork does not know of the topology or complexity of the existing syslog-message handling within your infrastructure. Our intent is to disturb that setup as little as possible, and only accept feeds of selected messages that may be of interest to the monitoring system.
The following port numbers have been historically used for various types of syslog-message handling.
- 514/udp: Used for submission of classic BSD-syslog messages over UDP; see RFC3164: The BSD syslog Protocol. Also used for RFC5424-formatted messages sent over UDP, as described in RFC5426: Transmission of Syslog Messages over UDP. You will find this listed as the
syslogentry in the
- 514/tcp: This port is sometimes abused for receiving syslog messages over TCP, though it is not officially sanctioned for that use. You will find this port listed instead as the passwordless
shellservice in the
/etc/servicesfile, which may partly why this usage is somewhat common — in today's security-conscious world, nobody wants to open up their systems to that kind of command execution from remote systems. Since usage of this port for syslog messages is not officially documented, the format of messages sent to this port is not defined anywhere. Nonetheless, it can serve as a useful
rsyslogd-daemon endpoint for debugging, as noted in the Syslog Monitoring Testing and Troubleshooting page.
- 601/tcp: Used for the BEEP protocol applied to syslog, as defined in RFC3195: Reliable Delivery for syslog. You will find this listed as the
syslog-connentry in the
- 6514/tcp: Used for RFC5424-formatted messages sent using TLS, as described in RFC5425: Transport Layer Security (TLS) Transport Mapping for Syslog. You will find this listed as the
syslog-tlsentry in the
- 10514/tcp: Historically used by rsyslog setups for RFC5424-formatted messages sent using TLS. The rsyslog documentation was changed in 2019 to refer to the standard port 6514 for this purpose instead, but you may find the old usage still grandfathered in, such as within SELinux profiles.
- 20514/tcp: Typically used by rsyslog setups for the RELP protocol; see Using TLS with RELP for more information.
Having all those choices makes for something of a confusing mess. Inasmuch as we don't want to get into the extra details of TLS connections here, we choose a port for our example setup that does not conflict with any of those. Using a separate port means there should be no confusion with our choice of protocol and data format with what is generally present on one of the standard ports just listed. In our example, we choose:
- 26514/tcp: Used for RFC5424-formatted messages sent via TCP, but without TLS in play.
Port 26514/tcp is currently unassigned by IANA, so there should be no confusion with any established use.
For simplicity in setup and understanding, we use port 26514/tcp both outside and inside the
tsyslogd container, as shown in this line in the setup below.
Note that there are likely to be special considerations if your GroundWork server runs SELinux. See Advanced setup below and the Syslog Monitoring Testing and Troubleshooting page for more information.
Configuring Docker-Compose to run the tsyslogd container
The following text must appear under the
services: category in your
gw8/docker-compose.override.yml config file on the GroundWork host:
That setup alone will not be enough to get you a running container. You must continue on and configure at least the Telegraf portion of the container, as discussed in the next section, before the container will stay up and running once started.
Configuring run-time operation of the tsyslogd container
Two configuration files provide control of how the
tsyslogd container operates. One of them configures Telegraf itself, configuring global options, the selection of plugins to run, and the config options for most of the plugins. The other file controls the detailed operation of the
classify plugin. It is this second file that a customer will modify to adapt to the kinds of syslog messages seen in their environment. These files are not currently provided as part of the GW8 install; instead, they are available from the Syslog Monitoring Configuration Files section of our Downloads page. Once downloaded, unroll the tarball (here, presuming you parked it in the
That will create a
~/gw8/config-syslog/ subdirectory containing the two config files. Make sure you put this subdirectory under control of whatever backup procedures you have in place, as it will contain critical setup data for your system.
As you put this capability into play, you may wish to gradually extend the kinds of messages it recognizes. And in the process of such deployment, mistakes will be made. To shake things out and prove to your satisfaction that syslog messages are being handled as you intend, see the Syslog Monitoring Testing and Troubleshooting page.
telegraf-syslog.conf file must be placed in the
gw8/config-syslog/ directory. The fields you will need to modify are these, all in the
OUTPUT PLUGINS section for the
To obtain a value for the
agent_id option, you must have the
uuid program installed, as described in the Prerequisites section above. Run the
uuid command without arguments, and use the output of that command as the
agent_id. You will end up with something like:
password options must be filled in with values configured in the Administration > Users screen within GroundWork Monitor. This should be an unprivileged user created for this particular purpose.
Once you have configured that much, the
tsyslogd container can be started and should remain running. However, further adaptation to your local conditions is still required. For that part, note that the
telegraf-syslog.conf file as you downloaded it comes preconfigured to refer to the
classify.toml file, which is the subject of the next section.
Configuring the classify plugin processing
Classification of syslog messages works in two stages, as described in the About Syslog Monitoring page. The first stage uses a part of the incoming data (the name of the host that generated the message) to group this message into a bucket of messages from hosts with similar functionality, so the same regular expressions can be used to match messages from all of them without tedious duplication in the config file. The second stage analyzes the syslog message by attempting to match it against a series of regular expressions that you provide, to determine which category of status should be assigned to this message. More detail is provided in the Classify Processor Plugin page.
classify.toml fle is where you will configure all the regular expressions used to match the syslog messages you expect to see. This file must be placed in the
gw8/config-syslog/ directory on the GroundWork Monitor host system. Once you have it there, you will want to configure these options:
selector_mapping(a mapping from hostnames to regex group names)
mapped_selector_regexes.MY_REGEX_GROUP(one variant for each distinct
MY_REGEX_GROUPname you have listed in the
default_regex_groupoptions). Each variant will include the category names to be used as the classification result if that regex group is chosen, along with the regular expressions that should be matched against syslog messages to see if they fall into those categories. Matching will occur in the order that you define regexes within the regex group. Each category within a given regex group can only be mentioned once, but you are free to re-order the categories under a regex group as needs dictate. Attempts to match regular expressions within each category will be made in the order in which those regexes are listed in the setup for that category.
drop_categories(which categories should be ignored, with messages so classified not being passed to GroundWork Monitor)
- aggregation options, as desired. A future version of this documentation will describe how you might want to collect and display consolidated statistics on syslog message handling within the
A very detailed description of all of those options, including their meanings, their effects on the classifier, and their acceptable formats in the config file, is presented in the Configuration documentation for the Telegraf
classify plugin. In practice, when using the
classify plugin in the context of handling syslog messages and sending them to GroundWork Monitor, there are only four category names that make sense to define for data that should reach GroundWork:
Any messages that don't fit one of those categories should be configured to be dropped.
Configuring the rsyslog service to receive syslog messages
Your OS distribution will presumably have come configured out-of-the-box so most system services can send syslog messages to the
rsyslogd daemon and have them received there. However, the transport used for such local messages has evolved over time, and that setup alone might not be adequate for our purposes here. There are two situations of particular note that require you to take some explicit action to configure the rsyslog service to accept incoming messages.
- If you need to debug any of your rsyslog service or
tsyslogdcontainer setup, you will want some simple way to send in crafted test messages. See the Syslog Monitoring Testing and Troubleshooting page for general discussion of that process. Here, we note that it is simplest to have the
rsyslogddaemon open a TCP port that can receive messages sent by the
- If the rsyslog service on your GroundWork server will act as a relay for syslog messages from remote devices and hosts, it will need to have some public port open for that purpose. You will probably need to discuss that with your network administrators, because firewall settings (not discussed here) will also be involved. The notes on port usage given for Selecting port numbers for the tsyslogd container also apply to selecting the right port for the rsyslog service (as opposed to the
tsyslogdcontainer). Obviously, the service and the container must use different port numbers on the GroundWork host, so they don't collide.
The basic directives for opening up a network port for rsyslog to accept incoming messages are simple, and just involve uncommenting some lines in the as-delivered
/etc/rsyslog.conf file. On a CentOS7 system, for instance, these are the relevant directives:
Uncomment one or both of those pairs, and set the port number differently if you need to for your situation. Pay attention to SELinux constraints on port numbers if you are running SELinux on your machine. On CentOS8, the equivalent lines look slightly different:
We said "basic directives" above because the rsyslog documentation describes a lot of options you may or may not wish to apply in your situation. Discussion of those options is beyond the scope of what we are describing here, but you should at least look through that material (for the version of rsyslog that you are running) to satisfy yourself that your system will operate as intended.
Once you have made appropriate changes, bounce the service to pick up changes:
Forwarding messages from rsyslogd to the tsyslogd container
The idea behind Syslog Monitoring in GW8 is that we do not wish to interfere with OS-provided syslog-message handling. The usual system files, such as
/var/log/messages, should continue to log particular kinds of syslog messages as would happen in an unmodified system. Forwarding of messags to the
tsyslogd container is therefore done as a strict extension to the existing setup, not replacing it. To that end, you will create a new
/etc/rsyslog.d/90-groundwork.conf file so as not to disturb the existing configuration. Once changes are made there, you can bounce the
rsyslogd process with a
service rsyslog restart command, and the extra configuration will be included in the operational setup.
The setup shown here for forwarding messages from remote servers does so by directly targeting the
tsyslogd container's externally visible port, bypassing the
rsyslogd daemon on the GroundWork Monitor host. This seems like the simplest and most efficient means to process such messages. An alternative setup would route such messages through the GroundWork Monitor host's
rsyslogd process. You can work out the details for doing that if you are so inclined.
The rsyslog feature provided by the Linux OS has detailed documentaton beyond just the simple
man rsyslogd and
man rsyslog.conf manual pages. A separate OS-provided documentation package can be installed on your machine to provide HTML pages with in-depth information on the release you have installed, or you can view that information online for the latest release, at https://www.rsyslog.com.
Messages from rsyslogd on the GroundWork Monitor server
If you wish to send syslog messages from the GroundWork Monitor server's own
rsyslogd daemon to the
tsyslogd container, here is the setup you will need in the
/etc/rsyslog.d/90-groundwork.conf file on the GroundWork Monitor host. Modify the facility.priority setting (shown here as
user.err) as appropriate for your own local conditions.
rsyslog service after first establishing this setup, and again in the future after making changes:
Messages from rsyslogd on remote servers
For messages that originate off the GroundWork server, you must make some decisions about the syslog-message routing topology that makes sense at your site. The easiest solution from a debugging point of view is not to send message directly to port 2
6514 on the GroundWork server, but instead to route them to the rsyslog daemon on the GroundWork server using its native facilities for receiving messages. The rsyslogd daemon can then act as a relay to the
tsyslogd container for selected messages. This provides a huge advantage: if you have any connectivity issues, you can factor out the
tsyslogd container as part of the remote-to-GW-server path, and debug the situation using ordinary rsyslog configuration for writing messages to files on the GW server. You can use the standard device documentation and rsyslog documentation to decipher how to get the messages properly forwarded to the GW-server rsyslog daemon. A second advantage is that using just the
rsyslogd daemon on the GW server to relay syslog messages to the
tsyslogd container guarantees that they will all arrive at the container in exactly the same format as regards framing and message structure (RFC5424). Otherwise, you will have more work to do for each individual device to guarantee that those parameters are set correctly on every sending device. It's better to work out all of that detail in the device-to-relay link, again factoring out the
tsyslogd container as part of what needs to be looked at while setting up sending in remote messages.
In the following diagram, we show the physical flow of messages from original sources and thence through the system, when using rsyslog on the GroundWork server as a relay for messages from other hosts and devices. Contrast this with the diagram in How Syslog Monitoring Works that shows only the logical flow, which reflects a setup where remote machines forward directly to the
If you do wish to send syslog messages from other machines directly to the
tsyslogd container (note again, this is not generally recommended), here is the setup you will need in the
/etc/rsyslog.d/90-groundwork.conf file on those machines. Obviously, you will need to replace
MY-GROUNDWORK-SERVER with the hostname or IP address of the GroundWork Monitor host machine. Modify the facility.priority setting (shown here as
user.err) as appropriate for your own local conditions.
rsyslog service after first establishing this setup, and again in the future after making changes:
Establishing firewall rules for accepting remote syslog messages on the GroundWork server
For a remote
rsyslogd daemon to send messages to the GroundWork server, you will probably need to set up firewall rules to accept such traffic. Details of that sent depend on the firewall you have in play, and what port you have selected for receipt of remote messages (either by
rsyslogd on the GroundWork server, which would be the recommended configuration, or by the
What we have presented above is a basic setup, so we could concentrate on the details specific to the
tsyslogd container. However, that configuration may not suffice for your own infrastructure, for a variety of reasons, such as:
- Your GroundWork server is running SELinux.
- Your network administrators want control over which ports are used to send data between machines.
- You will want to tune the set of facility.level specifications for messages that should be forwarded to the
tsyslogdcontainer, so it only sees messages of particular interest to the regular expressions you will be attempting to match within the
- You may want to use additional features of rsyslog, not shown here, to further filter the set of messages that are forwarded. See, for example:
- You may wish to use the rsyslog RELP protocol to reliably accept messages from remote servers. See, for example:
- You may wish to use message queueing within
rsyslogdto guarantee eventual message delivery to the
tsyslogdcontainer. See, for example:
- Your local syslog-handling policy might insist on using a TLS transport, for security reasons (e.g., authentication, authorization, observability of sensitive data on the wire) which are outlined in the various RFCs. See, for example:
Most of that sort of thing is outside the bounds of our documentaton here. You first need to know that the selection of message-forwarding topology and port-number usage is up to you to decide, and the example port 26514 used above can be altered if some other port number is better suited for your infrastructure. That said, it is best to insist on octet-counting framing and on RFC5424 message structure as the messages are received by the
tsyslogd container, as otherwise you are in for some long debugging sessions.
SELinux constraints on port numbers
Special consideration applies when your GroundWork server runs SELinux. It turns out that in that context, the
rsyslogd daemon will be subject to constraints on which port numbers it can open for listening for incoming messages and for connecting to syslog relays and collectors while forwarding messages. It can be infuratingly difficult to understand why you are experiencing a malfunction when this is in play, so we provide some guidance here.
You can check whether SELinux is imposing port-access contraints by seeing if SELinux is operating:
That's what you will see when SELinux is controlling the ports accessible by the
rsyslogd daemon. You can see which particular ports that daemon can access using the following command:
That sample output shows the ports associated with syslog handling in a standard CentOS7 SELinux security profile. You will notice that port 26514 is not among those ports, so in this configuration, the rsyslogd daemon will be unable to use that port to forward messages to the tsyslogd container. Your choices to solve this are:
- Disable SELinux. We mention this for completeness, but you probably have it running for good reason, so this choice is not recommended.
- Use one of the ports you see listed in the
semanageoutput on your machine. Those ports won't cause any conflict with SELinux, but they may cause conflict and confusion with ports you are otherwise using in your infrastructure for syslog message transport, or you may end up using such a port number in a way that does not align with the standard protocol and message format that is associated with that port. That could cause a lot of administrative confusion down the line.
Tell SELinux to allow the
rsyslogddaemon access to a port of your choosing. For example, to add access to the same port 26514 as we have shown above, you can say:
It will take perhaps 15 seconds for that change to percolate through the system and have the
rsyslogddaemon know about the change. To see that your addition has been successful:
Alternatively, look just for all local changes to the standard compiled SELinux policies:
To thereafter remove that port from special consideration:
semanage-portman page for full details on these commands.
Note that there is some unexplained leeway in the port usage by the
rsyslogd daemon in the SELinux context, even without any local adjustments to the SELinux policy. In particular, suppose you have the daemon listen on port 514/tcp using these lines in the
/etc/rsyslog.conf file, as they would appear on a CentOS7 system (equivalent directives on CentOS8 use a different syntax):
Then the daemon can receive syslog messages on that TCP port, even though it is not listed as being available in the output of a "
semanage port -l | fgrep syslog" command. We take advantage of this loophole in our configuration of rsyslog as shown, for debugging purposes described in Syslog Monitoring Testing and Troubleshooting.
Setting up alerts for syslog messages
Messages received by the
tsyslogd container that are classified and not dropped will end up being reported to GroundWork Monitor via the
syslog service on the hostname included in the syslog message. That, of course, should be the name of the host that sent the message. The status of that service will be set as determined by a combination of the regular-expressions you define and the messages that actually come through. That status can be used to drive notifications in the usual manner, using the features of GroundWork Messenger.
Configuration for the
classify processor plugin
The rocket-fast Syslog Server - Documentation on the
rsyslogd daemon and its configuration
RFC3164 - The BSD syslog Protocol, which documents the now-deprecated older syslog-message format and its uncontrolled variations
RFC5424 - The Syslog Protocol
RFC5425 - Transport Layer Security (TLS) Transport Mapping for Syslog