About Syslog Monitoring
What is Syslog Monitoring?
Syslog Monitoring is used for secondary processing of messages generated by various parts of your infrastructure and sent to a central daemon for logging. When the content of syslog messages is such that the state of the infrastructure can be inferred from those messages, it becomes possible to analyze those messages, generate a corresponding service state, and send that to GroundWork Monitor for display and possible alerting.
This feature in GroundWork Monitoring is intended for low to moderate volumes of syslog messages. It is not designed as a full-fledged replacement for various commercial products in the space of log-message handling. Rather, it is intended to provide basic functionality for recognizing common messages and folding that data into the monitoring.
Why Use Syslog Monitoring?
A portion of the original syslog messages may be interesting from a monitoring point of view. Typical use cases include:
- Sensing various conditions that might happen in a database (e.g., startup; shutdown; running out of tablespace).
- Sensing situations that might be seen by a router (e.g., noting changes to its configuration; noting changes in reachability of other network nodes).
- Sensing situations that might be seen by an Intrusion Detection System (e.g., attempts to log in as an administrative user; attempts to execute arbitrary code).
Certain aggregated-data statistics also become available, and these may be graphed to show how active your machines are in this regard and what kinds of states are seen over time.
How Syslog Monitoring Works
Syslog messages are generated by various sources on host machines in your infrastructure. These messages may originate in the kernel, in system daemons and service programs, and in user applications. Those messages are typically sent to the local rsyslogd daemon running on the same machine. That daemon is usually configured to write various flavors of messages to local system log files. For purposes of Syslog Monitoring, some portion of those messages will be configured to be sent to the tsyslogd container port on the GroundWork server. That container will analyze the messages it receives, categorize them according to the message content, and assign a service status to each message (essentially, OK, WARNING, CRITICAL, or UNKNOWN). That status is appended to the message, and the whole is passed to Foundation for storage and for display in the Status Summary dashboard. The calculated status can then also drive alerts.
In the following diagram, we show the logical flow of messages from original sources and thence through the system.
In GW7 handling of syslog messages, GroundWork Monitor replaced the OS-level syslog daemon with the syslog-ng program, and in so doing took over all duties of the standard system-provided syslog daemon. In contrast, the GW8 implementation of syslog message handling is more selective. The rsyslogd program has become the standard syslog daemon across Linux distributions, and we leave that in place, operating in its usual fashion and writing to its usual collection of log files. Only some administrator-chosen portion of the messages received by rsyslogd will be forwarded to the tsyslogd container for analysis and classification. This strategy simplifies administration of your machines.
Within the tsyslogd container, there are two stages in processing. The first stage is to determine the name of a regular expression (regex) group, depending on the value of some particular field (generally, the hostname). That field is the "selector" in the diagram below. In the second stage, which is what happens after a regex group has been chosen, a match of the message string is attempted against the series of regular expressions defined in that group. Once a match is found, the category that included the particular matching regex is assigned as the service status of this message, and the completed message is passed out.
Both the selector-to-regex-group mapping and all of the regex groups and their constituent categories and regular expressions are defined by the user, in the classify.toml file which is read by the tsyslogd container when it starts up. Using this flexibility, the customer adapts the Syslog Monitoring to whatever kinds of messages they expect to appear from their infrastructure, and decides what to do with each of them. Details of that configuration are presented in the Syslog Monitoring Installation page.
The tsyslogd container is implemented as a Telegraf container that uses three plugins in series:
- The
sysloginput plugin, which knows how to receive and parse incoming syslog messages. - The
classifyprocessor plugin, written by GroundWork, which implements the decision logic shown above. - The
groundworkoutput plugin, also written by Groundwork, which knows how to format and send data to GroundWork Monitor.
The classify plugin was written as a general-purpose classifier, with capabilities designed for robust syslog message handling but also probably useful in other contexts. Partly because of current limitations in Telegraf 1.x, its implementation also bundles in some aggregated-statistics collection instead of deferring that optional work to a separate plugin. Aggregated statistics can be useful for graphing how much syslog-message activity occurs over time.
Generating Alerts with Syslog Monitoring
Syslog Monitoring creates a service named syslog on each host from which it sees messages. The derived state of that service can be seen in the corresponding Status Summary screen. Changes in the state of the syslog service can be used to drive notifications from GroundWork Messenger.