About GroundWork Messenger

GroundWork Messenger is a free standing notification and escalation subsystem for configuring system-wide alert notifications for all monitors regardless of the monitoring method used. By using the GroundWork Messenger front-end, hosts, host groups, services, and service groups can be assigned to notification rules. When GroundWork Messenger receives alerts, it searches for matching definitions in its database. If a matching definition is found, contacts and methods are determined and notifications will be sent. 

GroundWork Messenger is powered by the GroundWork Optimized Request Transmitter (GORT) which works with GroundWork Foundation as updated by TCG ConnectorsTo have Nagios configured elements notify using the new GroundWork Messenger system powered by GORT does not require the use the notification commands previously use in the NoMa application. GroundWork Messenger listens to GroundWork Foundation. Everything that happens in Nagios shows up in Foundation, which allows GroundWork Messenger to get notifications on Nagios resources, as well as any resources monitored by any other source. So, the bottom line is if a state change occurs on Foundation and a notification rule is configured to pick it up, it will be notified upon.

Notifications and escalations using the Nagios method remain available for configuration and maintenance. This allows customers who have made investments in time or scripting for Nagios alerts to continue to use these methods. The states changes that result in Nagios notifications will also be sent to GroundWork Messenger, regardless of whether notification commands are used. The alert flow coming from monitoring sources such as Nagios and Cloud Hub (and any other source) is sent to the RESTful API for the Foundation database. In turn, GroundWork Messenger subscribes to alert messages via the REST API and sends any alerts it finds there that pass its filters. GroundWork Messenger can re-notify for continuing issues if desired, and can also delay the start of notifications, allowing you to suppress volatile services from generating too many notifications. 

Converting from NoMa

GroundWork Messenger is largely the same as NoMa from the point of view of configuration of notification rules, and so the basic rules will be easy to convert. The big difference is in methods. If you have invested in custom method scripts, these will need to be re-done using the cURL method in GroundWork Messenger. If you are using email, you can easily enter your email server details and convert NoMa emails. Slack and SMS are now native methods as well, and on the whole you will find it far easier to configure notifications in GroundWork Messenger than in NoMa. 

For assistance in converting any complex custom scripting from NoMa to GroundWork Messenger, please contact GroundWork Support.  

Configuring Notifications

The operations for GroundWork Messenger are simple. It uses notification rules which essentially define the who, what, when, and how of alert notifications. A rule is a set of directives which determine the hosts and services to be included or excluded in the notifying, the contacts and contact groups to receive notifications, the timezone and timeframe in which notifications should be sent, the specific events to be notified upon, and the method(s) to be used to distribute notifications. Notification rules can be enabled (activated) and disabled, and rules can be set to escalate after a specified number of notifications is reached.

We start with creating notification methods that are then applied in a notification rule.

Creating Notification Methods

Notification Methods are included to direct how notifications are actually delivered. When creating methods you will need to configure the settings (e.g., Email method SMTP server) and also the actual notification templates that will be sent (e.g., Host Alert notification). Methods are configured by going to Configuration > Notifications > Notification Methods, and are applied within a notification rule in Configuration > Notifications > Configuration.

GroundWork Messenger currently supports four methods; cURL (most commonly used for ticketing integration), Email, Slack and SMS. Referencing the image below, the default notification methods are initially named Curl Default, Email Default, Slack Default and SMS Nexmo Default.

To configure the settings for a method, you'll click the corresponding   icon. You will see that for each method you can Delete, Clone (copy),  Save or Cancel its configuration, and each method has different directives. A green box indicates the method has been configured, a red box indicates it has not yet been configured.

To configure the templates for a method, you'll click the corresponding boxes under the Resource TypeA template is a place to create a message that will be sent by the method for Host and Service  Alert, Recovery, Acknowledgment, and Downtime notifications. A green box indicates the notification template is enabled, a grey box means it is disabled. You can format the message as you like using text and macros, the macros available are different for each message type. See Appendix A: Message Type Macro Explanations.

Each of the notification methods details are outlined below.

cURL Method

The cURL notification method can be used to post results to an API. Most commonly the cURL method will be used to send notifications to ticketing systems, and provides a simple way to integrate any ticketing system which supports posting data via an API. In our example, we use the cURL method to send alerts to PagerDuty. The requirements include:

  • Access from the GroundWork Monitor server to https://events.pagerduty.com port 443
  • An API v2 integration added to your PagerDuty service
  • Access to the integration key

Documentation on adding an API integration can be found in the PagerDuty documentation.

  1. To configure the cURL notification method, go to Configuration > Notifications > Notification Methods.
  2. Click on the   icon for Curl, enter a Display Name and click Save
  3. Then, continue by configuring the host and service notification templates. For example, selecting the resource type Host Alert and entering a Name and the Commands to Run a script for PagerDuty (displayed below). To use macros within a script, click the dropdown box, select a macro, click the   icon and paste within the command.

    Example cURL code for testing - just omit the "curl" command when pasting into the template:

    curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' --header 'From: someone@gwos.com' -d '
    {
      "payload": {
        "summary": "[HOSTNAME] [SERVICENAME]",
        "timestamp": "2015-07-17T08:42:58.315+0000",
        "source": "https://groundworkserver.gwos/",
        "severity": "info",
        "component": "postgres",
        "group": "Production",
        "class": "deploy",
        "custom_details": {
          "ping time": "1500ms",
          "load avg": 0.75
        }
      },
      "routing_key": "1234567890987654321abcdefghgfedcba",
      "links": [
        {
          "href": "https://example.com/",
          "text": "Link text"
        }
      ],
      "event_action": "trigger",
      "client": "Sample Monitoring Service",
      "client_url": "https://monitoring.example.com"
    }
    ' 'https://events.pagerduty.com/v2/enqueue'
  4. You can continue with other templates, then you can apply the method within notification rules. Also see Appendix D: cURL templates for PagerDuty.

Special Characters in cURL Method

As this method directly uses the cURL command for its functionality, all the restrictions of the cURL command still apply, such as the use of special characters: `,$,#, etc. In the event that macros are in use for services which return special characters in their output, such as special characters in the [MONITORINGOUTPUT] macro, such characters may cause the curl command itself to fail, in which case it is a good practice to escape the text in any macros utilized. For example:

-X POST -H "Content-Type: application/json" --user user:pw 
-d "{\"NAGIOS_CONTACTGROUPNAME\": \"CONTACT-GW\",\"NAGIOS_HOSTDISPLAYNAME\": \"[HOSTNAME]\",\"NAGIOS_HOSTNAME\": \"[HOSTNAME]\",
\"NAGIOS_HOSTOUTPUT\": [MONITORINGOUTPUT]\",\"NAGIOS_HOSTSTATE\": \"[STATUS]\",\"NAGIOS_LASTHOSTSTATECHANGE\": \"[PRIORSTATUS]\",\"NAGIOS_LASTSERVICESTATECHANGE\": \"[PRIORSTATUS]\",\"NAGIOS_NOTIFICATIONAUTHOR\": \"GroundWork\",\"NAGIOS_NOTIFICATIONCOMMENT\": \"[COMMENTS]\",\"NAGIOS_NOTIFICATIONTYPE\": \"[RESOURCETYPE]\",\"NAGIOS_SERVICEDESC\": \"[SERVICENAME]\",\"NAGIOS_SERVICEOUTPUT\": \"[MONITORINGOUTPUT]\",\"NAGIOS_SERVICESTATE\": \"[STATUS]\",\"NAGIOS_TIMET\": \"[INCIDENTUTC]\"}" 
https://ticketsystem.gwos/api/openticket

Email Method

Emails are sent using any SMTP server. If you run a Mail Relay, you may need to configure it to accept mail from the system. This method can also support a Mail Relay with just a Display Name and SMTP server indicated. GroundWork Messenger also allows you to send email notifications with modern email services such as Gmail or Office 365 as a TLS compliant client, though you should make sure you are aware of rate limits such services may impose.

The entries in this UI replace the gw8\gw8.env configuration file that were used with NoMa in prior versions of GroundWork Monitor, and are no longer needed. 

Here's how to set it up:

  1. As indicated, click the red    icon to bring up the configuration screen. You can use Clone to make a copy of a method.

  2. Enter the fields in the Outgoing Mail Server screen shown and described in the image below. 

    Display NameName for the new method.
    SMTP ServerName of your SMTP server.
    Note: When using Gmail the server is smtp.gmail.com.
    UserA service user email account to authenticate and send email on the service (e.g., Gmail account). For SMTP servers that require you to log in, the User name is typically an Email address, and may be used to override what you put in the Sender Address field.
    PasswordAn app password for the user account, separate from the normal email client password. For the Password, you usually need to generate an app password for this from your Email service account settings, e.g., App Password for Google and App Password for Office 365.
    Sender AddressA Sender Address is required, and can be whatever Email address you want to have as the sender on your notifications. 
    Sender NameThe Sender Name can be customized and will default to GroundWork Messenger if you leave this field blank. 
    PortAn outbound network access on port 587 or similar port (usually open if using one of these hosted email services).
    Note: When using Gmail the port is 587.
    SSLCheck box if using SSL.
    Note: When using Gmail SSL is not used.
    Don't check the certificateCheck this box to bypass checking of the certificate when connecting. You will want to check this box if the server is internal and it is setup with SSL, as it may fail if certificate is checked and fails validation.
  3. Next, select each of the corresponding Host and Service templates to create a unique message to be sent out for each of the notification types. See Appendix B: HTML Email Templates for some examples. For example, click the Alert template under the Host resource type for your current Email method:

    then fill in the following fields:


    NameName for the new template.
    FormatSelect Text or HTML as the message format. HTML allows for the use of tags.
    Subject LineEnter text (and optionally macros) to be displayed as the subject line of the notification.
    Macros
    Select Macros if you want to use variable values in your email subject and message. You can copy and paste the macros from the dropdown, first select the macro, click the copy icon, and then paste the macro into the template wherever you want the value to appear. Alternatively, you can just type in the macro name in square brackets. See the definitions of the macros in Appendix A: Message Type Macro Definitions
    MessageEnter the text (and optionally macros) to be sent as the notification message.
    EnabledMake sure to check the Enabled box to allow your message to be sent out.
  4. Click Save. This Email notification method will send out this notification message for the resource type Host Alert. You can continue to define the rest of the templates, each are a little different. When complete, you can apply this method in notification rules.

Slack Method

This method uses the Slack application and channel to send notifications. You will need:

After your app is installed, and you have your token and channel, you can proceed with the steps below. 

  1. Enter the following fields shown in the image and described below. 


    Display NameName for the new configuration.
    OAuth TokenAccess token for Slack authorization.
    Channel NameSlack channel name.
  2. Then for this method, select each of the corresponding Host and Service templates, e.g., Host Alert, and create a message to be sent out for the notification. For example, click the Alert box under the Host resource type for your current Slack method, and enter the following fields:


    NameName for the new template.
    MacrosSelect Macros if you want to use variable values in your message. You can copy and paste the macros from the dropdown, first select the macro, click the copy icon, and then paste the macro into the template wherever you want the value to appear. Alternatively, you can just type in the macro name in square brackets. See the definitions of the macros in Appendix A: Message Type Macro Definitions
    TemplateEnter the text (and optionally macros) to be sent as the notification Slack message.
    EnabledMake sure to check the Enabled box to allow your message to be sent out.
  3. Click Save. This Slack notification method will send out message for the resource type Host Alert. You can continue to define the rest of the templates, each are a little different. When complete, you can apply this method in notification rules.

SMS Method

GroundWork Monitor can be used with Vonage (formerly Nexmo) for SMS alerts. Vonage is an international SMS provider, and has a very affordable and robust set of SMS and voice services. This method uses the Vonage SMS API, so you will need an account. Requirements vary, but you will want to figure out what kind of account and number you will need. To configure you will need a Vonage SMS account with the API credentials and sending number. The easiest (and least expensive) in the USA is to use a toll-free number. You will want to choose the options for Developer when you sign up, and then once you log in you can choose SMS, and click the button to get an API key. 

This "master" key and secret will show up under https://dashboard.nexmo.com/, and if you are in the USA, you will get a number assigned to you under https://dashboard.nexmo.com/your-numbers If you need to buy a number, you can. In the USA with a toll-free number, you should (as of this writing) not need to register a brand and campaign, as long as the volume you send is low. 

  1. To configure the SMS notification method, go to Configuration > Notifications > Notification Methods.
  2. Simply fill out the form with the API key, API secret, and sending number of your account.
  3. Continue to configure the templates as above, paying attention to the limit of 160 characters, and taking into account that some macro values such as MONITORINGOUTPUT may be longer than this.   

    SMS messages are charged to your Vonage account. GroundWork doesn't resell or broker Vonage accounts, so your use of it is your choice. Any charges you incur using this method with GroundWork Messenger are your own responsibility.  

    If you are interested in brokering SMS accounts for your users or GroundWork customers of your MSP business, you can create sub-keys and assign them to numbers and accounts. It's also possible to use voice services to do text-to-speech, etc. and other methods may be coming that integrate these services. 

Creating a Notification Rule

Known Issue for Installation Prior to 01/12/2022

GroundWork Messenger notification system includes the capability of limiting notifications to a certain time frame. The most common, and default time frame is 24/7, which has a default expiration date of 12/31/2021. This configuration is adjusted in Configuration > Notifications > Configuration > Time Frames. The Validity Information section for the Valid To field needs to be adjust to a later date e.g., 12/31/2099. This configuration change may result in queued notifications being sent. Note, for an installation prior to 01/12/2022 refer to the documentation GroundWork Messenger Timeframes Remediation. For a download and installation as of 01/12/2022, this issue has been fixed in the latest release 8.2.1-1.

In this section, we cover creating a notification rule. A rule incorporates various notification components. If you don't yet have Holidays, Contacts, Contact Groups, Timeframes, and Notification Methods defined yet, you can go to these sections first. They all should be self-evident and intuitive.

  1. Go to Configuration > Notifications > Configuration.
  2. Within the Notifications tab, click Create, (select the pencil icon to Edit an existing notification rule).
    notification rule
  3. Starting with the top section, Notification Information, enter the following fields shown in the image and described in the table below.

    NameIdentifies the notification in the list of notifications.
    Description(Optional) Descriptive text displayed in the list of notifications.
    ActivatedIf this directive is checked, the notification rule is enabled (activated).
    OwnerIdentifies the contact designated as the owner of a rule.  An administrator can edit all rules.
  4. In the section Hosts and Services, you must indicate which hostshost groupsservices, and service groups are to be included or excluded in a notification rule. You will need to know the name of your hosts and services to include and exclude in this rule, as there's no auto-fill or results list in GroundWork Messenger. 


    HostsIndicate the name of a monitored host, or hosts separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used. 
    HostgroupsIndicate the name of a monitored hostgroup, or hostgroups separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?)can also be used.
    ServicesIndicate the name of a monitored service, or services separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used.
    ServicegroupsIndicate the name of a monitored servicegroup, or servicegroups separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used.

    You can use wildcards in includes and excludes:

    • The " * " wildcard character indicates "all" possibilities, (e.g., all hosts). Any include fields left empty are assigned an implicit "NULL" and will automatically not match anything. It's safest to put a "*" here.
    • In specifying single or comma-separated lists of hosts, hostgroups, services, or servicegroups for inclusion or exclusion filtering, the " * " can be used as a multi-character wildcard and " ? "  can be used as a single-character wildcard, within any particular item.

      For example, to illustrate how a wildcard can be used in any location of a filter string: Perhaps you have a tiered subnet structure, like a.canada, b.canada, c.canada. In that case, you can use something like: *.canada.1* for the Hosts Includes field. This would filter and include hosts to be notified upon that start with any multi-characters, continue with a "dot" canada, and end with "dot" 1, followed by anything".

      An applied include constraint will only allow the notification rule to pass if the host or service belongs to at least one of the listed hostgroups or servicegroups. Conversely, an applied exclude constraint will only allow the notification rule to pass if the host or service does not belong to any of the listed hostgroups or servicegroups.


  5. Continuing with the notification rule, the Contacts and methods to notify section is used to define who is to receive notifications (Contacts, Contact Groups), how the notification will be distributed (Methods), and the when the notifications should be sent (Timezone and Timeframe).

    Select Contacts and/or Groups to notify, and select at least one Method. The result will be that the contacts you select will get notified by the methods you select. You can define several rules for the same resources to create different timeframes for texts vs. emails, etc.  You will need to set the timezone and timeframe for the entire rule. The contacts themselves may not be notified if their timeframes or holiday schedules do not include when the notification is generated.  



    Contacts

    Contacts are individual definitions indicating who should get notified and how and when they should get notified in the event of a problem on your network. Contacts displayed in a Notification rule have been previously defined in the Contacts tab.

    Groups

    Contact groups are definitions of one or more contacts and can be used to send alert or recovery notifications to a group of contacts. Contact Groups displayed in a Notification rule have been previously defined in the Contact Groups tab.

    Methods

    Notification Methods are definitions of how notifications are delivered. Methods are defined by going to Configuration > Notifications > Notification Methods.

    Timezone

    The timezone in which notifications should be sent. A list of timezones is presented in the dropdown list.

    Timeframe

    The Timeframe directive indicates the hours allowed for notifications to be sent. Timeframes displayed in a Notification rule have to be defined in the Timeframes tab.

    Tip

    When selecting elements in each of the ContactsGroups, and Methods, note that an element is only associated with the present notification rule if its background is gray, indicating it has been selected. To select a single element, click on the element. To select more than one element use the ControlCommand, or Shift keys while clicking.

  6. Next is the section called Which events to notify. Here you need to select (check or uncheck) the filter options for recovery, problem, and state transitions and other states of change for which to send notifications.

    This is a subtle and powerful selection set. The key to understanding this set of choices is that the Problems section relates to a state change from Up (for hosts) or OK (for services) to one of the problem states. State transitions between problem states are handled in the State Transitions section, and Recovery is for when the resource transitions back to UP or OK. 

    This, if you want to get notified when something goes Critical from OK, AND from Warning, you should check Critical in the Problems section and Warning to Critical in the State Transitions section. You can be very detailed here, but if you want to get notified on everything, leave all the boxes checked. 


    Note also that the Methods section allows you to configure different message formats for Alerts, Recovery, Downtime and Acknowledgement notifications. 

    RecoveryIf checked, rule will notify upon a Service OK event and or a Host UP event. 
    ProblemsIf checked, rule will notify upon a Warning, Unknown, Critical, Host Unreachable, and or Host Down events. It is important to understand that these are for the initial problem state and not for transitions and in State Transitions described below.
    State TransitionsIf checked, rule will notify upon events that have transitioned state from Warning to Critical, Warning to Unknown, Unknown to Critical, Unknown to Warning, Critical to Warning, and or Critical to Unknown.
    OthersIf checked, rule will notify upon if a Downtime started and/or a Downtime ended, and upon an Acknowledgement of an event.
    Numbers and moreSet to Notify after (a delay) a specified number of minutes, thus waiting the entered amount of time before notifications are sent. This is helpful for rapid state changes that may recover and eliminate the need to be notified.

    You can also set to Renotify a number of minutes after the first notification.

    The default Renotify after value of 0 will make GroundWork Messenger send only one notification per state change (this is what we recommend). 

  7. At this point, you can click Create to define the rule.
  8. If you want GroundWork Messenger to send notifications using the same filter rule to alternate contacts for ongoing issues, you can add Escalations. To enable them, you must first Create the rule (click Create, then click the pencil icon on the rule in the list). 
    • Then, select add Escalation toward the bottom of the screen, and set the Contacts and or Groups to be notified, and the Methods of communication for the escalation.
    • As in the first notification, in an escalation the Notify after number indicates when this notification should be escalated. Click Create, adding additional escalations or click Save again to save the notification rule.

      If you need to delete a contact group and it is associated with an escalation, you need to first delete the escalation, then delete the contact group.

      You may want to create multiple Notification rules instead of using Escalations. Using multiple rules may provide an easier work flow in that you can view and manage recurring notifications from the rules list, where Escalations are hidden within each rule.

Creating Notification Components

These components are defined in their respective tabs and applied within the Notifications tab for use in a notification rule. Note the last tab, Logs, is only used to see status of the installation, however to view actual notifications coming in see the Audit Log where filters are available.

Holidays

Holiday definitions indicate a period of time a contact should not receive notifications. The Holiday option is only displayed for previously defined contacts. Holidays are applied in the Contacts tab. You can always come back to this to create and then apply to a contact.

  1. Go to Configuration > Notifications > Configuration > Holidays.
  2. Click the Create button and enter the following information, (select the pencil icon to Edit an existing holiday):

    configure holidays

    NameName for holiday.
    StartStart day and time.
    EndEnd day and time.
    TimeframeSets time frame in which holiday is effective.
    ContactSets defined contact associated with the holiday adding holiday to Contact's definition.
  3. Click Create.

Contacts

Contacts are individual definitions indicating who should get notified and how and when they should get notified in the event of a problem on your network. Contacts can then be applied within NotificationsContact Groups and Holidays tabs.

  1. Go to Configuration > Notifications > Configuration > Contacts.
  2. Click the Create button and enter the following contact information, (select the pencil icon to Edit an existing contact):

    Full NameFull name of the person to receive the notification. This is displayed in various screens as the contact name.
    UsernameUser name, displayed in the notification as a notification recipient.
    TimezoneSets the time zone of the contact. GroundWork Messenger makes use of the Perl DateTime::TimeZone to provide proper time zone support (including winter/summer time) and simplifies worldwide support for working hours , (e.g., America/Los_Angeles).
    TimeframeSets the contacts availability to receive notifications, (e.g., 24x7).
    Contact addressesSets the contact's information for notifications: Email, Phone, Mobile.
    Holiday(Available to existing contacts) Indicates a period of time a contact should not receive notifications. Selecting this option opens the Holidays tab and returns to the Contacts tab.
  3. Click Save.

Contact Groups

Contact groups are definitions of one or more contacts and can be used to send alert or recovery notifications to a group of contacts. Contact groups can be created for an area of expertise (e.g., network-administrators) or geographic location (e.g., san_francisco-support). When a host or service has a problem or recovers, GroundWork Messenger, as configured, will find the appropriate contact groups to send notifications to, and notify all contacts. These Groups are applied to rules in the Notifications tab.

Deleting Contacts Groups associated with Escalations

If you need to delete a contact group and it is associated with an escalation, you need to first delete the escalation, then delete the contact group.

  1. Go to Configuration > Notifications > Configuration > Contacts Groups.
  2. Click the Create button and enter the following information, (select the pencil icon to Edit an existing contact group):

    Name (short)Name of the contact group.
    Name (long)Descriptive name.
    TimezoneSets the time zone of the contact group, relative to time periods and holidays. 
    GroundWork Messenger uses DateTime::TimeZone to provide proper timezone support (including winter/summer time) and simplify worldwide support for working hours , (e.g., America/Los_Angeles).
    Notification hours (Timeframe)Sets the times contacts in the group are available to receive notifications, (e.g., 24x7).
    Do not send notifications (to members)If checked, contact group is a view-only group and notifications are not sent. The contact group is ignored while evaluating notification rules in response to incoming alerts.
    If unchecked, notifications are sent to the contact group per notification rules.
    MembersSets the contacts to be included in the contact group, the individuals that will be notified.
  3. Click Create.

Timeframes

Known Issue for Installation Prior to 01/12/2022

GroundWork Messenger notification system includes the capability of limiting notifications to a certain time frame. The most common, and default time frame is 24/7, which has a default expiration date of 12/31/2021. This configuration is adjusted in Configuration > Notifications > Configuration > Time Frames. The Validity Information section for the Valid To field needs to be adjust to a later date e.g., 12/31/2099. This configuration change may result in queued notifications being sent. Note, for an installation prior to 01/12/2022 refer to the documentation GroundWork Messenger Timeframes Remediation. For a download and installation as of 01/12/2022, this issue has been fixed in the latest release 8.2.1-1.

Timeframes are used to set a notification window for the Notification itself, for Contacts, Contact Groups, and Holidays. There are 3 default timeframes available for 24x7, outside work hours, and work hours.

  1. Go to Configuration > Notifications > Configuration > Timeframes.
  2. Click the Create button and enter the following information (select the pencil icon to Edit an existing timeframe):

    NameName of time frame.
    Schedule InformationSets time frame including day of week, from and to times with invert option (outside of from and to times), and choice of monthly occurrence.
    Validity InformationValid dates and time of time frame.
  3. Click Create/Save.

Appendices

Appendix A: Message Type Macro Definitions

MacroDescription

AUTHOR

Username of the author of the most recent comment or acknowledgment

DOWNTIMESTATUS

Start and end

COMMENTS

Comments on the host

HOSTGROUPS

Name of the hostgroups for hosts

HOSTNAME

Name of the host

INCIDENTDURATION

Duration of the incident (mins/secs)

INCIDENTTZ

Date and time for incident in TZ (local)

INCIDENTUTC

Date and time for incident in Coordinated Universal Time (UTC)

INCIDENTUTCMS

Date and time for incident in Coordinated Universal Time (UTC), in milliseconds

MONITORINGOUTPUT

Status text from host and service checks

NOTIFICATIONRULE

Name of notification rule that generated the notification

NOTIFICATIONTIME

Time the notification was sent

PRIORSTATUS

State before Current state

RESOURCETYPE

Type of host or service message (alert, recovery, acknowledgement, downtime)

STATUS

Current state

SERVICENAME

Name of host service

SERVICEGROUPS

Name of service groups for services

UPDATETZ

Date and time for last update in TZ (local)

UPDATEUTC

Date and time for last update in Coordinated Universal Time (UTC)

UPDATEUTCMS

Date and time for last update in Coordinated Universal Time (UTC), in milliseconds

Appendix B: HTML Email Templates

If you want to use HTML tags to format your notification Email messages, you can switch the format from Text to HTML to enable tag support in your templates. Doing so means you can add color, links and formatting to the emails you get from GroundWork Monitor. 

To get you started, here's a few example templates. To use them, just copy and paste them into your GroundWork Messenger templates and select the HTML format. Be sure to add in your GroundWork Monitor server name where you see the text:

{my gw8 hostname here}

Using these templates, your emails will look something like:

Following is a full set of 8 colorful HTML templates. Click each to expand the code to copy/paste:

Email Host Templates:
<html>
<table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #dc3545'>
        <b>GroundWork Host<br>Alert Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Incident start time:</td><td><b>[INCIDENTTZ]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
<table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #3bb658'>
        <b>GroundWork Host<br>Recovery Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Outage duration:</td><td><b>[INCIDENTDURATION]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #17a2bb'>
        <b>GroundWork Host<br>Acknowledgement</b><br>
        This issue has been acknowledged<br> by [AUTHOR]<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #C7C9CA'>
        <b>GroundWork Host<br>Downtime</b><br>
        This host's downtime status has changed<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime:</td><td><b>[DOWNTIMESTATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
Email Service Templates:
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #dc3545'>
        <b>GroundWork Service<br>Alert Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Incident start time:</td><td><b>[INCIDENTTZ]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #3bb658'>
        <b>GroundWork Service<br>Recovery Notification</b><br>
        This service has recovered from an outage. 
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Serivce:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Outage duration:</td><td><b>[INCIDENTDURATION]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #17a2bb'>
        <b>GroundWork Serivce<br>Acknowledgement</b><br>
        This issue has been acknowledged<br> by [AUTHOR]<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>[STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE
<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #C7C9CA'>
        <b>GroundWork Service<br>Downtime</b><br>
        This service's downtime status has changed<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime:</td><td><b>[DOWNTIMESTATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>
CODE

Appendix C: Building your Slack App

Here's how to build your Slack app for GroundWork notifications. You need to be a Slack administrator for your organization to authorize the app you will create for this. 

  1. Sign in to your Slack admin console at https://app.slack.com/apps-manage, you don't need to be admin for this part.
  2. Click Build, then Create New App.
  3. Give it a name and assign it to your workspace. Then click Create App. This will land you in the Add features and functionality section. 
  4. Click Permissions, then Scopes
  5. Add the scopes:
    • chat:write
    • chat:write.pubic
    • incoming-webhook

      to your Bot token:
  6. Click Install App on the left menu, then Install to Workspace
  7. Select the channel you want to post to. 
  8. Copy your bot user token to use in the GroundWork Messenger configuration.

Appendix D: cURL templates for PagerDuty

The cURL template can easily integrate with any API that will accept cURL POST. Here's how to use the cURL method to send notifications from GroundWork Messenger to PagerDuty.

  1. Sign in to your PagerDuty Instance.
  2. Click Services to navigate to the Service Directory.
  3. Select the service from the list which you want to receive notifications for by clicking the service name.
  4. Click the Integrations tab, then add the new integration.
  5. The next screen configures the integration:
    • Enter GroundWork Messenger in the Integration Name field (or another name if you prefer).
    • Select the radio button to Use our API directly and ensure the dropdown menu has Events API v2 selected.
    • Click Add Integration.
  6. Once the integration has been added, you are taken back to the integrations page, which shows the Integration Key, you'll need this key when implementing the cURL templates.

Now that you have a key, you can either develop your own customized cURL commands, or use the templates provided below. In these templates, just replace {my gw8 hostname here} with the hostname of your GroundWork Monitor 8 installation, and {my pagerduty routing key here} with your Integration Key.

You can also modify the templates to suit unique requirements you may have. It is recommended to test your cURL commands to ensure PagerDuty accepts your syntax, and this can be accomplished by using cURL on the Linux CLI. If you receive a cURL response of accepted, and your test generates a PagerDuty alert, then your syntax is good.

Following are host and service templates examples. Click each to expand the code to copy/paste:

cURL Host Templates:
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Alert: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Recovery: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Acknowledgement: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Scheduled Downtime event: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
cURL Service Templates:
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Alert: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Recovery: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Acknowledgement: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE
-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Downtime: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'
CODE

Troubleshooting

Ensuring Notification Rules Trigger Alerts

There may be cases in which connectivity or permission issues on the receiving side of an alert require deeper troubleshooting.

First, check if an alert was triggered. The GroundWork Messenger notification trigger events are logged in the Administration > Audit Log page. For ease of looking through the logs you may use the filter capability to filter the Subsystem to GORT, this is the GroundWork Optimized Request Transmitter subsystem. Log entries for triggered notifications look like this:

Triggered Alerts Not Received

If you do not have log entries with triggered notifications (as shown above), then notification rules in Configuration > Notifications > Configuration should be looked at. If the configuration is correct to trigger an alert, the notification will still be triggered and logged, including any communication failure with the (SMTP/cURL/SMS) endpoint. Basically, if it’s not in the logs, the filters are not correct.

If the alert is triggered but not received, debug logging can be turned on to get more verbose information on the problem. Debug logging is very verbose in the case of GroundWork Messenger, and it is not recommended to keep debug logging on for any length of time beyond troubleshooting sessions and obtaining logs, as it may be taxing on the system resources if left on.

To enable debug logging

  1. Stop GroundWork Monitor 8 from the gw8 directory:

    docker-compose down
    CODE
  2. Edit the file docker-compose.override.yml:

    vi docker-compose.override.yml
    CODE
  3. Add debug entries into the services section and save the file:

    gort:
        environment:
          - LOG_LEVEL=4
    CODE
  4. Start GroundWork Monitor 8:

    docker-compose up -d
    CODE
  5. To review the logs as they are written, enter:

    docker-compose logs -f gort
    CODE
  6. To copy the gort log, specifically to provide to support, enter:

    docker-compose logs gort > gort.log
    CODE
  7. If support requests all logs, run the following:

    docker-compose logs > all_logs.log
    CODE
  8. Once you have finished gathering logs and troubleshooting, comment out the debug logs previously added to docker-compose.override.yml:

    vi docker-compose.override.yml
    CODE
    # gort:
      #  environment:
       #  - LOG_LEVEL=4
    CODE
  9. Restart GroundWork Monitor 8:

    docker-compose up -d
    CODE

Related Resources