About GroundWork Messenger

GroundWork Messenger is a free standing notification and escalation subsystem for configuring system-wide alert notifications for all monitors regardless of the monitoring method used. By using the GroundWork Messenger front-end, hosts, host groups, services, and service groups can be assigned to notification rules. When GroundWork Messenger receives alerts, it searches for matching definitions in its database. If a matching definition is found, contacts and methods are determined and notifications will be sent.

GroundWork Messenger is powered by the GroundWork Optimized Request Transmitter (GORT) which works with GroundWork Foundation as updated by TCG Connectors. To have Nagios configured elements notify using the new GroundWork Messenger system powered by GORT does not require the use the notification commands previously use in the NoMa application. GroundWork Messenger listens to GroundWork Foundation. Everything that happens in Nagios shows up in Foundation, which allows GroundWork Messenger to get notifications on Nagios resources, as well as any resources monitored by any other source. So, the bottom line is if a state change occurs on Foundation and a notification rule is configured to pick it up, it will be notified upon.

Notifications and escalations using the Nagios method remain available for configuration and maintenance. This allows customers who have made investments in time or scripting for Nagios alerts to continue to use these methods. The states changes that result in Nagios notifications will also be sent to GroundWork Messenger, regardless of whether notification commands are used. The alert flow coming from monitoring sources such as Nagios and Cloud Hub (and any other source) is sent to the RESTful API for the Foundation database. In turn, GroundWork Messenger subscribes to alert messages via the REST API and sends any alerts it finds there that pass its filters. GroundWork Messenger can re-notify for continuing issues if desired, and can also delay the start of notifications, allowing you to suppress volatile services from generating too many notifications.

Converting from NoMa

GroundWork Messenger is largely the same as NoMa from the point of view of configuration of notification rules, and so the basic rules will be easy to convert. The big difference is in methods. If you have invested in custom method scripts, these will need to be re-done using the cURL method in GroundWork Messenger. If you are using email, you can easily enter your email server details and convert NoMa emails. Slack and SMS are now native methods as well, and on the whole you will find it far easier to configure notifications in GroundWork Messenger than in NoMa.

For assistance in converting any complex custom scripting from NoMa to GroundWork Messenger, please submit a ticket.

Configuring Notifications

The operations for GroundWork Messenger are simple. It uses notification rules which essentially define the who, what, when, and how of alert notifications. A rule is a set of directives which determine the hosts and services to be included or excluded in the notifying, the contacts and contact groups to receive notifications, the timezone and timeframe in which notifications should be sent, the specific events to be notified upon, and the method(s) to be used to distribute notifications. Notification rules can be enabled (activated) and disabled, and rules can be set to escalate after a specified number of notifications is reached.

We start with creating notification methods that are then applied in a notification rule.

Creating Notification Methods

Notification Methods are included to direct how notifications are actually delivered. When creating methods you will need to configure the settings (e.g., Email method SMTP server) and also the actual notification templates that will be sent (e.g., Host Problems notification). Methods are configured by going to Configuration > Notifications > Notification Methods, and are applied within a notification rule in Configuration > Notifications > Configuration.

GroundWork Messenger currently supports four methods; cURL (most commonly used for ticketing integration), Email, ServiceNow, Slack and SMS. Referencing the image below, the default notification methods are initially named Curl Default, ServiceNow Default, Email Default, Slack Default and SMS Nexmo Default.

To configure the settings for a method, you'll click the corresponding icon. You will see that for each method you can Delete, Clone (copy), Save or Cancel its configuration, and each method has different directives. A green box indicates the method has been configured, a red box indicates it has not yet been configured.

To configure the templates for a method, you'll click the corresponding boxes under the Resource Type. A template is a place to create a message that will be sent by the method for Host and Service Problems, Recovery, Acknowledgment, and Downtime notifications. A green box indicates the notification template is enabled, a grey box means it is disabled. You can format the message as you like using text and macros, the macros available are different for each message type. See Appendix A: Message Type Macro Explanations.

Each of the notification methods details are outlined below.

cURL Method

The cURL notification method can be used to post results to an API. Most commonly the cURL method will be used to send notifications to ticketing systems, and provides a simple way to integrate any ticketing system which supports posting data via an API. In our example, we use the cURL method to send alerts to PagerDuty. The requirements include:

Access from the GroundWork Monitor server to https://events.pagerduty.com port 443
An API v2 integration added to your PagerDuty service
Access to the integration key

Documentation on adding an API integration can be found in the PagerDuty documentation.

To configure the cURL notification method, go to Configuration > Notifications > Notification Methods.
Click on the icon for Curl, enter a Display Name and click Save.

Then, continue by configuring the host and service notification templates. For example, selecting the resource type Host Problems and entering a Name and the Commands to Run a script for PagerDuty (displayed below). To use macros within a script, click the dropdown box, select a macro, click the icon and paste within the command.

Example cURL code for testing - just omit the "curl" command when pasting into the template:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' --header 'From: someone@gwos.com' -d '
{
  "payload": {
    "summary": "[HOSTNAME] [SERVICENAME]",
    "timestamp": "2015-07-17T08:42:58.315+0000",
    "source": "https://groundworkserver.gwos/",
    "severity": "info",
    "component": "postgres",
    "group": "Production",
    "class": "deploy",
    "custom_details": {
      "ping time": "1500ms",
      "load avg": 0.75
    }
  },
  "routing_key": "1234567890987654321abcdefghgfedcba",
  "links": [
    {
      "href": "https://example.com/",
      "text": "Link text"
    }
  ],
  "event_action": "trigger",
  "client": "Sample Monitoring Service",
  "client_url": "https://monitoring.example.com"
}
' 'https://events.pagerduty.com/v2/enqueue'

You can continue with other templates, then you can apply the method within notification rules. Also see Appendix D: cURL templates for PagerDuty.

Special Characters in cURL Method

As this method directly uses the cURL command for its functionality, all the restrictions of the cURL command still apply, such as the use of special characters: `,$,#, etc. In the event that macros are in use for services which return special characters in their output, such as special characters in the [MONITORINGOUTPUT] macro, such characters may cause the curl command itself to fail, in which case it is a good practice to escape the text in any macros utilized. For example:

-X POST -H "Content-Type: application/json" --user user:pw 
-d "{\"NAGIOS_CONTACTGROUPNAME\": \"CONTACT-GW\",\"NAGIOS_HOSTDISPLAYNAME\": \"[HOSTNAME]\",\"NAGIOS_HOSTNAME\": \"[HOSTNAME]\",
\"NAGIOS_HOSTOUTPUT\": [MONITORINGOUTPUT]\",\"NAGIOS_HOSTSTATE\": \"[STATUS]\",\"NAGIOS_LASTHOSTSTATECHANGE\": \"[PRIORSTATUS]\",\"NAGIOS_LASTSERVICESTATECHANGE\": \"[PRIORSTATUS]\",\"NAGIOS_NOTIFICATIONAUTHOR\": \"GroundWork\",\"NAGIOS_NOTIFICATIONCOMMENT\": \"[COMMENTS]\",\"NAGIOS_NOTIFICATIONTYPE\": \"[RESOURCETYPE]\",\"NAGIOS_SERVICEDESC\": \"[SERVICENAME]\",\"NAGIOS_SERVICEOUTPUT\": \"[MONITORINGOUTPUT]\",\"NAGIOS_SERVICESTATE\": \"[STATUS]\",\"NAGIOS_TIMET\": \"[INCIDENTUTC]\"}" 
https://ticketsystem.gwos/api/openticket

Email Method

Emails are sent using any SMTP server. If you run a Mail Relay, you may need to configure it to accept mail from the system. This method can also support a Mail Relay with just a Display Name and SMTP server indicated. GroundWork Messenger also allows you to send email notifications with modern email services such as Gmail or Office 365 as a TLS compliant client, though you should make sure you are aware of rate limits such services may impose.

The entries in this UI replace the gw8\gw8.env configuration file that were used with NoMa in prior versions of GroundWork Monitor, and are no longer needed.

Here's how to set it up:

As indicated, click the red icon to bring up the configuration screen. You can use Clone to make a copy of a method.

Enter the fields in the Outgoing Mail Server screen shown and described in the image below.

Display Name	Name for the new method.
SMTP Server	Name of your SMTP server. Note: When using Gmail the server is smtp.gmail.com.
User	A service user email account to authenticate and send email on the service (e.g., Gmail account). For SMTP servers that require you to log in, the User name is typically an Email address, and may be used to override what you put in the Sender Address field.
Password	An app password for the user account, separate from the normal email client password. For the Password, you usually need to generate an app password for this from your Email service account settings, e.g., App Password for Google and App Password for Office 365.
Sender Address	A Sender Address is required, and can be whatever Email address you want to have as the sender on your notifications.
Sender Name	The Sender Name can be customized and will default to GroundWork Messenger if you leave this field blank.
Port	An outbound network access on port 587 or similar port (usually open if using one of these hosted email services). Note: When using Gmail the port is 587.
Secure	Check box if using TLS. Note: Usually, this will be checked if using port 465. If using 587, this most likely should be left unchecked.
Don't check the certificate	Check this box to bypass checking of the certificate when connecting. You will want to check this box if the server is internal and it is setup with SSL, as it may fail if certificate is checked and fails validation.

Next, select each of the corresponding Host and Service templates to create a unique message to be sent out for each of the notification types. See Appendix B: HTML Email Templates for some examples. For example, click the Problems template under the Host resource type for your current Email method:

then fill in the following fields:

Name	Name for the new template.
Format	Select Text or HTML as the message format. HTML allows for the use of tags.
Address To	This will be either CONTACTEMAIL or CONTACTEMAIL2, configured in notification contacts.
Subject Line	Enter text (and optionally macros) to be displayed as the subject line of the notification.
Macros	Select Macros if you want to use variable values in your email subject and message. You can copy and paste the macros from the dropdown, first select the macro, click the copy icon, and then paste the macro into the template wherever you want the value to appear. Alternatively, you can just type in the macro name in square brackets. See the definitions of the macros in Appendix A: Message Type Macro Definitions.
Message	Enter the text (and optionally macros) to be sent as the notification message.
Enabled	Make sure to check the Enabled box to allow your message to be sent out.

Click Save. This Email notification method will send out this notification message for the resource type Host Problems. You can continue to define the rest of the templates, each are a little different. When complete, you can apply this method in notification rules.

Setting up failover SMTP server

Setting up the failover SMTP server helps, if the Outgoing (primary) mail server fails, then the Failover (secondary) mail server can be utilized to send emails. There is no timeout to check the primary server’s connection failure, this means that as soon as the SMTP primary server fails to accept a mail method alert request, the secondary mail server is used. There can be various reasons that count as failures, such as:

When the primary server is not reachable
The primary server is shut down
An Invalid mail server is specified

The failover method supports sticky or timed sticky failover types. In either case, the failover mail server is only used if the primary server becomes unavailable.

When the Sticky type is enabled, email requests will be routed to the secondary mail server which will remain as the destination server even if the primary mail server returns to service. The system will only revert to the primary mail server if the secondary becomes unavailable.

When the Timed Sticky type is enabled sessions will route requests back to the primary server after a set period of time. For example, consider that you have set the duration to 10 minutes. Now, if the primary server fails, the requests will be routed to the secondary server, and the secondary server will be used until the 10-minute timeout is reached. After 10 minutes, GWOS will check if the primary SMTP server is available; if it is, notifications will return to being sent through the primary mail server.

GWOS checks if the primary SMTP server is available after a certain duration (as set for timed sticky) only when the secondary server is active and sending emails.

To configure the failover SMTP server:

Navigate to the following Configuration --> Notification --> Notification Methods from the menu option.
Select Email Default from the list of Notification Methods.
Configure and test the primary SMTP server as normal in the Outgoing Mail Server dialog.
Select the Use failover SMTP server checkbox.
Select the Define failover server.

Enter the following in Failover Mail Server dialog, and select Save or Test (which also saves):

Field Name	Description
SMTP Server	Name of your failover SMTP server. For example: If you want to configure Gmail the SMTP server would be smtp.gmail.com
User	A service user email account to authenticate and send email on the service (e.g., Gmail account, in case Gmail account is being configured). For SMTP servers that require you to log in, the User name is typically an Email address and may be used to override what you put in the Sender Address field.
Password	An app password for the user account, separate from the normal email client password. For the Password, you usually need to generate an app password for this from your Email service account settings. For example, App Password for Google and App Password for Office 365
Sender Address	A Sender Address is required and can be any Email address you want to have as the sender on your notifications. This email address should be a valid one.
Sender Name	The Sender Name can be customized and will default to GroundWork Messenger if the field is left blank.
Failover type	Select one of the following to control the behaviour if a primary server fails: Sticky: Requests are routed to the failover server until a failure of the failover server or software restart occurs. Timed Sticky: Requests are routed to the primary server after a specific duration (default: 10 min) and revert to the use of the primary server if it has returned to service. Currently you cannot set the duration from the user interface. However, you can set the duration in the docker-compose.yml file.
Port	An outbound network access on TCP port 587 or a similar port (usually port 587 for TLS and port 465 for SSL is open if using one of these hosted email services). For example - when configuring Gmail SMTP server, the port is 587.
Secure	Check the box if using TLS. Usually, this will be checked if using port 465. If using 587, this most likely should be left unchecked.
Don’t check the certificate	Check this box to bypass checking the SSL certificate when connecting. You will want to check this box if the server is internal and it is set up with SSL, as it may fail if the certificate is checked and fails validation.

Note that internal SMTP mail server may have enterprise policies that mandate the construction of sender addresses, sender names or require configuration on the SMTP server side to allow for acceptance and/or forwarding of mail. Coordinate with your enterprise mail server administrator team to verify and configure.

Configuring duration to route request to primary server

To configure the Timed sticky duration, you must add the duration in docker-compose.yml file under gort parameter.

Below is an example for reference.

gort:

	image: groundworkdevelopment/gort:${TAG}

	# ports:

	# - "3002:3002"

	  env_file:

	  - gw8.env

	environment:

		- SMTP_STICKY_DURATION_MINUTES=1

CODE

Once you update the duration in the .yml file, run the following command to reflect these changes.

docker-compose up -d gort

ServiceNow Method

ServiceNow is a modern third-party cloud platform for ticketing and workflow automation. GroundWork's ServiceNow method interacts directly with the API of a running ServiceNow instance. To fully utilize this method, some setup will be required by an administrator of the ServiceNow instance.

On this ServiceNow instance, it is recommended to first gather:

A set of credentials for the instance to be used by this notification method
A GroundWork-specific ServiceNow User to be tagged as the caller in the table
An Assignment Group that will be associated with GroundWork ServiceNow notifications

If properly configured, GroundWork will be able to populate ServiceNow tables – such as the Incidents table – with details on host and service status changes. Here is an example of such an integration:

Configuring GroundWork for ServiceNow Notifications

To configure the ServiceNow notification method, go to Configuration > Notifications > Notification Methods.

Enter the fields in the ServiceNow configuration screen shown and described in the image below.

Display Name	Name for the new method.
URL	URL of your ServiceNow instance. If using a developer instance for staging, this will be in the form of https://devXXXXXX.service-now.com/api/now/table/incident
User Name	An instance user with access to the ServiceNow API. It is possible – though not recommended in a production environment – to use the admin user.
Password	The instance password corresponding to the previous field
Caller ID	The User ID of a ServiceNow user. Please note that this may not be displayed by default in ServiceNow dashboards which will display corresponding First Name and Last Name fields.
Category	A string corresponding to an existing Category definition. These should be customized on the instance, as any unmatched Category messages will fall back to a default Category.
Assignment Group	A pre-configured Group on the ServiceNow instance. While this can be arbitrary, linking it to an existing Group is likely preferred.
Description, Short Description	Only for testing. Use these temporary fields to send an example message to configured API endpoint.

Next, select each of the corresponding Host and Service templates to create a unique message to be sent out for each of the notification types.

Be sure to give each method a name, and click Enabled.
Click Save. This ServiceNow notification method will send out this notification message for the corresponding resource types. Any templates which are expected to trigger this method should be filled out.

Processing Tickets

If correctly configured, you should begin to see this method creating tickets in the Incidents table of your ServiceNow instance.

The Number field in the ServiceNow table is what GroundWork uses to track tickets internally. There are two areas where these tickets are visible within Groundwork:

It is possible to track creation of these tickets in the Audit Log
Any Host or Service with associated tickets will display a Tickets tab in the Status Dashboard when focused on applicable Host or Service

The Tickets tab:

By clicking on the Edit button in the row of the corresponding ticket, a user is able to change the status of a ticket between Open and Closed:

Slack Method

This method uses the Slack application and channel to send notifications. You will need:

A channel to post to in your Slack organization
Your Slack App and Token (see Appendix C: Building your Slack App)
Admin access to Slack (for approval)

After your app is installed, and you have your token and channel, you can proceed with the steps below.

Enter the following fields shown in the image and described below.

Display Name Name for the new configuration.
OAuth Token Access token for Slack authorization.
Channel Name Slack channel name.

Then for this method, select each of the corresponding Host and Service templates, e.g., Host Problems, and create a message to be sent out for the notification. For example, click the Problems box under the Host resource type for your current Slack method, and enter the following fields:

Name	Name for the new template.
Macros	Select Macros if you want to use variable values in your message. You can copy and paste the macros from the dropdown, first select the macro, click the copy icon, and then paste the macro into the template wherever you want the value to appear. Alternatively, you can just type in the macro name in square brackets. See the definitions of the macros in Appendix A: Message Type Macro Definitions.
Template	Enter the text (and optionally macros) to be sent as the notification Slack message.
Enabled	Make sure to check the Enabled box to allow your message to be sent out.

Click Save. This Slack notification method will send out message for the resource type Host Problems. You can continue to define the rest of the templates, each are a little different. When complete, you can apply this method in notification rules.

SMS Method

GroundWork Monitor can be used with Vonage (formerly Nexmo) for SMS alerts. Vonage is an international SMS provider, and has a very affordable and robust set of SMS and voice services. This method uses the Vonage SMS API, so you will need an account. Requirements vary, but you will want to figure out what kind of account and number you will need. To configure you will need a Vonage SMS account with the API credentials and sending number. The easiest (and least expensive) in the USA is to use a toll-free number. You will want to choose the options for Developer when you sign up, and then once you log in you can choose SMS, and click the button to get an API key.

This "master" key and secret will show up under https://dashboard.nexmo.com/, and if you are in the USA, you will get a number assigned to you under https://dashboard.nexmo.com/your-numbers If you need to buy a number, you can. In the USA with a toll-free number, you should (as of this writing) not need to register a brand and campaign, as long as the volume you send is low.

To configure the SMS notification method, go to Configuration > Notifications > Notification Methods.
Simply fill out the form with the API key, API secret, and sending number of your account. Destination Number is only used for the SMS test button, and will not have any effect on event-triggered notifications.
Continue to configure the templates as above, paying attention to the limit of 160 characters, and taking into account that some macro values such as MONITORINGOUTPUT may be longer than this.

SMS messages are charged to your Vonage account. GroundWork doesn't resell or broker Vonage accounts, so your use of it is your choice. Any charges you incur using this method with GroundWork Messenger are your own responsibility.
If you are interested in brokering SMS accounts for your users or GroundWork customers of your MSP business, you can create sub-keys and assign them to numbers and accounts. It's also possible to use voice services to do text-to-speech, etc. and other methods may be coming that integrate these services.

Creating a Notification Rule

Known Issue for Installation Prior to 01/12/2022

GroundWork Messenger notification system includes the capability of limiting notifications to a certain time frame. The most common, and default time frame is 24/7, which has a default expiration date of 12/31/2021. This configuration is adjusted in Configuration > Notifications > Configuration > Time Frames. The Validity Information section for the Valid To field needs to be adjust to a later date e.g., 12/31/2099. This configuration change may result in queued notifications being sent. Note, for an installation prior to 01/12/2022 refer to the documentation GroundWork Messenger Timeframes Remediation. For a download and installation as of 01/12/2022, this issue has been fixed in the latest release 8.2.1-1.

In this section, we cover creating a notification rule. A rule incorporates various notification components. If you don't yet have Holidays, Contacts, Contact Groups, Timeframes, and Notification Methods defined yet, you can go to these sections first. They all should be self-evident and intuitive.

Go to Configuration > Notifications > Configuration.
Within the Notifications tab, click Create, (select the pencil icon to Edit an existing notification rule).

Starting with the top section, Notification Information, enter the following fields shown in the image and described in the table below.

Name	Identifies the notification in the list of notifications.
Description	(Optional) Descriptive text displayed in the list of notifications.
Enabled	If this directive is checked, the notification rule is enabled (activated).
Owner	Identifies the contact designated as the owner of a rule. An administrator can edit all rules.
Timezone	The timezone in which notifications should be sent. A list of timezones is presented in the dropdown list.
Timeframe	The Timeframe directive indicates the hours allowed for notifications to be sent. Timeframes displayed in a Notification rule have to be defined in the Timeframes tab.

In the section Hosts and Services, you must indicate which hosts, host groups, services, and service groups are to be included or excluded in a notification rule. You will need to know the name of your hosts and services to include and exclude in this rule, as there's no auto-fill or results list in GroundWork Messenger. The preview pane on the right hand side of the dialog will show which hosts and services match a given pattern set.

Hostgroups	Indicate the name of a monitored hostgroup, or hostgroups separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?)can also be used.
Hosts	Indicate the name of a monitored host, or hosts separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used.
Servicegroups	Indicate the name of a monitored servicegroup, or servicegroups separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used.
Services	Indicate the name of a monitored service, or services separated by a comma, to be included or excluded in this notification rule. Wildcards (*,?) can also be used.

You can use wildcards in includes and excludes:

The " * " wildcard character indicates "all" possibilities, (e.g., all hosts). Any include fields left empty are assigned an implicit "NULL" and will automatically not match anything. It's safest to put a "*" here.
In specifying single or comma-separated lists of hosts, hostgroups, services, or servicegroups for inclusion or exclusion filtering, the " * " can be used as a multi-character wildcard and " ? " can be used as a single-character wildcard, within any particular item.

For example, to illustrate how a wildcard can be used in any location of a filter string: Perhaps you have a tiered subnet structure, like a.canada, b.canada, c.canada. In that case, you can use something like: *.canada.1* for the Hosts Includes field. This would filter and include hosts to be notified upon that start with any multi-characters, continue with a "dot" canada, and end with "dot" 1, followed by anything".

An applied include constraint will only allow the notification rule to pass if the host or service belongs to at least one of the listed hostgroups or servicegroups. Conversely, an applied exclude constraint will only allow the notification rule to pass if the host or service does not belong to any of the listed hostgroups or servicegroups.

Next is the section called Which events to notify. Here you need to select (check or uncheck) the filter options for recovery, problem, and state transitions and other states of change for which to send notifications.

This is a subtle and powerful selection set. The key to understanding this set of choices is that the Problems section relates to a state change from Up (for hosts) or OK (for services) to one of the problem states. State transitions between problem states are handled in the State Transitions section, and Recovery is for when the resource transitions back to UP or OK.

This, if you want to get notified when something goes Critical from OK, AND from Warning, you should check Critical in the Problems section and Warning to Critical in the State Transitions section. You can be very detailed here, but if you want to get notified on everything, leave all the boxes checked.

Note also that the Methods section allows you to configure different message formats for Problems, Recovery, Downtime and Acknowledgement notifications.

Problems	If checked, rule will notify upon a Warning, Unknown, Critical, Host Unreachable, and or Host Down events, or upon events that have transitioned state from Warning to Critical, Warning to Unknown, Unknown to Critical, Unknown to Warning, Critical to Warning, and or Critical to Unknown.
Recovery	If checked, rule will notify upon a Service OK event and or a Host UP event.
Other	If checked, rule will notify upon if a Downtime started and/or a Downtime ended, and upon an Acknowledgement of an event.
Numbers and more	Set to Notify after (a delay) a specified number of minutes, thus waiting the entered amount of time before notifications are sent. This is helpful for rapid state changes that may recover and eliminate the need to be notified. You can also set to continuously Renotify a number of minutes after the first notification. This is usually used alongside the Acknowledgement feature.

The default Renotify after value of 0 will make GroundWork Messenger send only one notification per state change (this is what we recommend).

Continuing with the notification rule, the Contacts and methods to notify section is used to define who is to receive notifications (Contacts, Contact Groups), how the notification will be distributed (Methods), and the when the notifications should be sent (Timezone and Timeframe).

Select Contacts and/or Groups to notify, and select at least one Method. The result will be that the contacts you select will get notified by the methods you select. You can define several rules for the same resources to create different timeframes for texts vs. emails, etc. You will need to set the timezone and timeframe for the entire rule. The contacts themselves may not be notified if their timeframes or holiday schedules do not include when the notification is generated.

Contacts

Contacts are individual definitions indicating who should get notified and how and when they should get notified in the event of a problem on your network. Contacts displayed in a Notification rule have been previously defined in the Contacts tab.

Contact Groups

Contact groups are definitions of one or more contacts and can be used to send alert or recovery notifications to a group of contacts. Contact Groups displayed in a Notification rule have been previously defined in the Contact Groups tab.

Methods

Notification Methods are definitions of how notifications are delivered. Methods are defined by going to Configuration > Notifications > Notification Methods.

Note that some methods require a corresponding contact to function, such as SMS and Email. Others will broadcast without the need for contact info, such as Curl, or Slack.

At this point, you can click Create to define the rule.
If you want GroundWork Messenger to send notifications using the same filter rule to alternate contacts for ongoing issues, you can add Escalations. To enable them, you must first Create the rule (click Create, then click the pencil icon on the rule in the list).

- Then, select add Escalation toward the bottom of the screen, and set the Contacts and or Groups to be notified, and the Methods of communication for the escalation.
- As in the first notification, in an escalation the Notify after number indicates when this notification should be escalated. Click Create, adding additional escalations or click Save again to save the notification rule.
  
  If you need to delete a contact group and it is associated with an escalation, you need to first delete the escalation, then delete the contact group.
  
  You may want to create multiple Notification rules instead of using Escalations. Using multiple rules may provide an easier work flow in that you can view and manage recurring notifications from the rules list, where Escalations are hidden within each rule.

Creating Notification Components

These components are defined in their respective tabs and applied within the Notifications tab for use in a notification rule. Note the last tab, Logs, is only used to see status of the installation, however to view actual notifications coming in see the Audit Log where filters are available.

Holidays

Holiday definitions indicate a period of time a contact should not receive notifications. The Holiday option is only displayed for previously defined contacts. Holidays are applied in the Contacts tab. You can always come back to this to create and then apply to a contact.

Go to Configuration > Notifications > Configuration > Holidays.

Click the Create button and enter the following information, (select the pencil icon to Edit an existing holiday):

Name	Name for holiday.
Start	Start day and time.
End	End day and time.
Timeframe	Sets time frame in which holiday is effective.
Contact	Sets defined contact associated with the holiday adding holiday to Contact's definition.

Click Create.

Contacts

Contacts are individual definitions indicating who should get notified and how and when they should get notified in the event of a problem on your network. Contacts can then be applied within Notifications, Contact Groups and Holidays tabs.

Go to Configuration > Notifications > Configuration > Contacts.

Click the Create button and enter the following contact information, (select the pencil icon to Edit an existing contact):

Full Name	Full name of the person to receive the notification. This is displayed in various screens as the contact name.
Username	User name, displayed in the notification as a notification recipient.
Timezone	Sets the time zone of the contact. GroundWork Messenger makes use of the Perl DateTime::TimeZone to provide proper time zone support (including winter/summer time) and simplifies worldwide support for working hours , (e.g., America/Los_Angeles).
Timeframe	Sets the contacts availability to receive notifications, (e.g., 24x7).
Contact addresses	Sets the contact's information for notifications: Email, Phone, Mobile. Email will correspond to CONTACTEMAIL macro, and Email2 will correspond to CONTACTEMAIL2 macro when configuring templates.
Holiday	(Available to existing contacts) Indicates a period of time a contact should not receive notifications. Selecting this option opens the Holidays tab and returns to the Contacts tab.

Click Save.

Contact Groups

Contact groups are definitions of one or more contacts and can be used to send alert or recovery notifications to a group of contacts. Contact groups can be created for an area of expertise (e.g., network-administrators) or geographic location (e.g., san_francisco-support). When a host or service has a problem or recovers, GroundWork Messenger, as configured, will find the appropriate contact groups to send notifications to, and notify all contacts. These Groups are applied to rules in the Notifications tab.

Deleting Contacts Groups associated with Escalations

If you need to delete a contact group and it is associated with an escalation, you need to first delete the escalation, then delete the contact group.

Go to Configuration > Notifications > Configuration > Contacts Groups.

Click the Create button and enter the following information, (select the pencil icon to Edit an existing contact group):

Name (short)	Name of the contact group.
Name (long)	Descriptive name.
Timezone	Sets the time zone of the contact group, relative to time periods and holidays. GroundWork Messenger uses DateTime::TimeZone to provide proper timezone support (including winter/summer time) and simplify worldwide support for working hours , (e.g., America/Los_Angeles).
Notification hours (Timeframe)	Sets the times contacts in the group are available to receive notifications, (e.g., 24x7).
Do not send notifications (to members)	If checked, contact group is a view-only group and notifications are not sent. The contact group is ignored while evaluating notification rules in response to incoming alerts. If unchecked, notifications are sent to the contact group per notification rules.
Members	Sets the contacts to be included in the contact group, the individuals that will be notified.

Click Create.

Timeframes

Timeframes are used to set a notification window for the Notification itself, for Contacts, Contact Groups, and Holidays. There are 3 default timeframes available for 24x7, outside work hours, and work hours.

Go to Configuration > Notifications > Configuration > Timeframes.

Click the Create button and enter the following information (select the pencil icon to Edit an existing timeframe):

Name	Name of time frame.
Schedule Information	Sets time frame including day of week, from and to times with invert option (outside of from and to times), and choice of monthly occurrence.
Validity Information	Valid dates and time of time frame. Be sure to set this to a large value if the timeframe is to be in permanent use.

Click Create/Save.

Appendices

Appendix A: Message Type Macro Definitions

Macro	Description
AUTHOR	Username of the author of the most recent comment or acknowledgment
DOWNTIMESTATUS	Start and end
COMMENTS	Comments on the host
HOSTADDRESS	Address of the host, if configured
HOSTALIAS	Alias of host configured in Nagios
HOSTGROUPS	Name of the hostgroups for hosts
HOSTNAME	Name of the host
HOSTNOTES	Notes field configured in Nagios
ID	Notification unique ID, incomplete implementation
INCIDENTDURATION	Duration of the incident (mins/secs)
INCIDENTTZ	Date and time for incident in TZ (local)
INCIDENTUTC	Date and time for incident in Coordinated Universal Time (UTC)
INCIDENTUTCMS	Date and time for incident in Coordinated Universal Time (UTC), in milliseconds
MONITORINGOUTPUT	Status text from host and service checks
NOTIFICATIONRULE	Name of notification rule that generated the notification
NOTIFICATIONTIME	Time the notification was sent
PRIORSTATUS	State before Current state
RESOURCETYPE	Type of host or service message (alert, recovery, acknowledgement, downtime)
STATUS	Current state
SERVICENAME	Name of host service
SERVICEGROUPS	Name of service groups for services
UPDATETZ	Date and time for last update in TZ (local)
UPDATEUTC	Date and time for last update in Coordinated Universal Time (UTC)
UPDATEUTCMS	Date and time for last update in Coordinated Universal Time (UTC), in milliseconds
CONTACTEMAIL, CONTACTEMAIL2, CONTACTPHONE, CONTACTMOBILE, CONTACTUSERNAME, CONTACTFULLNAME, CONTACTTIMEFRAME, CONTACTTIMEZONE	Configured contact details, available only for non-broadcast notification methods, and available to cURL. This can also be used to configure per-template behavior in the email method.

Appendix B: HTML Email Templates

If you want to use HTML tags to format your notification Email messages, you can switch the format from Text to HTML to enable tag support in your templates. Doing so means you can add color, links and formatting to the emails you get from GroundWork Monitor.

To get you started, here's a few example templates. To use them, just copy and paste them into your GroundWork Messenger templates and select the HTML format. Be sure to add in your GroundWork Monitor server name where you see the text:

{my gw8 hostname here}

Using these templates, your emails will look something like:

Following is a full set of 8 colorful HTML templates. Click each to expand the code to copy/paste:

Email Host Templates:

Email: Host Alert

<html>
<table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #dc3545'>
        <b>GroundWork Host<br>Alert Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Incident start time:</td><td><b>[INCIDENTTZ]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Host Recovery

<html>
<table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #3bb658'>
        <b>GroundWork Host<br>Recovery Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Outage duration:</td><td><b>[INCIDENTDURATION]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Host Acknowledgement

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #17a2bb'>
        <b>GroundWork Host<br>Acknowledgement</b><br>
        This issue has been acknowledged<br> by [AUTHOR]<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Host Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Host Downtime

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #C7C9CA'>
        <b>GroundWork Host<br>Downtime</b><br>
        This host's downtime status has changed<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?hostName=[HOSTNAME]&link=best'>[HOSTNAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime:</td><td><b>[DOWNTIMESTATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email Service Templates:

Email: Service Alert

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #dc3545'>
        <b>GroundWork Service<br>Alert Notification</b>
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Incident start time:</td><td><b>[INCIDENTTZ]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Service Recovery

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #3bb658'>
        <b>GroundWork Service<br>Recovery Notification</b><br>
        This service has recovered from an outage. 
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Serivce:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>was: [PRIORSTATUS] now: [STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Monitoring Info:</td>
    <td><b>[MONITORINGOUTPUT]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Outage duration:</td><td><b>[INCIDENTDURATION]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Service Acknowledgement

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #17a2bb'>
        <b>GroundWork Serivce<br>Acknowledgement</b><br>
        This issue has been acknowledged<br> by [AUTHOR]<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service State:</td>
    <td style='background-color: #FEDBBE'><b>[STATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Email: Service Downtime

<html>
    <table width='auto' style='background-color: #FEDBBE; min-width: 350px'>
    <caption style='font-weight: bold; background-color: #C7C9CA'>
        <b>GroundWork Service<br>Downtime</b><br>
        This service's downtime status has changed<br>
        at [UPDATETZ]
    </caption>
<tr>
    <td style='background-color: #fdac69'>Host:</td><td><b>[HOSTNAME]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Service:</td>
    <td><b><a href='https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best'>[SERVICENAME]</a></b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime:</td><td><b>[DOWNTIMESTATUS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Downtime Comments (if any):</td><td><b>[COMMENTS]</b></td></tr>
<tr>
    <td style='background-color: #fdac69'>Triggered by rule:</td><td><b>[NOTIFICATIONRULE]</b></td></tr>
</table>
</html>

CODE

Appendix C: Building your Slack App

Here's how to build your Slack app for GroundWork notifications. You need to be a Slack administrator for your organization to authorize the app you will create for this.

Sign in to your Slack admin console at https://app.slack.com/apps-manage, you don't need to be admin for this part.
Click Build, then Create New App.
Give it a name and assign it to your workspace. Then click Create App. This will land you in the Add features and functionality section.
Click Permissions, then Scopes.
Add the scopes:
- chat:write
- chat:write.pubic
- incoming-webhook
  
  to your Bot token:
Click Install App on the left menu, then Install to Workspace.
Select the channel you want to post to.
Copy your bot user token to use in the GroundWork Messenger configuration.

Appendix D: cURL templates for PagerDuty

The cURL template can easily integrate with any API that will accept cURL POST. Here's how to use the cURL method to send notifications from GroundWork Messenger to PagerDuty.

Sign in to your PagerDuty Instance.
Click Services to navigate to the Service Directory.
Select the service from the list which you want to receive notifications for by clicking the service name.
Click the Integrations tab, then add the new integration.
The next screen configures the integration:
- Enter GroundWork Messenger in the Integration Name field (or another name if you prefer).
- Select the radio button to Use our API directly and ensure the dropdown menu has Events API v2 selected.
- Click Add Integration.
Once the integration has been added, you are taken back to the integrations page, which shows the Integration Key, you'll need this key when implementing the cURL templates.

Now that you have a key, you can either develop your own customized cURL commands, or use the templates provided below. In these templates, just replace {my gw8 hostname here} with the hostname of your GroundWork Monitor 8 installation, and {my pagerduty routing key here} with your Integration Key.

You can also modify the templates to suit unique requirements you may have. It is recommended to test your cURL commands to ensure PagerDuty accepts your syntax, and this can be accomplished by using cURL on the Linux CLI. If you receive a cURL response of accepted, and your test generates a PagerDuty alert, then your syntax is good.

Following are host and service templates examples. Click each to expand the code to copy/paste:

cURL Host Templates:

cURL: Host Alert

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Alert: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Host Recovery

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Recovery: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Host Acknowledgement

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Acknowledgement: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Host Downtime

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Host Scheduled Downtime event: [HOSTNAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?selectedEntityType=Host&hostName=[HOSTNAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL Service Templates:

cURL: Service Alert

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Alert: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Service Recovery

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Recovery: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Service Acknowledgement

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Acknowledgement: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

cURL: Service Downtime

-X POST --header 'Content-Type: application/json' --header 'Accept: application/vnd.pagerduty+json;version=2' -d '
{
  "payload": {
    "summary": "Service Downtime: [HOSTNAME]:[SERVICENAME] is [STATUS]",
    "source": "https://{my gw8 hostname here}/",
    "severity": "info",
    "component": "monitoring",
    "group": "prod-datapipe",
    "class": "deploy",
    "custom_details": {
      "Detail": "[MONITORINGOUTPUT]",
      "Comments": "[COMMENTS]",
      "Triggered by notification rule": "[NOTIFICATIONRULE] at [NOTIFICATIONTIME]"
    }
  },
  "routing_key": "{my pagerduty routing key here}",
  "event_action": "trigger",
  "client": "GroundWork Monitor 8",
  "client_url": "https://{my gw8 hostname here}/status?serviceName=[HOSTNAME]:[SERVICENAME]&link=best"
}
' 'https://events.pagerduty.com/v2/enqueue'

CODE

Troubleshooting

Time Zone Considerations

There are multiple places where an administrator can configure time zones for notifications in GroundWork, and unexpected results can occur if not configured correctly:
GroundWork system-wide (gw8.env)
Per-rule
Per-contact

GroundWork Messenger will prioritize these in a specific order, and fall back to system-wide timezone values if no overrides are found.
The order is: Per Contact → Per Rule → gw8.env

This configuration will be reflected in the messenger macros which reference time-zone-corrected timestamps.

Ensuring Notification Rules Trigger Alerts

There may be cases in which connectivity or permission issues on the receiving side of an alert require deeper troubleshooting.

First, check if an alert was triggered. The GroundWork Messenger notification trigger events are logged in the Administration > Audit Log page. For ease of looking through the logs you may use the filter capability to filter the Subsystem to GORT, this is the GroundWork Optimized Request Transmitter subsystem. Log entries for triggered notifications look like this:

Triggered Alerts Not Received

If you do not have log entries with triggered notifications (as shown above), then notification rules in Configuration > Notifications > Configuration should be looked at. If the configuration is correct to trigger an alert, the notification will still be triggered and logged, including any communication failure with the (SMTP/cURL/SMS) endpoint. Basically, if it’s not in the logs, the filters are not correct.

If the alert is triggered but not received, debug logging can be turned on to get more verbose information on the problem. Debug logging is very verbose in the case of GroundWork Messenger, and it is not recommended to keep debug logging on for any length of time beyond troubleshooting sessions and obtaining logs, as it may be taxing on the system resources if left on.

To enable debug logging

Stop GroundWork Monitor 8 from the gw8 directory:
```
docker-compose down
```
CODE
Edit the file docker-compose.override.yml:
```
vi docker-compose.override.yml
```
CODE
Add debug entries into the services section and save the file:
```
gort:
    environment:
      - LOG_LEVEL=4
```
CODE
Start GroundWork Monitor 8:
```
docker-compose up -d
```
CODE
To review the logs as they are written, enter:
```
docker-compose logs -f gort
```
CODE
To copy the gort log, specifically to provide to support, enter:
```
docker-compose logs gort > gort.log
```
CODE
If support requests all logs, run the following:
```
docker-compose logs > all_logs.log
```
CODE
Once you have finished gathering logs and troubleshooting, comment out the debug logs previously added to docker-compose.override.yml:
```
vi docker-compose.override.yml
```
CODE
```
# gort:
  #  environment:
   #  - LOG_LEVEL=4
```
CODE
Restart GroundWork Monitor 8:
```
docker-compose up -d
```
CODE

Display Name	Name for the new configuration.
OAuth Token	Access token for Slack authorization.
Channel Name	Slack channel name.

About GroundWork Messenger

Converting from NoMa

Configuring Notifications

Creating Notification Methods

cURL Method

Special Characters in cURL Method

Email Method

Setting up failover SMTP server

Configuring duration to route request to primary server

ServiceNow Method

Configuring GroundWork for ServiceNow Notifications

Processing Tickets

Slack Method

SMS Method

Creating a Notification Rule

Creating Notification Components

Holidays

Contacts

Contact Groups

Timeframes

Appendices

Appendix A: Message Type Macro Definitions

Appendix B: HTML Email Templates

Email Host Templates:

Email Service Templates:

Appendix C: Building your Slack App

Appendix D: cURL templates for PagerDuty

cURL Host Templates:

cURL Service Templates:

Troubleshooting

Time Zone Considerations

Ensuring Notification Rules Trigger Alerts

Triggered Alerts Not Received

To enable debug logging

Related Resources