About Nagios Notifications

Notifications and escalations are how the GroundWork Monitor Nagios engine alerts its users when monitoring services change between states (OK, WARNING, CRITICAL, and UNKNOWN). The following is a brief discussion on setting up notifications using Nagios in GroundWork Monitor.

Contact notifications are communications to contacts or contact groups about the status of a monitored element. Notifications can be configured for circumstances including any hard state change, if a host or service remains in a non-OK state, and for acknowledgments. When do notifications occur? Notifications occur when a hard state change occurs and all filters are passed, and when a host or service remains in a hard non-OK state and the time specified by the <notification_interval> option in the host or service definition has passed since the last notification was sent out (for a specified host or service). Who gets notified? Each host may belong to one or more host groups and each host group has a contact group option that specifies what contact groups receive notifications for hosts in that particular host group. Additionally, each service definition has a contact groups option that specifies what contact groups receive notifications for that particular service. What filters must be passed in order for notification to be sent? This includes program wide filters, host and service filters, and contact filters.

Notifications in GroundWork Monitor are communications made to contacts about the status of a monitored element. Notifications can be configured for circumstances including any hard state change, if a host or service remains in a non-OK state, and for acknowledgments. Contact groups are associated with escalation trees which are then used for host and service notifications. The image below shows the various notification objects. 

When creating contacts you'll want to use a contact template. A template typically stores generalized contact information which is consistent across multiple contacts definitions such as time periods, specific host and service states for which notifications can be sent out, and commands used to notify of a host or service problem or recovery. These templates aid in creating contacts by conveniently applying commonly used directives so if something changes you have the advantage of updating the templates which then changes associated contacts using that template. GroundWork does provide two generic templates. Specific contact information such as e-mail addresses and phone numbers would be contained in the contact definition described next. Contact definitions inherit generalized information from contact templates.

After you have completed defining a contact template, you'll want to continue with creating contacts. Contacts contain individual settings defining who should get notified in the event of a problem on your network. Contact definitions also indicate which notification options will be used for the contact based on inheritance or selected template directives.

A contact group is a grouping of contacts for the purpose of sending out alert/recovery notifications to one or more contacts. Contact definitions are grouped into contact groups typically by area of expertise or geographic location. For example you might have one contact group called network-administrators and perhaps another contact group called sanfrancisco-support. Then, when a host or service has a problem or recovers, Nagios will find the appropriate contact groups to send notifications to and notify all contacts in those contact groups. After creating contact groups, you will be ready to associate these groups with host and service templates, hosts, services, and host groups definitions.

A host group definition is used to group one or more hosts together for the purposes of simplifying notifications. When a host goes down, becomes unreachable, or recovers, Nagios will find which host group(s) the host is a member of, get the contact group for each of those host groups, and notify all Contacts associated with those contact groups. Host groups allow for flexibility in determining who gets paged for what kind of problems. Each host that you define must be a member of at least one host group - even if it is the only host in that group. Hosts can be in more than one host group.

A service group allows for flexibility in determining who gets notified for what kind of problems. Service groups allow you to group services together for display purposes and can be referenced in service dependency and service escalation definitions to make configuration a bit easier. The prerequisites for service groups are defined hosts with services and services assigned to hosts. 

There are two methods for assigning contact groups for notifications; the first being a direct contact group assignment through a host or service template to an object (e.g. host, service) or directly in a host group or service group definition; and the second an escalation tree assignment to an object (host, host group, or service).

An escalation, which is optional, combines specified contact groups that are to be notified when a notification is escalated. An escalation tree is a grouping of multiple escalations which are then assigned to a host, host profile, host group, or a service. A service escalation is used to escalate notifications for a particular service. A host escalation is used to escalate notifications for a particular host.

GroundWork Monitor does not assume that just because a notification has been sent that the underlying problem is under control. It requires the recipient of the page to log into the system and acknowledge having received a notification. If that acknowledgment does not occur within a period of time identified by the notification interval, subsequent notifications will be sent out. The monitoring System Administrator can configure how many notifications get sent at each escalation level before escalating the problem to a higher level of support. Notifications are escalated if one or more escalation definitions matches the current notification that is being sent out. If a host or service notification does not have any valid escalation definitions that applies to it, the contact group(s) specified in either the host group or service definition will be used for the notification.

Configuring Nagios Notifications

Create a Contact Template

1. Select Configuration > Nagios Monitoring > Contacts, select the Contact Templates option, and New to create a new template, or you can Copy or Modify an existing template.
2. In the Contact Template Properties screen, enter a required contact template Name.
3. Next, enter the various host and service contact template properties which will be used for all contacts that use this contact template:
   • Host notification period:  [required] This directive is used to specify the name of the time period during which the contact can be notified about host problems or recoveries.
   • Host notification options: [required] This directive is used to define the host states for which notifications can be sent out to this contact.
   • Host notification commands: [optional] This directive is used to define a list of names of the commands used to notify the contact of a host problem or recovery. All notification commands are executed when the contact needs to be notified.
   • Service notification period: [required] This directive is used to specify the name of the time period during which the contact can be notified about service problems or recoveries.
   • Service notification options: [required] This directive is used to define the service states for which notifications can be sent out to this contact.
   • Service notification commands: [optional] This directive is used to define a list of names of the commands used to notify the contact of a service problem or recovery. All notification commands are executed when the contact needs to be notified.

4. Contact templates also enable you to define and manage optional custom object variables. This section is used to define macros which are ultimately associated with contacts. These macros may be referenced when you invoke notification commands.

5. Select Add to save the template. You can now use this template when creating new contacts.
   

Create a Contact

1. Select Configuration > Nagios Monitoring > Contacts, Contacts, and New to create a new contact (or you can Copy or Modify an existing contact).
2. In the Contact Properties screen, if a new contact, enter a contact Name, and an Alias as a description used to identify the contact. Optionally enter an email address or pager number.
   • Name: [required] This directive is the name used to identify the contact.
   • Alias: [required] This directive is used to define the name or description for the contact. Under the right circumstances, the $CONTACTALIAS$ macro will contain this value.
   • Email: [optional] This directive is used to define an email address for the contact. Depending on how you configure your notification commands, it can be used to send out an alert email to the contact. Under the right circumstances, the $CONTACTEMAIL$ macro will contain this value.
   • Pager: [optional] This directive is used to define a pager number for the contact. It can also be an email address to a pager gateway (e.g., pagejoe@pagenet.com). Depending on how you configure your notification commands, it can be used to send out an alert page to the contact. Under the right circumstances, the $CONTACTPAGER$ macro will contain this value.
3. Next, enter a required Contact Template most suitable for this contact. By default, many of the selected template directives are checked to be used in the new contact definition. You can uncheck any of the Inherit boxes on the left to override the selected template values. Clicking the Set Full Inheritance button will recheck all values and enable the contact definition to inherit all values from the template.
4. You can adjust any custom object variable values that appear in the controlling contact template, and add new custom object variables that are specific to this one contact. Under the right circumstances, these will appear as special macros that you can refer to in notification command definitions. See the Nagios documentation for details.
5. Finally, you may optionally select Contact Groups which are used to identify the name(s) of the contact groups that the contact belongs to. This directive may be used as an alternative to (or in addition to) using the contacts directive in the contact groups definition.
6. When finished, select Save.
   

   

Create a Contact Group

1. Select Configuration > Nagios Monitoring > Contacts, Contact Groups, and New to create a new contact group, or you can Copy or Modify an existing definition.
2. In the Contact Group Properties screen, enter a contact group Name, and an Alias as a description used to identify the contact group.
3. Next, using the Add and Remove buttons, select which contacts are to be part of the contact group.

4. When finished with your choices, click Save.
   

5.  You can now associate these contact groups with contacts, host and service templates, hosts, services, and host groups definitions. Contact groups can be applied to escalation trees to be used in setting up notifications for hosts and services.

Create a Host Group

Host Groups are an arbitrary collection of hosts into named sets. The usage of host groups simplifies access control, status displays, notifications, scheduling maintenance, multi-server commands, and reports. You will receive a warning message during a Pre Flight Test or Commit if a host is not a member of any host group. A host can be assigned to one or more host groups.

Tips

• Copy an existing host group keeping a previous definition's details and just adding a host group name, or you can Modify an existing host group updating its members, contact groups, or escalation trees.

• Members are the hosts to be included in the host group.

• Contact Groups define who should be notified whenever there are problems (or recoveries) with any of the hosts in this host group.

• The Host escalation tree directive is optional and indicates all hosts in this host group receive the same host escalation tree. And the Service escalation tree selection is also optional and indicates all services on each host in this host group receive the same service escalation tree.

1. Select Configuration > Nagios Monitoring > Hosts, expand the Host Groups drop-down menu and select New.
2. In the Host Group Properties screen, enter a host group Name which should be unique and not the same as a contact group, and enter an Alias.
3. Next, optionally enter Notes. This field is intended to describe common features of the host group (e.g., the business unit, the geographical location, or the team responsible for maintenance). This information is shown on the host group detail page within the Status dashboard.
4. Select the Members (hosts) from the right side list to be included in this host group, and choose the Contact Groups that should be notified whenever there are problems (or recoveries) with any of the hosts in this host group. We will skip the optional host and service escalation ID directives for now.
5. Select Add to save the new host group.
   

   

Create a Service Group

Service Groups are used to group one or more services together for the purposes of simplifying notifications. When a service goes down, becomes unreachable, or recovers, Nagios will find which service group(s) the service is a member of, get the contact group for each of those service groups, and notify all contacts associated with those contact groups. Additionally, the usage of service groups simplifies access control, status displays, scheduling maintenance, multi-server commands, and reports. A service can be assigned to one or more service groups.

Tips

• You can Modify an existing host group updating its members, contact groups, or escalation trees.
• Members are the services to be included in the service group, you can add services using Add Service(s) or Add Host(s).
• The Service escalation tree selection is also optional and indicates all services on each host in this host group receive the same service escalation tree.

1. Select Configuration > Nagios Monitoring > Services, Service Groups, and New.
2. In the Service Group screen, enter the properties:
   • Name:  [Required] Name of the service group, (e.g., dbservices).
   • Alias: [Required] This directive is is used to define a longer name or description used to identify the service group. It is provided in order to allow you to more easily identify a particular service group, (e.g., Database Services).
   • Notes: [Optional] This filed is intended to describe common features of the service group (e.g., the usage, functions, or specific application software being monitored). This information is shown on the service group detail page within the Status application.
   • Host/Service: [Required] This is a list of the descriptions of services (and the names of their corresponding hosts) that should be included in this group.
   • Service Escalation: This is a list of defined service escalations that can be assigned to this service group.
     
3. Click Add to go to the Service Group screen.
4. Select the Host drop-down box to select a host. Available services will be listed.
5. Choose a service to add and click Add Service(s). The host and service columns above will displayed your selections. To remove a selected service click the X corresponding to the line to be deleted.
6. Select the Service drop-down box to select a service, available hosts will be listed.
7. Choose a host to add and click Add Host(s). The host and service columns above will display your selections. To remove a selected host click the X corresponding to the line to be deleted.
8. Select Save to save the service group. Delete removes the current service group, and Rename is used to change the name of the current service group. You will be notified on the next screen that the service groups has been updated. After setting up host and/or service groups you're ready to setup notifications and escalations.
   

Create an Escalation

Notifications and escalations are how the Nagios™ engine within GroundWork Monitor alerts its users when monitoring services change between states (OK, WARNING, CRITICAL, and UNKNOWN). Escalations combine specified contact groups that are to be notified when a notification is escalated. An escalation tree is a grouping of multiple escalations which are then assigned to a host, host profile, host group, or a service. Escalations are optional. Below, we cover creating escalations and then configuring escalation trees.

1. Select Configuration > Nagios Monitoring > Escalations, expand the Escalations drop-down menu, expand the Host (or Service) drop-down menu, then select New. You can also choose to Modify an existing host or service escalation.
2. In the Host (or Service) Escalation Properties screen, enter an escalation name.

3. Next, setup the notification options:

   • First Notification: [required] This directive is a number that identifies the first notification for which this escalation is effective. For instance, if you set this value to 3, this escalation will only be used if the host is down or unreachable long enough for a third notification to go out or if the service is in a non-OK state long enough for a third notification to go out.

   • Last Notification: [required] This directive is a number that identifies the last notification for which this escalation is effective. For instance, if you set this value to 5, this escalation will not be used if more than five notifications are sent out for the host (or service). Setting this value to 0 means to keep using this escalation entry forever (no matter how many notifications go out).

   • Notification Interval: [required] Indicates when a notification should be sent. This directive is used to determine the time interval, (unless you've changed the interval_length directive from the default value of 60, this number will mean minutes), at which notifications should be made while this escalation is valid. If you specify a value of 0 for the interval, Nagios will send the first notification only, and will then prevent any more problem notification s from being sent out for the host. Specifying any other value will send continuous notifications at the time interval specified.

     

     If multiple escalation entries for a host overlap for one or more notification ranges, the smallest notification interval from all escalation entries is used.
4. Continue with the escalation options, you can optionally enter an Escalation period which is used to specify the name (e.g., Work Hours, 24x7) of the time period during which this escalation is valid. If this directive is not specified, the escalation is considered to be valid during all times.
5. The Escalation options directive lets you optionally select the criteria that determines when an escalation is used (e.g., For a host: Recovery, Down, Unreachable; and for a service: Recovery, Warning, Unknown, Critical). The escalation is used only if the host (or service) is in one of the states specified in this directive. If this directive is not specified the escalation is considered to be valid during all host (or service) states.
   • Example (host escalation): If you specify Down in this field, the escalation will only be used if the host is in a DOWN state; (service escalation): If you specify warning in this field, the escalation will only be used if the service is in a WARNING state.
6. Select Add to confirm the the new escalation.
   

Create an Escalation Tree

1. Next, create an escalation tree. Select Configuration > Nagios Monitoring > Escalations, expand the Escalation Trees drop-down menu, expand the Host (or Service) drop-down menu, and select New, (or Modify an existing).
2. Enter a name for the host (or service) escalation tree, and select Add, the Managing Escalations screen will be displayed.
   

3. Next, in the Managing Escalations screen you can add/remove defined escalations. After selecting each escalation the Assign Contact Groups screen will be displayed where you can assign contact groups to be associated with the escalation. These are groups of contacts that will be notified when the host or service notification is escalated. You can build up a number of escalations to accomplish elaborate notification schemes.
   
   For example, a Technical contact group could be assigned an escalation that sends out the first through the last notifications. Whereas a different escalation might start with the 5th notification and end with the 6th notification and go to a Senior Manager that might want to be notified only when a alarm hasn't been acknowledged within the 4 previous notifications.
   
   Select a defined escalation and click Add Escalation, you are adding an escalation to the escalation tree.
   

4. Next, << Add the contact groups to be associated with the escalation, and Save.
   
5. Next, you will be in the Detail tab where you can add additional escalations or modify contact groups. Next we'll assign host groups to set the default host escalation.
   
   

   You can now assign host groups which will set the default host escalation for all hosts in a host group. You also have the option to set a host escalation for an individual host.

6. Select the Assign Host Groups tab, then << Add the host groups to assign to the escalation tree.
7. Select Save and Close.
   

Apply an Escalation Tree

When creating or modifying Escalation Trees you have the option to assign host groups which sets the default host escalation for all hosts in the host group. And you can assign hosts which sets the host escalation for the host. Additionally, you can add an escalation tree to defined host groups and hosts.

To apply to a host:

1. Go to Configuration > Nagios Monitoring > Hosts > Hosts, and navigate to the host you want to edit.
2. Select Detail, and the Manage Host screen will be displayed.

3. Select the Escalation Trees tab, and select an Escalation Tree from the drop-down box appropriate for this host.

   

   To avoid amplified notifications (e.g., multiple notifications for the same event), a host escalation assigned to this host should not also be assigned to a host group in which the host is a member.

4. Choose an Escalation Tree from the drop-down box appropriate for all services on this host.

   

   When a service escalation is assigned here, all services on this host will use the same escalation. To use different escalations for different services, each service must have its own escalation. In that case do not assign a service escalation at this point.

5. Select Save.
   

To apply to a host group:

1. Go to Configuration > Nagios Monitoring > Hosts > Host Groups.
2. Click Modify to get a listing of defined host groups, and select a Host Group. The Host Properties screen will be displayed.

3. Next, select a Host Escalation Tree from the drop-down box applicable to the host group.

   

   All hosts in the host group will receive the same host escalation.

4. Select a Service Escalation Tree from the drop-down box applicable to this host group.

   

   All services on each host in the host group receive the same host escalation.
5. Click Save.
   

Enable Nagios Notifications

Notifications for Nagios monitoring are set to OFF system-wide by default. This means that host and service notifications will not be sent out. After enabling notifications, host and service notifications will be sent to any valid system contacts. In addition to the system-wide default setting for notifications, the Enable Notifications option can be set for all hosts and services assigned to a specific template by setting this option in a host or service template. Also, the Enable Notifications option can be set for individual host or service definitions by changing the definitions template inheritance. Below, we outline how to enable notifications. After enabling notifications, continue setting up system notifications and escalations by clicking the home icon at the top of this page and referring to links under Setting up Host and Service Notifications.

To enable system wide notifications:

1. Select Configuration > Nagios Monitoring > Control > Nagios main configuration, and click Notification.
2. Select the box next to Enable Notifications to enter a check which indicates notifications are on. At the bottom of the page, click Save and Next to save this change.
3. You'll need to commit this change to your Nagios configuration, select Control again.
4. Select Backup if necessary, and then Commit to overwrite the active Nagios configuration and restart Nagios. 
   

Related Resources

• Page:
  GroundWork Messenger