For an installation prior to 01/12/2022 refer to the documentation below. For a download and installation as of 01/12/2022, this issue has been fixed in version 8.2.1-1.

Summary

GroundWork is aware of an issue that may impact your ability to process notifications as a result of the default notification timeframes ending on New Year's Eve 2021. In GroundWork Monitor 8, no notifications tied to events occurring since then have been lost, they remain in the notification queue to be sent once the condition leading to them being unable to be sent (such as being outside of the allowed timeframe, in this case) has been resolved. If you normally receive many notifications, this may result in the notifications queued between 12/31/2021 23:59:59 and today being all sent at once after this configuration has been corrected.

The potential to receive queued notifications does not exist for GroundWork Monitor 7, as there is no queue system, you will only need to fix the timeframe issue.

This document covers altering the timeframes to allow notifications to trigger again, and also contains a way to discard messages queued between 12/31/2021 23:59:59 and now, if you wish to discard those events. If you have any questions please submit a support request.

Remediation Steps for GroundWork Monitor 8

For GroundWork Monitor 8, the combination of both conditions means no notification rules would apply, so all notifications from New Years should be in the Kafka queue. You may or may not want to get these queued notifications. There are 2 separate (but similar) remediation actions to take depending if you want to "throw away" the notification from New Years or not.

If you wish to receive all queued notifications

This will allow any queued notifications to be sent.

  1. Run the following from the command line to fix the time zone issue:

    cd /home/groundwork/gw8/
    docker-compose exec pg psql -d noma -c "update timeframes set dt_validto = '2099-01-01 00:00:00' where dt_validto = '2021-12-31 23:59:59'"
    CODE
  2. Then restart GroundWork (this will log out all users from the UI):

    docker-compose down
    docker-compose up -d
    CODE

If you wish to discard all queued notifications

  1. In GroundWork Monitor, go to Configuration > Notifications > Configuration, to create a "catch-all" rule - meaning a rule where all host/hostgroup/service/servicegroups are wild-carded and all checkboxes under "Which events to notify" are checked.
    • The idea is that any possible notification will match to this rule.
    • The method for this rule should be Email, and the recipient can either be a black-hole Email address (example: nobody@example.com), or the GroundWork administrator's address - depending if the admin wants to be able to review these messages or not. 
    • Leave this rule active, but set all other GroundWork Messenger rules inactive.
  2. Now with only the "catch-all" rule active, run the same commands to fix the timeframes:

    cd /home/groundwork/gw8/
    docker-compose exec pg psql -d noma -c "update timeframes set dt_validto = '2099-01-01 00:00:00' where dt_validto = '2021-12-31 23:59:59'"
    CODE
  3. Then restart GroundWork (this will log out all users from the UI):

    docker-compose down
    docker-compose up -d
    CODE
  4. Once GroundWork Monitor 8 is back up and running, allow about 30 minutes for the Kafka queue to empty out, then re-enable all other GroundWork Messenger rules. The "catch-all" rule can be disabled or enabled afterward, at your preference.

Remediation Steps for GroundWork Monitor 7

  1. This SQL statement will adjust the NoMa timeframes:

    source /usr/local/groundwork/scripts/setenv.sh
    export PGPASSWORD=MyPostgresPassword
    psql -d noma -c "update timeframes set dt_validto = '2099-01-01 00:00:00' where dt_validto = '2021-12-31 23:59:59'"
    CODE
  2. Your alert notifications will not have been sent from 12/31/2021 23:59:59 on, however, you can view all monitoring events in the Event Console.

Related Resources