What is an Anonymized Database?

Sometimes a GroundWork customer may run into a production problem where the nature of the failure suggests it has to do with the structure of the data in the configuration database, or how that structure interacts with downstream processing. To diagnose the problem, GroundWork will often request a copy of the monarch database, so we can replicate the customer setup in-house and run extensive testing on it. 

However, the customer may be reluctant to provide that data, inasmuch as it may contain a variety of confidential items (hostnames, IP addresses, passwords, and so forth). To solve that problem, we have developed a small set of scripts that can create a copy of the monarch database wherein all the potentially sensitive data has been masked.  For instance:

  • Hostnames are all converted to the generic form host-123, where the host number reflects the host ID already contained in the database.  That allows GroundWork to relate back to the customer which objects have a configuration problem, without us knowing the exact names involved.  If we are in that situation, GroundWork will tell the customer how to look up the particular items involved in their own database.
  • Host IP addresses are all replaced with a sequence that begins with 10.0.0.1, 10.0.0.2, and so forth, counting up from there.  No trace is left of the original IP addresses.

  • Lots of other fields have their content modified to remove details of its original nature, while still leaving a few traces (mostly a few selected punctuation characters) to suggest the general form of the original data.  In general, all alphabetics will be replaced with "x" characters.  So for example, the original command definition:

    $USER1$/check_tcp -H $HOSTADDRESS$ -p 23

    would be converted to the sanitized command definition:

    $xxxx9$/xxxxxxxxx -x $xxxxxxxxxxx$ -x 99

    which tells us very little about the original command.  This form will pass muster enough to allow GroundWork to run a Commit operation during testing, but not to have Nagios produce useful monitoring results with that setup.

The anonymized database you create by following the steps below will have a different database name (sanitized_monarch), in order not to affect your production monarch database.  Your production database will remain untouched, so capturing the anonymized version is a safe operation.  The only impact will be the use of some extra file space for the modified copy, and some modest CPU loading and extra i/o operations while the altered version is being created.

Anonymizing the Monarch Database

Preparation tasks

You must take the following steps before you can generate an anonymized database.

  • Obtain the PostgreSQL super-user password, namely the password for the postgres database user.  You will be prompted for that password several times while executing the following commands.

  • Obtain the anonymizer tools from the Tools section of the Downloads page, and place the tarball in the/tmpdirectory on your GroundWork server.  Then unroll the tarball so the scripts you need will show up in the/tmp/toolsdirectory:

    Run as the nagios user in GW7, as the gwos user in GW8

    cd /tmp
tar xfz anonymize-database-2.0.tar.gz
    CODE

Creating an anonymized GW7 database

Here are the commands to run the tools in a GW7 context.  Assuming that you have parked the tools locally in the /tmp/tools directory, run the following commands as either the root user or the nagios user:

cd /tmp/tools
./copy-monarch-database -c
./anonymize-monarch -a
./dump_gw_database sanitized_monarch -e -p -z

The path to the final database tarball, such as /tmp/tools/sanitized_monarch-entire.myserver.2021-05-30_20.46.22.sql.gz, will have been printed in your terminal window when the dump_gw_database program completed its work.

Creating an anonymized GW8 database

Here are the commands to run the tools in a GW8 context.  Assuming that you have parked the tools locally in the /tmp/tools directory, run the following commands as the user you normally use to manage your GW8 monitoring, typically the gwos user:

cd ~/gw8
docker cp /tmp/tools dockergw8_monarch_1:/tmp
docker-compose exec monarch /tmp/tools/copy-monarch-database -c
docker-compose exec monarch /tmp/tools/anonymize-monarch -a
docker-compose exec monarch bash -c "cd /tmp; /tmp/tools/dump_gw_database -h pg sanitized_monarch -e -p -z"
docker cp dockergw8_monarch_1:/tmp/sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql.gz /tmp

All but the last of those commands are to be typed precisely as shown.  For the last command, the server name and timestamp will be different for you.  The exact name of your file will have been printed in your terminal window when the dump_gw_database program completed its work.

Handling an Anonymized Database

PRESERVE THE DUMPFILE NAMING CONVENTION

The name of the dumpfile is critical to recognizing that it contains content which would be highly dangerous to restore on a production monitoring server (because it would wipe out all your carefully established setup).  If you ever rename the dumpfile, be sure to make your new name begin with sanitized_monarch, to flag that fact to anyone who encounters the dumpfile later on.

Examining an Anonymized Database

We chose options for the database dump that make it a compressed plaintext dumpfile.  We did that for three separate reasons:

  • That format generally results in the smallest possible file size, which is convenient for transferring to other systems.
  • The compression and its embedded checksums provides some amount of protection against otherwise-undetected corruption as the file is transferred between systems.
  • The underlying plaintext format allows the customer to look at the dump content and verify that they will not be exporting any sensitive data when they provide the dumpfile to GroundWork.

If you want to examine the content of the dumpfile, just make a copy somewhere and decompress it:

gunzip sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql.gz

The result of that command would be a plaintext sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql file that you could examine with an ordinary text editor or other tools of your choice.  If you didn't make a separate copy before decompressing the file, be sure not to accidentally alter the contents before you re-compress the file and provide it to GroundWork.

Loading an Anonymized Database

There is nothing inside the database dumpfile that identifies it as being the content of a particular database name such as sanitized_monarch.  Because of that, you can easily restore such a database on a test system, replacing its normal monarch database.

DO NOT REPLACE YOUR PRODUCTION DATABASE

You must never attempt to restore a sanitized database on a production machine.  It will completely wipe out the utility of your production monitoring.  The only ways to recover from such a mistake are to have a recent backup in hand from before you performed the restore, or to rebuild the entire configuration from scratch.  This is just part of the reason why we include the word "sanitized" in the dumpfile filename, so you have some warning that this is not just an ordinary dump.  Do not ever change that convention if you rename the file!

Note that as of recent GW7 releases, and continuing into GW8, Monarch will automatically take a backup at the end of every successful Commit operation.  (Unless you enter an explicit annotation at Commit time, these backups are annotated as Backup auto-created after a Commit by user "myadminname".)  So in a worst-case situation, you could restore the most recent of those backups via the Monarch UI screens.  That said, you would still lose any changes you had already made since that Commit.  And only a limited number of unlocked backups are kept in the system; see the Control > Setup > Max unlocked backups option.  So you had better realize your mistake quickly, and either revert it right away, or at least lock some of the backups containing your production data.

Only on a test machine, if you want to load an anonymized database into the monarch database, take the following steps:

  • If you might want to return after testing to the previous configuration, first use the Configuration > (Nagios Monitoring >) Control > Back up and restore screen to make an annotated backup of the present database.  Lock that backup to be sure it won't roll off and be lost during subsequent testing, as lots of additional backups get made during your testing.

  • Restore the sanitized database with commands similar to the following.

    Restoring a database in GW7 context, as the nagios user

    gunzip sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql.gz
/usr/local/groundwork/postgresql/bin/psql -U postgres -d monarch \
    -f sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql
    CODE

    Restoring a database in GW8 context, as the gwos user

    cd ~/gw8
# Assuming you have the compressed tarball parked in the /tmp directory ...
gunzip /tmp/sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql.gz
docker cp /tmp/sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql \
    dockergw8_monarch_1:/tmp
docker-compose exec monarch psql -h pg -U postgres -d monarch \
    -f /tmp/sanitized_monarch-entire.myserver.2021-05-31_03.21.42.sql
    CODE

  • Depending on the Monarch version of the system on which the sanitized database was captured and the Monarch version of your test system, it may be necessary to run a Monarch migration script.  Navigate to the Configuration > (Nagios Monitoring >) Control > Setup screen to check that, and follow the on-screen instructions if an error appears and migration is needed.  Under some circumstances, migration of a sanitized database has been observed to fail.  If that happens, there is currently no simple workaround; you will need to test instead on a machine that is running the same release of GroundWork Monitor as was in use where the sanitized database was captured.

    Migrating a database in GW7 context, as the nagios user

    /usr/local/groundwork/core/migration/postgresql/pg_migrate_monarch.pl
    CODE

    Migrating a database in GW8 context, as the gwos user

    cd ~/gw8
docker-compose exec monarch /usr/local/groundwork/monarch/bin/pg_migrate_monarch.pl
    CODE

Testing with an Anonymized Database

Once you have loaded the database, you can run ordinary operations, with limitations:

  • You may edit the configuration via Monarch UI screens.  This allows for testing of any potential bugs within Monarch itself, modulo the fact that sanitization will have changed the string lengths of object names.
  • You may run a Nagios Pre-flight or Commit operation.  Nagios will accept the sanitized configuration and pretend to monitor with it.  It won't produce useful results, because all the hostnames will have been replaced with generic names and all the command lines will have been replaced with stuff that won't execute.  But you will at least be able to see what gets generated by Monarch in the Nagios configuration files, though of course in obfuscated form.  And in a Commit, all the modified inventory will be sent downstream to Foundation, as usual, but now using the sanitized object names (host names, service names, hostgroup names, and servicegroup names).
  • As an automatic special adjustment, Nagios CGI screens will be visible to the admin user.  Permissions for all other users will have been suppressed because the user names will have been sanitized.
  • You may run a Build Instance action, but you cannot Build and Deploy to the child server because the Monarch Group name has been sanitized to something unrecognizable.  If you want to test a full build-and-deploy, you will need to find the Monarch Group of interest and rename it to the name of your test-system Child server.  At the same time, you will also need to replace the Monarch Group's Build Folder with an appropriate path, since the original value will have been sanitized to something generic.  Of course, even then all the configuration data sent to the Child will still be obfuscated.
  • You may run a Build Externals action to verify its normal operation, but the results will not be useful to GDMA clients.  Not only will the host and service names have been sanitized, but the content of the externals themselves will have been mashed to un-usability for monitoring purposes.

Related Resources