About Graphs

Best used to view service performance data. This dashboard option uses the integrated Grafana application, a dashboard builder for visualizing time series data on every metric for every service. The Grafana integration includes the GroundWork data source making it easy to select the individual hostgroups, servicegroups, hosts, and services you want to graph on your dashboards, other datasources can also be used. The dashboards themselves can contain not only graphs but histograms, heat maps and other representations. With Grafana, performance data is stored in InfluxDB, which saves every point it receives, until it is aged off after 13 months (or optionally longer). No summarization of data occurs. Grafana is a powerful graphing engine, this documentation is meant as a summary of how to create graphs in GroundWork Monitor, for extensive Grafana documentation please see Grafana Labs.

Creating a Dashboard

This is an example of how to create a simple Grafana dashboard in GroundWork Monitor using the default data source GroundWork. There is a limit of 16 services that can be graphed in one panel. Graphs can have multiple panels. Here we'll graph, within two panels, CPU usage and RAM usage for key GroundWork Monitor containers. 

  1. As an Admin role user, navigate to Dashboards > Graphs.
  2. Hover over the  Create icon, and select Dashboard.
  3. Select a panel type for the first panel in the graph, e.g., Graph.
  4. To edit the panel, click the title of the panel e.g., Panel Title and select Edit, this will open the Graph Editor where you can add metrics to the graph and change the graph style. 

  5. We start with the General tab which is focused on panel identity. You'll want to provide a panel title, a description, and optionally set the panel to be transparent. For now we'll skip the links section at the bottom as this is covered in the Add Links further down. For example:

    • Enter a Title, e.g., CPU on GroundWork Containers, this is the name displayed for the panel within the graph itself.
    • Enter a Description, e.g., CPU usage on key GroundWork containers, this adds panel information in the form of a hover over as seen in the image below.
    • Check the Transparent option which will render the panel transparent/no background.
  6. Next we'll go to the Metrics tab to configure what is to be graphed. Notice the data source is set as default which is the GroundWork data source enabling use of GroundWork monitoring data. To configure other data sources see Adding a Data Source further down in this document. The area below the data source is where you define queries for display within the graph panel. The first directive, Service by, indicates how you want to select the metrics to graph, either by Hosts, Host Groups, Service Groups or Custom Groups (if defined). For instance, if you select service by Host Groups you would then select all the monitored host group(s) you want to pull metrics from, and the host(s) within those host groups, and service(s) within those hosts, and indicate the metric(s) to graph. You can optionally select metric threshold(s) to be visible in a graph. It is important to keep in mind the maximum number of services you can graph is 16. For example:
    • For Service By select Host Groups.
    • For Host Group select DOCK-H:cadvisor.
    • For Host select some gw8-dockergw8_ hosts (we select 9).
    • For Service select cpu.usage.total which is also our reporting Metric.

  7. Let's focus now on adjusting the look and feel. First Axes, then Legend and Display.
    • In the Axes tab we leave the default and auto settings as is and add a Label for the Left Y-axis (e.g., CPU)
    • We make adjustments in the Legend tab to Show the legend As a Table and To the right, with the value Max to also be visible in the table. We also Hide zero values.
    • For the Display tab Draw options we keep it at Lines vs. Bars or Points, make the Fill a little denser and Line Width a little thicker setting both at 2, and we want the Hover tooltip to show all series with a sort order of decreasing.
    • And we're finish for now in the editor. Select the X on the right to exit the bottom panel.
  8. Then using the same process as steps 5, 6 and 7 above using similar information but measuring RAM, add a second panel by clicking the  + Add panel icon (left of the star). You can drag the panel around the graph to reposition if needed.
  9. When finished, click the  save icon from the tool bar near the top, enter a dashboard name, and click Save. That's it! A creation of a basic graph anyway, there's much more of course that you can do.

Adding Links

A Grafana dashboard can contain one or more panels. You can use links as shortcuts in panels to link other panels, dashboards, or external websites. Grafana supports and Panel Links which are accessible from a panel, and Dashboard Links which are accessible from a dashboard, Image 1 below displays the hover over panel description and panel links (in blue), and the dashboard links on the right (external submit links in this example).

Image 1

Grafana also supports Share Links in which you can obtain a panel or dashboard URL and use it within another location, for example Status dashboard where you can link a monitored entity to an internal or external link including a graph dashboard. The URL used in this scenario can be a panel or dashboard Share Link. Here we a Host Groups Hosts in Status to the Grafana dashboard CPU and RAM for Key Containers.

Image 2

  1. In the Graph Editor of a dashboard, select the General tab, scroll down and click +Add link.
  2. In the Type field:
    • Select dashboard to link to an existing dashboard, indicate the Dashboard name(you can start typing and available dashboards will display), and enter a Title which will appear as the link in the panel dropdown.
    • Select absolute to link to a URL, provide the URL, and a link Title.
  3. You have the option to include the time range and current template variables to directly jump to the same context in another dashboard to assure the right data is being displayed using URL params. Link can be set to opened in a new tab.
  4. Click the  save icon from the tool bar near the top. You can click the blue left arrow at the top to view your dashboard panel links.
  1. From within a dashboard, at the top of the screen click the  settings icon.
  2. Select Links.
  3. Click Add Dashboard Link (or +New).
  4. In the Type field you can select dashboards or link:
    • Selecting dashboards will display a list of all your dashboards or those With tags (you can add tags to a dashboard in Settings > General). If you select As dropdown the dashboards will be listed as a dropdown list, if not they will be list in a row.
    • Selecting link allows you to add any URL link, title it, attached a tooltip and an icon.
  5. For either type you can choose to display the time range, add variables, and open of the linked dashboard in a new tab. 
  6. When finished click Add and then Save. You can click the blue left arrow at the top to view your graph dashboard links.

In this section we use the example of linking to a Grafana panel or dashboard from the Status dashboard. To do this we use Grafana Share Links along with the Status Links feature. Essentially, Grafana provides shared links for panels and dashboard that can then be added as Status dashboard links. In Status you can add links to go to internal (GroundWork) and external (URL) sources, from any entity including custom groups, host groups, service groups, hosts, services. Later you may want to incorporate the use of Templates and Variables, for now we'll simply walk through how to link a Status entity to a Grafana dashboard.

  1. From the GroundWork UI, select Dashboards > Graphs.
  2. Create a new or open a Grafana graph that is to be linked to from Status, we use a dashboard named GroundWork server in this example).
  3. Obtain a Share Link (we use the dashboard link):
    • To obtain a Share Link for a panel, click the dropdown in the panel title and select Share, customize any options, and click Copy.
    • To obtain a Share Link for a dashboard, click the Share icon from the top toolbar, customize any options, and click Copy.
  4. Then go to Dashboards > Status > Host Group (e.g., DOCK-H:cadvisor), click the actions icon, and Add Link and configure the following:
    • Link to: Select Every Host in Host Group DOCK-H:cadvisor so every host in this host group will display the link to CPU usage.
    • Link Type: Select Internal GroundWork Link in new tab.
    • GroundWork Link: Then paste the copied Share Link from the Grafana dashboard eliminating the server name from the URL because the graph is an Internal link.
    • Link Display: Enter a title e.g., CPU and RAM use on key GroundWork containers.
    • Click Create.
  5. In Status for every host in the host group DOCK-H:cadvisor you should see a Status link that goes to the dashboard GroundWork server which displays the CPU and RAM panels, as shown in the screenshot at the beginning of this section. If you need to edit a Status Link, just return to a page displaying the link and click the  settings icon.

Templating Graphs with Variables

Grafana variables allows for more interactive and dynamic dashboards. You can use this feature to create generic dashboards in which you can quickly change servers or metrics to be displayed. Instead of hard-coding things like host group, host, and host-service in metric queries and in your panel titles, you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard which make it easy to change the data being displayed in your dashboard. For additional information see Grafana Labs Templates and Variables.

We'll take a look at one of the GroundWork default dashboards, Variable Hostgroups, as an example of how to use and set up variables.

  1. To access the GroundWork default dashboards, go to Dashboards > Graphs, select the  Dashboards icon and select Manage. Then click the GroundWork Defaults folder to list all default templates, and choose Variables Hostgroups. 
  2. In the Variables Hostgroups dashboard below, variables have been added enabling a Host Group, Host, Service, Metric, and Threshold to be selected and used to generate the graph. The image shows the Query to be by Host Group AZURE-M-Pay-As-You-Go, Hosts, Services, and Metrics as All, and Thresholds as Critical. 
  3. Let's see how the variables for this dashboard were configured. Click the settings icon at the top.
    • Select Variables and a list of configured variables for this dashboard will be displayed as shown here. 

    • The GroundWork data source supports the following template variables for host groups, hosts, service by host, service groups, and services by service group.

      VariableQuery Examples
      $hostGrouphostGroup, hostGroup $customHostGroup
      $serviceGroupserviceGroup, service $customServiceGroup
      $customHostGroupcustomHostGroup
      $customServiceGroupcustomServiceGroup
      $hosthost, host $hostGroup
      $serviceservice $host, service $serviceGroup
      $metricmetric $service
      $thresholdthreshold

    • Go ahead and click a variable (e.g., $hostGroup) to view its definition for this dashboard.
      • This page is pretty straight forward. You can see the Type is Query, and the Query is hostGroup, and it doesn't include Multi-value in this case as users can choose only one host group for the dashboard versus several.
      • The Preview of values at the bottom lets you see what you will see as values for the Host Group dropdown variable when viewing the dashboard.
  4. In addition to variable being using to graph specific monitors, you can also use variables in the Panel Title to add a title base on the variables selected. For example:
    • Go back to editing the graph (click the arrow in the blue box in the upper right).
    • Click the Panel Title dropdown and select Edit. 
    • In the General tab, for the Title you can enter something like HOSTGROUP: $hostGroup HOSTS(S): $host SERVICES: $service which will present a variable title for you dashboard. The title will be based on the variables selected replacing the generic Panel Title or other static title.

Adding a Data Source

The default data source GroundWork allows you to query GroundWork monitored custom groups, host groups and hosts, service groups and services, and also by metrics and thresholds. That is the GroundWork data map which is defined and exposed by the GroundWork API. You can see, in the image below, the GroundWork data source URL points to the container foundation and the API. 

Grafana supports many different storage backend for time series data. If you select Add data source you'll see a list of Grafana compatible data sources in which you can use to create graphs. There are two other databases available to you locally in GroundWork, InfluxDB and MySQL. To connect you'll need the URL to the GroundWork container and the port as http://influxdb:8086, along with indicating the InfluxDB details database as groundwork). Since Grafana is a container inside GroundWork, it is an http connection and encryption, whitelisting, or credentials is not needed as its all insulated in a containerized network. To connect to a data source on another system then you would need credentials, authentications, and certificates, and TLS.

Each data source has a specific Query Editor for what that data source exposes. The InfluxDB connection in Grafana can provide you with different metric selections versus the default GroundWork data source which has been simplified to focus on GroundWork entities and metrics. You can essentially get the same output from using the GroundWork data source it's just that using the data source InfluxDB allows you to use the query in different ways including the Query Inspector. See more on Data Sources in Grafana Labs.

Using Grafana Running Externally

It is now possible for an externally-running instance of Grafana to gather data from GroundWork. You can find a custom datasource plugin for this on the Downloads page. Alongside the datasource, there are a number of sample dashboard files which can be imported into your custom Grafana instance.

Quick containerized setup of Grafana

This section can be skipped if there is already an existing Grafana instance to be used.

The fastest way to run a new instance of Grafana on the same system as GroundWork is to use Grafana's docker image. The following docker compose file will run this instance of Grafana. Note there is minimal security enabled, and any production system should consider not using this basic configuration. While this setup is fine for testing, you should contact your Grafana administrator or support for a permanent solution.

This file must be placed in a directory other than GroundWork's docker-compose files. In addition, care must be taken to not create conflicting docker networks.

docker-compose.yml

# To validate the YAML formatting in this file, run:
# yamllint -d "{extends: default, rules: {line-length: {max: 300}}}" docker-compose.yml
---
#
# Test grafana docker compose
#

version: "3.4"

services:
  grafanaext:
    #image: grafana/grafana-oss:5.4.5
    image: grafana/grafana-oss
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      - UPGRADEALL=true
      - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=groundwork-grafana-datasource
      - GF_SECURITY_ALLOW_EMBEDDING=true
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

volumes:
  grafana-storage:
YML

This Grafana instance will be accessible at http://localhost:3030

To install the plugin:

  1. First, make sure you have downloaded and extracted the external datasource and dashboards from the Downloads page. The folder should be called groundwork-grafana-datasource

  2. Copy this folder into the Grafana plugins directory in the volume created above. The name of this volume will depend on your folder setup, but the command will be similar to:

    docker cp ./groundwork-grafana-datasource/ gext_grafanaext_1:/var/lib/grafana/plugins/
    CODE
  3. Restart the container with docker-compose restart

Installation of GroundWork datasource for existing external instance

Grafana requires an override to be set to allow use of custom third-party plugins. If the previously-mentioned containerized setup is being utilized, this is already set via the environment overrides in the docker-compose.yaml file. Otherwise, follow these steps:

  1. First, make sure you have downloaded and extracted the external datasource and dashboards from the Downloads page. The folder should be called groundwork-grafana-datasource
  2. Find the Grafana config file. Usually it is the defaults.ini in {installation directory}/grafana/conf/. For some installations, it will be found in /etc/grafana/grafana.ini

  3. In paths section, find or add the plugins setting:

    # Directory where grafana will automatically scan and look for plugins
plugins = full\path\to\groundwork-grafana-datasource
    CODE

  4. Then, in plugins section, add or edit the following setting:

    # Enter a comma-separated list of plugin identifiers to identify plugins to load even if they are unsigned. Plugins with modified signatures are never loaded.
allow_loading_unsigned_plugins = groundwork-grafana-datasource
    CODE
  5. Finally, restart Grafana

Configuration of custom datasource

Once Grafana has been overridden to allow custom datasources, and restarted, follow these steps:

  1. Navigate to Grafana's Data sources settings, under the Configuration icon:
  2. Click Add data source
  3. GroundWork datasource should show up in the search. Click to add one.
  4. Configure the datasource as follows, with the credentials of an existing GroundWork user:
  5. Click Save & Test. Feedback should indicate that the configuration was updated.

Importing default dashboards

Whereas internally-loaded Grafana that comes with GroundWork has embedded default dashboards, these are not included with an external setup, nor are they in a format that supports modern Grafana. These are basic json files, and are imported through a web interface.

To import GroundWork dashboards, first download and extract the archive found in Downloads onto the machine you are accessing GroundWork from.

Then import a dashboard into Grafana:

  1. From the Dashboards panel, click New, and then Import
  2. Click Upload JSON file, then select one of the previously downloaded files.
  3. Upon clicking Import, you should see the GroundWork dashboard.

Related Resources