This page reviews how to add and configure a Docker connection using GroundWork Cloud Hub. The connection requires a unique set of parameters. If you are connecting to a remote GroundWork server to send results, you will need your remote GroundWork server RESTACCESSAPI token.

Adding a Docker Connection

To access Cloud Hub configuration, log in to GroundWork Monitor as a member of the Admin role (e.g., user admin), and select Configuration > Cloud Hub. To add a new connection click the +Add button next to the desired connector icon. You will need to create a new connection in this way for each project to be monitored.

Connection Parameters

The data the GroundWork server receives comes from the remote server running Docker. The information is pulled from the cAdvisor API on a periodic basis based on the check interval that is set. In the configuration page you will need to enter both the GroundWork Server and Docker Connector parameters. The  fas fa-info-circle  link located at the top right side provides information and versioning for the selected Cloud Hub connector.

GroundWork Server

The GroundWork Server can simply be the same as the one you are running the Cloud Hub connector on, or it can be a remote GroundWork server. If it's the same as the one you are running on, leave the directive Use Local Connection checked. 

Otherwise, uncheck this box and fill in the hostname of the remote GroundWork server in the Hostname field, leave RESTAPIACCESS in the Username field, and paste in the the encrypted Token. The token can be obtained on the remote GroundWork server, for users with the Admin role, by going to Administration > Security under Webservices API Account: RESTAPIACCESS, Encrypted Token. Just copy the key from the remote server into the Token field on the Cloud Hub server.

Once you have the GroundWork Server side of the form filled out, click Test. If you have the credentials correct and you have access to the API, you will see a Success message. Otherwise an error will give you a hint as to what is wrong and let you try again. 

Using a remote server will populate the remote server with the Docker monitoring data, and this will not show in the local GroundWork Server.

Docker Server

Next you will need to fill in the Docker Connector parameters and test the connection. The host you monitor must be running a cAdvisor container, and may need to be prepared. See Setting up a Docker Host for Monitoring below. 

Start by entering a Display Name. This won't be used except to identify this connector instance on the Cloud Hub page.

Next enter the URI for the Docker host running cadvisor - this is usually of the form  http://hostname:port,  where the hostname or IP address of the system running docker with the port configured to listen on is set up.

Then enter the cAdvisor host and port, usually of the form hostname:port. The host is the hostname or IP address of the Docker host, and in this case the port is what is mapped to cAdvisor on the Docker host, often 8098.  Note the hosts are added to hostgroups in GroundWork that use this name, so it's recommended to use the short DNS name instead of the IP address in this case. 

You may next optionally specify a Host Prefix. What you put here will be prepended to all the container hosts you monitor on this connection, so you probably want to have this end with a dash ("-"). You can combine several connector instances with the same prefix if you want to. 

You can optionally set the Interval and Retry directives.

Then, validate the connection by clicking Test. A dialog will be displayed with either a Success message or, if the project cannot be contacted, an error message will be displayed with a hint as to why the connection failed.

Make sure to click Save in the upper right corner to save your correct connection parameters. 

After the credentials have been validated and the resources indicated, select the Metrics link (top navigation) to start customizing metrics for the connection, refer to the document Customizing Metrics.

Setting up a Docker Host for Monitoring

Modifying the Docker Service Instance

In order to monitor Docker hosts with cAdvisor over a network, you need to configure two separate network ports open on your firewalls, one for the Docker URI and one for cAdvisor. The specific ports can be your choice.

The following procedure is an example. Please follow your own internal procedures for managing Docker and cAdvisor to achieve the same result, and be sure to secure the network access accordingly.

In our example we will use tcp/2375 for the URI and the default tcp/8098 for cAdvisor. 

Requirements

You will need to be running Docker on a host in your network that is accessible from the GroundWork Server on ports 2375 and 8098. You will also need a cadvisor container image loaded on that host, in addition to whatever other containers you are running. You will need to restart Docker to configure it for monitoring. 

Please note, stopped containers will be down with no services.

You might need to pull a cadvisor container. Generally you do this using a docker command:

docker pull google/cadvisor

This pulls the latest cAdvisor image from Docker Hub for you, so you can run a cAdvisor container as noted below. You may however not have access to Docker Hub, and so here's how to get a useful cAdvisor container out of your GroundWork server deployment without accessing the Internet at all:

1. On the GroundWork 8 server, go to the gw8 directory:

   cd gw8

   CODE

2. Save the image: 

   docker save -o cadvisor_save.tar google/cadvisor:v0.33.0

   CODE
3. Transfer the image to your Docker host to be monitored (use scp or similar).

4. Load the image: 

   docker load -i cadvisor_save.tar

   CODE

   Now you have the image loaded, with tag v0.33.0. 

Docker configuration

On the Docker host (we assume a Linux System V host for convenience), as root (or use sudo):

1. Locate the systemd docker service unit file for your Linux distribution. The example below is for Ubuntu 16 (Xenial Xerus). Edit the systemd unit file for the docker daemon, for example:

   vi /etc/systemd/system/multi-user.target.wants/docker.service

2. Edit the ExecStart line to include the portmapping, for example (as root or with sudo):

   

   There is a bug in SUSE Linux with how the docker service is launched in systemd that requires the procedure to be modified. The following is the edit for ExecStart for SUSE only, or continue with Other than SUSE.

   SUSE only

   <pre class="code-java">[Service]
   Type=notify
   # the <span class="code-keyword" style="color: #910091;">default</span> is not to use systemd <span class="code-keyword" style="color: #910091;">for</span> cgroups because the delegate issues still
   # exists and systemd currently does not support the cgroup feature set required
   # <span class="code-keyword" style="color: #910091;">for</span> containers run by docker
   ExecStart=/usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc -H unix:<span class="code-comment" style="color: #808080;">// -H tcp://0.0.0.0:2375 $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS</span></pre>

   CODE

   Other than SUSE

   [Service]
   Type=notify
   # the default is not to use systemd for cgroups because the delegate issues still
   # exists and systemd currently does not support the cgroup feature set required
   # for containers run by docker
   ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock

   CODE

   You can of course allow connections from just the GroundWork server(s) instead of everywhere.

   

   Some versions of Docker will not start with the -H fd:// option on the ExecStart line. The error message is:
   failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
   If your version of Docker is more recent, this error may occur. In this case, change the -H fd:// to -H unix:// instead.

3. Restart systemd daemon and docker service:
   

   systemctl daemon-reload
   systemctl restart docker

4. Run the cadvisor container with the following (minimum) parameters, using the tag v0.33.0 instead of "latest" if you exported the container from GroundWork 8.1.3:

   docker run -d --privileged -p 8098:8080 \
   -v /:/rootfs:ro \
   -v /var/run:/var/run:rw \
   -v /sys:/sys:ro \
   google/cadvisor:latest

   

   The --privileged flag is only necessary if you wish to enable OOM functionality in cadvisor. It will log a warning if you do not use this.  

   You can also use the following docker compose yaml entries, if you are running under docker compose:

   #
   # GroundWork 8 docker compose
   #
   version: "3.4"
   services:
     cadvisor:
       privileged: true
       image: google/cadvisor
       ports:
         - "8098:8098"
       command:
         - '-port=8098'
         - '--housekeeping_interval=55s'
         - '--docker_only'
       healthcheck:
         test: wget --spider http://cadvisor:8098/healthz || exit 1
         interval: 10s
         timeout: 3s
         retries: 3
         start_period: 10s
       volumes:
         - /:/rootfs:ro
         - /var/run:/var/run:rw
         - /sys:/sys:ro
         - /var/lib/docker/:/var/lib/docker:ro
         - /dev/disk/:/dev/disk:ro

   CODE

   This second option has the advantage of a health check, which is a good thing. 
   
   You should now be able to monitor this host with the Docker Cloud Hub connector with the following settings:
   

   Docker API URI incl. protocol: http://hostname or ip address:2375
   
   Docker Host:Port: hostname or ip address:8098

Related Resources

• Page:
  AWS
• Page:
  Azure
• Page:
  Cloud Hub Troubleshooting
• Page:
  Cloudera
• Page:
  Customizing Metrics
• Page:
  Google
• Page:
  NeDi
• Page:
  NetApp
• Page:
  OpenStack
• Page:
  VMware