This page reviews how to add and configure a Docker connection using GroundWork Cloud Hub. The connection requires a unique set of parameters. If you are connecting to a remote GroundWork server to send results, you will need your remote GroundWork server RESTACCESSAPI token.
Adding a Docker Connection
To access Cloud Hub configuration, log in to GroundWork Monitor as a member of the Admin role (e.g., user admin), and select Configuration > Cloud Hub. To add a new connection click the +Add button next to the desired connector icon. You will need to create a new connection in this way for each project to be monitored.
The data the GroundWork server receives comes from the remote server running Docker. The information is pulled from the cAdvisor API on a periodic basis based on the check interval that is set. In the configuration page you will need to enter both the GroundWork Server and Docker Connector parameters. The fas fa-info-circle link located at the top right side provides information and versioning for the selected Cloud Hub connector.
The GroundWork Server can simply be the same as the one you are running the Cloud Hub connector on, or it can be a remote GroundWork server. If it's the same as the one you are running on, leave the directive Use Local Connection checked.
Otherwise, uncheck this box and fill in the hostname of the remote GroundWork server in the Hostname field, leave RESTAPIACCESS in the Username field, and paste in the the encrypted Token. The token can be obtained on the remote GroundWork server, for users with the Admin role, by going to Administration > Security under Webservices API Account: RESTAPIACCESS, Encrypted Token. Just copy the key from the remote server into the Token field on the Cloud Hub server.
Once you have the GroundWork Server side of the form filled out, click Test. If you have the credentials correct and you have access to the API, you will see a Success message. Otherwise an error will give you a hint as to what is wrong and let you try again.
Using a remote server will populate the remote server with the Docker monitoring data, and this will not show in the local GroundWork Server.
Version: Indicates the minimum GroundWork Monitor version needed. In other words, a version below the indicated value is incompatible.
Hostname: The host name or IP address where a GroundWork server is running. A port number should not be entered here. If GroundWork is running on the same server, leave this blank and click Use Local Connection.
Username: The provisioned Username granted API access on the GroundWork server.
Token: The corresponding API Token for the given Username on the GroundWork server, see Administration > Security under Webservices API Account: RESTAPI Encrypted Token.
SSL: Check this box if the GroundWork server is provisioned with a secure HTTPS transport.
Merge Hosts: If checked, this option combines all metrics of same named hosts under one host, regardless of the case of the hostname. For example, if there is a Nagios configured host named Demo1 and a Cloud Hub discovered host named demo1, the services for both configured and discovered hosts will be combined under the existing hostname Demo1.
Monitor: If checked, enables connection to be monitored. Gives you a way to know when the connector is having trouble reaching the endpoint by creating a service on the host it reports to.
Use Local Connection: This directive refers to where the Cloud Hub results are sent. If this field is checked, results will be posted to the same server as where Cloud Hub is running. Or, with this field unchecked, you can forward results to any accessible GroundWork server you define with the name and API key.
Ownership: Ownership is the owner of a connectors hosts and the ownership can be switched. When a Cloud Hub connector is instantiated the following options are available for ownership:
Always take ownership: The connector will assume ownership of all hosts it instantiates, even merged hosts. This will remain true even if another app merges the host.
Leave ownership if already owned: The connector host will remain with the existing owner until or unless the owner deletes the host.
Always defer ownership (default): This option leaves ownership unchanged on merged hosts, and allows other apps to take ownership.
Note that multiple apps can report on a single service, but only one can own the host.
See Ownership options.
Connection Status: Click Test to verify a connection using the GroundWork server entries.
Next you will need to fill in the Docker Connector parameters and test the connection. The host you monitor must be running a cAdvisor container, and may need to be prepared. See Setting up a Docker Host for Monitoring below.
Start by entering a Display Name. This won't be used except to identify this connector instance on the Cloud Hub page.
Next enter the URI for the Docker host running cadvisor - this is usually of the form http://hostname:port, where the hostname or IP address of the system running docker with the port configured to listen on is set up.
Then enter the cAdvisor host and port, usually of the form hostname:port. The host is the hostname or IP address of the Docker host, and in this case the port is what is mapped to cAdvisor on the Docker host, often 8098. Note the hosts are added to hostgroups in GroundWork that use this name, so it's recommended to use the short DNS name instead of the IP address in this case.
You may next optionally specify a Host Prefix. What you put here will be prepended to all the container hosts you monitor on this connection, so you probably want to have this end with a dash ("-"). You can combine several connector instances with the same prefix if you want to.
You can optionally set the Interval and Retry directives.
Then, validate the connection by clicking Test. A dialog will be displayed with either a Success message or, if the project cannot be contacted, an error message will be displayed with a hint as to why the connection failed.
Make sure to click Save in the upper right corner to save your correct connection parameters.
After the credentials have been validated and the resources indicated, select the Metrics link (top navigation) to start customizing metrics for the connection, refer to the document Customizing Metrics.
Display Name: This is the configuration’s name displayed in the list of Cloud Hub connectors on the Cloud Hub home page.
Docker API URI incl. protocol: This is the location of the Docker daemon you want to monitor. The default is the local unix socket on the current GroundWork server, so you will likely need to change this to the URL and port for the Docker host you configured for monitoring.
Docker Host:Port: This is the address and port of the cAdvisor instance you want to monitor.
Host Prefix: This is the prefix of the host (e.g., qa- or docker1-).
Interval (min): This is the metric gathering interval for collecting monitoring data from Azure and sending it to the GroundWork server. The value is in minutes.
Infinite Retries: Check this box if you want Cloud Hub to infinitely retry connection to Azure when the connection fails. When this box is checked, the Retry Limit field is disabled. When this box is unchecked, the Retry Limit field is enabled.
Retry Limit: This entry is the number of retries for the connection and sets a limit on how many attempts are made after a failure. The number set indicates how many connections are attempted before the connection is left in an inactive state. At this point, the connection is suspended and you will need to manually restart it. When a retry limit is exhausted, all hosts managed by this connection are set to the monitor status Unreachable and all services for the matched hosts are set to the status of Unknown.
Connection Status: Click Test to verify a connection using the Azure connector entries.
Setting up a Docker Host for Monitoring
Modifying the Docker Service Instance
In order to monitor Docker hosts with cAdvisor over a network, you need to configure two separate network ports open on your firewalls, one for the Docker URI and one for cAdvisor. The specific ports can be your choice.
The following procedure is an example. Please follow your own internal procedures for managing Docker and cAdvisor to achieve the same result, and be sure to secure the network access accordingly.
In our example we will use tcp/2375 for the URI and the default tcp/8098 for cAdvisor.
You will need to be running Docker on a host in your network that is accessible from the GroundWork Server on ports 2375 and 8098. You will also need a cadvisor container image loaded on that host, in addition to whatever other containers you are running. You will need to restart Docker to configure it for monitoring.
Please note, stopped containers will be down with no services.
You might need to pull a cadvisor container. Generally you do this using a docker command:
docker pull google/cadvisor
This pulls the latest cAdvisor image from Docker Hub for you, so you can run a cAdvisor container as noted below. You may however not have access to Docker Hub, and so here's how to get a useful cAdvisor container out of your GroundWork server deployment without accessing the Internet at all:
On the GroundWork 8 server, go to the gw8 directory:
Save the image:
docker save -o cadvisor_save.tar google/cadvisor:v0.33.0CODE
- Transfer the image to your Docker host to be monitored (use scp or similar).
Load the image:
docker load -i cadvisor_save.tarCODE
Now you have the image loaded, with tag v0.33.0.
On the Docker host (we assume a Linux System V host for convenience), as root (or use sudo):
Locate the systemd docker service unit file for your Linux distribution. The example below is for Ubuntu 16 (Xenial Xerus). Edit the systemd unit file for the docker daemon, for example:
Edit the ExecStart line to include the portmapping, for example (as root or with sudo):
There is a bug in SUSE Linux with how the docker service is launched in systemd that requires the procedure to be modified. The following is the edit for ExecStart for SUSE only, or continue with Other than SUSE.
<pre class="code-java">[Service] Type=notify # the <span class="code-keyword" style="color: #910091;">default</span> is not to use systemd <span class="code-keyword" style="color: #910091;">for</span> cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # <span class="code-keyword" style="color: #910091;">for</span> containers run by docker ExecStart=/usr/bin/dockerd --add-runtime oci=/usr/sbin/docker-runc -H unix:<span class="code-comment" style="color: #808080;">// -H tcp://0.0.0.0:2375 $DOCKER_NETWORK_OPTIONS $DOCKER_OPTS</span></pre>CODE
Other than SUSE
[Service] Type=notify # the default is not to use systemd for cgroups because the delegate issues still # exists and systemd currently does not support the cgroup feature set required # for containers run by docker ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sockCODE
You can of course allow connections from just the GroundWork server(s) instead of everywhere.
Some versions of Docker will not start with the
-H fd://option on the ExecStart line. The error message is:
failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
If your version of Docker is more recent, this error may occur. In this case, change the -H fd:// to -H unix:// instead.
Restart systemd daemon and docker service:
systemctl daemon-reload systemctl restart docker
Run the cadvisor container with the following (minimum) parameters, using the tag v0.33.0 instead of "latest" if you exported the container from GroundWork 8.1.3:
docker run -d --privileged -p 8098:8080 \ -v /:/rootfs:ro \ -v /var/run:/var/run:rw \ -v /sys:/sys:ro \ google/cadvisor:latest
The --privileged flag is only necessary if you wish to enable OOM functionality in cadvisor. It will log a warning if you do not use this.
You can also use the following docker compose yaml entries, if you are running under docker compose:
# # GroundWork 8 docker compose # version: "3.4" services: cadvisor: privileged: true image: google/cadvisor ports: - "8098:8098" command: - '-port=8098' - '--housekeeping_interval=55s' - '--docker_only' healthcheck: test: wget --spider http://cadvisor:8098/healthz || exit 1 interval: 10s timeout: 3s retries: 3 start_period: 10s volumes: - /:/rootfs:ro - /var/run:/var/run:rw - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro - /dev/disk/:/dev/disk:roCODE
This second option has the advantage of a health check, which is a good thing.
You should now be able to monitor this host with the Docker Cloud Hub connector with the following settings:
Docker API URI incl. protocol: http://hostname or ip address:2375 Docker Host:Port: hostname or ip address:8098