Application Performance Monitoring
TCG APM Connector
GroundWork Monitor, as of 8.1.2, includes an Application Performance Monitoring (APM) connector you can use to send metrics to GroundWork Monitor directly from your running applications. This article will explain how to enable this connector and get started using it, with examples in Golang since GroundWork uses the Prometheus client, many other languages can be used in similar ways.
Also see related blog postings:
fas fa-blog
Application Monitoring with the Prometheus Client and GroundWork Monitor
fas fa-blog Application Monitoring with Spring Boot, Prometheus, and GroundWork Monitor
Requirements
You will need a GroundWork 8.1.2 or above system, an application you can add code to, and a place to run the APM connector. As of 8.1.3, this can be done in a container on the GroundWork server itself.
Setting Up TCG
If you are running the connector as a standalone Go application, you will need to download and install the connector, and run it. If you want to run it on the GroundWork host as a container, you don't need to download it explicitly, as the container will automatically download when you start it (though you need Internet access from your GroundWork server to do this). You can also run it as a Linux service on the GroundWork host or another host, your choice.
Running as a Container in GroundWork
The simplest way is to add the APM connector as a container to the GroundWork host. To do so, follow these steps:
Un-comment (or, if you have upgraded, add) the following lines in the gw8/docker-compose.override.yml file, after "services:":
tcg-apm: image: groundworkdevelopment/tcg:${TAG} volumes: - tcg-var:/tcg entrypoint: ["/app/docker_cmd.sh", "apm-connector"]
CODEand, at the bottom of the file, make sure the volume is enabled globally:
volumes: tcg-var:
The indentation has to line up exactly with the other entries under "services:", as this is a yaml file. If your GroundWork server doesn't have Internet access, please contact GroundWork Support for download options for tcg containers. They are free and open source.
Restart GroundWork:
docker-compose down docker-compose up -d
CODE
Setting Up the Connector
Next, you will add the connector configuration to your GroundWork server.
Sign in to your GroundWork server as an administrative user.
Go to Configuration > Connectors.
- Click the Add fas fa-plus-square icon and select the APM connector from the menu.
Enter the following information:
Connector Name: Provide a unique display name
TCG Host Address: Enter a location. If you decided to run the connector as a container on the GroundWork host, the location will be
tcg-apm:8099
, this location is controlled in the tcg-config.yaml file in the connector installation.Interval: You can specify an interval for metrics gathering, down to 1 minute, and other parameters as with all connectors.
Resources: You may also need a resource, which is one or more applications that are publishing data with the Prometheus client on a URL and optionally a specific port. The example application referenced below does this on port 2222. If you run it on the GroundWork host, you can use the address
http://172.17.0.1:2222/metrics
, make sure to press enter after typing the resource name or it will not be saved.You need to specify the protocol (http or https), and the location. In this example 172.17.0.1 is the host address from the point of view of the containerized connector. The port is what the application publishes the metrics on. If you don't have an instrumented application up yet, just save the connection and come back to it later, when you know the URL and optional port to point it at. You can add several resources to a single connector (we recommend no more than 4, depending on how many metrics you plan to post).
You don't absolutely need a resource. If you are using the push option for your APM connection, you don't need to put anything in the resource section. The connector is listening for metrics to be posted by default. This is an advanced option, however. Please see the example application for details at https://github.com/gwos/tcg/tree/master/examples/apm/pull/go.
- Optionally you can set Headers for your resource, used to connect to it with the http protocol. Headers supplied can include needed tokens and key values if your resource requires authentication, for example, or needs specific headers to be allowed through a proxy or other connection requirements. Just click the Headers button on your resource to open the dialog and add your header name-value pairs.
- Optionally you can set a default Host Group and Host Name to use for this resource. Just click on the Defaults button to expose the dialog. If you don't supply one, this resource will add a host in the "Servers" Host Group named "APM-Host".
Save the connection. If all is well, you will see the connection status turn green. If not, you will see an error message and will need to resolve the error before moving on.
Before you will see the metrics for an APM pull resource in the status dashboard, the metrics must be manually added. See the section of documentation entitled "Adding Resource Metrics"
Adding Resource Metrics
- Go to Configuration > Connectors.
- Click the name of the APM connector you have just set up.
- Click the Metrics tab. If you have already set up metrics, you can access the following metrics selection screen by clicking Select Metrics
- Select and deselect the metrics you wish to monitor from all resources. In the case where there is a large number of metrics, you may find it helpful to use the regex filter. When finished selecting metrics, click Submit
- Before changes will take effect, they must be saved from the Metric Configuration screen. If no custom overrides are desired, simply click Save
- After saving the metrics configuration, no further steps should be necessary. Selected and configured metrics should begin showing up in the Status Dashboard.
Dynamic Inventory Creation From Prometheus Labels
GroundWork 8.3.0 introduced the ability to create custom inventory based on existing resources and values. While GroundWork-specific labels should be considered reserved in this context, such as "group, status", other labels can be processed to create host and service names. This can be used to organize inventory in the case where the admin has limited control over the Prometheus resource.
Consider the following metrics payload. Of note, there are arbitrary labels containing values — abc, def, ghi
.
$ curl http://192.168.50.22:2222/metrics
# HELP bytes_per_minute Finance Services bytes transferred over http per minute.
# TYPE bytes_per_minute gauge
bytes_per_minute{critical="48000",def="somevalue2",group="PrometheusDemo",resource="FinanceServicesGo",service="analytics",warning="45000"} 35679
bytes_per_minute{critical="48000",def="somevalue2",group="PrometheusDemo",resource="FinanceServicesGo",service="distribution",warning="45000"} 29104
bytes_per_minute{critical="48000",def="somevalue2",group="PrometheusDemo",resource="FinanceServicesGo",service="sales",warning="45000"} 20255
# HELP requests_per_minute Finance Services http requests per minute.
# TYPE requests_per_minute gauge
requests_per_minute{abc="somevalue1",critical="95",group="PrometheusDemo",resource="FinanceServicesGo",service="analytics",warning="85"} 64
requests_per_minute{abc="somevalue1",critical="95",group="PrometheusDemo",resource="FinanceServicesGo",service="distribution",warning="85"} 46
requests_per_minute{abc="somevalue1",critical="95",group="PrometheusDemo",resource="FinanceServicesGo",service="sales",warning="85"} 72
# HELP response_time Finance Services http response time average over 1 minute.
# TYPE response_time gauge
response_time{critical="2.8",ghi="somevalue3",group="PrometheusDemo",resource="FinanceServicesGo",service="analytics",warning="2.5"} 0.6
response_time{critical="2.8",ghi="somevalue3",group="PrometheusDemo",resource="FinanceServicesGo",service="distribution",warning="2.5"} 2.2
response_time{critical="2.8",ghi="somevalue3",group="PrometheusDemo",resource="FinanceServicesGo",service="sales",warning="2.5"} 2.6
Without using telegraf plugins to process these labels, GroundWork is blind to these, and the arbitrary labels will not be monitored or graphed. However, a brief configuration in the APM labels tab can map these labels to either hosts or services.
- From the APM connector screen, make sure that a resource has been added, and navigate to the Labels tab.
- Add any number of Host mappings and Service mappings. Using $1 in the destination will copy the value of the label into the corresponding name. Using sample data from above, the following configuration will work to create both host and service inventory.
Click Save once labels are properly configured. The following is an example of the result given the sample labels. Note the presence of new host based on label
abc
, and the presence of two new services based on labelghi
anddef
.
Sending Data from an Application
You will need to include the Prometheus client library in a Golang program. For example you can use the sample app we have provided:
import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/promhttp"
Here is a minimal example of creating a counter metric, only requiring the counter metric name:
simpleCounter = prometheus.NewCounter( prometheus.CounterOpts{ Name: "request_counter", })
When labeling your metrics, you can optionally control where the results show up in the GroundWork system. The following table summarizes the labels and their effect.
Label | GroundWork Property | Notes |
---|---|---|
resource | Host Name | Adding this label associates the metric to the host named in the label. If it does not exist, it will be created unless a Default is supplied. |
group | Host Group | This label determines the Host Group that the host gets added too. |
status | Service Status | This label can be any of the following values, representing valid service states in GroundWork:
If there is no status label, the service status will be OK, unless there are thresholds that are violated. |
message | Status Text | This label can be a quoted string of up to. 4096 characters. It is displayed on the Status Summary info panel for the service. |
service | Service Name | This label is a string that is used as the name of the service to which the metric is attached. If not supplied, the metric name will be used. |
device | Device ID | This typically the IP address of the host (see Host Name above). It is optional, but useful in dealing with merged hosts. If not supplied, the Host Name (resource) value will be used. |
warning | Warning Threshold | This is a numeric value that determines the Warning Threshold for this metric. If you don't set one with a label, you can set one in the Status Summary with the Edit button. |
critical | Critical Threshold | This is a numeric value that determines the Critical Threshold for this metric. If you don't set one with a label, you can set one in the Status Summary with the Edit button. Typically this is above the Warning Threshold for rising metrics, but if you set it below the Warning Threshold, the system will interpret this as a "falling" metric and set state to CRITICAL when it falls below this value. |
unitType | Type of Metric | This is a quoted string that assigns the units of measure for the posted metric, possible values are:
This optional value will be stored with the metric, but GroundWork does not use it for graphing or other purposes. It is useful only in data retrieved from GroundWork for analysis in other tools, for example. |
For further details, you can see this working with a useable example of a Go program that connects to the APM connector, https://github.com/gwos/tcg/tree/master/examples/apm/pull/go.
Running as a Linux Service
To run the APM connector as a standalone Go program is fine for experimental purposes, but in production you will want to run it as a Linux service. Here's how to do so on an Ubuntu system:
Requirements
You will need a Linux host that you can connect to on port 8099 from the GroundWork server. It can be the GroundWork server itself, if you like.
Download the Connector
Assuming you have Internet access from your server, you can download the connector directly to the server where you want to run it. If not, just download it first and then transfer it, and start at step 3 below.
- On the Downloads page, copy the link for the latest version of the APM Connector.
Log in to your Ubuntu system and download the connector, for example:
wget https://support8.gwos.com/gw/gw8/files/8.1.3/27729688/27729687/1/1607729558712/apm-connector-8.1.3-GA.tgz
Expand the tar in a directory of your choice:
tar zxvf apm-connector-8.1.3-GA.tgz
This will make an "apm" subdirectory.
Modify the
tcg-apm.service
file to point to your apm subdirectory, and (at least on Ubuntu) include the Install section as noted below. For example if you expanded it in/home/ubuntu
, the file will need to look like this:[Unit] Description= TCG APM Service Documentation= https://support8.gwos.com/gw/doc/latest/monitoring-methods/application-performance-monitoring [Service] # Configure to match TCG installation WorkingDirectory=/home/ubuntu/apm Group=docker # Set environment (path) Environment= 'TCG_CONFIG=/home/ubuntu/apm/tcg_config.yml # TCG up ExecStart= /home/ubuntu/apm/apm-connector # TCG down ExecStop=/bin/kill -2 $MAINPID [Install] WantedBy=multi-user.target
Copy the tcg-apm.service file into the systemd directory:
sudo cp tcg-apm.service /etc/systemd/system/
CODEEnable the service:
sudo systemctl enable tcg-apm
CODEStart the service:
sudo systemctl start tcg-apm
CODE- Now you can create a connection to this running connector, and configure it as in the Setting Up the Connector section above. Note the hostname will need to be supplied, and of course you will need to be able to connect to that host on port 8099.
Related Resources
-
Page:
-
Page:
-
Page:
-
Page:
-
Page: