Elastic Connector with Transit Connection Generator (TCG)

The Elastic Connector is implemented in the GroundWork Transit Connection Generator (TCG), and allows a unique type of service to be deployed into GroundWork Monitor, Elasticsearch Lucene queries.

While we won’t be describing how to create more than an example or two in Elasticsearch, we will show you exactly how to take your searches through data you get from logs, traps, and other systems into Elastic and bring it directly into GroundWork, unifying the monitoring with Open Source Software across the enterprise.

First, a short background on the components.

Elasticsearch, Logstash and Kibana

To put it simply, Elasticsearch is a database, Logstash is a forwarder, and Kibana is a query engine. Together, they are the Elastic Stack. You can run them together on your own hardware or in cloud services, and when you do you have the ability to aggregate logs and events, and store them for searching and analysis.

GroundWork 8.x bundles these open source components in containerized versions you can use for lightweight aggregation. We use them as a way to monitor the containers in GroundWork itself, which is a great example. For anything larger, however, you will need a separate Elastic cluster, typically composed of several hosts (or vms, or containers) to handle the traffic from multiple systems all reporting.

So much for the Elastic components. On the GroundWork side we use TCG to connect Elastic Stack to GroundWork. The queries you save in Kibana are available to a connected GroundWork server to be brought in as services, with the only metric being the number of matches returned. This may sound simple, but it is beautiful that way. The matches can be anything you can search up, such as failed logins in the last 10 hours, or cache misses less than 5 minutes old. The range of possibilities is huge. We simply enable you to see these issues, and get alerts and reports on them in context with the other monitoring you are doing with GroundWork connected applications.

Requirements

You will need:

GroundWork server: 8.1.0 or later
TCG Elastic Connector binary: For your platform
Supported platforms

Currently only Linux is supported.
An Elastic cluster for testing or lightweight aggregation: You can use the existing internal GroundWork Elastic Stack
Network connectivity between the Elastic cluster, Kibana, the TCG system, and the GroundWork server.

An Example

For an example, we will configure the existing Elastic Stack inside of GroundWork with a few queries, and connect them as services to the GroundWork server itself. We will run TCG on the GroundWork server host for simplicity, so in our example, it will all be running on one system. In production, it doesn't matter where you run TCG, as long as it meets the connectivity requirements. The GroundWork host is a good choice, but so is an independent host, or even a container or other instance.

Example preparation

In order for this specific case to work, you will need to adjust the setting for the GroundWork Elasticsearch and Kibana nodes so they are accessible from the local GroundWork host. Normally, you would not do this, since GroundWork doesn't allow access to these nodes from outside the container network, but this example will give you a good idea of what you need to do in terms of exposing the required hosts and ports to TCG. TCG itself uses a configurable port, defaulted to 8099.

To prepare your example on the GroundWork server:

Edit the docker-compose.override.yml file in the gw8 directory, and add the lines to the services section, including the services: line if it's not already present:
```
services: 
  elasticsearch:
     ports:
       - "9200:9200"

  kibana:
     ports:
       - "5601:5601"
```
As this is a .yml file, pay special attention to the indenting. You might get errors on the next step if the format is wrong.
Restart GroundWork:
```
$ docker-compose down
$ docker-compose up -d
```
Production system

Don't forget to comment these lines out and restart again when you configure your production system. This is just an example! If you do leave it in place, please consider fire-walling the ports on the host so other actors cannot access the Kibana and Elasticsearch instances running in GroundWork.

Install Steps

Download the TCG Elastic Connector to the same Linux system.
- To download TCG go to Downloads.
- The download contains three files in a tar archive.
  - Un-tar these files into a directory called elastic. We recommend this directory be a subdirectory of the tcg directory at the same level and user as the GroundWork gw8 directory. For example, if you run GroundWork as user gwos under the /home/gwos directory, extract files into /home/gwos/tcg/elastic, like this:
    /home ─── /gwos ─── /tcg ─── /elastic/ <connector files> └── /gw8
    Running more than one connector
    
    If you have the need to run more than one connector, you can. You do need to run them in separate directories, as the NATS pools will conflict if you run them in the same location.
  - The files you will have after expansion include:
    - elastic-connector : The binary of the connector itself
    - tcg-config.yaml : The default configuration file for the connector
    - tcg-elastic.service : The systemctl services file for making TCG run at system startup
To enable the Elastic Connector to run, first change the mode to executable on the binary:
```
chmod +x elastic-connector
```

It's convenient to run the connector as a system service, so you can start and stop it with the service command, and have TCG start automatically when the system restarts. To do this, follow these steps:

Modify the tcg-elastic.service file. Change the lines:

WorkingDirectory=
User=
Group=

# Set environment (path)
Environment='TCG_CONFIG=tcg-config.yaml'

# TCG up
ExecStart=elastic-connector

to reflect the location in which you have installed TCG. Using the example above, you would set this as:

# Configure to match TCG installation
WorkingDirectory=/home/ec2-user/tcg/elastic
Group=docker

# Set environment (path)
Environment='TCG_CONFIG=/home/ec2-user/tcg/elastic/tcg-config.yaml'

# TCG up
ExecStart=/home/ec2-user/tcg/elastic/elastic-connector

# TCG down
ExecStop=/bin/kill -2 $MAINPID

Copy the tcg-elastic.service file to the systemctl services directory and make root the owner:

$ sudo cp tcg-elastic.service /etc/systemd/system/
@ sudo chown root:root /etc/systemd/system/tcg-elastic.service

Enable the service to run at startup:

$ sudo systemctl enable tcg-elastic.service

Start the connector:

$ sudo service tcg-elastic.service start

You can see the results by looking at the log:
```
$ sudo journalctl -u tcg-elastic.service
```
At this point, the connector is started in a waiting state, which you will see indicated in the last line of the log file (empty GWConnections). It's listening on the configured port you listed in tcg-config.yaml, by default 8099. Next, you will need to connect GroundWork to this instance of TCG, and generate a connection to the Elastic and Kibana servers.
The first thing you will need (if you don't already have it for other connectors), is a credentialed user. This is easy to create, just like a normal user, under the Administration > Users menu. The username and password you use does not matter, however make sure that you assign it the Operator role. Also, make sure you check the box for a “credentialed user” in the dialog.
One credentialed user

There can be only one such Credentialed user in any GroundWork server, and it must be a local (not an LDAP-controlled) user account.
Within the GroundWork Monitor UI, go to Configuration > Connectors, and click the Add button, then select Elastic from the listed connector types:

Fill out the form as follows:

Field/Option	Purpose
Enabled?	This is a way to either prepare the connector for operation and not start it, or to stop it temporarily. It is effective when you click Save. Generally you leave this checked.
Connector Name	This is up to you, but you can't have two connectors named the same.
TCG Host Address	Since you can run TCG on any accessible Linux system, this is where you specify where that is, from the point of view of the GroundWork containers. Note we will use the host name where GroundWork is running in this example and not localhost, since GroundWork is running in a container and localhost is the local container, not the host. Make sure you specify the hostname or IP address and port number, like this: 172.17.0.1:8099 This IP will work, since the host docker is running on appears at this address from the container point of view. You can also use the host IP or dns name, just be sure to add the port.
Server Names	This is a list of Elastic cluster nodes, from the point of view of the connector. Since we are running on the same server, again we use the host name, and the default port of 9200. Note you need to press Enter on each node name to be able to save it.
Interval	The number of minutes between polling for query results. Minimum is 1. We recommend 5.
Timeout	The number of seconds to wait for a connection. Depending on how busy your systems are, this might be as long as 5-10 seconds. 10 is the conservative default.
Retries	This has a checkbox and a numerical input. If you uncheck it, the connector will keep retrying as long as it is running. If you want the related services to show they are not receiving updates when the Elastic cluster is not responding to the connector's attempts, however, leave it checked and leave a relatively low number of attempts. This will ensure your service will go to an Unknown state, and allow you to correct the issue with Elastic Stack.
Kibana Server	Since you can connect to any Kibana server, this is where you input its address and port. Note this is also from the connector's point of view: you are telling the connector where to look for its Kibana server. We use the host name and default port of 5601 in our example.
Kibana User Name and Kibana Password	On GroundWork with Kibana exposed, these are not used. If your Kibana server requires authentication, you should provide a valid username and password that can access the queries you need to use as services.
Always override time filter	This is an advanced option. It goes with the next two options:
Time Filter, from and Time Filter, to	Time range to use as an override from the query time range. You can leave this off unless you always want to use the same interval for historical queries with this connector.
Host Name Prefix	Keeping track of the hostnames generated by connectors is done by using prefixes. You can tell this connector to use a unique prefix, so the hosts it creates are identifiable.
Host Name Label Path	You can give the connector hostnames to create with a label in Kibana. Just leave this at the default for now.
Host Group Label Path	You can give the connector hostgroup names to create with a label in Kibana. Just leave this at the default for now.
Log Level	Unless you are having issues with this connector, leave this at the Error level.

Save the connector. If all is well, the dialog will close and the connector status will turn Green. If there are issues, then a message will appear with an idea about what is wrong, allowing you to correct it and move on.

If you see the status of the connection is red, or grey, please check all your inputs. Also you may need to go to GroundWork Connections and click Connect on the new connector to get it to turn green after a configuration issue.

Setting up metrics queries

At this point, your Elastic connector is set to capture the results of queries. For this to work, you need some active queries in Kibana. Here's how you create some to get started. If you are already familiar with Kibana, you can create a few queries of your own and skip to Defining metrics below.

Go to Dashboards > Log Analysis, this will log you in to Kibana.
Along the left side of the screen, click the last icon Management.
Select Index Patterns and click Create index pattern.
In the first step for Index pattern type logstash-*.
Click Next step, and select @timestamp from the time filter field drop-down, and click Create index pattern:
Click the top icon for Discover, and you can enter search text to create your own queries.
- We have included a couple of sample queries for you. To import these:
  - Download the export.ndjson and check the MD5 sum matches the comment on the attachment.
  - Click Open, then Manage searches.
  - Click Import, then select and import the file. You should see a message for successful import, and then there should be two new queries and a new search in the list:
  - Searches are useful in setting up dashboards and visualizations in Kibana. If you want to use a search in GroundWork as a service, save it as a query.
  - If you have not yet used Kibana, we suggest you explore these queries. Note the options on the host filter example, where you can specify which hosts the query applies to, and the fact a search can be saved as a query. It will benefit you in future to gain some familiarity with creating queries now. Kibana is a very powerful log analysis tool, and this example will help you gauge its value to your organization.

Defining metrics

Here's how to set up the queries you entered or imported as services in GroundWork.

In GroundWork Monitor, go to Configuration > Connectors and select the connector you created above.
Select the Metrics tab.
Click Add Metric (plus icon), this brings up the New Metric dialog.
Start typing the name of the query you want to add as a metric. All matching names will show in the drop-down list. Select the one you want.

You can fill out the form as follows:

Field	Purpose
Metric name	The actual name of the query you imported or created in Kibana, e.g., query_error
Metric Format String	An optional c-style format descriptor, e.g., %d.2
Display name	An optional name for the service that will report this metric, it will use the metric name if you don't enter one
Monitor	Enable or disable the monitoring of this metric
Graph	Enable or disable the graphing of this metric
Delta	Treat this metric as a delta from the last time (generally used for numbers that are increasing)
Default Warning Threshold	Set a number above which (or below which, if more than the Critical threshold) the service that reports on this metric will be in a Warning state. Note you can override this with the Edit function in individual cases from the Status Summary dashboard. This is optional, and a -1 will disable it.
Default Critical Threshold	Set a number above which (or below which, if less than Warning threshold) the service that reports on this metric will be in Critical state. Note you can override this with the Edit function in individual cases from the Status Summary dashboard. This is optional, and a -1 will disable it.
Description	Optional text describing this metric

Click Update Metrics to save your new metrics.
Browse to Status to see your new metrics start to populate on the interval you set for the connector. These metrics will show up against the GroundWork containers, since these are the only hosts logging in Elastic in this example.

There are many possible uses for the Elastic connector. For example, Elastic can receive log messages from specific hosts using Filebeat or Winlogbeat agents. You can then query for critical messages using Elastic, and report them into GroundWork using the Elastic connector.