This page reviews how to set up and monitor your Amazon EC2 infrastructure using the GroundWork Cloud Hub AWS connector. The connection requires a unique set of parameters (e.g., endpoint, credentials). If you are connecting to a remote GroundWork server to send results, you will need your remote GroundWork server RESTACCESSAPI token.

Adding an AWS Connection

To access Cloud Hub configuration, log in to GroundWork Monitor as a member of the Admin role (e.g., user admin), and select Configuration > Cloud Hub. To add a new connection click the +Add button next to the desired connector icon. You will need to create a new connection in this way for each project to be monitored.

Connection Parameters

The data the GroundWork server receives comes from the remote virtualization server. The information is pulled from the API on a periodic basis based on the check interval that is set. In the configuration page you will need to enter both the GroundWork Server and AWS Connector parameters, and select available Views (resources to be monitored). And just as a note, the   link located at the top right side provides information and versioning for the selected Cloud Hub connector.

connection parameters

GroundWork Server

The GroundWork Server can simply be the same as the one you are running the Cloud Hub connector on, or it can be a remote server. If it's the same as the one you are running on, leave the directive Use Local Connection checked. 

Otherwise, uncheck this box and fill in the hostname of the remote GroundWork server in the Hostname field, leave RESTAPIACCESS in the Username field, and paste in the the encrypted Token. The token can be obtained on the remote GroundWork server, for users within the Admin role, by going to Administration > Security, under Webservices API Account: RESTAPIACCESS, Encrypted Token. Just copy the key from the remote server into the Token field on the Cloud Hub server.

restapiaccess token

Once you have the GroundWork Server side of the form filled out, click Test. If you have the credentials correct and you have access to the API, you will see a Success message. Any error message will give you a hint as to what is wrong and let you try again. 

Using a remote server will populate the remote server with the AWS monitoring data, and this will not show in the local GroundWork Server. 

  • Version: Indicates the minimum GroundWork Monitor version needed. In other words, a version below the indicated value is incompatible.

  • Hostname: The host name or IP address where a GroundWork server is running. A port number should not be entered here. If GroundWork is running on the same server, you can enter localhost.

  • Username: The provisioned Username granted API access on the GroundWork server.

  • Token: The corresponding API Token for the given Username on the GroundWork server, see Administration > Security under Webservices API Account: RESTAPI Encrypted Token.

  • SSL: Check this box if the GroundWork server is provisioned with a secure HTTPS transport.

  • Merge Hosts: If checked, this option combines all metrics of same named hosts under one host. For example, if there is a Nagios configured host named demo1 and a Cloud Hub discovered host named demo1, the services for both configured and discovered hosts will be combined under the hostname demo1 (case-sensitive).

  • Monitor: If checked, enables connection to be monitored. Gives you a way to know when the connector is having trouble reaching the endpoint by creating a service on the host it reports to. 

  • Use Local Connection: This directive refers to where the Cloud Hub results are sent. If this field is checked, results will be posted to the same server as where Cloud Hub is running. Or, with this field unchecked, you can forward results to any accessible GroundWork server you define with the name and API key.

  • Ownership: Ownership is the owner of a connectors hosts and the ownership can be switched. When a Cloud Hub connector is instantiated the following options are available for ownership:

    Always take ownership: The connector will assume ownership of all hosts it instantiates, even merged hosts. This will remain true even if another app merges the host. 

    Leave ownership if already owned: The connector host will remain with the existing owner until or unless the owner deletes the host.

    Always defer ownership (default): This option leaves ownership unchanged on merged hosts, and allows other apps to take ownership.

    Note that multiple apps can report on a single service, but only one can own the host.

    See Ownership options.

  • Connection Status: Click Test to verify a connection using the GroundWork server entries.

Remote Server

Next you will need to fill in the AWS Connector parameters. If your GroundWork Monitor server is running in AWS, you can assign an IAM role to the server to grant access to the AWS services you wish to monitor in great detail. If not, you will need to gather the details of the connection to AWS, including the AWS Region Endpoint, AWS Access Key ID, and an AWS Secret Key ID.

Enter a Display Name and the AWS Region Endpoint which is where you access the CloudWatch API. 

Continue and enter the access key fields AWS Access Key ID and AWS Secret Access Key.

If you are running Cloud Hub on an AWS server, you don’t need to provide these credentials. You can simply check the box to Enable IAM Roles and the authentication is automatic. To do so, you must configure an IAM role, assign a policy that allows you to list the resources you wish to monitor, then assign this role to the AWS instance. 

A typical permissive policy for monitoring AWS resources is, for example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "sns:*",
                "elasticloadbalancing:DescribeSSLPolicies",
                "autoscaling:Describe*",
                "elasticloadbalancing:DescribeTags",
                "rds:List*",
                "logs:*",
                "elasticloadbalancing:DescribeLoadBalancerPolicyTypes",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:DescribeLoadBalancers",
                "ec2:Describe*",
                "rds:Describe*",
                "elasticloadbalancing:DescribeTargetGroupAttributes",
                "elasticloadbalancing:DescribeListeners",
                "cloudwatch:*",
                "elasticloadbalancing:DescribeAccountLimits",
                "elasticloadbalancing:DescribeLoadBalancerPolicies",
                "elasticloadbalancing:DescribeTargetHealth",
                "elasticloadbalancing:DescribeTargetGroups",
                "elasticloadbalancing:DescribeListenerCertificates",
                "elasticloadbalancing:DescribeRules",
                "elasticloadbalancing:DescribeInstanceHealth"
            ],
            "Resource": "*"
        }
    ]
}

You can of course configure your own policies. This is an advanced feature of AWS, and not covered by this documentation. 


You can optionally Enable Host Group Tagging. Check this option to enable support for Amazon EC2 and EBS tagging as a mapping mechanism for host groups, defined in the next field. For more information see Appendix A: Host group tagging. Optionally Enable SSL, and also optionally set the Interval and Retry directives.

Then, validate the connection by clicking Test. A dialog will be displayed with either a Success message or, if the project cannot be contacted, an error message will be displayed with a hint as to why the connection failed.

Make sure to click Save in the upper right corner to save your correct connection parameters. 

Then, you need to select the resources to monitor, Views. These features are the core components, or services that are managed by AWS. AWS Views include Storage View which shows instances according to the Amazon EBS resources where they are assigned, Network View shows instances as they are assigned to Virtual Private Cloud subnets and Relational Database Service (RDS) metrics, and Custom View enables a query that will retrieve all active custom CloudWatch metrics in the zone being connected. And again, click Save when finished.

After the credentials have been validated and the resources indicated, select the Metrics link (top navigation) to start customizing metrics for the connection, refer to the document Customizing Metrics.

  • Display Name: This is the configuration’s name displayed in the list of Cloud Hub connectors on the Cloud Hub home page.

  • AWS Region Endpoint: This is the Web Service endpoint for a region (e.g., us-west-2.amazonaws.com for region Oregon). Create a connector like this for each region deployed to point to the endpoint for that region. The endpoint is where you access the CloudWatch API.

  • Enable AIM Roles: 
    AWS Access Key ID
    This can also be a common user name set up for command line access (does not have to be the master account). It should be clear that the user name must be assigned rights to the region that you wish to monitor.

    AWS Secret Access Key
    This can also be a common password set up for command line access (does not have to be the master account). It should be clear that the password must be assigned rights to the region that you wish to monitor.

  • Enable HostGroup Tagging?: Cloud Hub can map your AWS resource tags to GroundWork host groups. This entry represents the key name for the GroundWork tag. All instances with this tag key will be mapped to the GroundWork Monitor host group (e.g., gwhostgroup) with the EC2 tag value (e.g., GWHostGroup). There may be multiple host groups with the same tag name and tag names are case-sensitive. Refer below to Appendix A: Host group tagging.

  • Enable SSL: Check this box if the Amazon server is configured for secure HTTPS.

  • Cloud Hub Interval (min): This is the polling interval for collecting monitoring data from the virtual instance and sending it to the GroundWork server. It defines how often Cloud Hub will query Amazon CloudWatch for change updates. The value is in minutes.

  • Infinite Retries: This entry is the number of retries for the connection and sets a limit on how many attempts are made after a failure. If you set this to -1, the retrying goes on forever. The number set indicates how many connections are attempted before the connection is left inactive (until you restart it).

  • Retry Limit: This entry is the number of retries for the connection and sets a limit on how many attempts are made after a failure. The number set indicates how many connections are attempted before the connection is left in an inactive state. At this point, the connection is suspended and you will need to manually restart it. When a retry limit is exhausted, all hosts managed by this connection are set to the monitor status Unreachable and all services for the matched hosts are set to the status of Unknown.

Appendices

Appendix A: Host group tagging

To help manage and group your EC2 instances and EBS images, AWS allows you to assign your own metadata to each resource in the form of tags. A tag is a key and value pair of user defined strings that can be assigned to AWS resources as mapping mechanism for host groups. Tags enable you to categorize your AWS resources in different ways, for example, by purpose, owner, or environment. This is useful when you have many resources of the same type — you can quickly identify a specific resource based on the tags you've assigned to it. For example, with a tag Key name of GWHostGroup, and a tag Value gwservers, all instances assigned the tag key GWHostGroup are placed in the host group called gwservers.

Tagging notes

  • Host Groups are deleted when empty as is the case for all such host groups automatically created by Cloud Hub.
  • A host can be in more than one group, so if you create tagging and only want to display the tagged group - you can allow access to specific Host Groups and Service Groups in the portal membership management. For example, restricting user operator to just the gwservers host group would result in the Cloud Hub discovered host groups to not display in such applications as Status.
  • Tag key and value strings are case-sensitive.
  • Resources (e.g., EC2 instances, EBS volumes) can have one or more (up to 50) tags each.

Enabling tagging

Cloud Hub can map your AWS resource tags to GroundWork Host Groups. To utilize this feature you need to enable tagging and set a Host Group tag name. To enable tagging, go to the connection configuration and check the Enable HostGroup Tagging? checkbox. In the next field, enter a Host Group Tag Name to match a tag key name configured in EC2.

Configuring tagging in the console

Along with the Cloud Hub configuration above, to tag EC2 instances or EBS volumes you will need to configure tags in the EC2 console.

Launch and sign in to the Amazon EC2 console. If necessary, in the upper right corner change the location of the resources to the desired region. In the navigation pane, select Tags. Click the Manage Tags button. Select the instances you would like to tag (e.g., checked in blue) and assign a tag key (e.g., GWHostGroup) and a tag value (e.g., gwservers), and select Add Tag.

The tag key (e.g., GWHostGroup) is then used in the Cloud Hub connector configuration (from above), and the tag value (e.g., gwservers) becomes the name of the host group for the associated instances. Alternatively, you can navigate to individual EC2 instances and assign tags specifically to that one instance from the Tag tab. Similarly, Elastic Block Store volumes can also be assigned tags.

Appendix B: Notes on AWS metrics

Elastic Block Store (EBS) metrics

  • EBS metrics are now attached to corresponding hosts. A host can have one or more EBS metrics. Previously, EBS metrics were gathered in their own groups which was causing unused volume metrics to be reported.
  • EBS metric naming has been changed to prefix service names on EC2 instance with: EBS.{volume-name}.{metricname}
  • EBS metrics are only added to EC2 instances when Storage View on the connection configuration page is enabled
  • EBS host groups and hosts have been removed
  • When unchecking Storage View, all EBS services are deleted
  • Supported EBS metrics:
    • EBS.VolumeIdleTime: Total seconds spent by in Idle time on EBS device
    • EBS.VolumeQueueLength: Total Read and Write Operations on EBS device
    • EBS.VolumeReadBytes: Total Bytes Read on EBS device
    • EBS.VolumeReadOps: Total Read Operations on EBS device
    • EBS.VolumeTotalReadTime: Total seconds spent by Read operations completed on EBS device
    • EBS.VolumeTotalWriteTime: Total seconds spent by Write operations completed on EBS device
    • EBS.VolumeWriteBytes: Total Bytes Written on an EBS device
    • EBS.VolumeWriteOps: Total Write Operations on EBS device

Relational Database Service (RDS) metrics

  • When Storage View is enabled, RDS hosts are added to a host group named AWS-RDS:storage
  • No unused RDS volume metrics are reported, such as volumes that exist in the zone but are not attached to any running instance
  • When unchecking Storage View, all RDS hosts and services are deleted, and the RDS host group is deleted
  • RDS metrics are only associated with an RDS host
  • Supported RDS metrics:
    • RDS.BinLogDiskUsage: Disk space occupied by binary logs on the master RDS node
    • RDS.DatabaseConnections: Number of database connections currently in use on RDS
    • RDS.DiskQueueDepth: Number of outstanding IOs (read/write requests) waiting to access RDS disks
    • RDS.FreeStorageSpace: Amount (bytes) of available Storage space on RDS service
    • RDS.FreeableMemory: Amount (bytes) of available RAM on RDS service
    • RDS.NetworkReceiveThroughput: Incoming (Receive) network traffic on the DB instance in bytes/second, includes both database and RDS traffic
    • RDS.NetworkTransmitThroughput: Outgoing (Transmit) network traffic on the DB instance in bytes/second, includes both database and RDS traffic
    • RDS.ReadIOPS: Average number of disk I/O read operations per second (RDS)
    • RDS.ReadLatency: Average amount of time in seconds taken per disk Read I/O operation (RDS)
    • RDS.ReadThroughput: Average number of bytes read from disk per second (RDS)
    • RDS:ReplicaLag: Amount of time a Read Replica DB Instance lags behind the source DB Instance
    • RDS.SwapUsage: Amount (bytes) of swap space used on the DB instance
    • RDS.WriteIOPS: Average number of disk I/O write operations per second (RDS)
    • RDS.WriteLatency: Average amount of time in seconds taken per disk Write I/O operation (RDS)
    • RDS.WriteThroughput: Average number of bytes written to disk per second (RDS)

CloudWatch custom metrics

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. Amazon CloudWatch is used to collect and track metrics for all standard AWS resources such as EC2 instances, Relational Database Service (RDS), and Elastic Block Store (EBS) storage volumes.

Additionally, CloudWatch can be extended to gather Custom Metrics generated by your applications and services and monitor these metrics for you. If your application is generating custom metrics, to have GroundWork Cloud Hub retrieve these metrics you'll need to turn on Custom View in your AWS connection configuration.

At the top of the screen click Refresh Custom to retrieve a list of custom metrics from your application.

It is best practice to use namespaces when registering your custom metrics with CloudWatch. This can help you separate your custom metrics from other metrics in the system. In the Cloud Hub configuration, you can treat these metrics like any other.

Related Resources