Statistics Logger? for TIBCO Enterprise Message Service?

Introduction

This tool can be used to log and monitor EMS (including EMS Appliance) server statistics. Server Statistics can be logged to local CSV files and can additionally be pushed to either AWS CloudWatch, InfluxDB or to the Prometheus monitoring tool.

The package contains the following Java classes:

• EmsStatsLogger: Regularly polls and logs EMS server statistics to CSV files, which can easily be graphed in tools such as Excel. A new CSV file is generated for each day.
• EmsStatsPromCollector: This class collects EMS server statistics for export to Prometheus. Statistics can also be optionally logged to a local CSV file. Prometheus is a powerful open source monitoring system (see https://prometheus.io)
• EmsStatsCloudWatchPusher: This class pushes EMS server statistics to AWS CloudWatch. Statistics can also be optionally logged to a local CSV file.
• EmsStatsInfluxPusher: This class pushes EMS server statistics to InfluxDB. Statistics can also be optionally logged to a local CSV file.

The following EMS server metrics are collected:

• Inbound/outbound message/byte rates
• Pending message count/size
• Connection/session counts
• Destination counts
• Message memory usage
• Disk read/write rates
• Sync/async datastore sizes
• Response time

The following EMS queue/topic metrics can be monitored:

• Inbound/outbound message/byte rates
• Inbound/outbound total message/byte counts
• Pending message count/size
• Receiver/Consumer counts

The EMS queue/topic metrics are documented EMS Admin API DestnationInfo, QueueInfo, TopicInfo and StatData classes, for further details see:

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/ServerInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/QueueInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/TopicInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/StatData.html

The package also contains the following sample Grafana dashboards for visualising the statistics stored in Prometheus and InfluxDB (for further details on Grafana see https://grafana.com)

• EMS Server Overview (graphs in/outbound message rates, pending message size/count, connection count and response time)
• EMS Details (graphs memory usage, sync/async datastore size and disk read/write rates)
• EMS Queues (graphs queue in/outbound message rates, pending message size/count, receiver counts)
• EMS Topics (graphs queue in/outbound message rates, pending message size/count, consumer counts)

Example Grafana dashboards are shown below:

An example AWS CloudWatch dashboard is shown below:

Download the tool here emsStatsLogger3.0.zip

Version History

Version 3.1

•  Added EmsStatsInfluxPusher to push stats to InfluxDB
•  Added Grafana dashboards for InfluxDB

Version 3.0

•  Added support for monitoring multiple EMS servers
•  Added support for SSL connections
• Added support for configuration with servers.xml file
• Added support for monitoring queues and topics
• Added support to auto cleanup old log files
• Changed emsStatsPromPusher to emsStatsPromCollector (Prometheus now directly scrapes the collector for stats)
• Updated Grafana dashboards, including new dashboards for Queues and Topics

Version 2.1

•     Added logfile option to override log file name
•     Added noappend option to prevent append to log file on restart

Version 2.0

•     Added emsStatsCloudWatchPusher

Software Requirements

• JRE 1.8 or higher
• TIBCO Enterprise Message Service 8.1 or higher (client)
• For emsStatsCloudWatchPusher:
  • AWS SDK for Java

To rebuild the source for the emsStatsPromPusher class, the following libs are also required:

• Prometheus simpleclient, simpleclient_common and simpleclient_pushgateway libs available at https://github.com/prometheus/client_java

Installation

Follow this procedure to install this tool into your environment:

• Unpack the emsStatsLogger.zip file.
• Edit the servers.xml file to add connection detail for the EMS servers to be monitored.
• Edit the run scripts to make appropriate versions for your environment:
  • Set the TIBEMS_ROOT variable to your TIBCO EMS installation directory (EMS client 8.1 or greater required)
  • Set the appropriate command line options for your environment

Installation for Prometheus/Grafana

• Download Prometheus and the PushGateway from: https://prometheus.io/download
• Edit the prometheus.yml file and configure a scrape config for the collector targets. There are three collector targets, one for EMS server stats, one for queue stats and one for topic stats.
• Configure the collector targets appropriately for where the EmsStatsPromCollector is running, the example below show default configuration if the collector is running on the same host as the Prometheus server.
• Remove the queue/topic collector target if not required.

1.  scrape_configs:   - job_name:       'TIB-EMS'     scrape_interval: 60s     scrape_timeout: 10s     honor_labels: true     static_configs:       - targets: ['127.0.0.1:9091', '127.0.0.1:9092', '127.0.0.1:9093'] 

You may add multiple scrape configs for different EmsStatsPromCollector instances, for example for different EMS environments.

Ensure the job names are different and match those configured using the command line when ruuning the collector.

By default EmsStatsPromCollector listens on HTTP port 9091 for exporting EMS server stats, 9092 for queue stats, and 9092 for topic stats.

The Prometheus server listens on TCP port 9090 by default.

Edit the runPromCollector.sh file and configure appropriate HTTP ports if necessary for your environment.

• Use the -nolog command line option to prevent logging to local file if required.

Note; you no longer need to run the Prometheus gateway, the prometheus server now scrapes the EmsStatsPromCollector directly.

Example command line to run prometheus:

1.  prometheus --config.file=prometheus.yml 

To Install Grafana:

• Follow the instructions here: http://docs.grafana.org/installation
• Take a copy of the grafana.ini file from the installation (e.g. /usr/local/etc/grafana) and edit appropriately, simple example shown below.
• Open the grafana URL in a browser and create a prometheus data source, then simply import the sample dashboards provided or create your own.
• Note by default grafana listens on TCP port 3000, default user/password is admin/admin

Example command line to run grafana:

1.  /usr/local/Cellar/grafana/4.6.3/bin/grafana-server -homepath /usr/local/share/grafana --config grafana-ems.ini

Example grafana configuration file:

1.  [paths]   data = data   logs = data/log/grafana_log  [server]   http_addr = 0.0.0.0   http_port = 3000  [dashboards.json]   enabled=true   path=grafana_dashboards

Installation for InfluxDB

Download influxdb-java-2.15 client and dependencies downloaded from:

https://jar-download.com/artifacts/org.influxdb/influxdb-java/2.15

InfluxDB client libraries required:

• converter-moshi-2.5.0.jar
• influxdb-java-2.15.jar
• logging-interceptor-3.13.1.jar
• moshi-1.5.0.jar
• msgpack-core-0.8.16.jar
• okhttp-3.13.1.jar
• okio-1.17.2.jar
• retrofit-2.5.0.jar

Installation for AWS CloudWatch

If running the tool on EC2:

• Install AWS SDK using the command
  • wget http://sdk-for-java.amazonwebservices.com/latest/aws-java-sdk.zip
• Create an IAM role and then give your EC2 instance access to that role as shown in:
  • https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-roles.html
• Edit the runCloudWatchPusher.sh file and set the appropriate configuration for your environment.
  • Use the -region option to send stats to a different region if required
  • Use the -nolog command line option to prevent logging to local file if required

You can also run the tool on-premise and push stats to AWS:

• Install the AWS SDK library which can be downloaded here:
  • https://github.com/aws/aws-sdk-java
• Edit the runCloudWatchPusher.sh file and set the appropriate configuration for your environment.
  • Use the -accessId -secretKey and -region options for you AWS environment
  • Use the -nolog command line option to prevent logging to local file if required

EmsStatsLogger

The EmsStatsLogger class logs EMS server statistics to an output CSV file once every poll interval. By default a new output log file is created for each day, this may be overridden using the logfile parameter.

 Usage: java EmsStatsLogger [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml   -interval  <secs>              - interval between log entries, default 30 seconds   -debug                         - Enable debug trace to stdout 

EmsStatsPromCollector

The EmsStatsPromCollector class inherits from EmsStatsLogger and in addition to logging EMS statistics to file also collects the statistics for monitoring with Prometheus. Logging to file can be disabled if not required.

You may set a job name to represent the EMS environment being monitored, eg: PROD, QA, DEV etc.

Ensure you add a prometheus scrape target for each environment/job and the job names match those configured in the prometheus.yml file.

 Usage: java EmsStatsPromCollector [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -interval  <secs>              - interval between log entries, default 30 seconds   -serverStatsPort  <port>       - The HTTP port to scrape for EMS server stats, default 9091. (0 disables)   -queueStatsPort   <port>       - The HTTP port to scrape for EMS queue stats, default 9092. (0 disables)   -topicStatsPort   <port>       - The HTTP port to scrape for EMS topic stats, default 9093. (0 disables)   -job              <job-name>   - The Prometheus job-name, default TIB-EMS   -nolog                         - Prevent all logging of stats to log directories, default false 

EmsStatsInfluxPusher

The EmsStatsInfluxPusher class inherits from EmsStatsLogger and in addition to logging EMS statistics to file also sends the statistics to InfluxDb. Logging to file can be disabled if not required.

 Usage: java EmsStatsInfluxPusher [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -dbURL     <DB URL>          	 - Infux DB URL (default http://localhost:8086)   -username  <username>          - Influx DB usnername (default admin)   -password  <password>          - Influx DB password (default admin)   -dbName    <DB Name>           - The Influx DB name (change for different environments) default TIB-EMS"   -nolog                         - Prevent all logging of stats to log directories, default false 

EmsStatsCloudWatchPusher

The EmsStatsCloudWatchPusher inherits from emsStatsLogger and in addition to logging EMS statistics to file also sends the statistics to AWS CloudWatch. If you are not running the tool on AWS you must supply the accessId, secretKey and region parameters for your AWS environment. Logging to file can be disabled if not required.

 Usage: java EmsStatsCloudWatchPusher [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -interval  <secs>              - interval between log entries, default 30 seconds   -accessId  <accessId>          - AWS access Id (default environment credentials)   -secretKey <secretKey>         - AWS secret key (default environment credentials)   -region    <region>            - AWS region (default current region)   -ns        <namespace>         - AWS namespace (default TIBCO/EMS)   -nolog                         - Prevent all logging of stats to log directories, default false