Statistics Logger for TIBCO Enterprise Message Service™

A tool to log and monitor EMS (including EMS Appliance) server statistics.  Display collected statistics in a dashboard to gain insights into your messaging system.  Server metrics can simply be logged to CSV files or pushed to AWS CloudWatch or to a Prometheus monitoring system for visualization with dashboards.

Compatible Products

TIBCO Enterprise Message Service™

Provider

TIBCO Software

Supported Versions

JRE 1.8 or higher

TIBCO Enterprise Message Service 8.1 or higher (client)

License

TIBCO Component Exchange License

Overview

This tool can be used to log and monitor EMS (including EMS Appliance) server statistics. Server Statistics can be logged to local CSV files and can additionally be pushed to either AWS CloudWatch or to the Prometheus monitoring tool.

The following EMS server metrics can be monitored:

  • Inbound/outbound message/byte rates
  • Pending message count/size
  • Connection/session counts
  • Destination counts
  • Message memory usage
  • Disk read/write rates
  • Sync/async datastore sizes
  • Response time

The following EMS queue/topic metrics can be monitored:

  • Inbound/outbound message/byte rates
  • Inbound/outbound total message/byte counts
  • Pending message count/size
  • Receiver/Consumer counts

For full details and reference pages to each, please see the corresponding wiki page.  Included in the package: a sample dashboard that gives you a visual representation of the collected statistics.

Release(s)

Release 3.0

Published: June 2018

V3.0 June 2018

There are currently no reviews for this content.

Why not be the first to review it - click here to login

Statistics Logger​ for TIBCO Enterprise Message Service™

Introduction

This tool can be used to log and monitor EMS (including EMS Appliance) server statistics. Server Statistics can be logged to local CSV files and can additionally be pushed to either AWS CloudWatch or to the Prometheus monitoring tool.

The package contains the following Java classes:

  • EmsStatsLogger: Regularly polls and logs EMS server statistics to CSV files, which can easily be graphed in tools such as Excel. A new CSV file is generated for each day.
  • EmsStatsPromCollector: This class collects EMS server statistics for export to Prometheus. Statistics can also be optionally logged to a local CSV file. Prometheus is a powerful open source monitoring system (see https://prometheus.io)
  • EmsStatsCloudWatchPusher: This class pushes EMS server statistics to AWS CloudWatch. Statistics can also be optionally logged to a local CSV file.

The following EMS server metrics are collected:

  • Inbound/outbound message/byte rates
  • Pending message count/size
  • Connection/session counts
  • Destination counts
  • Message memory usage
  • Disk read/write rates
  • Sync/async datastore sizes
  • Response time

The following EMS queue/topic metrics can be monitored:

  • Inbound/outbound message/byte rates
  • Inbound/outbound total message/byte counts
  • Pending message count/size
  • Receiver/Consumer counts

The EMS queue/topic metrics are documented EMS Admin API DestnationInfo, QueueInfo, TopicInfo and StatData classes, for further details see:

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/ServerInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/QueueInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/TopicInfo.html

    https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/StatData.html

 

The package also contains the following sample Grafana dashboards for visualising the statistics stored in Prometheus (for further details on Grafana see https://grafana.com)

  • EMS Server Overview (graphs in/outbound message rates, pending message size/count, connection count and response time)
  • EMS Details (graphs memory usage, sync/async datastore size and disk read/write rates)
  • EMS Queues (graphs queue in/outbound message rates, pending message size/count, receiver counts)
  • EMS Topics (graphs queue in/outbound message rates, pending message size/count, consumer counts)

Example Grafana dashboards are shown below:

 

An example AWS CloudWatch dashboard is shown below:

Download the tool here emsStatsLogger3.0.zip

Author: Richard Lawrence (rlawrenc@tibco.com)

Version History

Version 3.0

  •  Added support for monitoring multiple EMS servers
  •  Added support for SSL connections
  • Added support for configuration with servers.xml file
  • Added support for monitoring queues and topics
  • Added support to auto cleanup old log files
  • Changed emsStatsPromPusher to emsStatsPromCollector (Prometheus now directly scrapes the collector for stats)
  • Updated Grafana dashboards, including new dashboards for Queues and Topics

Version 2.1

  •     Added logfile option to override log file name
  •     Added noappend option to prevent append to log file on restart

Version 2.0

  •     Added emsStatsCloudWatchPusher

Software Requirements

  • JRE 1.8 or higher
  • TIBCO Enterprise Message Service 8.1 or higher (client)
  • For emsStatsCloudWatchPusher:
    • AWS SDK for Java

To rebuild the source for the emsStatsPromPusher class, the following libs are also required:

Installation

Follow this procedure to install this tool into your environment:

  • Unpack the emsStatsLogger.zip file.
  • Edit the servers.xml file to add connection detail for the EMS servers to be monitored.
  • Edit the run scripts to make appropriate versions for your environment:
    • Set the TIBEMS_ROOT variable to your TIBCO EMS installation directory (EMS client 8.1 or greater required)
    • Set the appropriate command line options for your environment

Installation for Prometheus/Grafana

  • Download Prometheus and the PushGateway from: https://prometheus.io/download
  • Edit the prometheus.yml file and configure a scrape config for the collector targets. There are three collector targets, one for EMS server stats, one for queue stats and one for topic stats.
  • Configure the collector targets appropriately for where the EmsStatsPromCollector is running, the example below show default configuration if the collector is running on the same host as the Prometheus server.
  • Remove the queue/topic collector target if not required.
  1. scrape_configs:
      - job_name:       'TIB-EMS'
        scrape_interval: 60s
        scrape_timeout: 10s
        honor_labels: true
        static_configs:
          - targets: ['127.0.0.1:9091', '127.0.0.1:9092', '127.0.0.1:9093']
    

     

You may add multiple scrape configs for different EmsStatsPromCollector instances, for example for different EMS environments.
Ensure the job names are different and match those configured using the command line when ruuning the collector.

By default EmsStatsPromCollector listens on HTTP port 9091 for exporting EMS server stats, 9092 for queue stats, and 9092 for topic stats.
The Prometheus server listens on TCP port 9090 by default.

Edit the runPromCollector.sh file and configure appropriate HTTP ports if necessary for your environment.

  • Use the -nolog command line option to prevent logging to local file if required.

Note; you no longer need to run the Prometheus gateway, the prometheus server now scrapes the EmsStatsPromCollector directly.

Example command line to run prometheus:

  1. prometheus --config.file=prometheus.yml
    

To Install Grafana:

  • Follow the instructions here: http://docs.grafana.org/installation
  • Take a copy of the grafana.ini file from the installation (e.g. /usr/local/etc/grafana) and edit appropriately, simple example shown below.
  • Open the grafana URL in a browser and create a prometheus data source, then simply import the sample dashboards provided or create your own.
  • Note by default grafana listens on TCP port 3000, default user/password is admin/admin

Example command line to run grafana:

  1. /usr/local/Cellar/grafana/4.6.3/bin/grafana-server -homepath /usr/local/share/grafana --config grafana-ems.ini

     

Example grafana configuration file:

  1. [paths]
      data = data
      logs = data/log/grafana_log
    
    [server]
      http_addr = 0.0.0.0
      http_port = 3000
    
    [dashboards.json]
      enabled=true
      path=grafana_dashboards

     

Installation for AWS CloudWatch

If running the tool on EC2:

You can also run the tool on-premise and push stats to AWS:

  • Install the AWS SDK library which can be downloaded here:

  • Edit the runCloudWatchPusher.sh file and set the appropriate configuration for your environment.
    • Use the -accessId -secretKey and -region options for you AWS environment
    • Use the -nolog command line option to prevent logging to local file if required

EmsStatsLogger

The EmsStatsLogger class logs EMS server statistics to an output CSV file once every poll interval. By default a new output log file is created for each day, this may be overridden using the logfile parameter.

Usage: java EmsStatsLogger [options]

   where options are:

  -config    <config file>       - Server config filename, default servers.xml
  -interval  <secs>              - interval between log entries, default 30 seconds
  -debug                         - Enable debug trace to stdout

 

EmsStatsPromCollector

The EmsStatsPromCollector class inherits from EmsStatsLogger and in addition to logging EMS statistics to file also collects the statistics for monitoring with Prometheus. Logging to file can be disabled if not required.
You may set a job name to represent the EMS environment being monitored, eg: PROD, QA, DEV etc.
Ensure you add a prometheus scrape target for each environment/job and the job names match those configured in the prometheus.yml file.

 

Usage: java EmsStatsPromCollector [options]

   where options are:

  -config    <config file>       - Server config filename, default servers.xml 
  -interval  <secs>              - interval between log entries, default 30 seconds
  -serverStatsPort  <port>       - The HTTP port to scrape for EMS server stats, default 9091. (0 disables)
  -queueStatsPort   <port>       - The HTTP port to scrape for EMS queue stats, default 9092. (0 disables)
  -topicStatsPort   <port>       - The HTTP port to scrape for EMS topic stats, default 9093. (0 disables)
  -job              <job-name>   - The Prometheus job-name, default TIB-EMS
  -nolog                         - Prevent all logging of stats to log directories, default false

EmsStatsCloudWatchPusher

The EmsStatsCloudWatchPusher inherits from emsStatsLogger and in addition to logging EMS statistics to file also sends the statistics to AWS CloudWatch. If you are not running the tool on AWS you must supply the accessId, secretKey and region parameters for your AWS environment. Logging to file can be disabled if not required.

Usage: java EmsStatsCloudWatchPusher [options]

   where options are:

  -config    <config file>       - Server config filename, default servers.xml 
  -interval  <secs>              - interval between log entries, default 30 seconds
  -accessId  <accessId>          - AWS access Id (default environment credentials)
  -secretKey <secretKey>         - AWS secret key (default environment credentials)
  -region    <region>            - AWS region (default current region)
  -ns        <namespace>         - AWS namespace (default TIBCO/EMS)
  -nolog                         - Prevent all logging of stats to log directories, default false

 

 

View the Wiki Page