Introducing the all-new TIBCO Community site!

For current users, please click "Sign In" to reset your password and access the enhanced features. If you're a first-time visitor, we extend a warm welcome—click "Sign Up" to become a part of the TIBCO Community!

If you're seeking alternative community sites, explore ibi, Jaspersoft, and Spotfire.

Jump to content
  • Statistics Logger? for TIBCO Enterprise Message Service?


    Bill Mclane

    Introduction

    This tool can be used to log and monitor EMS (including EMS Appliance) server statistics. Server Statistics can be logged to local CSV files and can additionally be pushed to either AWS CloudWatch, InfluxDB or to the Prometheus monitoring tool.

    The package contains the following Java classes:

    • EmsStatsLogger: Regularly polls and logs EMS server statistics to CSV files, which can easily be graphed in tools such as Excel. A new CSV file is generated for each day.
    • EmsStatsPromCollector: This class collects EMS server statistics for export to Prometheus. Statistics can also be optionally logged to a local CSV file. Prometheus is a powerful open source monitoring system (see https://prometheus.io)
    • EmsStatsCloudWatchPusher: This class pushes EMS server statistics to AWS CloudWatch. Statistics can also be optionally logged to a local CSV file.
    • EmsStatsInfluxPusher: This class pushes EMS server statistics to InfluxDB. Statistics can also be optionally logged to a local CSV file.

    The following EMS server metrics are collected:

    • Inbound/outbound message/byte rates
    • Pending message count/size
    • Connection/session counts
    • Destination counts
    • Message memory usage
    • Disk read/write rates
    • Sync/async datastore sizes
    • Response time

    The following EMS queue/topic metrics can be monitored:

    • Inbound/outbound message/byte rates
    • Inbound/outbound total message/byte counts
    • Pending message count/size
    • Receiver/Consumer counts

    The EMS queue/topic metrics are documented EMS Admin API DestnationInfo, QueueInfo, TopicInfo and StatData classes, for further details see:

        https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/ServerInfo.html

        https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/QueueInfo.html

        https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/TopicInfo.html

        https://docs.tibco.com/pub/ems/8.4.0/doc/html/tib_ems_api_ref/api/javadoc/com/tibco/tibjms/admin/StatData.html

    The package also contains the following sample Grafana dashboards for visualising the statistics stored in Prometheus and InfluxDB (for further details on Grafana see https://grafana.com)

    • EMS Server Overview (graphs in/outbound message rates, pending message size/count, connection count and response time)
    • EMS Details (graphs memory usage, sync/async datastore size and disk read/write rates)
    • EMS Queues (graphs queue in/outbound message rates, pending message size/count, receiver counts)
    • EMS Topics (graphs queue in/outbound message rates, pending message size/count, consumer counts)

    Example Grafana dashboards are shown below:

    An example AWS CloudWatch dashboard is shown below:

    Download the tool here emsStatsLogger3.0.zip

    Version History

    Version 3.1

    •  Added EmsStatsInfluxPusher to push stats to InfluxDB
    •  Added Grafana dashboards for InfluxDB

    Version 3.0

    •  Added support for monitoring multiple EMS servers
    •  Added support for SSL connections
    • Added support for configuration with servers.xml file
    • Added support for monitoring queues and topics
    • Added support to auto cleanup old log files
    • Changed emsStatsPromPusher to emsStatsPromCollector (Prometheus now directly scrapes the collector for stats)
    • Updated Grafana dashboards, including new dashboards for Queues and Topics

    Version 2.1

    •     Added logfile option to override log file name
    •     Added noappend option to prevent append to log file on restart

    Version 2.0

    •     Added emsStatsCloudWatchPusher

    Software Requirements

    • JRE 1.8 or higher
    • TIBCO Enterprise Message Service 8.1 or higher (client)
    • For emsStatsCloudWatchPusher:
      • AWS SDK for Java

    To rebuild the source for the emsStatsPromPusher class, the following libs are also required:

    Installation

    Follow this procedure to install this tool into your environment:

    • Unpack the emsStatsLogger.zip file.
    • Edit the servers.xml file to add connection detail for the EMS servers to be monitored.
    • Edit the run scripts to make appropriate versions for your environment:
      • Set the TIBEMS_ROOT variable to your TIBCO EMS installation directory (EMS client 8.1 or greater required)
      • Set the appropriate command line options for your environment

    Installation for Prometheus/Grafana

    • Download Prometheus and the PushGateway from: https://prometheus.io/download
    • Edit the prometheus.yml file and configure a scrape config for the collector targets. There are three collector targets, one for EMS server stats, one for queue stats and one for topic stats.
    • Configure the collector targets appropriately for where the EmsStatsPromCollector is running, the example below show default configuration if the collector is running on the same host as the Prometheus server.
    • Remove the queue/topic collector target if not required.
    1.  scrape_configs:   - job_name:       'TIB-EMS'     scrape_interval: 60s     scrape_timeout: 10s     honor_labels: true     static_configs:       - targets: ['127.0.0.1:9091', '127.0.0.1:9092', '127.0.0.1:9093'] 

    You may add multiple scrape configs for different EmsStatsPromCollector instances, for example for different EMS environments.

     

    Ensure the job names are different and match those configured using the command line when ruuning the collector.

    By default EmsStatsPromCollector listens on HTTP port 9091 for exporting EMS server stats, 9092 for queue stats, and 9092 for topic stats.

     

    The Prometheus server listens on TCP port 9090 by default.

    Edit the runPromCollector.sh file and configure appropriate HTTP ports if necessary for your environment.

    • Use the -nolog command line option to prevent logging to local file if required.

    Note; you no longer need to run the Prometheus gateway, the prometheus server now scrapes the EmsStatsPromCollector directly.

    Example command line to run prometheus:

    1.  prometheus --config.file=prometheus.yml 

    To Install Grafana:

    • Follow the instructions here: http://docs.grafana.org/installation
    • Take a copy of the grafana.ini file from the installation (e.g. /usr/local/etc/grafana) and edit appropriately, simple example shown below.
    • Open the grafana URL in a browser and create a prometheus data source, then simply import the sample dashboards provided or create your own.
    • Note by default grafana listens on TCP port 3000, default user/password is admin/admin

    Example command line to run grafana:

    1.  /usr/local/Cellar/grafana/4.6.3/bin/grafana-server -homepath /usr/local/share/grafana --config grafana-ems.ini

    Example grafana configuration file:

    1.  [paths]   data = data   logs = data/log/grafana_log  [server]   http_addr = 0.0.0.0   http_port = 3000  [dashboards.json]   enabled=true   path=grafana_dashboards

    Installation for InfluxDB

    Download influxdb-java-2.15 client and dependencies downloaded from:

    https://jar-download.com/artifacts/org.influxdb/influxdb-java/2.15

    InfluxDB client libraries required:

    • converter-moshi-2.5.0.jar
    • influxdb-java-2.15.jar
    • logging-interceptor-3.13.1.jar
    • moshi-1.5.0.jar
    • msgpack-core-0.8.16.jar
    • okhttp-3.13.1.jar
    • okio-1.17.2.jar
    • retrofit-2.5.0.jar

    Installation for AWS CloudWatch

    If running the tool on EC2:

    You can also run the tool on-premise and push stats to AWS:

    • Install the AWS SDK library which can be downloaded here:
    • Edit the runCloudWatchPusher.sh file and set the appropriate configuration for your environment.
      • Use the -accessId -secretKey and -region options for you AWS environment
      • Use the -nolog command line option to prevent logging to local file if required

    EmsStatsLogger

    The EmsStatsLogger class logs EMS server statistics to an output CSV file once every poll interval. By default a new output log file is created for each day, this may be overridden using the logfile parameter.

     Usage: java EmsStatsLogger [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml   -interval  <secs>              - interval between log entries, default 30 seconds   -debug                         - Enable debug trace to stdout 

    EmsStatsPromCollector

    The EmsStatsPromCollector class inherits from EmsStatsLogger and in addition to logging EMS statistics to file also collects the statistics for monitoring with Prometheus. Logging to file can be disabled if not required.

     

    You may set a job name to represent the EMS environment being monitored, eg: PROD, QA, DEV etc.

     

    Ensure you add a prometheus scrape target for each environment/job and the job names match those configured in the prometheus.yml file.

     Usage: java EmsStatsPromCollector [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -interval  <secs>              - interval between log entries, default 30 seconds   -serverStatsPort  <port>       - The HTTP port to scrape for EMS server stats, default 9091. (0 disables)   -queueStatsPort   <port>       - The HTTP port to scrape for EMS queue stats, default 9092. (0 disables)   -topicStatsPort   <port>       - The HTTP port to scrape for EMS topic stats, default 9093. (0 disables)   -job              <job-name>   - The Prometheus job-name, default TIB-EMS   -nolog                         - Prevent all logging of stats to log directories, default false 

    EmsStatsInfluxPusher

    The EmsStatsInfluxPusher class inherits from EmsStatsLogger and in addition to logging EMS statistics to file also sends the statistics to InfluxDb. Logging to file can be disabled if not required.

     Usage: java EmsStatsInfluxPusher [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -dbURL     <DB URL>          	 - Infux DB URL (default http://localhost:8086)   -username  <username>          - Influx DB usnername (default admin)   -password  <password>          - Influx DB password (default admin)   -dbName    <DB Name>           - The Influx DB name (change for different environments) default TIB-EMS"   -nolog                         - Prevent all logging of stats to log directories, default false 

    EmsStatsCloudWatchPusher

    The EmsStatsCloudWatchPusher inherits from emsStatsLogger and in addition to logging EMS statistics to file also sends the statistics to AWS CloudWatch. If you are not running the tool on AWS you must supply the accessId, secretKey and region parameters for your AWS environment. Logging to file can be disabled if not required.

     Usage: java EmsStatsCloudWatchPusher [options]     where options are:    -config    <config file>       - Server config filename, default servers.xml    -interval  <secs>              - interval between log entries, default 30 seconds   -accessId  <accessId>          - AWS access Id (default environment credentials)   -secretKey <secretKey>         - AWS secret key (default environment credentials)   -region    <region>            - AWS region (default current region)   -ns        <namespace>         - AWS namespace (default TIBCO/EMS)   -nolog                         - Prevent all logging of stats to log directories, default false 


    User Feedback

    Recommended Comments

    There are no comments to display.


×
×
  • Create New...