Introducing the all-new TIBCO Community site!

For current users, please click "Sign In" to reset your password and access the enhanced features. If you're a first-time visitor, we extend a warm welcome—click "Sign Up" to become a part of the TIBCO Community!

If you're seeking alternative community sites, explore ibi, Jaspersoft, and Spotfire.

Jump to content
  • TIBCO® Messaging - Apache Kafka Distribution Extended Monitoring


    Vikas Kumar 9

    Table of Contents

     

    TIBCO® Messaging - Apache Kafka Distribution - FTL Bridge Release 2.0.0 added the capability to collect and display a wide range of monitoring statistics to keep an eye on your Apache Kafka clusters and clients.  With the information and extended monitoring template in this article, you'll be able to extend your monitoring to include more Apache Kafka statistics as well as adding monitoring of your Apahe ZooKeeper installation, all using the same InfluxDB database and Grafana-based dashboards you've already deployed.

    New Statistics

    The tables below cover the new statistics collected, their attributes, how they are stored within InfluxDB, and any extra information related to that statistic.

    New Apache Kafka Statistics

    Metric NameAttributesInfluxDB AliasNotes
    kafka.server.MessagesInPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    MessagesInPerSecPerTopicTopic=*
    kafka.server.BytesInPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    BytesInPerSecPerTopicTopic=*
    kafka.server.BytesOutPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    BytesOutPerSecPerTopicTopic=*
    kafka.network.RequestsPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    ProduceRequestsPerSec

    Request=Produce
    kafka.network.RequestsPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    FetchConsumerRequestsPerSecRequest=FetchConsumer
    kafka.network.RequestsPerSec

    Count

    MeanRate

    OneMinuteRate

    FiveMinuteRate

    FifteenMinuteRate

    FetchFollowerRequestsPerSecRequest=FetchFollower
    kafka.contrioller
    .OfflinePartitionsCount
    ValueOfflinePartitionsCount 

    New Apache ZooKeeper Statistics

    Metric NameAttributesInfluxDB Alias
    org.apache.ZooKeeperService

    StartTime

    Version

    TickTime

    MinSessionTimeout

    MaxSessionTimeout

    JuteMaxBufferSize

    AvgRequestLatency

    MinRequestLatency

    MaxRequestLatency

    ClientPort

    PacketsSent

    NumAliveConnections

    OutstandingRequests

    MaxClientCnxnsPerHost

    FsyncThresholdExceedCount

    PacketsReceived

    zookeeper            
    org.apache.ZooKeeperService.InMemoryDataTree

    LastZxid

    NodeCount

    WatchCount

    zookeeper

    Installation

    To use these new capabilities, you will need to replace two files in your Apache Kafka Distribution - FTL Bridge installation; the first adds the new statistics to the JSON template file and the other adds additional configuration fields to the .csv file.  Both of these are used to generate the monitoring configuration file that drives the entire statistics collection and archiving process. 

    Regardless of your platform, you'll need to know the directory where the Apache Kafka Distribution - FTL Bridge was installed.  By default, for Linux and Mac this directory will be /opt/tibco/akd/bridge/2.0 (where 2.0 is the version number and uld change as new versions are released), for Windows this directory will be C:\tibco\akd\bridge\2.0.  If you installed in a non-default directory, change any paths specified below to reflect your actual installation directory. 

    To add the new statistics, copy the tibkmon_Template.json and tibkmon_config.csv files attached to this article into the monitoring/tibkmon_JSON_generator directory of your Apache Kafka Distribution - FTL Bridge installation.

    Configuration

    There are two steps needed to complete the configuration and get you ready to collect the new statistics.  First, edit the tibkmon_config.csv file and add the appropriate information for accessing your ZooKeeper installation - this covers the same settings that you've already configured for the Apache Kafka brokers such as JMX host, port, user name, and password (just with teh appropriate settings for your ZooKeeper installation which might or might not be the same depending on your overall system configuration).  When you are finished, save this file and continue to the next step.

    If there are specific questions you would like to see answered or new topics covered, please add a comment because that will help us prioritize future content.


    User Feedback

    Recommended Comments

    There are no comments to display.


×
×
  • Create New...