How to Configure TIBCO Messaging Castle - Powered by Apache Kafka 3.7.1 with KRaft in A Kubernetes Environment

TIBCO Messaging Castle - Powered by Apache Kafka now supports the use of KRaft controllers instead of Zookeeper. A feature of using KRaft is that a combined process can run with both the controller and broker running as one PID. Deciding to run a combined process or two separate processes depends on certain factors.

Separate KRaft controllers are generally recommended for production Kafka deployments.

While combined controller-broker setups are useful for development and testing due to their simplicity, they have limitations that make them less suitable for larger, mission-critical environments.

Use Separate KRaft Controllers When:

• High Availability and Scalability: Isolating controllers allows them to focus solely on metadata management, improving their responsiveness and reducing the risk of impacting broker performance during heavy loads or controller failovers. This is crucial for large clusters with high throughput requirements.

• Fault Isolation: Separating concerns helps contain failures. If a broker node experiences issues, it won't directly impact the controller's availability, ensuring cluster stability.

• Security: Dedicated controllers can be placed in more secure network zones, limiting access to critical metadata operations.

• Resource Optimization: You can tailor hardware resources for each role, optimizing performance and cost efficiency. Controllers typically need less memory and CPU than brokers, so combining them might lead to overprovisioning.

Use Combined KRaft Controller-Broker When:

• Development and Testing: Simplifies setup and reduces resource requirements, making it ideal for local testing and small-scale development environments.

• Resource-Constrained Environments: In scenarios where resources are limited (e.g., edge deployments), a combined setup can be a pragmatic choice.

• Small, Low-Traffic Clusters: For clusters with minimal traffic and data volumes, the overhead of separate controllers might outweigh the benefits.

In summary, separate KRaft controllers offer improved performance, scalability, fault tolerance, and security, making them the preferred choice for production Kafka clusters. Combined setups are suitable for specific scenarios where simplicity and resource constraints are primary considerations.

This document to provide guidelines for configuring the KRaft controllers as separate processes from the brokers.

Configuration Overview

Running TIBCO Messaging Castle – Powered by Apache Kafka (AKD) on different Kubernetes Container Platforms involves:

• Configuring a Kubernetes cluster for TIBCO Messaging Castle – Powered by Apache Kafka (AKD). The Kubernetes Container can be configured on Azure Kubernetes Service (AKS), Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GK), RedHat OpenShift (OC), or a generic Kubernetes environment.

• Creating a single Docker® image for both the Castle Broker and the KRaft Controller, where the containers will be hosted in a Docker Registry

• Creating persisted volumes for the Broker and the Controller

• Configuring and creating Kubernetes containers based on the Docker® image for the components

• Configuring Load Balancer(s) in Kubernetes to access AKD (optional), and to access JMX ports for both the brokers and controllers (optional)

• Ensuring TIBCO Messaging Castle – Powered by Apache Kafka is instrumented (JMX) for TIBCO Message Monitor with the AKD extension.

• Helm Charts are used to deploy AKD.

• NOTE: This configuration does not create a secure environment, and designed to show the steps for running TIBCO Messaging Castle – Powered by Apache Kafka (AKD) on different Kubernetes Container Platforms with KRaft controllers. SSL and SASL can be added to the configuration (not documented).

Supported Versions

The steps described in this document are supported for the following versions of the products and components involved:

• TIBCO Messaging Castle – Powered by Apache Kafka 3.7.1 or later is required TIBCO Messaging Castle – Powered by Apache Kafka can be downloaded from

edelivery.tibco.com.

• Docker Community/Enterprise Edition should be most recent version.

• Kubernetes 1.28.x or Red Hat OpenShift Container Platform 4.1.6. Recommend latest versions of the container platform

• HELM 3.92 or later

• Msgmon 1.1 (optional)

tibakd_kubernetes_files_3.7.1.zip How to Configure TIBCO Messaging Castle - Powered by Apache Kafka 3.7.1 with KRaft in A Kubernetes Environment.pdf