Flogo Use-Case - Event-based Integration

Preface

TIBCO Integration consists of a broad range of capabilities served by several products. Namely, TIBCO BusinessWorks 5, TIBCO BusinessWorks 6/CE and TIBCO Flogo. Each of these products by itself can handle a broad range of use-cases. It should not be a surprise that each use-cases could also be implemented in multiple of these products. However, there are reasons why these products are in the stack as each do optimally serve certain use-cases better than the other. We should understand when to use which. In this series of articles, we will deep dive into some of the use-cases best suited for TIBCO Flogo. Some of the use-cases includes Web API-based Integration, Messaging/Event-based Integration, Data-as-a-Service Architecture, Simple Event Processing, Stream Event Processing, Standard Protocol-based IoT Integration, Edge Processing, API Ecosystem for Mobile/Web, and Mobile/Web Event-driven Integration.

Event-based Integration

The focus of this article is on Event-based integration. There are many messaging systems in the market today each serving its targeted use-case and audiences. TIBCO Flogo has connectors to a lot of these messaging systems. In an event based architecture, it is common to share a single event with many different consumers, each needing the event and data associated with it to perform its specific function. However, with many types of messaging systems today, especially in large enterprises, it is also possible to have multiple being operated at the same time.

In this scenario, consider the few requirements below

• You have multiple different types of messaging systems that you need to share events across. The messaging systems cannot be consolidated because some specific clients can only communicate with specific types of messaging systems. 
• Some modern messaging systems provide connectors or bridges to other messaging systems. Each type of messaging system has a different message representation/format of the message that it can work with. The connectors and/or bridges provided might have a very specific way of transforming the message from one system to another. You want to have control over how the messages are transformed between the messaging systems.
• The current connectors and/or bridges available have limited capabilities when it comes to scaling, performance, and observability. 

Flogo is extremely well suited to handle scenarios like these. 

• It has connectors to Apache Pulsar, Apache Kafka, TIBCO EMS, TIBCO eFTL, MQTT, Azure Service Bus, Amazon SNS, Amazon SQS to name a few and with more to come.
• It has the capability to allow full control on how the message should be transformed as it is consumed and published out the other messaging systems.
• It has the capability to be scaled up or down, extremely lightweight and performant, and provides observability out of the box. 

Apache Kafka to Apache Pulsar Implementation

To illustrate the implementation of a scenario like this, we will use Flogo to get messages from Apache Kafka and propagate it to Apache Pulsar. 

In this scenario, we assume an enterprise message schema below used in all messaging systems. We have the sample payload below.

{

  "environmentCode": "DEV",

  "messageId": "ABC-123",

  "correlationid": "XYZ-123",

  "receivedDateTime": "2024-01-01 12:00:00",

  "messageProperty": [ {"name": "CustomerId", "value": "A123"} ],

  "messagePayload": [ {"type": "JSON", "payload": "..."} ]

}

This enterprise schema can be defined in the SCHEMAS section of the Flogo flow. Do note that by entering the sample payload into the schema text box in Flogo, it will automatically be converted into a JSON schema on exit.

This scenario requires us to connect to both Apache Kafka and Apache Pulsar. To do this, we define two connections in the CONNECTIONS section of the flow.

The KafkaConnection allows the configuration of connection parameters to Apache Kafka.

Also note the Auth Mode supported.

The PulsarConnection allows the configuration of connection parameters to Apache Pulsar.

Also note the authorization type supported.

Connection properties that can be interchangeable between environments are also automatically created in the APP PROPERTIES of the flow.

The image below shows a simple flow to accomplish this use-case. What you need is a Kafka Consumer trigger to consume the message from a Kafka topic. The message consume can then be propagated into an Apache Pulsar Producer Activity. We can then mark the message being consumed using the Kafka Commit Offset activity. The flow also has optional Log Message to log the start and end of each message propagation.

The Kafka Consumer Trigger allows you to configure the settings and behaviors of this consumer such as the Kafka Connection, the topic name, the consumer group ID, the Value Deserializer (String, JSON, Avro), Initial Offset (Newest, Oldest) and others.

Likewise, the Apache Pulsar Producer Activity allows you to configure the settings and behavior of this activity. Notable settings include the Pulsar Connection, the Topic name, Compression Type (LZ4, ZLIB, ZSTD), and Message Format (String, JSON).

Mapping the Kafka message information over to the Pulsar message follows the same method and rules of mapping any other Flogo key and properties.

To test this Kafka to Pulsar flow, we have also created an Apache Kafka Producer and Apache Pulsar Consumer flow.

Apache Kafka Producer

The Apache Kafka Producer flow will act as the first step in the test where it will produce messages into an Apache Kafka topic. It is triggered by a simple repeating timer and uses the Kafka Producer activity.

The Kafka Producer activity is configured to use the same Kafka Connection with mapping into the MessageEnvelope JSON schema using hard coded values.

Apache Pulsar Consumer

The Apache Pulsar Consumer flow is used in the last step of the test where it consumes messages that have been propagated by the Kafka to Pulsar flow. It will also acknowledge the receipt of the message after consumption.

The configuration of the Apache Pulsar Consumer is below.

The code written in Flogo  2.25.0 using the Visual Studio Code Extension is provided below for your reference.

KafkaProducer.flogo PulsarConsumer.flogo Kafka2Pulsar.flogo