Hybrid Database Access with TIBCO Cloud Integration

Table of Contents

• Introduction
• TIBCO Cloud Integration ? Hybrid Agent
• Setting up the Hybrid Agent
• Build an Application in the TIBCO Cloud
• Configure, Deploy & Run

Introduction

Whilst every Business wants to move to a 100% cloud environment, large Enterprises are faced with a long road to realization ? they can?t just switch off their data centers overnight and expect everything to work in a Cloud environment as it did before. Couple this with data sovereignty issues and most Enterprise Customers that I work with a move towards a Hybrid Cloud model.

In this model ? much of the Business Logic, Micro-Services, and of course SaaS applications live in the Cloud somewhere ? but databases often reside in a data center and are unlikely to move to the Cloud for the foreseeable future.

This article walks you through how to use TIBCO Cloud Integration to connect to an on-premise Database instance to access the data ? allowing you to build out a hybrid implementation where your Business logic is in the Cloud and the data is in your data center. In this article, I?m using MongoDB but the same techniques work for Microsoft SQL Server, Oracle, MySQL, Postgres, etc.

TIBCO Cloud Integration ? Hybrid Agent

The Hybrid Agent is a lightweight app that runs on your data center to facilitate communication between on-premises apps and TIBCO Cloud? Integration. With the TIBCO Cloud Integration ? Hybrid Agent, you can:

• Establish tunnel connections to apps running on TIBCO® Cloud Integration. In TIBCO® Cloud Integration, you can configure an app to use the tunnel to connect to on-premises resources like a database server (in this case MongoDB), a Kafka Broker, a JMS server, or a REST server etc.
• There is no need to expose your database or service as a public Internet service, so there is no requirement to change the firewall rules or to host the TIBCO Cloud ? Proxy Agent in the DMZ (Demilitarized Zone).
• Stream logs directly from a TIBCO® Cloud Integration app container.
• Securely communicate information on registered remote apps, such as monitoring statistics, app status, and other app details. Remote apps are displayed in TIBCO® Cloud Integration on the Apps list. You can see and monitor your remote apps the same way as your cloud apps.

The full documentation for using the Hybrid agent can be found at this location: https://integration.cloud.tibco.com/docs/using/hybrid-agent/index.html

The main benefit of using the Hybrid Agent is that it removes the need for organizations to open firewalls to the Internet or mess about with whitelisting ? whilst providing secure TLS-based communication between the data center and the TIBCO Cloud. Communications are also initiated from the Data Centre to create the tunnel, providing an extra layer of security.

Setting up the Hybrid Agent

There are a few steps to get the Hybrid Agent up and running:

• Get logged into the TIBCO Cloud (https://cloud.tibco.com ? 30-day trial available)
• Create a Proxy Agent Access Key
• Set up Inbound whitelist rules (if required)
• Download the Hybrid Agent
• Configure the Agent with your Access Key
• Start the Agent, mapping the ports required
• Build and Test your App (which has its own steps covered later!)

Creating a Proxy Agent Access Key

Once you?ve logged into your TIBCO Cloud Account, Navigate to Settings and look for ?Proxy Agent Access Keys? on the left-hand side of the page ? when you are there ? you?ll see a page similar to the screenshot below:

To generate a new key ? just click on ?Generate Access Key? and give it a name (something recognizable!). This will generate the key ? which will only be displayed once ? so make sure you copy it to your clipboard as you?ll need it later!

Downloading the Agent

To download the Agent ? you?ll need to navigate to TIBCO Cloud Integration, click on ?Environment & Tools? click on ?New Agent? and choose the appropriate agent for your intended Operating System.

Downloading the Agent CLI

Configuring the Agent

First of all, we have to create an Agent configuration. In my case, I?m running on a Mac, and I want to create a new agent called ?mongodb_agent?, so all I need to do is to run the following command from the terminal:

 ./tibagent configure agent mongodb_agent

You will then need to provide your credentials to log into the correct account and Organisation within the TIBCO Cloud.

Once completed, you now need to configure the Agent with the access key that you generated earlier replacing <key> with your access key, replacing <key id> with your key ID, and of course the agent name as we created above:

 ./tibagent configure connect ? accessSecret <key> ? accessKey <key id> mongodb_agent

Starting the Agent

The final step in this stage is to start the agent, with the appropriate Ports that you want to tunnel between your environment and The TIBCO Cloud.

MongoDB has a number of default ports, which are mainly 27017, 27018, and 27019. On my machine, I have set up a MongoDB replica set with 3 replicas each known as mongoset1, mongoset2, and mongoset3. Each of these instances is mapped in my Hosts file ? so that they can be accessed by name rather than IP address.

I?m going to set up a ?Change Data Capture? process to monitor changes in the MongoDB database ? and therefore we are mandated by MongoDB to use replica sets. A request/reply pattern for querying the database does not mandate the use of replica sets, so how you start and map the ports may be different.

To start the Hybrid agent, mapping the appropriate ports, we will use the following command:

 ./tibagent start agent mongodb_agent -spec 27017:mongoset1:27017 -spec 27018:mongoset2:27018 -spec 27019:mongoset3:27019

This will start the Hybrid Agent and create the tunnel between the 3 MongoDB replicas and the TIBCO Cloud.

You can confirm that the Hybrid Agent is running by navigating back to the TIBCO Cloud, into ?Integration Apps?, then clicking on ?Environment & Tools?, choosing ?Agents? and then on the ?Filter By? panel on the left, click ?Hybrid?:

You?ll see that if we hover over the agent ? we can see the ports that are mapped and that the tunnel is up and running.

Now to go and build an app to test it all out!

Build an Application in the TIBCO Cloud

I?m going to use TIBCO Cloud Integration to build an application that listens to changes made to our MongoDB instance. This application will be running in the TIBCO Cloud ? but it will listen to changes that are being made to our MondoDB instance in our data center (or more accurately, my laptop!), it?s only going to be a simple application that writes to a log ? but you can take that simple app and go from there.

Configure a connection

First, we need to configure a Connection to MongoDB, and this is done by logging into TIBCO Cloud Integration and navigating to the ?Connections? tab where you will see all the currently configured connections:

Press the ?Create? button to create a new connection and navigate to the ?MongoDB Connector? which is marked with the ?Develop? tag. You can use the search button to narrow down the list. The ?Develop? tag tells us that it is a connector for TIBCO Flogo. ?Connect? means that it is a connector for TIBCO Scribe. We?ll be using Flogo in our examples.

When you have selected the connector, you will need to configure the details:

The main area to configure is the URL, and you can see that I have the 3 machine names and appropriate ports for each machine ? plus I have defined the replica set that my instances use. If you need to add the username/password ? you will need to change the ?Credential? type to use the appropriate security algorithm before filling out the details for the user.

Create the App

Click on the ?Apps? tab in the top left of the screen, which takes you to the main dashboard. Click on ?Create/Import? on the top right and then choose ?Flogo? in the dialog box, finally clicking on ?Create New App?:

Give your new app a name, by overwriting the name that is automatically generated in the top left corner:

Then, click the ?Create? button where we will be asked how we want to create our application. Click on ?Trigger? and look for the MongoDB Trigger:

You?ll be asked which connection you want to use ? and select the connection you created earlier, then click ?Create?

You will now have an empty application that is triggered by mongoDB changes ? so now we want to add some logic.

Click on ?New Flow? and this time we want to choose ?Flow? in the dialog and give it a name ? for instance, ?mongoDBListener?, followed by ?Create?:

Select your existing trigger and press ?Continue?:

Choose ?Copy Schema?:

You will then be taken to a screen similar to the following:

This is showing you the schema from the output of the trigger ? but we don?t need to do anything with that for now, as we?ve copied the schema from the trigger. Next, we need to configure what database and collection we want to listen to, and what triggers we want to monitor. To do that click on the trigger button which is on the left-hand side of the screen and has a red ?X? icon on it. It is marked with an X because there is work to do. When you click on it, you will see the following:

What we need to do is to configure the database name, collection, and which events we want to listen to. For example, my database is ?customers? and I will leave ?Collection? as empty so I can listen to all events on all collections.

I want to listen to all events, so I will leave them as default, then click on ?Save? when ready.

You will see that ?Map to Flow Inputs? is marked with a yellow triangle. This means that we have to configure how we want to map the output of the trigger to the input of the flow we are creating. Click on the ?Map to Flow Inputs? link and you will see:

The ?Flow Input? panel is what the flow itself is expecting. On the right-hand side ? is where you map the incoming data from the trigger to the flow. So to do this, I expand the ?{} output? branch of the ?Flow Input?, then also expand the ?{} trigger? output in the center and click on ?{} output? to insert the mapping into the panel above. What you should end up with is the following:

Click ?Save? when happy and then click on the ?x? button on the top right of the dialog to close this window.

We now have a flow that is ready to be built. If you hover over the main canvas, you can see a ?+? button appears ? click on this to add an activity in that location:

You will see a list of activities that you can click on, or you can use the search area to look for a specific activity. For now, all we are going to do is ?log? the output of the trigger from MongoDB. So type in ?log? into the search box, and choose the ?Log Message? activity:

Here we can decide what we will output on our log message ? so click on the ?Input? panel, which shows you that it is expecting the ?message? input to be defined (it?s marked in red), and you can see the output from the upstream activities ? in this case, it?s our trigger. You could type in the formula that you want to use, or click around on the functions, etc. For now, we can use the following formula:

 string.concat(?Received ?,$flow.output.OperationType," Event")

Which will just output the type of trigger that we received:

We?re ready to try out our application!

Normally we would be able to use the ?Test? function in the top right corner of the screen to run through and debug the application, but because we are using the Hybrid Agent we have to do one more configuration step to deploy our application for testing.

Configure, Deploy & Run

First ? we need to configure our App with the correct Access Keys so that it can listen to our MongoDB instance. So from the Application details screen, click on the ?Environment Controls? link:

Choose the key that you set up in Step 1 and click ?Push updates?

Go back the the ?Flows? section and then click ?Push? ? this compiles and deploys the application into a container in The TIBCO Cloud ? but does not start it until you are ready.

When the application is deployed ? you can then use the ?Instances? counter to decide how many instances of the application you want to start. 1 is enough for our purposes ? click on the counter and press the ?Scale? button:

At this point ? you can click on the ?Logs? tab to view the logs ? and click on the ?Real-time? slider to see the logs as they are generated:

Look for the line:

 ?[flogo.trigger.mongodb.mongodb-event-listener] ? ===listening===? 

This tells us that the application has successfully started and has created a connection through the tunnel into the local MongoDB application.

If you get any failure at all ? then we will have to debug further.

Test the App

To test the application ? all we need to do is to add a new entry into the mongodb collection. To do this, I have used ?mongosh? to connect to the mongo cluster and used the following commands:

 mongodb-replicaset:PRIMARY> use customers switched to db products mongodb-replicaset:PRIMARY> db.customers.save({firstname: ?Dave?}) WriteResult({ ?nInserted? : 1 }) mongodb-replicaset:PRIMARY> db.customers.find() { ?_id? : ObjectId(?5ffdb8db3405ea9985f1e92f?), ?firstname? : ?Dave? } mongodb-replicaset:PRIMARY> db.customers.save({firstname: ?David?}) WriteResult({ ?nInserted? : 1 }) mongodb-replicaset:PRIMARY> db.customers.save({firstname: ?Davie?}) WriteResult({ ?nInserted? : 1 }) mongodb-replicaset:PRIMARY> db.customers.save({firstname: ?Davey?}) WriteResult({ ?nInserted? : 1 }) mongodb-replicaset:PRIMARY> db.customers.save({firstname: ?Abhijeet?}) WriteResult({ ?nInserted? : 1 }) mongodb-replicaset:PRIMARY>

And when I look at the log output from our Flogo application we see the logs updating and our ?LogMessage? activity shows that it received an ?Insert? operation.

So now you know how to configure the TIBCO Hybrid Agent to create a tunnel between your private cloud and the TIBCO Cloud platform, how to configure a connection to MongoDB that resides within your private cloud, and how to create a basic application that listens for updates to MongoDB.

Your next task would be to expand that application to perform whatever you need in order to process the MongoDB updates.

To find out more about TIBCO Cloud Integration ? go to https://cloud.tibco.com