Image Reader Mod for TIBCO® Data Science - Team Studio

This offering consists of a custom mod that provides the ability to read images using TIBCO Data Science - Team Studio. The mod allows users to read images from a cluster using a no-code UI.

Compatible Products

TIBCO® Data Science Team Studio

Provider

TIBCO Software

Compatible Versions

TIBCO Data Science - Team Studio 6.5

License

BSD

Overview

The Image Reader MOD for TIBCO® Data Science - Team Studio allows users to read images from a cluster using Apache Spark in the backend. 

This ability is particularly useful for manipulating image data and building Computer Vision models using the Team Studio platform. 

The output of the mod will be several metadata columns and a data column containing a string of comma separated pixels.

If you want to start using Mod/Custom operator in Team Studio environment, please follow this Knowledge Base article for installation guidelines.

 

Release(s)

Release 1.0.0

Published: August 2020

Initial release 

Includes:

  • jar file (mod/custom operator library)
  • documentation

 

Release 1.0.1

There are currently no reviews for this content.

Why not be the first to review it - click here to login

Computer Vision

Introduction

Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human visual system can do. [Source]

In this page, I will talk about the basics of computer vision and the current industry solutions that solve this AI problem.

Image structure

Images are a set of integers representing the intensity of different pixels values between 0 and 255. Images can either be Black and White (BW) or colour (RBG).

For instance, examine the BW image below. We can see how one small portion within the larger image looks like. Notice that in the matrix of integers, 0 represents a completely dark pixel and 255 represents a white pixel. All other pixel values that fall in between 0 and 255 are shown in different shades of gray.

Image result for image pixels

Image Source: https://seis.bristol.ac.uk/~ggjlb/teaching/ccrs_tutorial/tutorial/chap1/c1p7_i2e.html

Colour images are also very similar to the BW example shown above except for one key difference. Instead of only one channel representing the intensity between Black and White, colour images will have 3 different channels – Red, Green, Blue. These channels are superimposed upon each other to create images that accurately depict objects, people etc. A 4th channel – called alpha - representing transparency factor may also be available.

Transformations

We can make several different transformations on an image file to alter the image’s appearance or layout. Some of those are –

  • Cropping
  • Resize
  • Rotate 
  • Adjust Brightness, Contrast, Hue, Saturation
  • Blur

Several libraries are commonly used for these transformations –

Deep Learning with Images – Intro to Convolution Neural Networks

A convolution neural network (CNN) is a type of feed forward neural network that is very effective in computer vision related tasks. A standard CNN has 2 types of layers in addition to the dense & dropout layers found in a normal feed forward neural network. They are –

  • Convolution layer
  • Pooling layer

We can see the working of a convolution layer in the video snippet below. In an operation, we take a matrix of weights and multiply them with a sub-matrix within an image. The result is saved to a new output matrix and the process is repeated after shifting by one cell. This operation will create a smaller dataset for use in subsequent layers. 

Image Source: Google developer site

The pooling layer is required in a CNN to down sample the features that we detected in a previous convolution layer. Typically, Max pooling and Average pooling layers are used in this layer. 

More information on convolution and pooling layers is available here. A representative picture of a sample convolution neural network is as below.

https://lh4.googleusercontent.com/Tgb2909NT3E7Cren4qg6W_fEJSeUuAbt5NRC5APAEgVWSaBHOFsbdl9nrAfpSnWUAEgIvNwYVZ0_LNn-d2n2YjRFEOotwzSZkJcvchivDQMQLfPziw6vj30CFT20z_3Hm9eYaQE

Image Source: 

Earlier layers of a neural network tend to capture low-level features of an image – for example, eyes in an image of a person’s face. As we train a deeper CNN, the network will pool all low-level features and so understand and generate high-level features – like a complete face. See example below.

https://lh4.googleusercontent.com/NC8JHaYtZ6Y82AancSIlEOcR26QiFDjPPRjGsP9-MxOSSAviZ9i93wpRa1bllEQLGtg63dmKPW_-yahpR7UYmPYyAvi1XhpbVOJno19U5Qssl-qXAI4d8buemtLc8CWNuwbKr5Q

Source: https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

Challenges in Computer Vision Currently

The accuracy of the computer vision models we can train is highly dependent on the quality of our data. The common challenges we encounter with image data currently are –

  • Occlusion : Some object is in the way of the object we would like to detect in the image
  • Viewpoint variation : Pictures in the same class were shot from different angles leading to a difficulty in capturing patterns
  • Illumination : Pictures in the same class have varying levels of brightness
  • Deformation : The object we are trying to detect is deformed in some of the pictures
  • Background objects : There are background objects that look similar to the object we are trying to detect in some of the pictures
  • Intraclass variation : Objects in pictures of the same class look different in different pictures. For example, if we are trying to detect a dog – the breed of dog might lead to variations in shape, size and colour.

Applications of Computer vision

Image recognition tasks are being used in a variety of enterprise and consumer focused technology products. Some examples of image recognition tasks are –

  • Image Classification: Trying to classify images based on the content of the image. For example – Identifying faulty/ not faulty chips in a manufacturing usecase. 

 

Image result for image classification neural network

Image Source: CS231.github

  • Object Detection: Identifying the exact location of objects or people in an image. The most popular example for this task is the Tesla Autopilot driving system, which identifies objects/people and uses these to make decisions.

 

Image result for object detection

Source: Sharif Elfoly medium article

  • Image Captioning: Identifying what is happening in the picture and then translating it into natural language. For example, a convolutional neural network would be able to look at the image show below and translate it into human language.

https://lh5.googleusercontent.com/Q0PJl5bl-4pU_34KyHRMXrPBBGP3NfQuhZVBF_gkwf18fn2yU1UV1xXaLKf1yNrMwqsNxYcjRe4AvEO0msr7AoJcnpjfygy_ddSLa0N9qxTSxbwzZU4sTLoZwhndXvyPvAa7eP8

Image Source: TF website

Computer vision services in the industry

Here is a list of some of the popular sources available for undertaking computer vision tasks -

  • Open source tools – OpenCV, TensorFlow, PyTorch etc.

These tools provide the fundamental building blocks for computer vision. We can read, transform and manipulate image data. We can use tools like TensorFlow & PyTorch to also CV models. 

  • Pre-trained AI models eg: Inception, VGG

These are open source CV models that can be used for techniques such as Transfer Learning that help new users achieve greater accuracy in shorter period of time.

  • Models-as-a-Service eg: AWS Rekognition, Azure Cognitive Services, Google AI platform

These CV models are made available by large multinational corporations via web servers.

  • AI development options eg: AWS Sagemaker, Azure ML, GCP AI platform etc.

These services will allow us to use the open source tools mentioned above to train and infer new computer vision models

TIBCO Solutions for Computer vision

Dr Data Science – A picture is worth a million bytes

A video walkthrough of the all the content mentioned above and a demo using a standard MNIST handwritten numbers dataset is shown in the video.

Accessing pre-trained models from cloud services

With the main cloud providers all offering image recognition services, we wanted to utilise their existing functionality in Spotfire® if possible. This would not only mean Spotfire could be used to run image recognition models but also connect to cloud services. 

Summary of cloud providers image services

To build this image recognition Spotfire tool we would need to be able to do the following:

  1. Read images into Spotfire and extract metadata i.e. image name, dimensions, etc
  2. Submit images to a cloud service and handle the returned results
  3. Visualise the results in Spotfire

To achieve this I would utilise Spotfire extensive access to API’s and languages such as IronPython/C#Python and JavaScript

Detailed information on this demo has been made available here. And a video presentation of this demo has also been made available here

Accessing image data at scale using TIBCO Team Studio

Mod (custom operator) for reading image data can be downloaded for free from this community exchange page.

 

Author:

Venkata Jagannath is a lead data scientist working on the Data Science team in TIBCO, in the US East region. He works on problems that are at the intersection of Artificial Intelligence, Big Data and Cloud computing. While not at work, he likes to work on his pet projects related to personal finance, home improvements etc.

View the Wiki Page