Anomaly Detection - Technology and Applications

Last updated:
7:51am Oct 06, 2021
Table of Contents

INTRODUCTION

What are Anomalies?

Anomaly detection is a way of detecting abnormal behavior. One definition of anomalies is data points that do not conform to an expected pattern compared to the other items in the data set. Anomalies are from a different distribution than other items in the dataset. Anomalies in data translate to significant (and often critical) actionable information in a wide variety of application domains. The figure below shows a simple example of anomalies (o1, o2, O3) in a 2D dataset. The autoencoder technique described here first uses machine learning models to specify expected behavior and then monitors new data to match and highlight unexpected behavior.;

(Anomalies are similar, but not identical, to outliers. Outliers are points with a low probability of occurrence within a given data set. They are observation points that are distant from other observations. However, they don't necessarily represent abnormal behavior. Outliers in data warrant attention because they can distort predictions and affect model accuracy, if you don’t detect and handle them. For more information on detecting outliers in Spotfire, see this Wiki article: Top 10 methods for Outlier Detection. )

Overview Webinars

Overview Whitepapers and Solution Briefs



INDUSTRY USE CASES

Here are a few examples from our practice:

Baseball

Baseball is one of the oldest sports in the United States, with a history dating back to the 19th century. Since 1880, there have been 101 different teams who have played a grand total of 2,829 different seasons. By looking at the data, we wanted to statistically uncover which of these 2,829 seasons were anomalies, which teams had seasons unlike any other. To accomplish this, we utilized a method called SAX (Symbolic Aggregate Approximation) encoding. The advantage of using SAX is that it is able to act as a dimensionality reduction tool, it is tolerant of time series of different lengths, and it makes trends easier to find.  For details, see this blog:  Using Time Series Encodings to Discover Baseball History’s Most Interesting Seasons

Preventing Machine Breakdowns with Connected Sensor Data

Many different types of equipment, vehicles and machines are now instrumented with sensors. Monitoring these sensor outputs can be crucial to detecting and preventing breakdowns and disruptions. Unsupervised learning algorithms like Auto encoders can be used to detect anomalous data signatures that may predict impending problems.  When sensor time series traces exhibit repeating patterns, special techniques, such as MASS, or the the one used in the Sensor Anomaly Detection at the Edge solution on this page, (shown in the image below) can be used. 

  

Listening for Abnormalities in the Sounds Machines Make

A good mechanic can tell whether your car is OK - or not - by listening to the sounds it makes.  A really good one can tell you what is wrong with it. 

Abnormal sounds can be an indicator that a machine needs maintenance.  The video below shows an example of an application that uses audio data from any device and learns to identify anomalous sounds made by machines.  Datasets of known abnormalities can then be created and the models can be deployed for real-time scoring.     

Identifying Abnormal Product

Many manufactured products undergo some form of testing to determine suitability for use.  Univariate and linear multivariate Statistical Process Control methods can be used to detect anomalous product based on this data.  However, with increasing component and system complexity, multivariate anomalies that also involve significant interactions and nonlinearities may be missed by these more traditional methods.  These anomalies can be implicated in reliability and system failures.  AI-based algorithms, such as autoencoders, can often be used to identify these complex anomalies. Once the anomalies are detected, their fingerprints can be generated so they can be classified and clustered, enabling investigation of the causes of the clusters. As new data streams in, it can be scored in real-time to identify new anomalies, assign them to clusters and respond to mitigate potential problems.

Defects and Abnormalities in Images

Connected digital cameras today capture large amounts of raw image data.  People are very good at rapidly identifying abnormalities in images.  But, it is expensive and time-consuming for humans to extract the critical information from large numbers of images; they often remain unprocessed.  AI algorithms are increasingly used to automate this process.  These use cases often involve some combination of unsupervised learning (where similar images are clustered together), human verification that images contain abnormalities and supervised learning, to train models that automate the identification of abnormalities of interest.  Examples include identification of cancer cells and manufacturing defects in images.  An example of how this is done with semiconductor wafermap spatial test and fail patterns can be found here.   

Cyber Threat Detection

Networked computers today are under constant threat of ransomware and other forms of cyber-attack.  System Threats can be detected through analysis of computer log data, utilizing unsupervised learning models such as LSTM autoencoders for anomaly detection.  LSTM autoencoders identify anomalies in the sequence of log events.  

Bank Stress Test

Economic and performance data can be used for "stress testing" the capital reserves of bank holding companies to identify data anomalies.  Details of one implementation can be found here:  

Data Quality Management and Anomaly Detection - A Bank Stress Test Use Case

Fighting Financial Crime

In the financial world, trillions of dollars’ worth of transactions happen every minute. Identifying suspicious ones in real time can provide organizations the necessary competitive edge in the market. Over the last few years, leading financial companies have increasingly adopted big data analytics to identify abnormal transactions, clients, suppliers, or other players. Machine Learning models are used extensively to make predictions that are more accurate.  Learn about and download the Risk Management Accelerator

Healthcare claims fraud

Insurance fraud is a common occurrence in the healthcare industry. It is vital for insurance companies to identify claims that are fraudulent and ensure that no payout is made for those claims. The economist recently published an article that estimated $98 Billion as the cost of insurance fraud and expenses involved in fighting it. This amount would account for around 10% of annual Medicare & Medicaid spending. In the past few years, many companies have invested heavily in big data analytics to build supervised, unsupervised and semi-supervised models to predict insurance fraud.  Learn about and download the TIBCO Cloud Risk Investigation App.  



TECHNIQUES for Anomaly Detection

Companies around the world have used many different techniques to fight fraud in their markets. While the below list is not comprehensive, three anomaly detection techniques have been popular.

Visual Discovery

Anomaly detection can also be accomplished through visual discovery. In this process, a team of data analysts/business analysts etc. builds bar charts; scatter plots etc. to find unexpected behavior in their business. This technique often requires prior business knowledge in the industry of operation and a lot of creative thinking to use the right visualizations to find the answers.

Supervised Learning

Supervised Learning is an improvement over visual discovery. In this technique, persons with business knowledge in the particular industry label a set of data points as normal or anomaly. An analyst then uses this labelled data to build machine learning models that will be able to predict anomalies on unlabeled new data.

Unsupervised Learning

Another technique that is very effective is Unsupervised learning. In this technique, unlabeled data is used to build unsupervised machine learning models. These models are then used to predict new data. Since the model is tailored to fit normal data, the small number of data points that are anomalies stand out. Some examples of unsupervised learning algorithms are:

Autoencoders

Unsupervised neural networks or auto encoders are used to replicate the input dataset by restricting the number of hidden layers in a neural network. A reconstruction error is generated upon prediction. Higher the reconstruction error, higher the possibility of that data point being an anomaly.

Clustering

In this technique, the analyst attempts to classify each data point into one of many pre-defined clusters by minimizing the within cluster variance. Models such as K-means clustering, K-nearest neighbors etc. used for this purpose. A K-means or a KNN model serves the purpose effectively since they assign a separate cluster for all those data points that do not look similar to normal data.

One-class support vector machine

In a support vector machine, the effort is to find a hyperplane that best divides a set of labelled data into two classes. For this purpose, the distance between the two nearest data points that lie on either side of the hyperplane is maximized. For anomaly detection, a One-class support vector machine is used and those data points that lie much farther away than the rest of the data are considered anomalies.

Time Series techniques

Anomalies can also be detected through time series analytics by building models that capture trend, repeated patterns (such as seasonality, machine cycles) and levels in time series data. Here is an introduction to the Detection of Anomalies in Repeating Time Series using the MASS algorithm.  It includes a Spotfire example.  

PDF iconmatrix_profiles_and_mass_v2.pdf


A Design Pattern for Human-Centered Anomaly Detection and Classification  

For many applications, it is not enough to determine that an item is an anomaly, but also important to know how it is anomalous.  It is important to enable the subject matter expert (SME) to remain in control throughout this process.  Aided by AI, they use their knowledge of the business to help determine how anomalies will be classified and how accurate the models will be.  Human Centered AI (HCAI) provides a framework for balancing computer automation and human control.  Here is a Design Pattern that we use for generating anomaly detection models consistent with HCAI principles.  It achieves this by using a combination of Visual Discovery, Supervised and Unsupervised learning techniques.      

  1.  Detect anomalies

  2.  Determine unique 'fingerprint' for each anomaly 

  3. Cluster anomalies together with similar fingerprints

    •  SME refines assignment of items to clusters to determine the Classes of practical significance for the use case

  4.  Train supervised learning model for each Class of interest

    •  SME reviews false positives and false negatives and refines model until it achieves desired accuracy

  5.  Deploy supervised learning models to Classify new items that belong to each class of interest.

  6. Monitor model health and re-train if accuracy degrades or new classes of anomalies are detected.  This process can be automated or guided by the SME.   

This design pattern is used in the Spotfire Anomaly Detection template and our Wafermap Pattern Classification solution.


Autoencoders Explained

Autoencoders use unsupervised neural networks that are both similar to and different from a traditional feed forward neural network. It is similar in that it uses the same principles (i.e. Backpropagation) to build a model. It is different in that, it does not use a labelled dataset containing a target variable for building the model. An unsupervised neural network also known as an Auto encoder uses the training dataset and attempts to replicate the output dataset by restricting the hidden layers/nodes.

The focus on this model is to learn an identity function or an approximation of it that would allow it to predict an output that is similar the input. The identity function achieves this by placing restrictions on the number of hidden units in the data. For example, if we have 10 columns in a dataset (L1 in above diagram) and only five hidden units (L2 above), the neural network is forced to learn a more restricted representation of the input. By limiting the hidden units, we can force the model to learn a pattern in the data if there indeed exists one.

Not restricting the number of hidden units and instead specifying a ‘sparsity’ constraint on the neural network can also find an interesting structure.

Each of the hidden units can be either active or inactive and an activation function such as ‘tanh’ or ‘Rectifier’ can be applied to the input at these hidden units to change their state.

Some forms of auto encoders are as follows –

  • Under complete Auto encoders
  • Regularized Auto encoders
  • Representational Power, Layer Size and Depth
  • Stochastic Encoders and Decoders
  • Denoising Auto encoders

A detailed explanation of each of these types of auto encoders is available here.



TIBCO SOLUTIONS

Spotfire Anomaly Detection Template - Autoencoders using TensorFlow

This template uses an autoencoder machine learning model to specify expected behavior and then monitors new data to match and highlight unexpected behavior.  It features automated machine learning to optimize model tuning parameters.  The Time Series release includes time series analysis, so it can be used as a form of 'control chart', and has input component drill-down to find the most important features influencing a reconstruction error and clustering analysis to group and analyze similar groups of anomalies.  

Download the template from the Component Exchange. See documentation in the download distribution for details on how to use this template

Time Series Analysis using the Anomaly Detection Template

Using AI to detect complex anomalies in time series data:  Here is a Dr Spotfire session on using Deep Learning Autoencoders for Anomaly Detection in Manufacturing and Industrial Applications. The Spotfire Template for Anomaly Detection is used in this presentation. In a dynamic manufacturing environment, it may not be adequate to only look for known process problems, but also important to uncover and react to new, previously unseen patterns and problems as they emerge. Univariate and linear multivariate Statistical Process Control methods have traditionally been used in manufacturing to detect anomalies. With increasing equipment, process and product complexity, multivariate anomalies that also involve significant interactions and nonlinearities may be missed by these more traditional methods. This is a method for identifying complex anomalies using a deep learning autoencoder. Once the anomalies are detected, their fingerprints are generated so they can be classified and clustered, enabling investigation of the causes of the clusters. As new data streams in, it can be scored in real-time to identify new anomalies, assign them to clusters and respond to mitigate potential problems. These tools are no longer the exclusive province of data scientists. After an initial configuration, the method shown can be routinely employed by engineers who do not have deep expertise in data science. 

Anomalies and their component signatures in a time series dataset

Click on the image below to see a demo of the Autoencoder deployed to our Hi Tech Manufacturing Accelerator for real-time monitoring:

Autoencoder Model deployed for real-time monitoring


Spotfire Python Data Function - Autoencoder using TensorFlow

Spotfire allows for inbuilt Python and R data functions. An autoencoder is a versatile deep learning model that is used in multivariate regression, anomaly detection, and dimension reduction. This implementation uses TensorFlow with the Keras API; both are popular Python deep learning libraries. The data function allows a user to configure different datasets, configure different neural network architectures, train and save the neural network model, and score new data using the trained models. The Spotfire DXP includes further analysis of model features contributing towards reconstruction errors and uses reconstruction errors to find a statistical golden batch of data. More information on this asset is available here.


Isolation Forest Python Data Function for TIBCO Spotfire

Isolation Forests are known to be powerful, cost-efficient models for anomaly detection.  They isolate anomalies using binary trees, work well in high dimensional problems that have a large number of irrelevant attributes, and in situations where the training set does not contain any anomalies.  This data function will train and execute an Isolation Forest machine learning model on a given input dataset. It can be downloaded from the TIBCO Community Exchange here.   


Local Outlier Factor Python Data Function for TIBCO Spotfire

This data function uses the unsupervised local outlier factor method to perform anomaly detection on a dataset.  The local outlier factor is based on a concept of a local density.  By comparing the local density of an object to the local densities of its neighbors, one can identify regions of similar density, and points that have a substantially lower density than their neighbors are considered to be outliers.  The data function can be downloaded from the TIBCO Community Exchange here.   


Autoencoder with AWS Sagemaker using TIBCO Team Studio

Autoencoders are deep learning models that can be efficiently designed and trained using Cloud Services. This TIBCO Team Studio workflow takes Sensor data, performs data preprocessing, stores the data and trains a model in S3, and outputs model results into Spotfire and other data sources. It uses AWS CLI, Boto3 Python SDK, and Sagemaker Python SDK to access AWS resources via Python notebooks.

 

AWS Collaboration - Autoencoders, Time Series Analysis and more

re:Invent Presentations

  • Reinvent 2019 presentation - Hot Paths to Anomaly Detection: Sensor data on the event stream can be voluminous. In NAND manufacturing, there are millions of columns of data that represent many measured and virtual metrics. These sensor data can arrive with considerable velocity. In this session, learn about developing cross-sectional and longitudinal analyses for anomaly detection and yield optimization using deep learning methods, as well as super-fast subsequence signature search on accumulated time-series data and methods for handling very wide data in Apache Spark on Amazon EMR. The trained models are developed in TIBCO Data Science and Amazon SageMaker and applied to event streams using services such as Amazon Kinesis to identify hot paths to anomaly detection. This presentation is brought to you by TIBCO Software, an APN Partner.
 
  • Reinvent 2018 presentation - AI and Data Science Innovation with Amazon SageMaker. TIBCO products can interact with the data on the cloud and build any type of neural networks using TensorFlow. Specifically, TIBCO Data science working with cloud resources like AWS allows users to build unsupervised neural networks for anomaly detection on data of any size. In this example, we use AWS products (s3, EMR, Redshift and Sagemaker) to build an autoencoder using muiltiple nodes in a cluster. TIBCO brings real-time AI to business challenges with the TIBCO Connected Intelligence Cloud. In this session, we show real-time AI in action; utilizing Amazon SageMaker, TIBCO Connected Intelligence Cloud, and open source—with at-scale, in-database compute; visual composition and notebooks; Slack-style collaboration among users; and model lifecycle deployment via low-code tooling such as TIBCO Live Apps. We include case studies in equipment surveillance, dynamic pricing, risk management, route optimization, and customer engagement. Here are the slides

AWS ML Marketplace


Microsoft collaboration - Sensor anomaly detection at the edge

In collaboration with Microsoft we have developed a containerized solution for Anomaly Detection.  The TIBCO anomaly detection solution includes Microsoft Cognitive Services container deployment with anomaly detection, text mining and root cause analysis.

Watch a presentation and demo of this solution: 

Business News around TIBCO presence at MSFT Build:


Statistical Process Control

Control charts are widely used in Manufacturing, Energy, Telco, Technology and many other sectors.  They are a form of anomaly detection used to monitor key metrics, detect deviations from the baseline, and generate automated alerts.  TIBCO supports many types of Shewhart (univariate) and multivariate charts; integrated limits generation, storage and deployment; selection of rules to detect out-of-control points; tagging and annotation; management and operations dashboards; periodic or real-time alerts; process capability studies and root cause drill-downs.  More details about TIBCO SPC solutions can be found here:  

Process Control & Anomaly Detection section on the Manufacturing Solutions page



REFERENCES:

General (non-TIBCO) references:

TIBCO Corporate assets on Anomaly Detection

TIBCO Overview Webinars

TIBCO Community pages on Anomaly Detection

TIBCO Community Exchange and AWS Marketplace software downloads

 

Feedback (1)

I really like this template as it allows business analyts or citizen data scientists to find anomolies in their data using methods that would normally be unavailalbe to them.  The template is simple to use with step by step instructions along the way.

dmeade 7:42am Jun. 13, 2017