TIBCO as part of the Open Subsurface Data Universe (OSDU) project
One of the biggest challenges of the Oil & Gas industry is the fragmented technology landscape to capture, store and manage subsurface and wells data. To overcome this challenge, OSDU (Open Subsurface Data Universe) has developed an industry standard on how subsurface and wells data needs to be collected, described and served to various applications and services. OSDU also provides a reference implementation for such a data platform and standardizes on data schemas and a set of unified APIs for bringing data into the data platform.
TIBCO Spotfire simplifies access to these datasets and APIs by providing a ready-to-use dashboard app for viewing and analyzing OSDU data. This blog provides an example of how Spotfire leverages the OSDU APIs to connect and import OSDU data, and apply visual data science and geoanalytics to surface insights that can be used to drive tech and business actions.
- Spotfire 10.7 or later comes by default installed with the Native Python data function. (This is the preferred version. Spotfire 10.8 was used to create the current example)
- Spotfire 7.13 through Spotfire 10.6 client and server. Spotfire Server should have the Community Python data function extension installed to enable execution of Python code.
- If you don't have Spotfire you can download a trial version here. We are running sessions on how to use Spotfire with OSDU - see our community site for more information.
- Working knowledge of Python data functions in Spotfire dashboards. Please refer to the above links and also please take a look at this link for data functions in general.
- Acquire AWS credentials - Please contact email@example.com from AWS OSDU team.
In this example, we will be utilizing “OSDU Release 1” which is the first release to include publicly available Open API documentation (Swagger APIs) to build our dashboards. For our example, the following APIs are fundamental to work with the OSDU data.
- Authentication & Authorization API– perform authentication and authorization to get an ID token and access token for accessing protected resources (such as OSDU API endpoints)
- Search API – find data and metadata ingested and indexed in the platform using ElasticSearch Query DSL for a full-text, field match or geospatial queries and faceting.
- Delivery API – retrieve various types of resources available in the platform, such as reference data, master data, work products or work product components.
In OSDU, there are multiple types of resources which are described with JSON schemas and are grouped into group types. The data in these group types are addressed using a cloud-agnostic reference mechanism called SRNs (SDU Resource Name) that are guaranteed to be unique across the platform. For our example, we would be utilizing the following group types
- Master Data: Well and Wellbore
- Example: srn:master-data/Well:7519:
- Work Product Components: WellLog, WellboreMarker and WellboreTrajectory
- Example: srn:work-product-component/WellboreMarker:7519_csv:
Running the Spotfire Analysis:
Please download the dashboard attached to this blog (scroll down on this page to the "attachments" section) and open it in the Spotfire application. When prompted to Trust the scripts in the dashboards, navigate to “Menu->File->Manage trust and scripts” and click “Trust All”. All the data functions used to create this example can be found under “Menu->Data->Data function properties”. Every data function here is either linked to an action button or a hyperlink in the dashboard.
Now, please follow the step by step approach to see how we can consume the Authentication, Search and Delivery APIs in the data functions to download the data into Spotfire:
- Enter your user credentials in the below fields in the dashboard and click the “Authentication” button. As long as the Authentication Status is OK you are good to go. In this step, we utilize the Authentication API in AWS Authentication data function to get an ID token and Access token for accessing protected resources in AWS using the entered credentials. These credentials are stored in the Spotfire Document Properties and used accordingly in other data functions.
- Zoom in and Zoom out in the map chart to set your boundary extents. All the wells in this boundary extent will be retrieved in the next step
- After setting the boundary, please click on the hyperlink in “Click here to get well list” to get all the well names that reside in the marked boundary. In this step we leverage the Search API in Get_Well_From_Extent data function to perform this operation
- Next, we select one or more wells in the Table1 that we populated in the last step and click the hyperlink in “Click here to get Wellbore SRN and review Table2” to retrieve the WellBore SRNs for the selected wells. Here, we use the Search API in the Get_WellBore_SRN data function to retrieve the WellBore SRNs.
- Once we have the Wellbore SRNs, we can now retrieve the Work Product Components such as well logs, WellBoreMarker and WellBoreTrajectory data. In order to achieve this, select a SRN from Table2 and click on one of the Work Product Component buttons at a time to get the respective data. In this step, we use the Search and Delivery APIs in the Get_WellLog_LAS, Get_Marker_Data and Get_Trajectory_Data data functions to retrieve the data.
- You will also notice that each visualization gets updated with the latest data when a data function is successfully executed.
These dashboards, when completed, require minimal or no customization to deploy to multiple operators adhering to the same APIs and data schemas.
Whether creating simple dashboard metrics, predictive applications, or dynamic real-time analytics applications, Spotfire® delivers numerous capabilities at-scale, including visual analytics, data wrangling, predictive analytics, location analytics, and streaming analytics.
- Keep an eye on our OSDU community Site for the OSDU Spotfire App updates and next releases
- This is work ongoing, please send your feedback or suggestions to firstname.lastname@example.org
- If you have a Spotfire question please add it here
- Spotfire Interactive Demos
- Oil Well Equipment Surveillance
- Spotfire Enablement Hub
- Spotfire 7.x to 10.x Delta Training
- YouTube Channel
- Spotfire Technical Blog
- Spotfire X webinar series
- Spotfire Releases
Acknowledgments & References:
Special thanks to the TIBCO Data Science team who are working on these analyses using Spotfire (Visual Analytics; Python): Michael O'Connell, Vinoth Manamala, Andrew Berridge, Catalina Herrera.
Blog contact author: Vinoth Manamala, email@example.com
|Michael O'Connell, Ph.D., is the chief analytics officer at TIBCO, where he helps clients with analytics software applications that drive business value. He has written a bunch of scientific papers and software packages on statistical methods. He also likes listening to electronic music; watching basketball, football and cricket; going to art galleries and walking around neighborhoods.|
|With a passion for data and analytics, Catalina Herrera is a Data Scientists at TIBCO and has spent her entire career helping industry push beyond digitization to business transformation. She’s held both educational and technical positions, worked with state-of-the-art technology solutions across multiple industry verticals, and served as a data scientist and a SaaS technology evangelist. Today she works with Fortune 100 companies and global technology leaders on digital transformation initiatives.|
|Vinoth Manamala is a Senior Data Scientist working in the Data Science Team at TIBCO, based in Toronto. He is passionate about solving complex industry problems using Big Data and machine learning. He brings his experience to his current role where he works with organizations to use their data assets to their fullest potential in support of their business operations. He is an avid sportsperson; plays cricket and badminton, and also loves to travel for nature and food.|
Andrew Berridge is a Senior Staff Data Scientist at TIBCO. He specializes in extending, customizing and developing Spotfire and driving its future in all industries. He is the author of TIBCO Spotfire, A Comprehensive Primer 2nd Edition (Packt). He loves restoring classic cars and playing in orchestras in his spare time.