Welcome!

Artificial Intelligence Authors: Yeshim Deniz, William Schmarzo, Liz McMillan, Pat Romanski, Elizabeth White

Related Topics: @ThingsExpo, @CloudExpo, @DXWorldExpo

@ThingsExpo: Article

Difference Between #BigData and Internet of Things | @ThingsExpo #IoT #M2M

What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?

A recent argument with folks whose intelligence I hold in high regard (like Tom, Brandon, Wei, Anil, etc.) got me thinking about the following question:

What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?

I think the heart of that question really boils down to this:

What are the differences between big data (which is analyzing large amounts of mostly human-generated data to support longer-duration use cases such as predictive maintenance, capacity planning, customer 360 and revenue protection) and IoT (which is aggregating and compressing massive amounts of low latency / low duration / high volume machine-generated data coming from a wide variety of sensors to support real-time use cases such as operational optimization, real-time ad bidding, fraud detection, and security breach detection)?

I don’t believe that loading sensor data into a data lake and performing data science to create predictive analytic models qualifies as doing IoT analytics.  To me, that’s just big data (and potentially REALLY BIG DATA with all that sensor data).  In order for one to claim that they can deliver IoT analytic solutions requires big data (with data science and a data lake), but IoT analytics must also include:

  1. Streaming data management with the ability to ingest, aggregate (e.g., mean, median, mode) and compress real-time data coming off a wide variety of sensor devices “at the edge” of the network, and
  2. Edge analytics that automatically analyzes real-time sensor data and renders real-time decisions (actions) at the edge of the network that optimizes operational performance (blade angle or yaw) or flags unusual performance or behaviors for immediate investigation (security breaches, fraud detection).

If you cannot manage real-time streaming data and make real-time analytics and real-time decisions at the edge, then you are not doing IOT or IOT analytics, in my humble opinion.  So what is required to support these IoT data management and analytic requirements?

The IoT “Analytics” Challenge
The Internet of Things (or Industrial Internet) operates at machine-scale, by dealing with machine-to-machine generated data.  This machine-generated data creates discrete observations (e.g., temperature, vibration, pressure, humidity) at very high signal rates (1,000s of messages/sec).  Add to this the complexity that the sensor data values rarely change (e.g., temperature operates within an acceptably small range).  However, when the values do change the ramifications, the changes will likely be important.

Consequently to support real-time edge analytics, we need to provide detailed data that can flag observations of concern, but then doesn’t overwhelm the ability to get meaningful data back to the core (data lake) for more broad-based, strategic analysis.

One way that we see organizations addressing the IoT analytics needs is via a 3-tier Analytics Architecture (see Figure 1).

Figure 1: IoT Analytics 3-Tier Architecture

We will use a wind turbine farm to help illustrate the 3-tier analytics architecture capabilities.

Tier 1 performs individual wind turbine real-time performance analysis and optimization.  Tier 1 must manage (ingest and compress) real-time data streams coming off of multiple, heterogeneous sensors. Tier 1 analyzes the data, and processes the incoming data against static or dynamically updated analytic models (e.g., rules-based, decision trees) for immediate or near-immediate actions.

Purpose-built T1 edge gateways leverage real-time data compression techniques (e.g., see the article “timeseries storage and data compression” for more information on timeseries databases) to only send a subset of the critical data (e.g., data that has changed) back to T2 and T3 (core).

Let’s say that you are monitoring the temperatures of a compressor inside of a large industrial engine.  Let’s say the average temperature of that compressor is 99 degrees, and only varies between 98 to 100 degrees within a 99% confidence level.  Let’s also say the compressor is emitting the following temperature readings 10 times a second:

99, 99, 99, 98, 98, 99, 99, 98, 99, 99, 100, 99, 99, 99, 100, 99, 98, 99, 99…

You have 10,000 of readings that don’t vary from that range.  So why send all of the readings (which from a transmission bandwidth perspective could be significant)?  Instead, use a timeseries database to only send mean, medium, mode, variances, standard deviation and other statistical variables of the 10,000 readings instead of the individual 10,000 readings.

However, let’s say that all of a sudden we start getting readings outside the normal 99% confidence level:

99, 99, 99, 100, 100, 101, 101, 102, 102, 103, 104, 104, 105, …

Then we’d apply basic Change Data Capture (CDC) techniques to capture and transmit the subset of critical data to T2 and T3 (core).

Consequently, edge gateways leverage timeseries compression techniques to drive faster automated decisions while only sending a subset of critical data to the core for further analysis and action.

The Tier 1 analytics are likely being done via an on-premise analytics server or gateway (see Figure 2).

Figure 2:  IoT Tier 1 Analytics

Tier 2 optimizes performance and predicts maintenance needs across the wind turbines in the same wind farm.  Tier 2 requires a distributed dynamic content processing rule generation and execution analytics engine that integrates and analyzes data aggregated across the potentially heterogeneous wind turbines. Cohort analysis is typical in order to identify, validate and codify performance problems and opportunities across the cohort wind turbines.  For example, in the wind farm, the Tier 2 analytics are responsible for real-time learning that can generate the optimal torque and position controls for the individual wind turbines. Tier 2 identifies and shares best practices across the wind turbines in the wind farm without having to be dependent upon the Tier 3 core analytics platform (see Figure 3).

Figure 3: Tier 2 Analytics: Optimizing Cohort Performance

Tier 3 is the data lake enabled core analytics platform. The tier 3 core analytics platform includes analytics engines, data sets and data management services (e.g., governance, metadata management, security, authentication) that enable access to the data (sensor data plus other internal and external data sources) and existing analytic models that supports data science analytic/predictive model development and refinement.  Tier 3 aggregates the critical data across all wind farms and individual turbines, and combines the sensor data with external data sources which could include weather (humidity, temperatures, precipitation, air particles, etc.), electricity prices, wind turbine maintenance history, quality scores for the wind turbine manufacturers, and performance profiles of the wind turbine mechanics and technicians (see Figure 4).

Figure 4:  Core Analytics for Analytic Model Development and Refinement

With the rapid increase in storage and processing power at the edges of the Internet of Things (for example, the Dell Edge Gateway 3000 Series), we will see more and more analytic capabilities being pushed to the edge.

How Do You Start Your IoT Journey
While the rapidly evolving expertise on the IoT edge technologies can be very exciting (graphical processing units in gateway servers with embedded machine learning capabilities with 100’s of gigabytes of storage), the starting point for the IoT journey must first address this basic question:

How effective is your organization at leveraging data and analytics to power your business (or operational) models?

We have tweaked the Big Data Business Model Maturity Index to help organizations not only understand where they sit on the maturity index with respect to the above question, but also to provide a roadmap for how organizations can advance up the maturity index to become more effective at leveraging the wealth of IOT data with advanced analytics to power their business and operational models (see Figure 5).

Figure 5:  Big Data / IoT Business Model Maturity IndexMaturity Index

To drive meaningful business impact, you will need to begin with the business and not the technology:

  • Engage the business stakeholders on day one,
  • Align the business and IT teams
  • Understand the organization’s key business and operational initiatives, and
  • Identify and prioritize the use cases (decisions/goals) that support those business initiatives.

If you want to monetize your IOT initiatives, follow those simple guidelines and you will dramatically increase the probability of your business and monetization success.

For more details on the Internet of Things revolution, check out these blogs:

The post Difference between Big Data and Internet of Things appeared first on InFocus Blog | Dell EMC Services.

More Stories By William Schmarzo

Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business” and “Big Data MBA: Driving Business Strategies with Data Science”, is responsible for setting strategy and defining the Big Data service offerings for Hitachi Vantara as CTO, IoT and Analytics.

Previously, as a CTO within Dell EMC’s 2,000+ person consulting organization, he works with organizations to identify where and how to start their big data journeys. He’s written white papers, is an avid blogger and is a frequent speaker on the use of Big Data and data science to power an organization’s key business initiatives. He is a University of San Francisco School of Management (SOM) Executive Fellow where he teaches the “Big Data MBA” course. Bill also just completed a research paper on “Determining The Economic Value of Data”. Onalytica recently ranked Bill as #4 Big Data Influencer worldwide.

Bill has over three decades of experience in data warehousing, BI and analytics. Bill authored the Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements. Bill serves on the City of San Jose’s Technology Innovation Board, and on the faculties of The Data Warehouse Institute and Strata.

Previously, Bill was vice president of Analytics at Yahoo where he was responsible for the development of Yahoo’s Advertiser and Website analytics products, including the delivery of “actionable insights” through a holistic user experience. Before that, Bill oversaw the Analytic Applications business unit at Business Objects, including the development, marketing and sales of their industry-defining analytic applications.

Bill holds a Masters Business Administration from University of Iowa and a Bachelor of Science degree in Mathematics, Computer Science and Business Administration from Coe College.

IoT & Smart Cities Stories
The many IoT deployments around the world are busy integrating smart devices and sensors into their enterprise IT infrastructures. Yet all of this technology – and there are an amazing number of choices – is of no use without the software to gather, communicate, and analyze the new data flows. Without software, there is no IT. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, Dave McCarthy, Director of Products at Bsquare Corporation; Alan Williamson, Principal ...
Apps and devices shouldn't stop working when there's limited or no network connectivity. Learn how to bring data stored in a cloud database to the edge of the network (and back again) whenever an Internet connection is available. In his session at 17th Cloud Expo, Ben Perlmutter, a Sales Engineer with IBM Cloudant, demonstrated techniques for replicating cloud databases with devices in order to build offline-first mobile or Internet of Things (IoT) apps that can provide a better, faster user e...
In his keynote at 19th Cloud Expo, Sheng Liang, co-founder and CEO of Rancher Labs, discussed the technological advances and new business opportunities created by the rapid adoption of containers. With the success of Amazon Web Services (AWS) and various open source technologies used to build private clouds, cloud computing has become an essential component of IT strategy. However, users continue to face challenges in implementing clouds, as older technologies evolve and newer ones like Docker c...
Disruption, Innovation, Artificial Intelligence and Machine Learning, Leadership and Management hear these words all day every day... lofty goals but how do we make it real? Add to that, that simply put, people don't like change. But what if we could implement and utilize these enterprise tools in a fast and "Non-Disruptive" way, enabling us to glean insights about our business, identify and reduce exposure, risk and liability, and secure business continuity?
The Founder of NostaLab and a member of the Google Health Advisory Board, John is a unique combination of strategic thinker, marketer and entrepreneur. His career was built on the "science of advertising" combining strategy, creativity and marketing for industry-leading results. Combined with his ability to communicate complicated scientific concepts in a way that consumers and scientists alike can appreciate, John is a sought-after speaker for conferences on the forefront of healthcare science,...
To Really Work for Enterprises, MultiCloud Adoption Requires Far Better and Inclusive Cloud Monitoring and Cost Management … But How? Overwhelmingly, even as enterprises have adopted cloud computing and are expanding to multi-cloud computing, IT leaders remain concerned about how to monitor, manage and control costs across hybrid and multi-cloud deployments. It’s clear that traditional IT monitoring and management approaches, designed after all for on-premises data centers, are falling short in ...
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The deluge of IoT sensor data collected from connected devices and the powerful AI required to make that data actionable are giving rise to a hybrid ecosystem in which cloud, on-prem and edge processes become interweaved. Attendees will learn how emerging composable infrastructure solutions deliver the adaptive architecture needed to manage this new data reality. Machine learning algorithms can better anticipate data storms and automate resources to support surges, including fully scalable GPU-c...
As IoT continues to increase momentum, so does the associated risk. Secure Device Lifecycle Management (DLM) is ranked as one of the most important technology areas of IoT. Driving this trend is the realization that secure support for IoT devices provides companies the ability to deliver high-quality, reliable, secure offerings faster, create new revenue streams, and reduce support costs, all while building a competitive advantage in their markets. In this session, we will use customer use cases...