Executive Network
Click Here, Learn More
Siamak Nazari
HP Fellow, HPE
Supernova Award Category:
Internet of Things
The Company:
HPE is a Fortune 500 technology company that enables customers to go further, faster. With the industry’s most comprehensive portfolio, spanning the cloud to the data center to workplace applications, HPE’s technology and services help customers around the world make IT more efficient, more productive, and more secure. HPE is a business-focused organization with four divisions: Enterprise Group, which works in servers, storage, networking, consulting and support; Services; Software; and Financial Services.
My business unit, HPE Storage, produces and sells efficient application-integrated data storage solutions designed to start small and scale without limits. Our technology portfolio includes HPE 3PAR, HP StorageWorks, and HPE XP.
The Problem:
HPE provides Tier 1 storage arrays that customers deploy in enterprise data centers worldwide. These arrays are intended to be easy to maintain, but under the covers are highly complex systems. By collecting diagnostic data in an “Industrial Internet of Things” model, we can leverage large-scale analytics to better assess how well the arrays operate at customer sites.
One major challenge was scale. We needed to get meaningful lifecycle analytics from tens of thousands of storage arrays at customer data centers worldwide. This meant collecting and analyzing a half million data files, representing hundreds of millions of data points, every 24 hours. Traditional technologies either were too cost-prohibitive to scale, or required trade-offs in terms of analytical capabilities. We also needed diverse functional teams to have access to the same data views at both an aggregate and granular level to improve product management, sales, marketing, customer satisfaction, and troubleshooting.
The Solution:
We conducted pilots with conventional technologies including a popular RDBMS from a Fortune 100 technology company, and a columnar database. Those pilots were unsuccessful, as the technologies failed to deliver the speed and functional breadth to analyze all the data. Our data architecture and infrastructure team then deployed Arcadia Enterprise on Hadoop. Since Arcadia was built to run in big data platforms, it met our requirements around scale, flexibility, performance, and security. We were able to design data-centric applications with it to get critical real-time business insights. While Hadoop was new to our division, our deployment was a success because Arcadia Enterprise simplified many aspects of the effort. It let us avoid slow and resource-intensive processes for moving data. The visualizations gave us the ease-of-use and granular data access we needed to drill down into details. And its query acceleration enabled us to get results very quickly, even at our volume of data.
The Results:
A single global view consolidates systems, configurations, events, and log data across all deployed systems. We could see actual utilization, system duty cycles, upgrades, feature uptake, historical utilization trends, failure notifications and trends, component vendor comparisons, and unusual system behaviors. With a single unified view, we could run complicated queries and get answers within seconds.
Arcadia Enterprise provides a drag-and-drop toolset for visual analysis to let company analysts easily create interactive data applications, with no coding required. These data applications provide a common view of all underlying behaviors within the monitored data. Because Arcadia Data runs securely in place on Hadoop, any application created with Arcadia could provide continuous interactive granularity – working directly from the end user’s browser.
Passing Hadoop’s native security services through Arcadia ensures that data is accessible only to appropriately authorized users. This prevents fragmentation of security management, reducing the security risk and improving manageability of the underlying data platform.
Metrics:
The results were transformational. In the past, event analysis scripts could take hours or days to run and had to be customized for each analytical problem. Now we employ a single unified view on all our data, and we can run complicated queries and get answers within seconds.
Every hour, over 50GB of raw data are collected and processed in under 30 minutes. The system persists one week’s worth of event data. On average, 33,000 enterprise storage server units create 310K events per hour, 7.4 billion events per day, and 52 billion events per week.
The benefits were nearly immediate. For the first time, product line leaders and end users alike were able to directly access the business intelligence (BI) they needed across the company’s entire enterprise storage server installed base, including the following:
- Ability to look at and fix underutilized and poorly provisioned systems
- Access to historical and real-time details on equipment and component reliability
- View of customer usage patterns for tracking product usage
- Insight into installed features that remained unused
- Continuous updates of capacity and utilization for planning purposes
- Details on disk failures across product lines and configurations, compared by component vendor
- Better understanding of license usage across the installed base
- Potential sales opportunities based on better knowledge of systems and licensing
The Technology:
Arcadia Enterprise 4.1
Disruptive Factor:
We were able to put our IoT data to work by:
- Securely sharing data with appropriately authorized users via Hadoop’s native security services, which prevents the fragmentation of security management typical of the traditional data warehouse and BI stack.
- Gaining continuous interactive data granularity – we could drill down in streamlined succession from a bar chart to a specific system to one of its components to the raw record of a specific event, all right from a web browser.
- Rapidly propagating discoveries and creating a “data network effect” – as users from different teams learned from each other, via our data, they gained insights for planning product roadmaps, improving customer satisfaction, and achieving revenue objectives.
The adoption of Hadoop and Arcadia Enterprise had a tremendous impact on how our teams collaborate with each other. Users from different teams could work together as they parse through the mounds of data and extract insights. Discoveries by one group are rapidly propagated via the Arcadia Enterprise applications to create a “data network effect” – users are learning from each other, via data, to continuously improve the productivity and value of deployed systems in the field.
Shining Moment:
One interesting aspect of this project was the educational journey we took. We started with what we thought was a straightforward analytics implementation, and it actually steered us into a big data solution. We now appreciate the power we get out of modern architectures, as we can analyze billions of events in a shorter time than before. Being able to query on both historical and current data in seconds, versus multiple hours, helps us to drive our competitive advantage.