Supernova Award Category
The Problem
Opower needed a Big Data platform that could support cloud-based analytics for over 95 utility partners, including 28 of the 50 largest U.S. electric utilities, and over 50 million household and business customers in nine countries. In addition, Opower sought scalability to handle hundreds of billions of data points from energy meters, third-party data feeds, and event data.
Opower’s key challenges were maintaining a stable environment to predictably meet service level agreements (SLAs), minimizing hardware expenses, and diagnosing performance problems in a timely manner.
The Solution
Eric Chang, Data Infrastructure Technology Lead at Opower, tried various solutions, including extensive cluster tuning, employing scheduling best practices, capacity planning, etc., but to no avail.
Opower needed a solution that could dynamically handle resource contention in real time, making the thousands of necessary decisions each second to ensure specific jobs and users got guaranteed access to cluster resources. This level of control would allow mission critical jobs to complete on time even if multiple, concurrent workloads were running on the same cluster.
Opower deployed Pepperdata’s software in Spring, 2014, and saw immediate throughput gains on their existing infrastructure. Additionally, Opower could now configure Pepperdata policies to guarantee resource allocation for high priority workloads — thus ensuring SLAs. This allowed them to safely run more workloads on their existing cluster.
The results
With Pepperdata software deployed, Opower’s Hadoop administrators now have the ability to monitor every facet of cluster performance in real time. Visibility into CPU, memory, disk I/O, and network usage by job, task, user, and group makes it easier to proactively identify potential performance problems before they occur and take preventive actions.
When performance bottlenecks do occur, Opower’s Hadoop administrators are able to quickly diagnose and fix the problem. Troubleshooting activities that used to take days are now typically completed in a matter of minutes.
Opower is also using Pepperdata software to dynamically adjust job resources to reflect their service level priorities, ensuring that their clusters are devoting sufficient resources to appropriate jobs.
Metrics
In using Pepperdata, critical jobs now run faster, more reliably, and more efficiently on Opower’s existing servers, allowing the infrastructure team to scale services with fewer hardware resources increasing ROI. Further details on specific metrics can't be disclosed due to privacy agreements.
The Technology
Pepperdata is the only solution for Hadoop and Spark that enables true SLA enforcement for mission critical jobs, applications, or users. With Pepperdata, you can run multiple, concurrent workloads on a single cluster with consistent, reliable performance. Pepperdata’s software installs on any Hadoop cluster in less than one hour, and requires no modifications to schedulers or workflows.
Disruptive Factor
Automatically adjusting for thousands of system changes each second, Pepperdata software installs on any existing Hadoop cluster and is delivering business results at the world’s largest companies:
• Improving ROI on capital equipment via efficiency gains of 30-50%
• Guaranteeing SLAs on time-sensitive jobs, allowing customers to transfer workloads from expensive proprietary systems to cost-efficient Hadoop installations.
• Increasing cluster uptime and delivering better results to users.
We are the only solution available that can handle resource contention dynamically and in real time on existing infrastructure with no changes to scheduler or workflow.
Shining Moment
Pepperdata believes its shining moment comes from being able to deliver real and meaningful results for its customers. Pepperdata is the only solution on the market today that allows customers to guarantee SLAs for critical jobs and that automatically handles resource contention in real time without requiring any modifications to a Hadoop scheduler or job flow.