Neustar is a real-time cloud based information services company based in Sterling, Virginia. The company’s 1,500 employees provide trusted, real-time information and analytics to marketing, Internet, communication services and entertainment companies around the world, helping clients promote and protect businesses and ensure optimal network performance.
With almost twenty years managing complex data registries for communications networks, Neustar is a proven expert in managing the flow of data through those networks. Today, its complete services combine data registry management expertise with best-in-class data analytics and privacy-compliant, permissible-use information services, delivered to clients in real-time, one interaction at a time.
In 2011, Neustar’s CEO challenged her leadership team to extend their business to capture new opportunities in the information services space. They needed to realize the value of the immense volume of data flowing through the company’s data registries, and optimize that value for clients in a privacy-friendly manner while simultaneously creating new businesses providing data analytics and real-time information services using authoritative, accurate, permissible-use data.
At the time, Neustar’s data architecture was insufficient to meet the challenge. Because of storage costs and capacity limitations, the company was storing only 20 terabytes of its network data (less than 10% of the total data available). They were only retaining that for 60 days, on a rolling basis. The Neustar team took on the challenge of capturing and storing 100% of the data and to store that for at least one year.
The team needed to accomplish three things to meet the challenge: store different types of data for longer time windows, speed data processing workloads, and provide a common data platform, easily accessed by all of its data analysts.
Neustar leveraged the open source Hadoop Hortonworks Data Platform (HDP) to manage its massive data processing workloads. Hadoop enabled Neustar to move from limited sampling and aggregation of data to full capture of detailed records throughout the project.
Before deploying HDP, Neustar typically limited data retention by developing a business need hypothesis and then retaining only the data it needed to confirm or refute it – identifying the data to be retained for deeper analysis based on the hypothesis rather than on insight drawn from examining the incoming raw data.
HDP allowed Neustar to replace this approach with a parse on demand solution, which allowed them to pre-parse data only after that data had proven value and deliver analysis on demand. Neustar data analysts needed fast access to all of the raw data, not just a subset. HDP also enabled the data to be available for longer to uncover subtle, long-term trends that might not surface in one or two months.
Neustar began its next-gen data platform project with seven core beliefs with which Hortonworks aligned. Now in its second year with Hortonworks, Neustar has seen tangible benefits:
- New Data Platforms Unlock Innovation - Adopting a new platform requires effort, but is rewarded with innovation and competitive advantage.
- Implementation is a “Contact Sport” - Since Neustar engineers maintain the solution, they needed to participate in the initial implementation.
- Open Source Motivates the Team - Open source is needed to energize the next generation of engineers.
- Rethink Assumptions - Hadoop enabled Neustar to move from limited sampling and aggregation for 45 days of data to 100% capture of detailed records for over a year. Data sources previously not retained are now easily ingested and queried.
- Focus Data Teams - Previously, application exhaust data was only used within the confines of a single app for the benefit of its clients. With Hadoop, insight is enriched through the mashing of data across apps.
- Increase Technology Skills – New platforms will increase the team’s technology skills only if the vendor shares the state of the art with the company.
- Security and Privacy Focus - Any solution must facilitate the strictest security and “Privacy by Design” principles for clients and consumers.
Neustar-Hortonworks alignment has produced more than just good will. Neustar and its clients have realized significant bottom-line benefits from adopting HDP and partnering with the Hortonworks team to make the most of the new architecture.
- By committing to HDP in March 2012, Neustar eliminated a large license refresh fee, and dramatically decreased annual support fees.
- By moving to HDP, Neustar has met the challenge of capturing and storing 100% of the raw data flowing through its networks. Hadoop’s storage efficiency improved Neustar’s data capacity by 150 times over its architecture before Hadoop.
- The team surpassed the goal of retaining one year’s data. Now they can save and retain data for up to two years in a secure, privacy compliant manner.
- More data has translated to more insight. The Hadoop ecosystem enables Neustar to create products that help clients improve the effectiveness of promoting and protecting their businesses, enabling them to correlate Internet traffic patterns to consumer behavior by geography and demographics.
- Data savvy product managers are thinking outside the box as more data is readily available in their Hadoop cluster. The newly accessible data assets are enable the company to offer clients value-added insights not possible before. The trend of enriching Neustar’s application data with trusted, authoritative third party data is continuing to accelerate the development of increasingly innovative services.
Hortonworks Data Platform - http://hortonworks.com/hdp/
Neustar’s HDP deployment helped the company accomplish several key core objectives:
- With HDP, Neustar increased storage capacity by 150x
- Neustar captured 100% of its data, and stored it for two years
- Neustar saved millions of dollars by moving workflows to Hadoop
- The company launched new data services, giving deeper insights into long term trends that shape future company initiatives
In 2011, Neustar’s CEO Lisa Hook challenged her leadership team to extend their business to capture new opportunities in the burgeoning information services and analytics space. Hook knew that the immense volume of data flowing through the company’s data registries was valuable, and saw the opportunity to optimize that value for clients in a privacy-friendly manner. With Hortonworks solutions, Neustar was able to realize Hook's vision without interrupting services.
