Supernova Award Category
The Problem
Prior to TrueCar’s data management system overhaul, the company operated various systems through a collection of five different data warehouses and more than 200 distinct databases. It was a system that worked well enough, but we anticipated a day when a new data-centric way of computing would become necessary. At the time, the company only had 20TB of data, but what my fellow tech leaders and I saw was that as the data grows, Hadoop would change how we collect analyze and distribute results. The initial challenge was in building our own Hadoop cluster in order to command our data. Even after approving the Hadoop investment, the other challenge we faced was in finding qualified developers.
The Solution
Having the approval to ramp efforts for Hadoop, the TrueCar team ordered petabytes of hardware to achieve an efficient 23 cents per gigabyte for data storage. With further investment, the TrueCar team began to train internal developers to work in Hadoop; one developer with results triggered wider training. Utilizing this growing developer base, the team successfully migrated its core Vehicle Intelligence System (VIS) to Hadoop via the Hortonworks Data Platform. The new VIS on Hadoop has become critical to fulfilling TrueCar’s mission to deliver data transparency.
The results
Owning our own data with a Hadoop cluster, we’ve been provided the opportunity to build new insights and entirely new business models. Thanks to the migration from our VIS to Hadoop, our new system stores raw data coming in from thousands of sources, cleans and transforms the data, processes it according to business rule and distributes it out to consumer-facing applications or anyone else who wants it. Using the Hortonworks Data Platform has given TrueCar the ability to scale. If it were not for our initial investment in our own storage and conversion to Hadoop with the Hadoop Data Platform, TrueCar would be unable to distribute its own data and would be forced into maintaining data warehouse agreements with other vendors. Nurturing the growth of our Hadoop developer team, at over 25 Hadoop developers today, we are now receiving in-bound requests from new Hadoop developers to work with our Hadoop cluster and to join our team. TrueCar is now at the cutting edge of technology compared to its competition, all thanks to our ability to collect and distribute our data.
Metrics
TrueCar takes in 12,000 data feeds, 65 billion data points and manages about 700 million car images.
Its data has grown 24-fold in the past year with system processing. TrueCar’s Hadoop success story is seen in its impressive numbers of 20 million buyer profiles and 600 terabytes of data in active use at any time. This count is reliant to the fact that it is able to provide almost real-time by updating the information, including prices, every 15 minutes.
The Technology
Hortonworks Data Platform, Spark, Elasticsearch
Disruptive Factor
This technology change came at a time when TrueCar was looking to go public, while growing at a 30% year over year rate. Our VIS provides comprehensive automotive details hitting the market today, which quickly adapts as we learn more on which data to include in our value analysis. Thanks to our introduction of the Hortonworks Data Platform, we have been able to take the latest data from vehicles, consumers and dealers in an “acquire everything” approach, integrate the data within 15 minutes and make the results easily accessible. This approach allows Truecar to accurately identify data, assess value and predict and prescribe the “who, what and where” of automotive assets.
Shining Moment
At the 2015 Hadoop Summit, we presented a keynote on our Hadoop implementation. The slide below is one of our “shining moments” which documents our growth with our data platform.