Matthew Singer

Senior Staff Hardware Engineer, Twitter

Supernova Award Category: 

Tech Optimization and Modernization

The Organization: 

Twitter is an online news and social networking service created in March 2006 by Jack Dorsey, Noah Glass, Biz Stone and Evan Williams, and launched in July of that year. The service rapidly gained worldwide popularity. As of 2018, Twitter had more than 321 million monthly active users. Twitter, Inc. is based in San Francisco, California, and has more than 25 offices around the world.

The Problem: 

There are hundreds of millions of Tweets every day, which then turns into over 1 trillion events per day – that represents a lot of data. Twitter uses Hadoop for storing that data and performing advanced analytics to generate important business insights. Twitter's Hadoop clusters collectively have more than an exabyte of physical storage. Because of their affordability, hard disk drives are the workhorses of Twitter’s Hadoop clusters. A typical Hadoop cluster at Twitter can have over 100,000 hard disk drives, which translates into nearly 100 PB of logical storage. But while hard drive capacities have increased over time, their IOPS has remained essentially flat. That has resulted in a storage bottleneck.

The Solution: 

To look at how they might overcome this bottleneck, Intel worked in collaboration with an Intel engineering team to conduct a series of experiments. Typical data flow into and out of a Hadoop cluster includes two key components: The Hadoop Distributed File System (or HDFS), which is the way that data is stored in Hadoop, and temporary data managed by Yet another Resource Manager (or YARN). These two data flows often occur at the same time, causing contention to access the hard drives. The engineering teams discovered that by selectively caching the YARN temporary files on a fast Intel SSD, the two data streams were no longer competing for the same resources, so hard drive utilization dropped and Hadoop could process data faster.

The Results: 

Removing the storage I/O bottleneck enabled Twitter to reduce the total number of racks in its cluster, which gives the company a smaller data center footprint. They achieve that density by moving from 12 smaller to 8 larger hard drives per system, which reduces the number of hard drives in a cluster without negatively impacting performance. And by removing the hard drive performance bottleneck, they were able to take advantage of more CPU horsepower, so they moved from 4-core processors to 24-core processors. The combination of the larger hard drives and higher-core-count processors means they can reduce the number of systems, hard drives, and racks in each Hadoop cluster in their data center, which leads to reduced maintenance costs and much better energy efficiency. It’s also great for the environment that we can produce the same results with about 75% less energy. Caching temp data and bumping up processor core counts is expected to result in up to 50 percent faster runtimes, and the increased density results in 30 percent lower TCO. This sets Twitter up to be ready for continued growth of its data, while still delivering the great experience their users expect.

Metrics: 

75% energy reduction for Twitter's Hadoop clusters

50 percent faster runtimes

30 percent lower TCO

The Technology: 

Intel Xeon Gold 6262V processors; Intel SSD DC P4610; Intel Cache Acceleration Software; Hadoop.

Disruptive Factor: 

A key breakthrough for Twitter was their willingness to challenge long-held assumptions, freeing the team up to make unexpected discoveries. For example, the Twitter team never anticipated that htere was a way to remove 75% of the hard drives from the system without harming performance.

Shining Moment: 

Matt Singer shared Twitter's story at Intel's Data-Centric Day on April 2, 2019, in San Francisco (keynote replay available at https://newsroom.intel.com/video-archive/video-intels-data-centric-innov...).

Related links:

https://twitter.com/mattbytes/status/1113129097553514502

https://www.intel.com/content/www/us/en/service-providers/enhanced-hadoo...

About Your Organization

Twitter, Inc. (NYSE:TWTR) is what's happening in the world and what people are talking about right now. On Twitter, live comes to life as conversations unfold, showing you all sides of the story. From breaking news and entertainment to sports, politics and everyday interests, when things happen in the world, they happen first on Twitter.