Supernova Award Category

Data to Decisions

The Problem

How do you analyze billions of telemetry data points from NASA’s Curiosity Rover, 150 million miles away, in real time?

Mars is the only known planet to be inhabited solely by robots. Those robots are continually sending billions of data points to earth. The ability to collect, send and analyze those data points in real time is critical to the success of Mars Curiosity and Soil Moisture Active Passive (SMAP) Earth satellite missions.

For example, If NASA’s JPL could correctly predict thermal parameters, Curiosity’s drive time on Mars could increase dramatically, which could lead to new and groundbreaking discoveries. Conversely, a mistake could seriously affect the $2B mission. In the past, JPL operations personnel painstakingly collected thermal telemetry data into PowerPoint that experts later analyzed. This took weeks and did not allow for interactive analysis or comparing data from several Curiosity instruments. This added cost, risk, and led to lost opportunities. The operational system was already built and changing it was both too costly and too risky.

The Solution

NASA’s JPL rebuilt its analytics systems around the Elastic Stack which now monitors and processes its flight telemetry and system dashboards from Curiosity and SMAP and gives an easy way for engineers to monitor and measure instruments given back from spacecraft.

All of Curiosity’s operations are planned a day in advance based on data received the previous day. With the Elastic Stack, JPL can get quick insights over multiple days’ worth of data, or even the entire mission, in just a few seconds. This move to real time analytics vastly speeds up the time in which decisions can be taken by mission control and made the Elastic Stack a clear front runner when evaluating options.

Using the Elastic Stack enables JPL to spot patterns and anomalies in the datasets far more quickly. Correlations which could provide mission-critical insights are more likely to become apparent, leading to a greater rate of scientific discovery and less danger of malfunction or failure.

The results

The analytics coupled with the approach of open development, moments of engagement, and rapid prototyping has become an innovation factory and will pay dividends for years to come. Some highlights of the impact:

-For the first time, operators are able to compare a flying spacecraft to how it behaved during test, leading to new insights.

-The Analytics enabled infusion of new technologies at low risk and scalable and affordable operations.

-An open environment helped developers to bring in new technologies and share code across many missions and multiple domains where sharing was previously not done.  Modern technology and practices are proliferating between groups that would otherwise not interact or share and code is reused for different purposes.

-Interactive and visual analysis of massive amounts of spacecraft data is leading to important new insights into the spacecraft. Analysts now easily work together from remote locations and can now interact with and query all the data at once. This included analytic Streams which were used during SMAP’s January 2015 launch and helped answer a question in real time that, if left unanswered could have forced the launch to be aborted.

Metrics

Beginning with deployment of the Curiosity rover in 2012, Elasticsearch enabled NASA to stream footage of the rover’s landing on Mars to thousands of viewers in a quick, efficient manner, increasing cost-effectiveness by 101,000 times and enabling the stream 175 terabytes of data in just a few hours.  Communication between scientists and Mars rovers is difficult to sync – there are only two windows throughout the day when it can occur, taking approximately seven to 20 minutes per transmission. The new high-performance cloud capabilities enable NASA/JPL to seamlessly collaborate between mission-control and information technology (IT), producing real-time results – 2.4 billion data points copied from GovCloud in real time and able to work inside the Elastic stack.  Further, new cloud developments provide NASA/JPL with an agile, flexible, and efficient space for scientists located around the globe, and enables them to work simultaneously – accelerating time by 840 percent.

The Technology

The Elastic Stack, which encompasses the open source products Elasticsearch, Logstash, Kibana and Beats and commercial features for security, alerting, machine learning, graph analytics and more.

Disruptive Factor

This is the first time in JPL history that a mission has been able to compare spacecraft behavioral data during the ground testing and actual operations in a simple visual manner, with all the data at their fingertips. This will change the way missions use test and operations data in the future and Streams will be used on many new missions. For example, it could enable the Mars 2020 Rover to gain significantly more drive time on Mars with no change to the Rover itself.

Shining Moment

JPL also uses the Elastic Stack for the Soil Moisture Active Passive (SMAP) mission. SMAP uses high powered radar and radiometer to analyze and capture high resolution data on soil moisture across the globe. A full moisture map of the Earth’s surface is generated every three days – data creation at an order of magnitude greater than that of the Mars Rover. While the Rover has so far generated around one billion data points, the SMAP project is already pushing 20 billion.