Supernova Award Category

The Problem

Lineage operates warehouses and must design them to maximize the amount of product it can store while allowing operators to move product in and out. It also must arrange its warehouses to minimize the travel time of forklift operators picking orders and putting away product. These warehouses are generally kept at 0F, requiring hundreds of megawatts of industrial refrigeration. As a result, the design of the refrigeration system and its control systems have enormous implications on how much energy the warehouses consume. Despite the obvious potential for data-driven decision making, Lineage had historically made decisions like the rest of the agricultural supply chain – via notional data and “gut feeling.” Warehouses were designed based on notions of product flow and rough estimates of throughput requirements. Additionally, refrigeration controls operated much as thermostats in homes, leaving enormous energy savings on the table.

The Solution

Lineage’s use of Cloudera Enterprise and Apache Hadoop. 

The results

Lineage’s use of Cloudera solutions and Hadoop have changed the company a great deal and it is now designing new buildings and redesigning existing buildings with millions of warehouse management systems (WMS) transactions. In addition, it is upgrading its facilities according to weather and other data that allows it to understand demand. Prior to Cloudera, it was prohibitive for Lineage to get data from different sources; one would need to ask 15 system administrators and receive data in 15 different formats. Now Lineage is able to access that information and insight by simply pressing a button. Lineage warehouse management systems now record everything that happens inside of a warehouse, and ingesting and standardizing data were the primary objective of its Cloudera and Hadoop implementation. A good example of how Cloudera enabled change involved freight pooling. With freight pooling, Lineage was able to consolidate freight shipments because, as a warehouse, it knows that whenever it handles something it knows where it’s coming from and going to. Until its enterprise data hub (EDH) came online, all of that information was in segregated servers. Now, however, Lineage is able to cut out millions of truck-miles while still getting everything where it needs to go.

Metrics

Having grown organically over the last eight years, Lineage had disparate WMS, inconsistent schemas with duplicate and mismatching tables. Lineage’s goal was to build a framework for standardizing data ingest and consolidating data that could be easily replicated across hundreds of tables in its 10-15 schemas within two independent WMS systems.The framework would serve as a baseline for future integrations. This was achieved by writing custom table and schema mapping scripts and MR jobs to create a consolidated view of the dataset. Cross-links between tables and custom views were generated using Impala.

Notable examples were real-time views of incoming and outgoing product inventory, arrangement within pellets, pellet arrangement by rack within the warehouse, case-pick slotting and reshuffling. Daily summarizations of the state of the warehouse, expected load, historical load and movement were used as key performance indicators. Previously this had been an ad hoc process.

Another notable improvement involved a flume pipeline for ingesting power and sensor data for monitoring cold storage temperatures to maintain the state of produce. For example, when a barrel of strawberries with a -30F storage environment is completely frozen, power is cycled to save costs, a process known as fly wheeling. Airflow optimization algorithms using fluid dynamics, were ported to run on HDFS. Lineage saw significant improvements in the default 96-hour freeze time for a barrel of strawberries.

The Technology

 - Cloudera Manager

- Flume (Apache)

 - Impala

 - Pig (Apache)

 - Sqoop (Apache)

 - EDH, Basic, Flex

Disruptive Factor

Using Cloudera has truly brought Lineage into the modern age of today’s connected world, allowing it to join big data, its insights and analytics, with warehousing and logistics. As a result of Lineage’s engagement with Cloudera, the company saw significant increases in warehouse utilization by designing racks to match customers’ product profile and order patterns. For instance, Lineage bin-packed a warehouse using data from its Hadoop cluster for a 32 percent increase in warehouse capacity. Dashboards have been implemented at individual warehouses for staff to monitor refrigeration temperatures and regulate power. Cloudera also helped Lineage develop a game plan for analyzing and optimizing transport routes for vendors storing inventory based on historical and current origin-destination routes, an ingest plan for weather forecast data, as well as a robotics telematics data ingest and application development.

Shining Moment

One of Lineage’s more interesting use cases involved squid freezing. Lineage had never integrated data prior to Cloudera so it was never in a position to notice, but it turned out that by routing different squid fishing companies dynamically to facilities based on where Lineage knew it had capacity, it realized an effective 36 percent increase in freezing capacity. The data basically built the company industrial freezers (the commodity it sells) for free.