Supernova Award Category

Tech Optimization and Modernization

The Problem

Data warehouse modernization and transformation is considered a strategic imperative today for most large enterprises. The legacy enterprise data warehouses are choking under the weight of an abundance of new, unstructured and fast data sources. This is creating a concern over the ever increasing cost of ownership and operation.

One key requirement of modernization is migrating the workloads from legacy data warehouses to the big data warehouse environment. Manual identification and migration of EDW, ETL, analytical and reporting workloads is complicated, costly, time-consuming and error-prone.

To address these problems, Impetus came up with the Workload Migration Solution that revolutionizes all aspects of the manual approach to migration through automation.  While many companies offer help for moving the data, we were the first to automate the migration of the workloads.  We have automated the entire process from assessment and ROI business case development to data quality validation and execution.  The Workload Migration Solution lifts, shifts and transforms the warehouse workloads automagically to big data; with speed, reliability and zero coding.

The Solution

Impetus has been a pioneer in helping enterprises migrate to Big Data. While working with many customers, we experienced the high cost, complexity and low reliability associated with manual migration. In response, we developed a solution that can take the pain out of this migration process and reliably automate it. Since the key target was production-level accuracy, we also developed an expert system to automate testing of the migrated workloads.

The solution helps to automatically identify the right migration candidates, migrate the legacy schema to an optimized big data model, translate logic to the preferred engine and certify the migration.

While doing this, the solution ensures that:

• Since the target system is a completely different paradigm, it needs to account for and correct technology gaps in the target environment.
• The end user experiences transparency and maintains control in the automated migration process.

The results

Yes. Validation received from our customers clearly indicates that we are about to disrupt the way migration is approached. Some questions from our customers are:

1. Can you really assess all my EDW workloads, consider my SLAs and recommend migration candidates and the migration strategy?
2. Can you really take all ETL logic and translate them to production-ready Big Data equivalents, with zero coding, zero vendor lock-in and minimal risk?
3. Can you account for cases where my data model changes from source to target?
4. Do you support such a vast span of source data stores, migration candidates and target engines?

And we have not only said YES to these customers but have also been their partner in their migration journey. Here are a few testimonials:

• Before: The enterprise didn’t know what kind of workloads are the best migration candidates.

After: Assessed, migrated and freed up 40% Teradata capacity with minimal disruption for end users.

• Before: The enterprise wanted to automatically convert the Ab Initio logic to Hadoop equivalent.

After: Migratedlegacy ETL (Ab Initio) onto Hadoop and saved 50% cost against manual conversions.

• Before: The enterprise believed that manual logic translation isrisky, tedious& error-prone.

After: Automatically translated 90% of Teradata BTEQ queries to HQLs.

This has empowered our customers to have an accelerated, risk-free consolidation of the EDW and advance the data innovation agenda.

Metrics

We conducted an assessment of the offload potential of a 70TB enterprise data warehouse database. The Impetus Workload Migration solution was used to perform an automated data analytics based identification of the candidate workloads for Hadoop migration from the Oracle database (SQL/PL SQL scripts).

The key outcomes were:

• Offload of 30-40 TB
• Maintain all SLA’s
• Savings of $300K+ annually, $ 1.2MM over 5 years

The learnings from the assessment will be further utilized to offload 100% of Oracle Exadata (400TB) within next few months.

Using the Impetus Workload Migration solution, our customer enterprises have gained similar ROI opportunities and benefits.

The Technology

The Impetus Workload Migration solution is powered by such open-source technologies as Hadoop, Hive, Sqoop, Solr, Kafka, Flink, Spark, NiFi,etc.and hence comes with zero vendor lock in.

Disruptive Factor

We apply powerful automation to migrate the data models, data sets and the logic built over them in SQL Workloads, Reports and ETL Tools. Further, we validate them so that the migrations can be pushed directly to the production servers. Experientially, this is lessscience and more of an art and hence comes with multiple challenges:

• Workload Assessor
  • 360 degree assessment based on business, performance and risk SLAs.
  • Right positioning of the assessed workload.
• Workload Logic Translator
  • Supports innumerable coding patterns.
  • Resolves gaps that may exist in the target system in comparison to the source system.
  • Takes care of the cases where the data model changes across the source and target system.
• Migration Validator
  • Ensuring that the migration is error-free and the user doesn’t end up wasting time in iterative validation cycles.
  • Ensuring that translated artifacts are trust-worthy and production ready.
  • Ensuring that performance SLAs are met.

We have solved almost all these challenges and are now changing the status quo in an industry which believes that Big Data migration is costly, complex and risky. We are surely on the verge of a major disruption through the industry’s first, patent pending, intelligent Translation Engine to enable repeatable, extensible and verifiable migration workflows.

Shining Moment

An interesting moment was when we delivered following metrics to disbelieving executives on the possibility of a completely automated migration solution:

• 4xFaster
• 100%Reliable
• 5xIncrease In Developer Productivity
• 300%ROI
• 100%Enterprise-Ready

A senior banking executive,after seeing the results,said“I see you guys as the automagical warehouse transformation guys.I didn’t believe it could be done but you have shown that you are able to deliver on your claims.”