Overview
ZIR AI is on a mission to help the world find meaning through search as a search engine as a service. The company develops a neural network that truly comprehends human languages at a very deep level, allowing users to search by meaning versus by keywords. This has three advantages: much higher search relevance, removing language as a barrier to finding information and performing with extremely low latency.
Supernova Award Category
Data to Decisions
The Problem
With more than 6,543 languages in the world today, communication barriers are everywhere, despite the need to transfer information instantly and accurately. One ubiquitous type of communication comes in the form of an online keyword search, which, literally, does not translate across languages, often failing the user performing the search. Increasing the quality of search with accurate meaning requires more developers’ time, increasing maintenance costs, and the building of the infrastructure that enables relevance and accuracy, which can all be cost-prohibitive on the route to removing language barriers; so, ZIR AI searched for a solution that could support its data-heavy platform for accurate, immediate, and meaningful search without compromising speed or performance.
The Solution
Before selecting Aerospike as the real-time data platform to support data-laden search, ZIR AI benchmarked top competitors against each other to determine the most efficient, effective option that could deliver on high performance and low latency, while ingesting massive amounts of data. The company evaluated DynamoDB, CockroachDB, and Aerospike (to name a few): DynamoDB ingested 4 documents per second (800 write qps), CockroachDB ingested 7 documents per second (1,400 write qps), and Aerospike ingested an impressive 50 documents per second (10,000 write qps). Aerospike offered the best performance, lowest latency, best user-defined functions (Aerospike’s superpower), and highest consistency of throughputs, as well as the lowest cost with the most transparency into those costs.
The results
Aerospike is at the heart of ZIR AI’s platform. With keyword search, documents are changing constantly. And with Aerospike, ZIR AI can continuously run index optimization in real-time as documents get added, deleted, fragmented, and modified. Every single piece of data is encoded and stored in Aerospike with strongly consistent uptime and global distribution. ZIR AI is able to access a highly available source of truth for rebuilding semantic indexes. ZIR AI experiences a seamless interface with Aerospike — queries are matched against ZIR AI’s semantic index then joined with metadata from Aerospike.
ZIR AI is able to perform multiple, critical functions with only one database platform, rather than numerous computing platforms. ZIR AI can perform digital retention for all incoming data, no matter the size, as well as query-serving, which requires always-low latency (where Aerospike shines). Without needing to change platforms, ZIR AI can also engage in offline index rebuilding and optimization and quota computations with data browsing.
Metrics
ZIR AI, with the Aerospike database platform, is able to excel at high performance and low latency — two crucial metrics. Aerospike consistently delivers 99 percent of the time in less than 10 milliseconds, which is the highest bar anywhere.
Additionally, with Aerospike, ZIR AI can take in massive amounts of data. Aerospike ingests 50 documents per second (10,000 write qps), while competitors CockroachDB ingests 7 documents per second (1,400 write qps) and DynamoDB ingests 4 documents per second (800 write qps).
The Technology
The Aerospike Real-time Data Platform enables enterprises to act instantly across billions of transactions while reducing server footprint by up to 80 percent. The multi-cloud platform powers real-time applications with predictable sub-millisecond performance up to petabyte-scale with five-nines uptime with globally distributed, consistent data. It’s used to fight fraud, dramatically increase shopping cart size, enable global digital payments, and deliver hyper-personalized user experiences.
Disruptive Factor
The ZIR AI engine matches understanding and removes language as a barrier to seeking knowledge with the highest search efficacy on the market — the engine derives meaning from a keyword search with the “between-the-lines” inferences and nuances that transcend text to unearth the deeper meaning of a language. This technology disrupts the traditional keyword search markets by providing highly relevant results in a language-agnostic way. The key differentiators of the product are low latency, high search relevance, and multilingual function.
Shining Moment
Neural information retrieval techniques have truly come into existence in the last decade, significantly disrupting the traditional search markets. However, these techniques were not fast enough in terms of latency, which is required for search interactivity. We are very proud that we were able to enable these algorithms to perform with extremely low latency, thus delivering on user interactivity.