Google Cloud Vertex AI updates focus on the practical with Context Caching, grounding services

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Google Cloud updated Vertex AI with new models, context caching, provision throughput and a bevy of grounding updates with Google Search, third party data and an experimental "grounding with high-fidelity mode."

The Vertex AI enhancements were detailed by Google Cloud CEO Thomas Kurian in a briefing. Kurian's briefing highlights Google Cloud's cadence of Vertex AI updates from Google Cloud Next to Google I/O to those latest additions.

Kurian highlighted customers including Uber, Moody's and WPP leveraging Vertex AI and generative AI. Google Cloud has been touting the 2 million token context window for Gemini 1.5 Pro as well as the performance of Gemini 1.5 Flash. Gemini 1.5 models are generally available.

While touting its own models, Google Cloud, which is on a $40 billion annual revenue run rate, is also emphasizing choice as Kurian noted that Anthropic's Claude Sonnet 3.5 as well as new open-source models such as Mistral and Gemma 2 are available. Google Cloud also launched Imagen 3.

While models are nice, the Vertex AI updates outlined by Kurian really revolve around features that are aimed at enterprises that want to scale generative AI.

Kurian said Context Caching should lower costs for enterprises since you won't have to feed context into the model on each request. Kurian said:

"By introducing a context cache, you only need to give the model the input prompt. You don't have to feed in the context on each request. This is obviously super helpful in lowering the cost of input into the model. It becomes particularly helpful if you've got a long context window and you don't want to keep feeding that long context every time. It is also helpful in applications like chain of thought reasoning, where you want the model to maintain the context rather than must feed it and on every request. So that's a big step forward."

A live demo of Context Caching highlighted how you could analyze the Google Cloud Next keynote with Gemini 1.5 Flash and put together video clips and summaries designed for particular roles such as developers and CIOs.

A few key points about Context Caching:

• Data is in memory based on customer settings.
• Customers have two flavors with Context Caching--one based on how much you want to cache and another based on duration and time. 
• With Context Caching, Google Cloud can more easily leverage its larger context windows in models.
• Context Caching can reuse content for multiple prompts and lead to a 75% savings in cost and performance.
• Behind the scenes Context Caching is maintained in Google Cloud VPC for data residency and isolation.

Provisioned Throughput is another addition to Vertex AI. Provision Throughput is designed to provide better predictability and reliability in production workloads. Customers can "essentially reserve inference capacity," said Kurian.

"Customers can still use pay as you go, but if they want to reserve a certain amount of capacity for running a large event or a big ramp of users platform customers can reserve capacity for a time," said Kurian, who said social media companies are using Provisioned Throughput. Snap is a big Google Cloud customer. Customers can use Provisioned Throughput to use new models when generally available or to get assurance on service levels and response times.

On the grounding front, Google is taking multiple approaches where it can leverage third party data and Google Search for timely prompts. The third-party grounding service will include data providers such as Moody's and Thomson Reuters.

Grounding with high-fidelity mode is an experimental feature that includes a grounding score as well as a source. Kurian highlighted an example of a prompt asking about Alphabet's first quarter revenue and an answer with a grounding score. If the question was about Alphabet's third quarter 2024 earnings, the system looks at the context and notes that the quarter hasn't happened yet.

Kurian said:

"High-fidelity grounding is not just a feature of the grounding service. It's a feature of the model itself. The model has been adapted to provide more factuality in its response. In order to do that, the longer context window helps, because you can tell the model to pay attention to what I'm giving you on the input context, and don't get distracted by other things."

• Foundation model debate: Choices, small vs. large, commoditization
• Google Cloud Next: The role of genAI agents, enterprise use cases
• Google Cloud Next 2024: Google Cloud aims to be data, AI platform of choice
• Google Unveils Ambitious AI-Driven Security Strategy at Google Cloud Next'24
• Equifax bets on Google Cloud Vertex AI to speed up model, scores, data ingestion

Customer use cases

Nick Reed, Chief Product Officer at Moody's, said Google Cloud's grounding features are enabling his company to scale generative AI into production. "Grounding gives us an ability to be able to pair the power of LLMs with knowledge and as proprietary owners of knowledge facts and data," said Reed. "The ability to be able to combine those two things together empowers our customers to be able to use the outputs of the generative AI calls that we're building in actual decision-making processes. We're moving out of the experimental phase into a much more production and decision phase."

14 takeaways from genAI initiatives midway through 2024

Reed said the plan for Moody's is to use Google Cloud's grounding services as a distribution mechanism for its content because end users will be able to trust answers. Reed said:

"It feels like 2023 was the kind of year of experimentation and people were getting used to how this stuff works. Now we're starting to move into a world where we're saying I want to actually use this to drive efficiency in my organization and to be able to use it more directly in decision making and customer touching processes."

Stephan Pretorius, CTO of advertising firm WPP, said Gemini models are being used by the company for speed to market in marketing, which has become a critical KPI for enterprises.

"We use Gemini 1.5 Flash particularly for a lot of the operational automation tasks that we have in our in our business. Everything from brief refinement to strategy development and some ideation concepting tasks," said Pretorius, who said WPP is looking to automate marketing services and its content supply chain.

The WPP CTO added that Context Caching will be critical for the company since "it's not cheap to put 2 million tokens into a query." "If you only have to do that once and you can then cache it, then the entire pipeline of things that you do beyond that becomes a lot more cost effective," said Pretorius.

My take

With its Vertex AI updates, Google Cloud is addressing practical real-life use cases that are evolving from pilot to production. The Gemini 1.5 models have large context windows for bragging rights, but the main effort is to address individual use cases and scenarios.

The big win here is the Context Caching that can drive costs down for enterprises with a close second being the grounding with high-fidelity mode. The first addresses pain points today, but the second may become critical over time since it leverages Google Cloud's key advantage--access to real-time search data.

Kurian said the company is focusing on reducing latency with a family of models and fleshing out techniques like Context Caching that bring costs down. Add it up and what Google Cloud is ultimately building is an abstraction layer that manages context, grounding and other processes all the way down to the GPU and TPU.

More on genAI dynamics:

• OpenAI and Microsoft: Symbiotic or future frenemies?
• AI infrastructure is the new innovation hotbed with smartphone-like release cadence
• Don't forget the non-technical, human costs to generative AI projects
• GenAI boom eludes enterprise software...for now
• The real reason Windows AI PCs will be interesting
• Copilot, genAI agent implementations are about to get complicated
• Generative AI spending will move beyond the IT budget
• Enterprises Must Now Cultivate a Capable and Diverse AI Model Garden
• Financial services firms see genAI use cases leading to efficiency boom

GenAI will change software pricing models, says Progress Software CEO

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Progress Software CEO Yogesh Gupta realizes that generative AI is going to change the way software is priced and his company is experimenting accordingly. One thing is clear: GenAI is going to upend current software pricing models especially those based on seats.

Speaking on Progress Software's second quarter earnings call, Gupta was asked how the company will drive revenue growth with genAI. Progress Software's second quarter revenue fell 2% from a year ago to $175 million, but the company raised its outlook for the year. Progress Software projected fiscal 2024 revenue of $725 million to $735 million with earnings of $1.98 a share to $2.10 a share.

Progress Software is seeing gains as enterprises leverage its data platform and tools for genAI. Gupta's answer was notable only because other software executives aren't as straightforward about it. Enterprise software companies have been talking about the delay in genAI profit euphoria in their most recent quarters. No enterprise is going to pay multiple vendors copilot taxes unless there are returns attached. Progress Software is using AI to add capabilities to its products, drive efficiencies for customers and improve its own processes and operations. Progress Software is using AI internally for technical and customer service, marketing content and customer contract analysis to drive value. 

• BT150 zeitgeist: Dear SaaS vendors: Your customers are pissed 
• Copilot, genAI agent implementations are about to get complicated
• Generative AI spending will move beyond the IT budget

Here's Gupta's answer in full for context:

"I think we in this industry are trying to figure out what is the best way to price our products for the value we are generating. In reality, software pricing has been done by some metric like data consumption, capacity, users, seats or servers. Pricing is based on a measurable quantity. The business value if it is meaningful and significant is often not related to those things.

We are experimenting is the best answer I can give you because I think that is the honest answer. Business models for genAI are going to evolve over the next year or two in our industry. For some products it's easy. Personal productivity is easy. You're Microsoft and you come out with Copilot for Office. Everybody who has Office, if you use copilot, is charged more on a per person basis. We are not like that. This is business value. You might have a fairly small number of users, and you might suddenly see hundreds of thousands or millions of dollars of benefit every year.

That model isn't on a per seat basis, and it isn't on data volume basis. It is truly on the value delivered to the end customer basis. How you tie that value together is really the key question. I wish I had a specific concrete answer for you. I don't other than the fact that we are experimenting as we speak. We are working with customers and others to basically figure out what is the best way to price these things so that they get the value and pay us for the return we're generating for them."

Hats off to Gupta for that answer, which identifies the software industry's biggest conundrum. Enterprise software vendors will need to show growth, but customers are pushing back on the upsell tactics and price increases when the value isn't there yet. Licensing models, subscriptions, pay-per-usage contracts don't account for genAI. That's why there's a rush to lock in customers now before the revenue models change.

For Progress Software, pricing models may be challenging since it has a mix of license, maintenance, subscription and services revenue. The company is a frequent acquirer and recently bought data platform MarkLogic. It also has digital experience, DevOps and infrastructure management software

Indeed, the move from licensing to subscriptions and consumption models typically meant meager growth for 12 to 18 months. This transition to genAI friendly pricing may have a similar time frame.

Meanwhile, enterprises are going to need some budget predictability. Software pricing models with genAI could be more dynamic, personalized and usage based as well as volatile. There may also be models where customers pay based on the value created instead of just access. Those value-based models will create a new set of winners and losers. Value audits will be the new licensing audit for enterprises.

• ServiceNow CFO outlines method to Pro Plus SKU pricing
• Microsoft raises Dynamics 365 prices starting Oct. 1
• Adobe raises Creative Cloud prices, adds generative AI credits
• Smartsheet preps new pricing model June 24
• Amazon Q generally available with new pricing plans
• Salesforce raises list prices across clouds by about 9%

On the buy side of the equation, you can expect generative AI to act as an enterprise advocate by analyzing competition, pricing and bundling and unbundling opportunities. 

Like Gupta, I'm not sure what software pricing models will emerge with genAI. The only certain thing is that the current models don't work. I could see a day where the vendor AI argues with the buyer AI at the negotiating table.

VMware updates VMware Cloud Foundation, adds import features

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

VMware added a bevy of features to its VMware Cloud Foundation (VCF) platform including import tools that make the migration from vSphere and vSAN to VCF easier.

The updates come as VMware is under fire from customers. When Broadcom completed the VMware acquisition it shifted to subscription-based pricing in a move that raised costs for enterprises.

BT150 zeitgeist: Dear SaaS vendors: Your customers are pissed

Nutanix has been able to add VMware customers, but acknowledged last quarter that Broadcom was responding with aggressive pricing. Hewlett Packard Enterprise also entered the virtualization space by building open-source kernel-based virtual machines into HPE Private Cloud, but that move will take time to play out.

On Broadcom's fiscal second quarter conference call, CEO Hock Tan said that VMware's transition to a new model is on track. VMware revenue in the second quarter was $2.7 billion, up from $2.1 billion in the first quarter. Tan said the company has signed up nearly 3,000 of its largest 10,000 customers to deals, mostly multi-year contracts.

Tan reiterated that VMware would deliver $4 billion a quarter in revenue, but declined to give a time frame.

The VMware Cloud Foundation 5.2 release includes the following:

• VCF Import from vSphere, vSAN. The import tool enables the integration from vSphere and vSAN into VCF without downtime. With easier migration, VMware can simplify its selling cadence and move customers from point products.
• VCF Edge, which extends cloud capabilities to edge location and better manage distributed infrastructure.
• Cloud management enhancements including Tanzu Kubernetes Grid (TKG) as an independent service to manage container-based workloads, a streamlined process to adopt virtual networking and new interfaces for cloud administrators and DevOps.
• Security features for live VCF patching as well as flexible patch schedules.
• Broadcom also added new features to VMware vSphere Foundation including self-service tools, TKG and an improved user experience.

Holger Mueller, Constellation Research analyst, said VMware is motivated to move customers to VCF and it's possible that disgruntled enterprises will stick with the company despite competition from Nutanix and possibly HPE. "I expect the same story with VMware as with CA," said Meuller. "There's alarm, disgruntled customers and 85% of enterprises stay. That's a good outcome for tech."

Mueller added:

"So far, Broadcom leadership has shown its first commitment to R&D and innovation. And it is better than what neutral observers were expecting. This innovation now puts VMware customers in the pickle--trust innovation will keep flowing or bet that this update is a one time burst of capabilities. It's up to Broadcom to show a steady cadence of innovation."

BT150 zeitgeist: Dear SaaS vendors: Your customers are pissed

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Enterprise customers are miffed about software-as-a-service vendors trying to double and sometimes triple prices, layering AI-services that may not deliver returns and prioritizing revenue quotas over value.

That's one of the big takeaways from our June BT150 meetup with CxOs.

A few nuggets:

Salesforce has been particularly aggressive about upselling AI and running that cross-selling playbook to the hilt. "We're getting a lot of pressure from Salesforce to buy AI," said one CXO. "Salesforce loves to pitch Data Cloud and anything with data also requires MuleSoft. They're trying to tie it all together which is what I would do if I were them too, but it's very apparent they see AI as a threat and want to lock their customer base in." This sales practice shouldn't be that surprising given Salesforce's goal is to cross-sell you clouds to continue to grow. 

OutSystems was another vendor that was dinged by our CxOs. When the OutSystems contract was up, one CxO was told by sales rep his price would triple. After some back and forth, the solution was apparent to the enterprise customer. Get a new vendor. "These vendors are just coming up with insane things when the contracts expire," he said.

Workday, Adobe and a bevy of others were also called out for similar tactics. Some vendors will play ball and see the win in a long-term relationship and others won't. And we've previously documented the VMware and SAP angst.

BT150 2024:

• BT15O CXO zeitgeist: Vendor procurement, cool projects and margin compression
• BT150 CXO zeitgeist: Data lakehouses, large models vs. small, genAI hype vs. reality
• BT150 CXO zeitgeist: Low marks for SAP RISE, process automation, change management, AI risk
• BT150 CXO zeitgeist: AI trust, AI pilots to projects, VMware angst, projects ahead
• BT150 nominations | 2024-2025 class

What is going on here? A few thoughts on why CxOs are increasingly annoyed with their software vendors, which appear to be aggressive.

• The value provided by the software is mostly horizontal and not necessarily differentiating to the customer's industry. The copilot use cases beyond coding and haven't delivered scalable value just yet. Nevertheless, SaaS vendors are under pressure to show they are monetizing generative AI.
• Software vendors are playing the annual contract value (ACV) game, but customers want to cut budget. Vendors are happy to sell you more stuff (even if it won't be used) so they can show growth. Customers are looking to hold the line on enterprise IT spending or cut to fund more interesting projects.
• The margin compression is just starting for software vendors and they can smell the upheaval. My hunch is that terms like "elongated sales cycles"
• Enterprise buyers want to consolidate vendors and standardize contracts, but not at the expense of lock-in.
• AI is more of a threat to software vendors than a turbocharger. When you buy SaaS you're getting easier upgrade cycles, more innovation and a consumption model. In many cases, you're also getting a UI. What happens when the UI becomes a natural language generative AI bot? Suddenly the enterprise vendors that grew by relegating rivals to plumbing suffer the same fate.
• There's little competition right now. Enterprise software is the land of giants and smaller startups that aren't ready to scale. As a result, the bench looks thin, and vendors are looking to press their advantage.
• It's unclear how this plays out, but some CxOs are leaning toward build over buy. Others are plotting their escapes from overzealous vendors. The real game may not play out for 2 years or so when a) generative UI upends the SaaS market; b) new entrants smell the opportunity to disrupt incumbents.

One thing is clear: No software vendor is going to see ACV going up without delivering value and lowering customer costs. A vendor's ACV chart in an earnings report isn't the enterprise buyer's problem.

Anthropic adds more collaboration features to Claude for Pro, Team customers

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Anthropic's secret sauce could be collaboration instead of churning out the latest greatest large language model (LLM).

The Anthropic vision that's developing is interesting because LLMs are going to need something more than chat, content generation and personality to drive revenue growth and profits.

Shortly after the launch of Claude 3.5 Sonnet, a high-performing LLM, Anthropic said Claude.ai Pro and Team users can organize chats into projects to bring activity together in one place for collaboration. The feature is called Projects.

In a blog post, Anthropic noted that Claude's collaboration tools will make it part of the mix to generate ideas, provide results with context and be more strategic. With Claude's collaboration features, Pro and Team customers can add documents, code and ideas in Claude 3.5 Sonnet's 200K context window.

Key items about Claude's collaboration features being added:

• Claude can be grounded with internal documents, transcripts, codebases and past work. This grounding gives Claude the background it needs to hit the ground running on projects.
• Customers can give custom instructions for each Project based on tone, industry and roles.
• Artifacts will appear on the side as previously launched. Artifacts are handy for coding and live previews.
• Claude Team users can share information on project activity feeds, so teammates are continuously updated.
• Anthropic said it will continue to add collaboration features and integrations with various applications and tools.

My take: Anthropic is on to something with its LLM and collaboration spin. Yes, these collaboration features exist everywhere, but for enterprise and business users looking to subscribe to generative AI services Anthropic's approach can win some converts vs. OpenAI and others. Anthropic has another advantage: It can natively build LLMs and collaboration in a native way. Existing collaboration apps are going to have to take a more bolt-on approach. 

• Anthropic launches Claude 3.5 Sonnet, Artifacts as a way to collaborate
• Anthropic launches Claude Team plan, iOS app
• Anthropic CEO Amodei on where LLMs are headed, enterprise use cases, scaling
• AWS ups its investment in Anthropic as giants form spheres of LLM influence
• Foundation model debate: Choices, small vs. large, commoditization

Oracle says TikTok ban could hit cloud revenue

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Oracle said a TikTok ban could hurt its cloud revenue should the social network be banned in the US. The company revealed the risk factor in its annual report filed with the SEC.

In April, President Biden signed a law that would make it illegal to provide cloud services to TikTok unless its parent ByteDance could separate its operations from the Chinese government. The bill demands that ByteDance sell TikTok in nine months, or one year if extension approved.

TikTok runs on Oracle cloud services under an effort called Project Texas to keep US user data secure.

• Oracle partners with Google Cloud, takes on OpenAI workloads
• Oracle, Microsoft launch Oracle services within Azure data centers
• Oracle adds generative AI features into Fusion Cloud, expands Azure partnership into Europe
• Oracle launches OCI Generative AI service, plans to embed throughout databases, applications
• Palantir will move workloads to Oracle Cloud as both court governments and enterprises
• Microsoft uses Oracle Cloud Infrastructure for Bing conversational workloads

Oracle said that if it can't provide those services to TikTok its "revenues and profits would be adversely impacted." However, Oracle also is seeing strong demand for Oracle Cloud Infrastructure (OCI) and it's possible that capacity could be redeployed. OCI revenue for fiscal 2024 was $6.9 billion.

Oracle also noted that TikTok's compliance with US laws may increase its expenses. Constellation Research analyst Holger Mueller said:

"Losing TikTok workloads will dent Oracle Cloud utilization, but should hamper the overall growth of Oracle Cloud. Being the only cloud with available Nvidia capacity will keep Oracle Cloud revenue growing - with TikTok revenue - or not. Also given that we are in an election year and TikTok's popularity, it is unlikely neither the old or any new administration will ban TikTok completely in 2024."

While the TikTok note will garner attention, there were multiple other tidbits that stood out in Oracle's annual report. Here's a look.

• As of May 31, Oracle owns 29% of Ampere, which makes Arm-based processors for cloud workloads including AI inference.  Convertible debt investments in Ampere mature in June 2026 and convert to equity. Oracle invested $600 million in convertible debt issued by Ampere in fiscal 2024 and has options to buy more equity from co-investors through January 2027. The upshot, Oracle could obtain control of Ampere if options are exercised by the company or co-investors.
• Oracle said it has managed through supply chain shortages in part by "committing to higher purchases and balances of hardware products that we market and sell to our customers and that we use as a part of our cloud infrastructure to deliver our cloud offerings, relative to our historical positions." However, that move to secure manufacturing capacity increases inventory and obsolescence risks.
• Multi-cloud partnerships were cited as a risk factor. Oracle said: "Use of our competitors’ technologies can influence a customer’s purchasing decision or create an environment that makes it less efficient to utilize or migrate to Oracle products and services. For example, we offer our customers multicloud services whereby our customers can combine cloud services from multiple clouds with the goal of optimizing cost, functionality and performance. OCI’s multicloud services work with a number of our competitors’ products, including Microsoft Azure, Amazon Web Services and Google Cloud Platform. This multicloud strategy could lead our customers to migrate away from our cloud offerings to our competitors’ products or limit their purchases of additional Oracle products, either of which could adversely affect our revenues and profitability." Obviously, multicloud partnerships are expected to drive more revenue than risk.
• Oracle spent $8.9 billion on research and development in fiscal 2024, up from $8.6 billion in 2023 and $7.2 billion in 2022. Oracle ended fiscal 2024 with 47,000 employees in R&D.
• 37% of Oracle's applications revenue are cloud services in fiscal 2024, up from 32% in 2023.
• Oracle ended the fiscal year with 159,000 full-time employees with an average tenure of eight years. About 58,000 of those employees were in the US.

Shopify unveils Target partnership, new AI features

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Shopify outlined a partnership with Target that could scale distribution for its premier merchants as well as AI-enhanced features across its unified commerce platform.

The partnership with Target highlights how Shopify is increasingly becoming an enterprise commerce platform. In recent quarters, the company has noted that it is increasingly moving upstream.

• Target launches Store Companion, genAI app for employees

Under the Target deal, curated Shopify merchants will expand Target Plus, the company's third-party marketplace. True Classic and Caden Lane were cited as Shopify merchants that would be available on Target Plus.

In addition, Target will be the first mass retailer to bring select Shopify merchants products into physical stores too. Harley Finkelstein, president of Shopify, said the Target deal will help its high-growth merchants expand.

The Target deal is part of Shopify's strategy to scale into larger retail accounts, B2B and other commerce markets. Finkelstein said:

"The past years show that we can cater to both start-ups and large companies. And we continue to invest in both to expand our merchant base. Our business model focuses on accelerating the success of our merchants and driving long-term value rather than short-term gains. We are a product-led company, and we will invest in those products and strategies that ultimately offer greater value for our merchants and thereby for Shopify."

Shopify eyes offline, B2B markets for commerce growth | Constellation ShortList™ Campaign to Commerce: All-In-One-Commerce Clouds

To that end, Shopify also launched its Summer '24 Edition, which includes more than 150 updates. Among the key updates:

• Shopify Markets has been revamped as a central command center that can customize buyer experiences for selling internationally, B2B expansion and tailoring in-person offers via Shopify POS (point-of-sale). With the revamp, Markets will combine views for multiple stores and workflows.
• AI commerce tools including image editing in the Shopify mobile app and editing tools across Online Store Editor and Email Editor. Shopify also outlined how its Magic feature suggests recommendations across product categories. 

• Sidekick, an AI commerce assistant that provides context and guidance for products, orders and customers.

• One-tap digital receipts for in-person shopping and automatic detection on whether a product can be returned.
• A unified analytics experience across stores and categories.

14 takeaways from genAI initiatives midway through 2024

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Generative AI projects in the enterprise have moved beyond the pilot stage with many use cases going into production. Scaling has been a bit of a challenge, but the maturation of how CxOs are approaching genAI is underway.

There's no census of generative AI projects in the enterprise, but directionally you can follow the trend. Datadog in a recent report found that GPU instances (a good proxy for genAI) are now 14% of cloud computing costs, up from 10% a year ago. Datadog's report was based on AWS instances so that percentage may be higher once Google Cloud, Microsoft Azure and increasingly Oracle Cloud is considered.

In recent weeks, I've made the rounds and heard from various enterprise customers talking about genAI. Here's a look at what's happening at the midway point of 2024 with genAI projects and what'll hopefully be a few best practices to ponder.

The big decisions are being made now by the business with technology close behind as a consideration. Use cases will proliferate, but they'll be scrutinized based on cost savings and revenue growth.

"It's not understanding AI. It's understanding how it works," said Jamie Dimon, CEO of JPMorgan Chase. By the end of the year, Dimon estimated that JPMorgan Chase will have about 800 AI use cases across the company as management teams become better at deploying AI. He added:

"We use it for prospect, marketing, offers, travel, notetaking, idea generation, hedging, equity hedging, and the equity trading floors, anticipating when people call in what they're calling it for, answering customer, just on the wholesale side, but answering customer requests. And then we have – and we're going to be building agents that not just answers the question, it takes action sometimes. And this is just going to blow people's mind. It will affect every job, every application, every database and it will make people highly more efficient."

Build a library of use cases. Vikram Nafde, CIO of Webster Bank, said at a recent AWS analyst meeting that it makes sense to build a library of genAI use cases. This library is effectively a playbook that can scale use cases across an enterprise. He said:

"Almost everyone has 10 use cases to try. I have hundreds of use cases. I've created this library of use cases. How do I prioritize? How do I engage my inner business? How do I engage more? Which are the ones that are worth an experiment? There are other costs and not only in terms of money or resource, but like people process and so forth."

Nafde said genAI use cases are viewed through a broader lens. Sometimes, a genAI use case is just a phase of a broader project. Use cases also must play well with multiple datasets and AI technologies and are vetted by an enterprise-wide AI council. Related: Target launches Store Companion, genAI app for employees 

GenAI is a tool to solve problems, but isn't more than that. One change in the last year is that generative AI is being seen more as a tool to use to solve business problems instead of this magical technology. Unum CTO Gautam Roy said at a recent AWS analyst event:

"We don't think about AI or genAI as something separate. We are going to use it to solve problems. The first question we ask is what problem are we solving? Not everything is changed by AI. Sometimes it's a process change. Maybe it's an education or training issue. It may not be a technology issue. We ask what the problem is we are solving and then use innovation and technology to solve it."

This post first appeared in the Constellation Insight newsletter, which features bespoke content weekly and is brought to you by Hitachi Vantara.

Data platforms are converging as data, AI, business intelligence converge. Databricks is working to show it has the data warehousing chops to complement its AI and data platform. Snowflake is leveraging its data warehouse prowess to get into AI.

Speaking at Databricks Summit, Brian Ames, senior manager of production AI and data products at General Motors, said the company has stood up its data factory and plans to layer in generative AI capabilities in the next year. "GM has a ton of data. That's not the problem. We had a beautiful on prem infrastructure. Why change? Well, two reasons. Number one was data efficiency. More importantly, the world changed. And GM understood that if we didn't have AI and ML in our arsenal, we could find ourselves at a competitive disadvantage," he said.

Smaller enterprises will be using AI everywhere via platforms. Dimon noted that smaller banks will be using AI too through AWS, Fiserv and FIS. You can assume the same thing for other smaller enterprises across verticals.

Generative AI projects require a lot of human labor that's often overlooked, said Lori Walters, Vice President, Claims and Operations Data Science at The Hartford. "We spend a lot of time talking about the cost to build, about the training costs and the inference cost. But what we're seeing is the human capital associated with genAI is significant. Do not underestimate it," she said.

AI is just part of the management team now. Multiple enterprises have created centralized roles to oversee AI. Often, the executive in charge of AI is also in charge of data and analytics. Again, JPMorgan Chase's approach to AI is instructive. There's a central organization, but each business in the banking giant has AI initiatives.

Daniel Pinto, Chief Operating Officer and President at JPMorgan Chase, said the company has moved to transform its data so it's usable for AI and analytics. There will also be a central platform to leverage that data across business units. "AI and, particularly large language models, will be transformational here," said Pinto.

Compute is moving beyond just the cloud. Judging from the results from hardware vendors, on-premise AI optimized servers are selling well. Enterprises are becoming much more sophisticated about how they leverage various compute instances to optimize price/performance. This trend has created some interesting partnerships, notably Oracle and Google Cloud.

Optionality is the word. Along with the on-premise, private cloud and cloud options, enterprises will mix multiple models and processors. In a recent briefing, AWS Vice President of Product Matt Wood said: "Optionality is disproportionately important for generative AI. This is true today, because there's so much change in so much flux."

GenAI projects will have to be self-funded. At a recent Cognizant analyst meeting, there were a bevy of customers talking about transformation, technical debt and returns. Cognizant CEO Ravi Kumar said the technology services firm's customers are preparing for generative AI, but need to do work in quantifying productivity gains to justify costs. Kumar's take was echoed repeatedly by the firm's customers. "Discretionary spending in tech over the last 25 years has happened in the low interest rate regime where there was no cost of capital. Today you need a business use case for new projects," said Kumar.

I don't doubt that business case argument for genAI. Enterprise software companies have been talking about the delay in genAI profit euphoria in their most recent quarters. No enterprise is going to pay multiple vendors copilot taxes unless there are returns attached.

• Copilot, genAI agent implementations are about to get complicated
• Generative AI spending will move beyond the IT budget
• Enterprises Must Now Cultivate a Capable and Diverse AI Model Garden
• Secrets to a Successful AI Strategy
• Return on Transformation Investments (RTI)

The transformation journey to the cloud isn't done and may need to be accelerated to reap genAI rewards. Pinto said the JPMorgan Chase is revamping its technology stack.

"It's been a big, long journey, and it's a journey of modernizing our technology stacks from the layers that interact with our clients to all the deeper layers for processing. And we have made quite a lot of progress in both by moving some applications to the cloud, by moving some applications to the new data centers, and creating a tool for our developers that is a better experience. And we are making progress. The productivity of this organization today is by far higher than it was several years ago, and still a long way to go. We have optimized the infrastructure that we use, the cost per unit of processing and storage."

• JPMorgan Chase: Digital Transformation, AI and Data Strategy Sets Up Generative AI
• JPMorgan Chase: Why we're the biggest tech spender in banking
• JPMorgan Chase CEO Dimon: AI projects pay for themselves, private cloud buildout critical

• GenAI projects may be sucked up into the transformation, digital core vortex

LLMOps is emerging and will converge with MLOps in the future. Hien Luu, Senior Engineering Manager at DoorDash and responsible for building out scalable genAI at the company, gave a talk at Databricks' Summit on LLMOps. Luu said LLMOps is becoming critical due to costs because working with LLMs and GPUs isn't cheap. He expects that MLOps and LLMOps platforms will converge.

Focus on long-lasting use cases and business value instead of infrastructure. Luu's big advice is: "Things are going to evolve rapidly so keep that in mind. Identify your goals and needs based on your company's specific environments and use cases. For now, focus less on infrastructure and more on long-lasting value components."

GenAI can speed up your analytics and data insights. Volume, surfacing insights, inflexibility and time are all analytics challenges, said Danielle Heymann, Senior Data Scientist at the National Institutes of Health. Speaking at Databricks Summit, Heymann said genAI is being explored to streamline data handling, uncover patterns, adjust and evolve and accelerate processing time and conduct quality and assurance functions. NIH National Institute of Child Health and Human Development are using genAI to process grant applications and streamline review processes. GenAI is being used to classify data and conduct QA with a bot.

More on genAI dynamics:

• OpenAI and Microsoft: Symbiotic or future frenemies?
• AI infrastructure is the new innovation hotbed with smartphone-like release cadence
• Don't forget the non-technical, human costs to generative AI projects
• GenAI boom eludes enterprise software...for now
• The real reason Windows AI PCs will be interesting
• Copilot, genAI agent implementations are about to get complicated
• Generative AI spending will move beyond the IT budget
• Enterprises Must Now Cultivate a Capable and Diverse AI Model Garden
• Financial services firms see genAI use cases leading to efficiency boom