Results

OpenAI takes stab at gauging productivity gains, AI economic impact

OpenAI plans to measure productivity gains for ChatGPT over the 12 months via the company's economic research team and academic partners.

The research from OpenAI is worth noting. Anthropic has a similar group and recently outlined its use popular use cases and economic impact.

Technology vendors have historically produced research to argue that they boost productivity. Whether it's AI PCs, productivity software or enterprise systems, these research efforts all have some element of marketing involved.

That said, OpenAI has scale--more than 2.5 billion messages per day globally--and insights into use cases. There's also something to be said for collecting data since AI is going to cost jobs over time.

OpenAI's first research volley on economic productivity has a bevy of use cases and stats worth pondering. Here are a few data points:

  • Learning and upskilling is the most popular use case with 20% of usage.
  • 18% of ChatGPT use cases revolve around writing and communication.
  • 7% of ChatGPT usages is programming, data science and math.
  • Design & creative ideation is 5%.
  • 24% of US ChatGPT users are between the ages of 18 and 24 with 32% between 25 and 34.
  • Among OpenAI's enterprise customers 20% are in finance and insurance, followed by 9% in manufacturing and 6% in education services.
  • OpenAI's o1 model increased lawyers' productivity between 34% to 140% across six workflows.

The upshot to OpenAI's first missive is that time savings across multiple industries is valuable. Most of the value today is eliminating the need to hire another human.

Although the data is worth noting, AI usage will need to be tied to more business metrics and economic impact. The big looming question is this: Will AI's benefits be outweighed by likely job losses and downstream effects?

More:

 

Data to Decisions Future of Work Innovation & Product-led Growth Tech Optimization Next-Generation Customer Experience Digital Safety, Privacy & Cybersecurity openai ML Machine Learning LLMs Agentic AI Generative AI Robotics AI Analytics Automation Quantum Computing Cloud Digital Transformation Disruptive Technology Enterprise IT Enterprise Acceleration Enterprise Software Next Gen Apps IoT Blockchain Leadership VR Chief Information Officer Chief Executive Officer Chief Technology Officer Chief AI Officer Chief Data Officer Chief Analytics Officer Chief Information Security Officer Chief Product Officer

SAP delivers 24% cloud revenue growth in Q2

SAP said its cloud revenue in the second quarter was up 24% with total revenue up 9%.

The company reported second quarter net profit of €2.46 billion, or €1.45 a share, on revenue of €9.03 billion, up 9% from a year ago. A weaker US dollar was a currency headwind in the quarter. Adjusted earnings were €1.50 a share.

Cloud ERP revenue in the second quarter was €4.42 billion, up 20% from a year ago.

As for the outlook, SAP laid out the following:

  • Cloud revenue at constant currency of €21.6 billion to €21.9 billion, up 26% to 28%.
  • Cloud and software revenue of €33.1 billion to €33.6 billion, up 11% to 13%.
  • The company also said its current cloud backlog growth in constant currencies will slightly decelerate in 2025. In the second quarter, current cloud backlog was €18.05 billion, up 22% from a year ago.

CEO Christian Klein touted SAP's efforts with its Joule AI agent and SAP Business Data Cloud. CFO Dominik Asam said:

"Our performance was supported by continued customer demand and disciplined cost control. As we move into the second half, we remain cautiously optimistic, keeping a close eye on geopolitical developments and public sector trends."

Constellation Research analyst Holger Mueller said:

"SAP had a good quarter, and manages to grow - albeit nominally. The expected reductions in cloud backlog means SAP is seeing customers go live on the cloud and expanding capacity. The cloud revenue slowdown maybe part of the seasonality where Q1s are strong and the second and third quarters are affected by vacations in Europe. Interesting to see Spain having an outstanding quarter."

Key takeaways from SAP's earnings call from Klein:

  • SAP is including Business Data Cloud in deals with Adobe and BAE Systems as wins. Klein added that SAP "is deepening our partnership with Palantir in the context of Business Data Cloud."
  • "The debate on digital sovereignty and the best way to achieve it has picked up speed in recent weeks. SAP stands out as the only vendor that can offer sovereignty over the entire stack, from the infrastructure to the application. Our platform runs on any hyperscaler and many local providers, but we also operate data centers of our own across the world. Our unique capabilities ensure that customers stay in control of their data at all times. They can be sure, regardless of how their local sovereignty requirements evolve, we will be able to meet them."
  • "By the end of the year, we expect the total number of available AI agents to reach 40. The agents will work across business functions."
  • "SAP also uses business AI internally to boost productivity. This is reflected in the solid expansion of our operating profit. We are decoupling expenses from revenue growth thanks to our transformation program."
  • "Uncertainty in global markets from earlier this year remains. A few individual industries have been impacted by uncertainty, and we are seeing extended approval workflows on the customer side in the US public sector and among manufacturers affected by tariffs. Whatever the market environment may bring, SAP is really well prepared."

 

Data to Decisions Next-Generation Customer Experience Chief Information Officer

GM's Q2: A look at the technology, AI takeaways

General Motors said it is expanding its software services revenue, adding AI talent and honing its development practices to bring down warranty costs.

The auto giant reported second quarter earnings of $1.89 billion, or $1.91 a share, on revenue of $47.12 billion, down nearly 2% from a year ago.

Here are some of the enterprise technology and AI takeaways from GM CEO Mary Barra and CFO Paul Jacobson.

Customer experiences via OnStar, Super Cruise and software services. Barra said GM has begun pricing vehicles to include a period of basic OnStar services. The move has increased subscriptions giving GM "even more ways to engage directly with our customers through the life of the vehicle," said Barra.

So far, GM has booked $4 billion in deferred revenue from software services. Super Cruise revenue is projected to top more than $200 million in 2025.

Investing in AI talent. GM's Barra said the addition of Sterling Anderson as chief product officer highlights the company's AI talent investment. Anderson was chief product officer from Aurora, an autonomous trucking company.

"We are also embracing AI across the enterprise, which is why we recruited Google and Cisco veteran, Barak Turovsky to lead our efforts under Apple veteran, Dave Richardson, who leads software and services engineering," said Barra.

The company is looking to AI to improve vehicle performance, customer experience and operations. GM has also partnered with Nvidia to build digital twins, robotics platforms and virtual testing tools.

Using over-the-air updates to bring down warranty claims. Jabobson said GM's higher warranty claims over software issues in electronic vehicles resulted in higher warranty claims. The company is also shifting some supply of components to improve quality.

"Our expanded use of over-the-air updates, lower number of incidents per vehicle and increased robustness in our infotainment system updates are all contributing to this improvement," said Jacobson. "Additionally, we are leveraging our enhanced diagnostics in developing new prognostic tools to identify issues sooner, develop repair procedures faster and minimize unnecessary repairs."

 

Data to Decisions Future of Work Next-Generation Customer Experience Chief Information Officer

Udemy launches MCP server, aims to embed learning content into workflows

Udemy is betting that learning content will be better utilized when integrated into workflows and AI-driven applications.

The company said it will launch a Model Context Protocol (MCP) server that's designed to give enterprises the ability to embed personalized learning into work tools.

According to Udemy, education content can be embedded into AI tools such as OpenAI's ChatGPT, Anthropic's Claude, Perplexity and Cursor.

Udemy's approach is worth noting given that enterprise training and upskilling programs are generally disjointed and segmented from day-to-day work. It wouldn't be surprising if Coursera, which offers business training courses, and Accenture, which acquired Udacity, Ascendient Learning and TalentSprint to build out its LearnVantage platform, launch MCP servers to embed into productivity tools.

Udemy MCP Server will include the following:

  • On-demand support via micro-courses and curated lessons.
  • Integration with pre-built MCP connectors for existing AI tools.
  • Content matching tools to embed learning content into coding environments and CRM systems.
  • Contextualization by role and business priorities.

The company said Udemy MCP Server will launch in August with Udemy Business customers and partners getting early access.

Data to Decisions Future of Work Chief Information Officer

OpenAI's CEO of Applications pens intro missive: 5 takeaways

Fidji Simo, incoming CEO of Applications at OpenAI and Instacart CEO, penned her first missive and laid out an optimistic vision for AI.

Describing herself as a "pragmatic technologist", Simo said OpenAI has to get AI right so that it benefits as many people as possible. "Every major technology shift can expand access to power—the power to make better decisions, shape the world around us, and control our own destiny in new ways. But it can also further concentrate wealth and power in the hands of a few—usually people who already have money, credentials, and connections," said Simo.

She added that OpenAI has to be intentional about how it builds and shares its AI. Here are the takeaways from Simo's introductory memo.

Simo believes in OpenAI's potential. Yes, there's an AI hiring spree going on, but the most interesting item in Simo's hiring is this: Simo is a CEO of a publicly listed company that has a lot of runway ahead and has decided to leave to ride shotgun to an entrenched CEO. There aren't many executives that would make that move. Simo technically is CEO of Applications, but Sam Altman leads the charge.

The vision aligns with OpenAI's verticals. AI can change healthcare outcomes. AI can democratize education. It's not surprising that OpenAI is playing in those same areas.

Simo is likely to be a good buffer between OpenAI, Altman and sometimes tone deaf nerds. Simo's memo noted that she hired a tutor for biology and genetics and has a business coach. Humility and AI tech bros typically aren't mentioned in the same sentence. Before becoming CEO of Instacart, Simo was Vice President and Head of Facebook and also founded the Metrodora Institute, a multidisciplinary medical clinic and research foundation dedicated to the care and cure of neuroimmune axis disorders.

OpenAI is aiming to be a mass market AI coach that can be used for knowledge, health, creative expression, economic freedom, time and support.

A roadmap for applications will emerge to address those broad themes outlined in Simo's memo. As LLM capabilities all converge to be good enough, it'll be critical to create applications and ecosystems for staying power. Simo's job will be to develop OpenAI apps that can compete with frenemy Microsoft, Google and a bevy of others across the consumer and enterprise markets.

Data to Decisions Future of Work Innovation & Product-led Growth Next-Generation Customer Experience Chief Executive Officer

Verizon bets on AI experiences, Google Cloud to bring down churn

Verizon has rolled out AI customer experiences and is betting that the move will win accounts in a hotly contested wireless services market. The company announced a partnership with Google Cloud in April to deliver AI experiences with Gemini models and Verizon went live June 24.

Hans Vestberg, CEO of Verizon, said the company was leveraging AI to make customer experiences "simpler, faster and more rewarding." The June rollout includes a personalized expert for complex issues using Google Cloud and Gemini. A new Customer Champion will make sure issues are resolved.

Verizon also launched 24/7 live support and infused the new My Verizon app with an AI-powered Verizon Assistant and Savings Boost.

When Verizon announced better-than-expected second quarter earnings, Vestberg and CFO Anthony Skiadas were asked about returns on those Google Cloud investments.

"The wireless market remains competitive, and we continue to take a strategic and segmented approach, maintaining our financial discipline," said Vestberg. "As expected, postpaid phone churn remained elevated this quarter, reflecting the lingering effects of our pricing actions and ongoing pressure from federal government accounts. We're actively focused on improving retention by strengthening our value propositions, and leveraging our AI-powered customer experience innovations."

In the second quarter, consumer wireless retail postpaid churn was 1.12% and wireless churn overall was 0.90%. A year ago, retail postpaid phone churn was 0.85%. A year ago, Verizon didn't split out its metrics for business and consumer. Verizon reported adjusted earnings of $1.22 a share on revenue of $34.5 billion, up 5.2%, in the second quarter.

Vestberg said Verizon has seen upgrades down in year-over-year comparisons for 8 out of the last 9 quarters. Verizon did see an uptick in the second quarter and it is combining customer wireless plans with fixed wireless access and fiber broadband. Fixed wireless access accounts now have topped the 5 million subscriber mark.

Skiadas said Verizon is focused on keeping customers and courting new ones while preserving profits. "Volume growth is only valuable when aligned with our disciplined financial framework," he said. "We have taken a series of actions to address our elevated churn. On June 24, we launched initiatives designed to improve the customer experience, including leveraging AI for more personalized support. In addition, we continue to enhance our value proposition and build customer loyalty through the best value guarantee. We provide exclusive access to the best events and experiences and our refresh app helps customers maximize the value of their plans."

Vestberg emphasized that Verizon isn't going to sacrifice financials to acquire customers. He added that Verizon will turn up promotions when there are opportunities to acquire high-quality customers and let others go. With churn, Vestberg said he is optimistic about AI-driven experiences and "very encouraged about what the team is doing and how they are working on the loyalty and retention of our customers."

Verizon was also clear that retaining customers was paramount. Vestberg noted the following:

  • AI tools are critical, but Verizon has also retooled customer care processes. "When it comes to the process, we now will have a customer care employees following any request or a complaint from our customers all the way. So we actually finish it out with the same person starting and ending and also having uptake," said Vestberg.
  • 24/7 customer service is also enabled by AI. "We're giving our customer care employees, an AI tool, so they can treat our customer better and know their problems better because this could be stressful," he said.
  • Stores matter. Vestberg said 93% of the population in the US has a Verizon store within 30 minutes.
Data to Decisions Future of Work Next-Generation Customer Experience Chief Information Officer

The rise of good enough LLMs

Large language models (LLMs) have reached the phase where advances are incremental as they quickly become commodities. Simply put, it's the age of good enough LLMs where the innovation will come from orchestrating them and customizing them for use cases.

This development is great for enterprises, which will be able to buy perfectly serviceable private label, rightsized and optimized models without worrying about falling behind in 10 minutes. Foundation models are reaching the point where for some use cases there won't be much improvement with upgrades. Do you really care if a new LLM is 0.06% better in math or reasoning relative to another you plan to use in the call center on the cheap?

Get the Insights newsletter

Use cases for documentation, summarization, extracting information and spinning up content aren't going to see big improvements with model advances. In other words, thousands of enterprise use cases can leverage generic LLMs. The model giants are almost confirming that they’ve hit the wall since they’re competing on personality, snark and faux empathy in LLMs.

The enterprise value is going to be delivered by the orchestration, frameworks and architecture that mix and match LLMs based on specialties. 

Here's a look at some developments that point to the age of incremental for LLMs.

Zoho launches Zia LLM, which is essentially a disruptive private label LLM family. Zoho is a company that makes you really question your SaaS bill. And now Zoho is infusing its platform with its Zia LLM, AI agents and orchestration tools. Zoho said by controlling its own LLM layer it can optimize use cases, control costs and pass along savings to customers.

Raju Vegesna, Chief Evangelist at Zoho, said the company isn't initially charging for its LLM or agents until it has a better view of usage and operational costs. "If there a big operational resource needed for intensive tasks we may price it, but for now we don't know what it looks like so we're not charging for anything," he said.

Just like consumers are going to private label brands to better control costs, enterprises are going to do the same. The path of least resistance for Zoho customers will be to leverage Zia LLMs.

Amazon Web Services launches customization tools for its Nova models. Nova is a family of AWS in-house LLMs. I overheard one analyst nearly taunt AWS for having models that don't make headlines. That snark misses the point if Nova offers good enough performance, commoditizes that LLM layer and controls costs.

The Nova news was part of AWS Summit New York that focused on fundamentals, architecture and meeting enterprise customers where they are. As most of us know, the enterprise adoption curve is significantly slower than the vendor hype cycle.

AWS said Nova has 10,000 customers and the cloud provider plans to land more with optimization recipes, model distillation and customization tools to balance price and performance. Nova will also get customization on-demand pricing for inference.

With those two private label moves out of the way there's are developments out of China and Japan worth noting.

Moonshot AI, a Chinese AI startup backed by Alibaba, launched Kimi K2, a mixture of experts model that features 1 trillion total parameters and 32 billion activated parameters in a mixture-of-experts architecture. Moonshot AI is offering a foundation model for researchers and developers and a tuned version optimized for chat and AI agents.

Kimi K2 is outperforming Anthropic and OpenAI models in some benchmarks and beating DeepSeek-V3. The real kicker is that Moonshot is delivering models that cost a fraction of what US proprietary models for training and inference.

Research from Japanese AI lab Sakana AI indicates that enterprises may want to start thinking of large language models (LLMs) as ensemble casts that can combine knowledge and reasoning to complete tasks.

Sakana AI in a research paper outlined a method called Multi-LLM AB-MCTS (Adaptive Branching Monte Carlo Tree Search) that uses a collection of LLMs to cooperate, perform trial-and-error and leverage strengths to solve complex problems.

Add it up but the new age of incremental for LLMs is why I’m wary of the gap between AI euphoria and enterprise value. It’s not a slam dunk that buying the latest Nvidia GPUs, paying $100 million to AI researchers to create super teams and building massive data centers is going to yield the breakthroughs being pitched.

It appears that LLM giants know where the game is headed as they build ecosystems and complementary applications around their foundation models. For instance, Anthropic is best known for its Claude large language model (LLM), but its enterprise software ambitions are clear as the company builds out its go-to-market team. OpenAI is focusing on ChatGPT advances for headlines, but the big picture is more about AI agents

Data to Decisions Future of Work Chief Information Officer

Systems integrators look to AI agents as transformation, reinvention engine

Systems integrators and services companies are launching AI agents, releasing frameworks and trying to help enterprises build multi-agent systems. The big question is whether AI agents turn out to be a boon or a bust for systems integrators in the long run. 

In recent days, we've heard from multiple systems integrators with more on tap talk about agentic AI. The extension of integrators into agentic AI makes sense given that they have the expertise to work across systems and processes. Consider:

Kyndryl, a services provider focused more on infrastructure, released the Kyndryl Agentic AI Framework, which orchestrates and dispatches AI agents that respond to shifting conditions. The framework is a way for Kyndryl to move up the stack to higher level offerings because it moves the integrator beyond infrastructure to workflows and processes.

According to Kyndryl, its Agentic AI Framework leverages algorithms, self-learning, optimizations and AI agents to run applications and processes.

Wipro said on its first quarter earnings call that enterprises are shifting discretionary funds to data and AI modernization. "AI is no longer a niche. It's becoming essential to how businesses operate at scale," said Wipro CEO Srinivas Pallia.

He added:

"Our AI capabilities are integrated into both industry and cross-industry solutions. By combining domain expertise with AI, we are able to deliver value through solutions such as hyper-personalized wealth management and predictive industrial insights. We have deployed over 200 AI-powered agents using advanced technologies from leading hyperscalers. These agents enables smarter lending, intelligent claims processing and autonomous network management."

At AWS Summit New York, there were multiple partners talking about the foundation needed for AI and agentic AI adoption. Deloitte's Chris Jangareddy, Managing Director of the company's AI, GenAI and Data Engineering, said the company will have nearly 180 agents on AWS Agent Marketplace.

According to Jangareddy, these agents are aimed at business problems, processes and specific tasks. Deloitte's AI agents are designed to be reusable Lego locks that will ultimately make up multi-agent systems. One offering is AI Advantage for CFOs that serve as a digital twin for CFOs, he said. The agents are built on Deloitte's institutional knowledge base of queries that are now prompts.

"These are not licensed, but are for clients," said Jangareddy, who noted that Deloitte is looking to transform its model from traditional billing to an outcome-based approach.

In a demo, Deloitte outlined Zora AI, which is part of an effort to produce AI agents that are product offerings. Deloitte views AI agents as digital labor that focuses on executing on processes. Zora AI is also integrated with SAP Joule.

AWS’ Brian Bohan, Director, Global Lead, Consulting Partner Center of Excellence, said during a talk that companies automating multiple business processes with agentic AI are seeing 30% to 40% productivity gains. He expects more efficiency to be unlocked.

Why? The cost of models is falling as are training and inference expenses. However, many AI projects aren't scaling due to a lack of architecture, data infrastructure and expertise. "There's just the complexity of integration," said Bohan.

Bohan added that change management, workflow optimization and the pace of innovation are all challenges. Enterprises will get to multi-agent systems across functions like finance, procurement and supply chain.

It's clear that systems integrators see AI agents as a booming business as well as a way to transform their businesses. The flip side of this transformation is that AI agents may ultimately hamper the systems integrator model.

Constellation Research analyst Holger Mueller said:

"As with any new technology, enterprises are looking at system integrators for help adopting them and AI is no difference here. The question is whether AI is so strategic that enterprises need AI skills inhouse, or can they rely on the integrator model. The experience depth is low for anyone as no one has more than two years in genAI experiences. Or more than 10 projects. It is likely going to be strategic for enterprises to have their own AI capacity and competency, especially once we move to inter-enterprise agents and the uptime and capability of frontline and backend agents determine success."

Data to Decisions Future of Work Innovation & Product-led Growth Next-Generation Customer Experience Tech Optimization Digital Safety, Privacy & Cybersecurity ML Machine Learning LLMs Agentic AI Generative AI Robotics AI Analytics Automation Quantum Computing Cloud Digital Transformation Disruptive Technology Enterprise IT Enterprise Acceleration Enterprise Software Next Gen Apps IoT Blockchain Leadership VR Chief Information Officer Chief Executive Officer Chief Technology Officer Chief AI Officer Chief Data Officer Chief Analytics Officer Chief Information Security Officer Chief Product Officer

Delta starting to scale its AI-driven dynamic pricing system

Delta Air Lines is pricing about 3% of its domestic fares with an artificial intelligence system and plans to get to 20% by the end of 2025.

Speaking on Delta second quarter earnings conference call, Delta President Glen Hauenstein gave an update on the company's plan to leverage AI-driven dynamic pricing.

In January, Delta outlined its AI plans and pricing initiatives with Fetcherr, an Israeli startup that uses generative AI to create optimized offers.

Hauenstein said:

"We're optimizing revenue through our partnership with Fetcherr, leveraging AI-enhanced pricing solutions. While we are still in the test phase, results are encouraging. You have to train these models and give them multiple opportunities to provide different results. We like what we see and we're continuing to roll it out, but we're going to take our time and make sure that the rollout is successful."

Hauenstein added that the more data and cases Delta feeds to Fetcherr, the more it learns and optimizes offers. If Delta gets to the 20% mark, it should be able to scale dynamic pricing at a faster clip.

On its conference call, Delta emphasized that it continues to roll out its technology including Delta Concierge, a virtual personal assistant built into the Fly Delta app launching later this year.

Delta is also using AI to optimize maintenance and resource availability.

But the biggest wins appear to be revenue optimization via partnerships with Fetcherr. In the second quarter, Delta operating revenue was up about 1% from a year ago to $15.5 billion. Hauenstein said demand stabilized late in the second quarter and business travel was solid. "During the quarter, demand trends stabilized at levels that are flat to last year. Our teams did a great job optimizing revenue performance in this environment by leveraging Delta's structural advantages and engaging customers beyond flight to generate a revenue premium to the rest of the industry," he said. "Diverse, high-margin revenue streams continue to show resilience, growing mid-single digits year-over-year and driving double-digit operating margins. Premium revenue grew 5% over the prior year, outpacing main cabin."

The plan for Delta is to expand profit margins on multiple fronts. The company restored its financial guidance that it cut in the first quarter with earnings of $5.25 a share to $6.25 a share and free cash flow of $3 billion to $4 billion.

Data to Decisions Next-Generation Customer Experience B2C CX Chief Customer Officer Chief Information Officer Chief Marketing Officer Chief People Officer Chief Human Resources Officer

AWS' practical AI agent pitch to CxOs: Fundamentals, ROI matter

CxOs are being barraged with constant change where AI time frames are compressed to days before there's a new development. The breakneck pace can freeze enterprise technology buyers since they can't spend on every new development, need to show returns and don’t want AI tech debt.

At AWS Summit New York, the focus was putting the fundamental approaches in place to give enterprises the structure to adopt AI agents.

Angie Ruan, CTO Capital Access Platforms division at the Nasdaq, summed up the current AI situation. "Technology used to operate over a decade. If you weren't upgrading something in five years you were behind. Later it became 18 months. Last year it was six months. Today my mindset is you have five days before you don't know what's going on and you're behind," said Ruan. "I've have never seen a pace as fast as AI."

Ruan added that there's a balancing act. "Stay calm, be strategic and be agile so you can be ready to pivot and take very practical delivery steps," she said.

Practical real-world returns for AI projects--generative, agentic and everything in between--was a recurring theme at AWS Summit New York. AWS rolled out a bevy of updates and features including Amazon Bedrock Agent Core, customizable Nova models and lots of talk about frameworks for reliability, security, observability and agility.

In the end, Swami Sivasubramanian, AWS VP of Agentic AI, used his keynote to return to enterprise fundamentals. Sivasubramanian's talk in New York had a lot to do with the balance of innovation and foundational approaches that can change models and underlying technologies. To AWS, a strong foundation and approach enable and accelerate innovation rather than constrain it.

For enterprises, a focus on fundamentals was long overdue. Do you really expect a business to swap out an LLM every time there’s a latest and greatest model that scores 0.06% better on math, coding or reasoning?

“AI agents are a tectonic change. They are a shift in how software is deployed and operated, and how software interacts with the world. Making that possible involves building on the foundations of today. In fact, in a world with agents, the foundation has become more important than ever. Things like security, right sizing, access, control and permissions and data foundations enable the right data to be used at the right time with the infrastructure that offers the right price performance,” said Sivasubramanian.

It's a message that enterprises were receptive to. “We're realistic about what AI can and cannot do. This isn't the silver bullet, but that's true of all AI systems. The value comes when you pair it with strong engineering practices,” said Matt Dimich, VP Platform Enablement at Thomson Reuters.

AWS is setting itself up for AI agent production systems where stability matters and models for most use cases are good enough to last a while. As agentic AI becomes more enterprise ready, basics such as identity, authentication and stability matter.

Constellation Research analyst Holger Mueller said:

“Amazon is making its step into the agent platform business with Agent Core. The good news for Amazon and its customers is that the traditional small, atomic services approach that comes from the AWS DNA, may be exactly the right thing for enterprises to build their first agents. AWS is enabling AI agents in a modular, individual and use case driven way - picking from Agent Core what they need. Adoption in the next few months will be interesting to watch. On the infrastructure side, the S3 vectors announcement is huge, as it makes digital assets stored in S3 available for AI.”

The AI ROI mismatch

Rohit Prasad, SVP and Head Scientist for AGI at Amazon, said enterprises have been struggling with an expectations gap between AI deployments and real returns.

"As exciting as AI is today, ultimately the real world is the real benchmark," said Prasad. "You hear about these models that come out every day. If you're an enterprise CIO you're thinking about the practical applications. How do I make real world applications happen at scale?"

Prasad said the focus on AGI, a topic that borders on obsession in the AI industry, is often a misdirected. "I want to level set on AGI. I think the whole conversation about who gets to AGI first or whether you can get to it is meaningless," said Prasad, noting that Amazon is chasing AGI and building out a full layer stack. "I don't think there will be a switch when we are AGI. Let's focus on whether we can make AI useful in real life. And can we make the complex simple?

AWS announced the ability to customize its Nova models for enterprise use cases. AWS will provide optimization recipes, model distillation and customization to balance cost and performance.

Prasad noted that every enterprise needs to think about AI returns in terms of workflows and processes. "It comes down to measurement. You can only improve on things you measure. Look at the success criteria for every workflow."

He added that metrics can't be stationary because your organization constantly changes. "Just go with very open eyes that in lot of applications at scale, what you measure, what you on a daily basis, also needs to evolve over certain time period," said Prasad.

In terms of AI agent value, measurement will be critical. Prasad said:

"The bar to evaluate the agent should be the same as the bar that is used to evaluate a human from a perspective of safety reliability. I think it's the same thing you want in a reliable human being. I want you to be reliable, which means it's a function of accuracy and consistency and robustness to the environment. If you want to be safe, you should have the values that you want your brand to be about, what your values to be humanity and the society is. So AI agents should be held to the same bar."

Measuring AI value

Erin Kraemer, Senior Principal Technical Product Manager at AWS Agentic AI, said that AI has the potential to fundamentally change how value is delivered.

The problem? Most companies don't properly measure AI impact. "One of the missteps that I'm seeing is how we're measuring AI impact right now and how we're talking about it," said Kraemer. "I'm not sure we're doing it the right way. Organizations that figure out how to thoughtfully apply AI to meaningful problems and measure success, are the ones that are going to adapt quickly and position themselves for the future."

Amazon's approach is to focus on controlled inputs and continual improvement to solve problems whether it's scaling infrastructure, managing product catalogs or admin tasks.

Key takeaways:

Focus on business outcome metrics over volume when it comes to AI, she said. Too much conversation about AI volume revolves around volume-based metrics, especially when it comes to code.

Kraemer said business outcomes trump volume. "I'm going to argue that, rather than volume, value should ultimately be our metric of success. So in my mind, volume, it's an output focus, and it's not even probably the right outcome."

Indeed, Kraemer said the stat that irks her is the commonplace 30% of code is written by AI. "The 30% number. I hate this number so very much. It's a fundamentally flawed number. It tells us very little about what's going on, our systems, our customers," said Kraemer.

Focus on the bottlenecks. She said enterprises need to see AI through business outcomes. Specifically, AWS looks to AI to address bottlenecks in processes. "If bottlenecks tend to be around human reasoning, there's a reasonably good chance that AI is a well-placed solution to that," said Kraemer.

Specifically, human bottlenecks have been an issue for Amazon throughout its history. She said:

"We love automation a lot. We like streamlined processes. We have some pretty massive, complex systems to handle those processes, but for a lot of our work, where we ultimately get stuck is in humans. Human reasoning capability has persistently been our bottleneck. It's not the worst bottleneck to have, but whether it's software upgrades, cleaning up catalog content defects in our shipping network, we either had to build very complicated and sometimes fragile systems, or we literally could not build systems that could scale through bottlenecks. What we're seeing with AI is a technology that's starting to blow by some of these bottlenecks."

Problem-specific metrics demonstrate real value. For code-related AI, Kraemer asked: "Are we fixing defects faster? Are we improving the security posture? Are we able to build things to delay our customers at a rate that we were never able to do before."

Amazon is looking at AI through a customer experience too. Here's a look at specific metrics AWS is using to gauge AI returns.

Software development:

  • Defect resolution speed.
  • Development velocity.
  • Infrastructure cost savings. AWS saved "roughly $260 million in AI-assisted Java upgrades," said Kraemer.
  • Developer time savings. AWS saved an estimated 4,500 developer years of effort on Java upgrades.

Customer experience:

  • Catalog quality improvements.
  • Contact per order decreases.
  • Customer satisfaction.

Knowledge work:

  • Time saved using AI to answer more than 1 million internal developer questions.
  • Research time and data to decision time.

Amazon's approach to AI internally

A panel representing technology leaders from various Amazon units--Amazon Ads, Alexa, technology infrastructure and other areas--talked about AI being integrated into their products and metrics for success.

Here are a few examples:

  • Amazon Connect uses genAI to enhance customer engagement and automation with data context as well as entity resolution.
  • AI is generating images and video for Amazon Ads and its AI services.
  • Amazon Business is using AI to automate business verification, improve accuracy and reduce manual review time. Search relevance and bulk buying reviews are also designed to improve procurement experience for Amazon Business customers.
  • AWS Marketplace is using AI for seller onboarding and funding approvals and offering a comparison engine for product insights.
  • Alexa is getting a rebuild for more natural interaction and agentic AI actions.

The metrics for these projects revolve around cost, friction elimination and customer experience. As you deploy these key performance indicators and metrics, keep an experiment-based mindset focusing on customer needs and iterate.

Lak Palani, Senior Manager, Product Management Tech at Amazon Business said:

"My recommendation is straightforward. Don't use AI just for the sake of using it. Find the right business cases where AI will really add value. Start small, measure results and remember it's an iterative process. Then you can scale success. Stay super focused on the business value and customer experience."

There’s a method to AWS' meat-and-potatoes focus on agentic AI and fundamentals: Enterprise adoption of AI agents will trail the technology advances and vendor marketing speak. AWS is meeting customers where they are right now.

 

Data to Decisions Next-Generation Customer Experience Innovation & Product-led Growth Future of Work Tech Optimization Digital Safety, Privacy & Cybersecurity amazon ML Machine Learning LLMs Agentic AI Generative AI Robotics AI Analytics Automation Quantum Computing Cloud Digital Transformation Disruptive Technology Enterprise IT Enterprise Acceleration Enterprise Software Next Gen Apps IoT Blockchain Leadership VR Chief Information Officer Chief Executive Officer Chief Technology Officer Chief AI Officer Chief Data Officer Chief Analytics Officer Chief Information Security Officer Chief Product Officer