AWS' Matt Wood on model choice, orchestration, Q and evaluating LLMs

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Matt Wood, Vice President of AI at AWS, outlined how enterprises will mix and match multiple models depending on use case, the need for orchestration and how regulated industries may have an advantage in adopting genAI.

Wood spoke with industry analysts including me and Doug Henschen at AWS' New York offices. Here are some of the key themes to note.

AWS is differentiating on ensuring customers retain IP and confidentiality. Unlike competitors, Wood says AWS "does not mix training data for models with customer data" and "does not allow any human review" of training data. The key competitor cited was Azure OpenAI.

Wood said:

"There is a schism appearing in some customers' minds, where you have to be willing to give up some level of IP protection or confidentiality or privacy of your data in order to be successful with generative AI. And that just is not the case on AWS. Customers are unwilling to do that in any of the industries, particularly those in regulated industries."

Regulated industries are traditionally known as technical laggards, but these enterprises are all over generative AI. Regulated industries such as financial services, insurance and health care are "moving slightly faster than the average" on generative AI.

Wood said:

"A lot of the regulatory approaches and compliance that those organizations have been working on for the past 20 years actually set them up very well to work with generative AI. All the governance, privacy and security data standards allow you to be able to get to utility and value with generative AI very quickly."

"Companies are starting to develop muscle around model evaluation" and they are "no longer tightly coupling to single models," said Wood. Doug Henschen's take: "That's consistent with what we hear from GCP and even Microsoft, but Wood insists AWS will be "Switzerland" and Bedrock will remain differentiated on model diversity."

• AWS ups its investment in Anthropic as giants form spheres of LLM influence
• AWS Q4 revenue growth 13% as Amazon's results shine

Wood said model choice is the only way to go in the long run. "Other providers are very married to a very small subset of models. And what that means is that customers end up having to approach a model a bit like a Swiss Army knife, which sounds great, except If It's a contractor turned up to fix your house and all they had was a Swiss Army knife, you would not be very happy."

Ultimately, enterprises will toggle between speed, use case and cost when managing model portfolios. Wood added that this portfolio management of models is already carrying over to compute. He said if training speed is needed Nvidia GPUs get the call, but if cost is more of a factor AWS' Trainium chip is an option. Wood said interest among the customer base is split about 50/50 between the need for speed and cost.

• Amazon Generative AI Stack Offering Overview
• Amazon's Vogels says 'cost awareness is a lost art' as AWS launches optimization tools

Model orchestration will become critical. "What we've seen is that a big part of success in the actual broad production use of generative AI is being able to access the right model for the right use case," said Wood. "Different models operate and have different use cases and have different sweet spots. The real kind of superpower for generative AI is the combination of those models and the compounding effect on the aggregate intelligence of the system."

Orchestration of models in Bedrock will evolve over time so enterprises can leverage multiple models. Wood said orchestration of models today requires hardcoded rules to data, but as models improve to handle more tasks generative AI will be able to play point guard better.

Q leverages multiple models based on specialty and use cases. "Amazon Q is the easy button for Generative AI" and it provides all the GenAI many companies want to use, said Wood. He specifically touted "a step-function change" in digital transformation and migration projects, such as Windows to Linux moves. Henschen said cloud migration will be an important use of GenAI and an accelerator and cost saver for customers looking to migrate off legacy platforms and onto modern, cloud platforms.

Wood noted that Q runs on a variety of models via a series of expert agents.

• These overlooked AWS re:Invent launches could solve pain points
• AWS launches Amazon Q, makes its case to be your generative AI stack

Palantir will move workloads to Oracle Cloud as both court governments and enterprises

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Palantir will move its workloads to Oracle Cloud Infrastructure in a partnership that also includes the two companies jointly selling to governments as well as enterprises.

The partnership will combine Palantir's AI platform and Oracle Cloud Infrastructure. The deal has multiple parts to it, but the gist is that the two companies plan to jointly target governments and enterprises with Oracle distributed cloud and AI infrastructure and Palantir's AI Platform (AIP) and Foundry decision support platform.

Both companies are seeing strong growth among governments looking to keep data in-country.

Enterprise generative AI use cases, applications about to surge | Palantir posts strong Q4, sees enterprise traction in US | Palantir's commercial business scales with help of AI boot camps

Here's a look at the moving parts of the partnership:

• Palantir will move its Foundry workloads to OCI.
• Palantir's Gotham and AIP will be available in OCI's public cloud as well as Oracle Cloud Infrastructure Dedicated Regions, Oracle Alloy, Oracle EU Sovereign Cloud, Oracle Government Cloud and Oracle Roving Edge.
• Oracle will workloads and an increase in ongoing cloud revenue.
• Both companies will jointly sell and support cloud and AI services across government and commercial accounts.

The Oracle and Palantir combination will line up against C3 AI in government and enterprise accounts. C3 AI has partnerships with Microsoft Azure, AWS and Google Cloud.

CEOs aim genAI at efficiency, automation, says Fortune/Deloitte survey

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

CEOs are ramping up generative AI adoption as they shift from pilots to active usage, according to the Winter 2024 Fortune/Deloitte CEO survey.

The survey, based on 107 CEOs mostly in the US, found 56% of respondents rank efficiency and productivity as the primary benefit. According to the survey, 58% of CEOs say they are already implementing genAI to automate manual tasks, up from 40% in October, and 45% say they are reducing operating costs.

Going forward, CEOs are looking to balance costs and innovation. Fortune/Deloitte said 51% of CEOs are implementing genAI to accelerate innovation.

A few key generative AI adoption stats from the survey:

• 50% of CEOs say they are using AI to automate content generation.
• 42% say genAI is writing code.
• 56% are using genAI to increase efficiencies.

Related: Middle managers and genAI | Why you'll need a chief AI officer | Enterprise generative AI use cases, applications about to surge

Other takeaways include:

• Geopolitical instability was rated as the primary disruptor by 65% of CEOs surveyed.
• 27% of CEOs express optimism about the global economy, up from 7% in the fall of 2023.
• 22% of CEOs have high optimism about their companies.

Low code platforms will become strategic to CxOs

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Generative AI and automation will mean low-code coding platforms will become strategic for enterprise transformation efforts. By 2025, Constellation Research estimates more than two-thirds of enterprises will have a standardized low-code tool in house.

That's a big takeaway from Constellation Research analyst Holger Mueller, in a recent published report Key Trends for Low Code in 2024 and Beyond. Mueller wrote:

"Enterprises need to own their digital transformation destiny by creating software assets that enable what matters. Enterprise Acceleration is an all-encompassing enterprise strategy that aligns people and technology so they can become more agile and move faster. The emergence of generative artificial intelligence (GenAI) only exacerbates the need for low-code platforms that enable enterprises to build the next-generation platforms they need in order to be winners in the era of digital transformation."

In many ways, low-code platforms will be critical as a bridge between human built code and code via software robots and agents.

Mueller outlined the following CxO considerations when thinking about low code strategies this year and beyond.

• Begin with a low-code mindset. Enterprises can't ignore the importance of low-code platforms as an enabler to digital transformation effort.
• Standardize low-code platforms, but make sure governance is so restrictive that it hampers developers.
• Select low-code platform vendors that can leverage generative AI because processes are going to be automated at a rapid rate.
• Know how low-code platforms interact with your standard applications. CxOs will have to balance enterprise application platforms and low-code offerings.
• Understand your total cost of ownership with low-code platforms to avoid higher software portfolio costs.

More low code platform resources and news:

• Constellation ShortList™ Enterprise Low-Code Tools and Platforms
• Constellation ShortList™ No Code/Low Code for SAP Systems
• Constellation ShortList™ Citizen Developer Low Code Tools and Platforms
• News:
• Business process automation platform debate about to heat up
   

Data scientists, cybersecurity analysts in high demand, says CompTIA

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

CompTIA is projecting that data scientists and analysts, cybersecurity analyst and engineers and software developers will see the most job growth in 2024 as hiring trends normalize.

Those in-demand roles are expected to post the most job growth over the next decade.

The projections in CompTIA's State of Tech Workforce 2024 report forecasts 300,000 net new IT workers this year, a gain of 3.1%. IT job growth in 2023 was 1.2%.

CompTIA acknowledged that the forecast doesn't account for the most recent trending data from monthly reports. CompTIA's forecast includes tech employment in the tech sector as well as tech-related positions.

According to CompTIA, its tech workforce report is based on data from the US Bureau of Labor Statistics and Lightcast.

CompTIA said in its report:

"Calculating future workforce needs over the next decade is a function of several variables. There is a growth component, which may entail organizations adding headcount due to expansion or possibly to support new emerging technologies. There is a retirement factor, with a portion of the workforce transitioning away from the labor market permanently. And lastly, there is a segment leaving the workforce for some other reason, also referred to as separations. These may stem from a career change, layoffs, a return to school, family pursuits, or other. Changes to any of these variables will affect future workforce projections."

Indeed, it's unclear whether positions today will equate to demand in a decade. Constellation Research analyst Holger Mueller said:

"We are seeing the last hurrah for data scientists and cybersecurity specialists – as CxOs need to make sense of their data (for data scientists). But traditional hiring for data scientists will give way to AI specialists. The college kid who played with prompts may  easily be more valuable for the job than any data scientist, even if tied down in an AI project. As for cybersecurity, we will see the same. Software will do more and more, and AI will help identify threats and mitigate them. Both jobs are moving from operator led to self-driving – sooner than traditional projections make us think."

Among the key data points in the report:

• CIO and IT director job growth in 2024 is projected to be 3.6%, a rate on par with Web developers and interface designers.
• Since 2018, the IT services and custom software services category has accounted for 50% of job gains.
• 45% of the tech workforce works for technology companies with the remainder spread across multiple industries.
• Net tech employment in the US was an estimated 9.62 million workers.
• The replacement rate for tech occupations during 2024 to 2034 is expected to be 6% annually, or 350,000 workers each year.
• The US Bureau of Labor Statistics doesn't break out emerging tech roles yet, but new specialties such as jobs involving artificial intelligence are contributing to workforce growth.
• California is No. 1 in net tech employment by a wide margin followed by Texas, New York and Florida.
• Top metros by net tech employment are New York City, Washington DC, Los Angeles and San Francisco.

Quantinuum, Microsoft claim quantum reliability breakthrough

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Quantinuum and Microsoft have claimed a breakthrough in fault tolerant quantum computing that can lower error rates and improve reliability.

Fault-tolerant quantum computing has been the Holy Grail for the industry as vendors and researchers race to minimize errors. Quantinuum, which recently raised $300 million, is the product of the merger between Honeywell Quantum and Cambridge Quantum Computing in 2021.

According to Quantinuum and Microsoft, teams from both companies were able to create four logical qubits that demonstrate error rates 800 times lower than corresponding physical error rates. The teams were able to run 14,000 independent instances without errors. Here's the full technical paper. 

The system used in the effort featured Quantinuum's 32-qubit H2 quantum processor from Honeywell with Microsoft's error correction. In a blog post, Microsoft said the breakthrough "finally moves us out of the current noisy intermediate-scale quantum (NISQ) level to Level 2 Resilient quantum computing."

The industry is looking to quantum use cases that can be used with supercomputing systems in a hybrid way that incorporates AI, supercomputing and quantum computing.

Microsoft said the technology behind the Quantinuum collaboration will be available in Azure Quantum Elements in the months ahead.

Constellation Research analyst Holger Mueller said:

"Giving the fragile state of Qubits, a key focus is to reduce the error rate creating logical Qubits. The collaboration is both a proof of Honeywell’s H2 platform as well as Microsoft's work on error correction. Good to see the cooperation, but it also shows that to get to Quantum use cases in the enterprise, even the largest players can't do it alone."

More quantum computing:

• Quantinuum raises $300 million, valued at $5 billion
• IBM launches latest Heron quantum processor, IBM Quantum System Two
• AWS launches Braket Direct with dedicated quantum computing instances, access to experts
• Will IonQ make quantum computing enterprise relevant in 2025?
• Quantum Computing Platforms
• Quantum Computing Software Platforms
• Quantum Full Stack Players
• Why your quantum computing vendors are going to look familiar

Intel Foundry had $7 billion operating loss in 2023

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

Intel outlined its Intel Foundry results as it recast its financial reporting segments and added a new CFO as the unit aims for break even.

The chipmaker outlined its new reporting segments and historical results. It also named Lorenzo Flores as chief financial officer of Intel Foundry. Flores was CFO of Xilinx.

Intel's new operating model creates a foundry relationship between Intel Foundry, the manufacturing unit, and Intel Products, the product business units. In an SEC filing, Intel said Intel Foundry operating revenue in 2023 was $18.9 billion, down from $27.5 billion in 2022. The operating loss for Intel Foundry in 2023 was $7 billion, up from $5.2 billion in 2022. Intel Foundry's external revenue was $953 million in 2023.

• Intel Foundry sets roadmap, aims to be No. 2 foundry by 2030
• Intel's Q4 strong amid PC rebound, outlook light: Here are the takeaways

Pat Gelsinger, Intel CEO, said the products and foundry business can drive long-term sustainable growth. Intel Products will include the company's client computing group, data center and AI group, and network and edge units. Altera, Mobileye and Other will be in an All Other category. Intel Foundry will be reported separately and include external revenue as well as sales from Intel Products.

Intel is aiming for 60% non-GAAP gross margins and 40% non-GAAP operating margins in 2030.

Intel CEO Pat Gelsinger said on a conference call with analysts that "we see this clear path to driving Intel Foundry to break even over the next few years." 

"What we see as a self-sustaining model with solid returns for Intel Foundry and the consolidated company by 2030," he added. "We are also making fundamental moves to standard intellectual property blocks and processes, where we became so disconnected from the industry. We are also addressing the rising capital requirements by optimizing for cost efficiency and extending the life of our assets. We are addressing the scale challenges by bringing in external designs into our factory network and monetizing the IP in both our products and our foundry."

Constellation Research analyst Holger Mueller said the burden of profit proof is on Intel. Mueller said:

"Intel goes 'fabless' – though only for the balance sheet. It remains to be seen if investors will fall for valuing the 'fabless' Intel the same as other truly fabless competitors like e.g. Nvidia or Qualcomm. Intel needs to finance the losses of the fab business – unless this is the start of spinning that off – something we saw in the past e.g. with IBM and GlobalFoundries. Intel is not producing competitive chips and for the sake of competition and budgets we can only hope that the old Intel will roar back and create some much needed competition in the years ahead."

The big breakout from the new reporting structure is Intel Foundry. Intel said the unit's operating losses are expected to peak in 2024 and the company will hit break-even operating margins midway to its 2030 goals. Intel Products already has solid operating margin en route to the 2030 targets.

An ode to middle managers

Larry Dignan

Editor in Chief of Constellation Insights
Constellation Research

Larry Dignan is Editor in Chief of Constellation Insights at Constellation Research, where he leads editorial coverage focused on enterprise technology, digital transformation, and emerging trends shaping the future of business. He oversees research-driven news, analysis, interviews, and event coverage designed to help technology buyers and vendors navigate complex markets with clarity and context. ...

Read more

It's open season for middle managers across technology and multiple industries, but it remains to be seen how short-sighted this trend ultimately becomes.

Yes folks, this is an ode to middle managers--the people who manage projects, serve as a buffer to executives, and lead teams in smaller batches. Today, we've gone from the COVID-19 era of over hiring to cutting out layers of management. In true American fashion, the pendulum swings all the way to the other side with little to no balance in the middle.

According to Live Data Technologies, manager-level or higher layoffs made up nearly half of all observed layoffs in 2023.

In 2024, it appears to be a similar layoff story via Layoff.fyi.

Consider:

• Mark Zuckerberg, CEO of Meta, said on the company's fourth quarter earnings call that he has "come around to thinking that we operate better as a leaner company." Zuckerberg just closed out 2023, dubbed a year of efficiency, and noted that "in 2024, we do have a big recruiting backlog because part of the layoffs that we did included teams basically swapping out certain talent profiles for others." Translation: Engineers and salespeople in. Everyone else out. Meta ended 2023 with 67,317 employees, down 22% from 2022. As the end of 2021, Meta had 71,970 employees up 23% from 2020.
• Sundar Pichai, CEO of Google parent Alphabet, said "teams are working to focus on key priorities and execute fast, removing layers and simplifying their organizational structures."
• UPS Carol Tome said the company is going to fit its organization to focus on "what's wildly important." UPS is cutting 12,000 positions to save $1 billion and most of those reductions are managers.
• Citi CEO Jane Frasier said it will cut more managers to simplify the org structure and save more than $1 billion. Citi restructurings aren't exactly a new trend since its core competency appears to be adding management layers and penning restructuring announcements later.

I realize it's a bit contrarian to defend middle managers--especially when one of my recent reads was Bullshit Jobs: A Theory--but there is going to be some fallout to this middle management purge.

This post first appeared in the Constellation Insight newsletter, which features bespoke content weekly and is brought to you by Hitachi Vantara.

Here are some of the ramifications from the middle manager cuts.

Morale. The UKG Workforce Institute found that 86% of managers’ report feeling burnout. That job burnout rate is higher than employees and CXOs. However, 75% of employees feel that their manager cares for and has empathy for them.

You can't tell me that a super flat organization where 200 people report to one executive who doesn't have time for them is going to boost morale.

Decline of project management. Many of these middle manager types getting cut happen to be project managers. Simply put, project managers have a tough gig. They're caught in the middle between the technology and business sides of an enterprise. These project managers are more human and happen to be bilingual in tech and business. Over time, enterprises are going to realize that you need a human in the middle of the Venn diagram to get things done.

Customer success teams at your friendly neighborhood vendor will suffer. Anecdotally, enterprise vendors have been cutting customer success managers who aren’t technical or salespeople. These customer success structures have always been a bit wonky. One setup could have a technical lead, a sales-ish type person focused on success (and a never-ending upsell cycle) and a manager who is more neutral and may want the customer to actually succeed. Guess what happened? The relationship person was cut (no credit for revenue and/or code). Enterprise customers have been asking where their core contacts went over the last 18 months. It's quite possible that customers are stuck with a tech-oriented contact with little people skills and another who is focused on relationship building with a heavy dose of upsell.

SaaS companies are going to lose seats. Live Data Technologies also had a post on the shrinking total addressable market for SaaS companies. Guess who SaaS users were? Yep, those managers who were cut over the last year and change.

A few companies will realize that the talent that has been purged can be very useful in the right cases. AuthID CFO Ed Sellitto said on the company's recent earnings call that it is building out its customer success team. "This team is laser focused on deploying and refining proven customer implementation methodology to help customers meet projected go live dates, establish full-service rollouts across contracted use cases and meet and ultimately exceed their minimum contracted commitments," said Sellitto.

Enterprises could quantify middle managers. If project managers and folks who run teams can be credited for some slice of the overall revenue pie the economics may look dramatically different. To date, the sales department runs off with the quantification (as if that person sold everything with no support whatsoever) and middle managers look like overhead. The person who keeps the product launch running on time is taken for granted. Maybe a slice of revenue attribution for the supporting cast is warranted.

The problem for managers is the same as it was in the 1980s: It's hard to quantify middle managers. In the age of AI, soft metrics aren't going to fly.

Executives are going to feel all the burnout middle managers faced. Middle managers were facing extreme burnout. Well, now that the middle tier is gone and all those gripes, employee morale issues and balancing act between KPIs and reality will hit higher-level executives. Once those execs in flattened orgs feel the pain look for enterprises to start adding a layer or two back. The harsh reality is someone must absorb the blows and executives are going to find it difficult to yell at algorithms that replaced those human managers just cut.

What the large language models think of middle managers

Given I was going to take the unlikely position of supporting middle management, it made sense to ask generative AI--the technology that'll replace these people--what it thought. Here's a look at what Gemini and ChatGPT gave me when I asked for an ode to the middle manager.

Gemini:

Oh, middle manager, you unsung hero,

You keep the wheels of industry turning.

You bridge the gap between upper management,

And the front-line workers who do the earning.

You're responsible for implementing strategies,

And making sure that work plans are followed.

You're also responsible for the execution,

Of your own and your team’s work.

You're often beholden to others' expectations,

Deadlines, and schedules.

But you always try your best,

To make sure that the job gets done.

You're the glue that holds the organization together,

And you deserve to be recognized for your hard work.

So, here's to you, middle manager,

We appreciate all that you do.

And the ever-wordy ChatGPT:

Ode to the Middle Manager, steadfast and true,

Keeper of balance, in skies often not blue.

In the realm of the cubicles, amidst fluorescent light,

You stand as a beacon, a guide through the night.

With a coffee in one hand, reports in the other,

You navigate the waters, a skilled helmsman and brother.

Between the titans above and the sea below,

You chart the course, steady and slow.

In meetings, you're the diplomat, the voice of reason,

Bridging gaps, through every season.

Your inbox, a testament to battles fought,

Each email is a volley, each decision a thought.

You translate the visions, from lofty to real,

Into actionable plans, with zeal you seal.

The dreams of the high, to the hands of the low,

Through you, they take shape, and begin to grow.

You're the unsung hero, in the corporate fray,

Juggling demands, come what may.

The mentor, the coach, on the office floor,

Guiding your team, to strive for more.

When the storm clouds gather, and morale is low,

You're the first to arrive, and the last to go.

With a joke, a smile, or a listening ear,

You dispel the shadows and bring back the cheer.

For the middle manager, no song is sung,

No praises shouted; no bells are rung.

But know this truth, as clear as day,

Without you, the ship would lose its way.

So, here's to you, in your unassuming might,

The linchpin that holds, the day and night.

In the heart of the maze, you find the path,

Middle manager, you deserve our gratitude and laughter.