Altimetrik CEO Sundaresan on AI optimization traps, architecture and model choices

Altimetrik CEO Raj Sundaresan said enterprises need to avoid AI "optimizations traps," trendy approaches that don't age well if you don't put architecture first.

Sundaresan riffed on AI optimization traps, token-maxxing, the penchant for forward deployed engineers and the need for multi-model approaches in a blog post.

Constellation Research caught up with Sundaresan for a discussion on enterprise AI and how CxOs should approach implementations. Here's a look at the key points.

Enterprise architecture is often overlooked. Sundaresan said enterprises rushed into AI experiments and proofs of concept without first establishing integration with existing systems. These efforts were often pushed by boards of directors and CEOs that wanted to show AI progress.

Sundaresan said:

"Everyone wants to jump into solutions and impress the board, but they're missing this big thing around architecture. If you want to bring everything together for production scale you have to make it fit within your existing architecture. You need a well-defined architecture to bring AI into the mix."

Optionality is critical to enterprise AI projects. "One thing we have to think through is optionality in any architecture. Enterprises don't want to lock into technologies where you're stuck. When you think about AI and models you need to have optionality built in to choose the right model for the right workload and right decision," said Sundaresan. "You don't want your knowledge group in a walled garden of a model."

• AI doesn't replace engineering rigor, says Altimetrik CEO
• Constellation ShortList™ Innovation Services and Engineering
• Constellation ShortList™ AI Services: Global

Brownfield realities trump greenfield fantasy. Sundaresan said AI projects can't be treated like a greenfield initiative because there's a brownfield reality. "You cannot have an AI office somewhere that's centralized. If you do that you're not thinking about the brownfield. If AI is isolated it fails miserably when you have scale," said Sundaresan.

The fix is to think holistically about how AI needs to be brought into an enterprise.

Beware of optimization traps like token-maxxing and FDE-maxxing. Sundaresan highlighted two primary optimization traps where enterprises focus too much on input or vendor-driven activity over business outcomes. "The optimization trap is where you jump into something and since it works in an area, you invest a lot more in that same area," said Sundaresan. "The input should not be a measure for what your outcome should be."

Token-maxxing is an obvious disconnect with outcomes. "The amount of code you generate is not always proportional to the business implication it has. Similarly, the amount of PRs (pull requests) is not a measure of a business success," said Sundaresan.

The use of FDEs can also be an optimization trap. If you rely too much on FDEs, you can often see flashy prototypes that don't translate to production-grade systems. Sundaresan said FDEs from frontier model companies may build to a frontier model’s needs. "I would say there's two types of FDEs. One is from the set of the frontier labs. These folks would leave some workable code with some demonstrable piece of work that the next FDEs could scale to production," he said. "The trend I see is maxxing out FDEs without getting to production grade. You need to think through a lot more from an architecture perspective to make it work."

The mixture of models matters. Sundaresan advocates for a mixture of model approach that should be orchestrated by a control plane owned by the enterprise and uses operational knowledge to route requests for the use case. He added that this mixture of models is about creating a mixture of experts.

"With a mixture of models, you should have control within your architecture to orchestrate based on the decision you're expecting," said Sundaresan. "You could go to a frontier lab model, or an open weighted model, or a small language model or a deterministic machine learning algorithm. You need some level of orchestration and judgement built within the architecture."

Sundaresan said Altimetrik leverages the full range of models including small language models and open models. The big takeaway is that you just don't need a frontier model for many use cases. Sometimes you may not even need a model when it's deterministic.

Is there an optimal mix of models like an investment portfolio? Sundaresan said there's no ideal mix of models and the selection process is based on needs, regulation, outcome and learning over time. "We see a lot more need for small language models and domain-specific models," he said. "Sometimes you don't need a large language model at all. The portfolio and mix is based on the need."