Beyond RAG: Where Enterprise AI Value Will Come From

What follows is a summary of a conversation Mike and Esteban had in different channels about RAG.

Retrieval-augmented generation (RAG) solved an important early problem for enterprise AI. Large language models could produce useful responses, but they lacked access to current, proprietary, and company-specific information. RAG provided a practical way to retrieve relevant material from enterprise systems and supply it to a model without retraining the model itself. It helped organizations move quickly, connect general-purpose models to internal knowledge, and improve the grounding of their answers.

Should enterprises continue treating RAG as the foundation of their AI architecture? Retrieval will remain necessary, but RAG is only one mechanism for providing context. Until recently, AI was used mostly to find the right information, or to advance analytics. As it shifts to agentic AI the more important decision is determining which knowledge belongs in the model, which information should remain outside it and be supplied dynamically, and how the enterprise will manage the knowledge required by both.

Enterprise AI investment is accelerating much faster than deep operational adoption. Companies expect AI spending to rise from approximately 0.8 percent of revenue in 2025 to 1.7 percent in 2026, according to BCG. Menlo Ventures estimates that enterprise generative AI spending reached $37 billion in 2025, more than three times the previous year’s level. Yet an analysis of S&P 500 filings found that only 11 percent of companies had deeply integrated AI into business processes in 2025, with another 10 percent using it directly in production or service delivery. Enterprises are spending rapidly while the operating architecture remains unsettled.

Stable knowledge belongs closer to the model as it makes it faster, cheaper, and aids in optimizing context extraction. Domain terminology, durable business practices, established relationships, common rules, and knowledge that changes slowly can be incorporated through training, adaptation, or specialization. Current priorities, live operating data, temporary instructions, agent memory, and dynamically assembled governance controls belong more naturally in the context supplied during inference.

Retrieval supports that context layer, but it should not become the organizing principle for the entire architecture. Every additional component around a model creates latency, maintenance requirements, cost, and another potential point of failure. Retrieval may return incomplete, stale, contradictory, or irrelevant information. It can compensate for knowledge missing from the model, but continued compensation can eventually conceal the need to improve the model itself.

The danger is that enterprises keep adding retrieval capabilities to general-purpose models while avoiding the harder work of deciding what those models should know and how they should operate. RAG then becomes an increasingly complex attachment to an insufficiently adapted model. It remains useful, but the architecture stops progressing.

The rapid evolution of open-weight models, smaller specialized models, fine-tuning methods, and model-development tools gives enterprises more choices than they had during the first wave of generative AI. Organizations can now make more deliberate decisions about which capabilities should be built into a model, which information should be supplied dynamically, and where public frontier models still provide sufficient value.

This boundary may become one of the primary sources of value in smaller and weight-managed models. Their advantage does not come only from lower cost or greater control. It comes from the enterprise’s ability to define more precisely what the model should know, how it should behave, and which information should remain dynamic. A smaller model with well-defined domain knowledge and a disciplined context layer may outperform a broader model wrapped in an increasingly complex retrieval architecture for a specific enterprise task.

The objective is not to incorporate everything into the model. It is to give the model enough stable domain knowledge to perform consistently while using context for what is current, situational, or specific to the task. A well-designed model should not depend on context to compensate continually for fundamental weaknesses. Context should improve resolution, completeness, and efficiency during inference.

This boundary will also change over time. Information repeatedly retrieved during inference may eventually prove stable and valuable enough to incorporate into the model. Observability can identify recurring gaps, frequent operator corrections, repeated retrieval patterns, and areas where context is producing poor or inconsistent results. Those signals should guide the next model iteration.

That creates a continuing relationship with knowledge management. Enterprises must still capture institutional knowledge, determine which sources are authoritative, maintain current information, and retire what is no longer useful. Some knowledge will inform future versions of the model. Some will remain in operational systems and be supplied dynamically. Some may never belong in either place because it is unreliable, redundant, obsolete, or insufficiently governed.

The investment in knowledge management therefore does not disappear as models improve. It becomes more important because the enterprise must decide where knowledge belongs, how it will be maintained, and which systems are responsible for keeping it current. Building better models does not eliminate the need to govern enterprise knowledge, and better retrieval does not resolve weaknesses in how the organization captures and maintains what it knows.

Experienced operators play a central role in making these distinctions. Technology can retrieve information and identify patterns, but it cannot independently determine which knowledge is durable, which exceptions matter, which policies should remain dynamic, or when repeated retrieval is masking a deeper weakness in the model. Those judgments require people who understand the domain, the workflow, the operating history, and the consequences of an incorrect decision.

Their interventions should also become part of the feedback loop. When experienced operators repeatedly correct the same output, add the same missing information, or override the same rule, the enterprise has learned something about the boundary between model and context. The lesson may require a model update, a change in the context layer, a new governance control, or an improvement in the underlying knowledge-management process.

The resulting enterprise architecture is composed of three related but distinct capabilities. The model provides stable domain understanding and the ability to reason or act within defined boundaries. The context layer supplies current information, operating conditions, relevant memory, and dynamic controls during inference. Knowledge management captures, governs, and maintains the institutional information that feeds both.

The enterprise advantage will come from managing the relationship among models, context, and knowledge with greater precision. Organizations that make the distinction deliberately can improve performance, control costs, simplify maintenance, and replace models as better options emerge. They can also create smaller, more specialized, and more manageable systems built around their own domain knowledge and operating requirements.

Those that continue treating RAG as the complete answer to enterprise context risk extending an early transitional architecture beyond the point where it delivers strategic value. The question is no longer whether retrieval belongs in the architecture. It is whether the enterprise knows what retrieval should support, what the model should contain, and how its knowledge should evolve over time.