As the pulse of AI investment moves forward, the major focus of vendors and users alike revolves around Generative AI and AI Agents.  Managers of all walks continue to try to grasp how to harness these technologies, but this remains a daunting task.  A common focal point of the allure of these advanced approaches revolves around the ROI that may be achieved, but reality is that estimating these returns (if they exist), entails another daunting endeavor.

GEN AI and AI AGENTS

GEN AI refers to a platform that combines various functionalities of AI subcomponents, namely predicative AI, NLP along with computer vision.   You know the tools in this space (ChatGBt, Claude, Gemini, CoPilot etc), which enable the generation of quick information corresponding to search criteria (e.g. prompting LLMS).  Data that LLMs access can be open source based or RAG (internal organizational data can be included).  The application of GEN AI is used for speed of information generation and a decision support mechanism for users to work smarter and faster.  Applications include investigative and information/knowledge creation activities for customer support, industry standards, marketing content, organizational descriptions, computational support, etc.  However GEN AI does not think, it processes prompts and generates content from available data.

AI Agents on the other hand differ from GEN AI in that they address more commoditized routines and processes that can be replicated by algorithms.  A more simplistic description is they are similar to earlier versions of code based BOTS that facilitate a particular activity.  Agentic AI is more appropriate for standardized operations, where tasks are well defined and repetitive.  Examples are numerous and include automated customer service, financial applications, HR applications etc.

The focus of the Agentic approach is not decision support for humans but more replicating tasks and activities of organizational operations that are carried out by humans.  Agents can be optimized by training them in performing processes, where errors or bugs can be corrected. More advanced Agentic applications integrate LLMs that guide and provide input for multiple Agents to carry out more complex, processes.

Measuring Returns (first GEN AI)

Given the difference in functionality of these approaches, estimations of ROI require a focus corresponding to the task they address.

ROI for GEN AI is generally more difficult to estimate given its supportive nature in various applications.  As previously stated, GEN AI provides valuable information for users in a timely manner. Consider an example of an LLM that provides sales support by feeding a sales rep critical information about a potential customer that enables them to fine tune their presentation (e.g. products and services relevant to the prospect’s interests).  Returns to this AI involve factors such as enhanced probability of winning/closing a deal faster.  It also includes freeing up time for the sales rep to pursue more deals. Estimating ROI for the organization requires connecting (GEN AI) to sales performance.  In other words, did the LLM consistently cause the sales team to close more deals faster?  This introduces the issue of whether other elements were involved in increasing sales as well (e.g. sales rep experience, understanding of the customer, type of interaction with prospects, etc.)

The result is a fuzzy picture of value to GEN AI.  This scenario is applicable to other applications as well.

Consider time saved in creating marketing content with GEN AI.  ROI needs to consider not just the timely production of the content but the effectiveness of the content and how many additional resources are required to implement GEN AI (editors who must examine the content).  Again ROI is difficult to estimate.

Next (Agentic ROI)

Simply put, for GEN AI, ROI must realize that LLMs generally support users/humans as opposed to fully automating a work process.  Additional resources are required to monitor the quality/accuracy of the information it generates, which must be included in estimating returns to investment. ROI for AI Agents involve a more measurable approach.  Agents seek to automate work processes and often fully automate them.  The term managers must understand is the idea of “processes”.  These can be as detailed as directing and addressing phone queries during customer support or can fully automate more complex and interactive processes such as creating reports according to users’ needs. ROI is a more tangible and direct calculation and generally involves a displacement of labor or a reduction in the “human input” that was needed to perform the process activities. It can sometimes be estimated in simpler “time saved” metrics.  In other words…how have our operational costs changed due to automation?

These ROI calculations may help explain why MIT’s recent study found 95% of AI pilots fail but there is more success with back-office applications 1  (which are more routine Agentic based).

Managing Risk and the Adjustment to ROI

Whether organizations are investigating returns to GEN AI or AI Agents, they must consider resources saved and resources required.  Managers often focus on the former (eg. reductions in time and work hours) for GEN AI, and reduction in personnel for AGENTS. AI implementations require additional layers of resources which address efficiency and risk management. Resources include people, accurate data, computational power, governance policies etc.

GEN AI provides support to produce usable information but the technology is not flawless.  Hallucinations or simple inaccuracies can lead to major damages if they are not edited by skilled and knowledgeable personnel.  This is an addition to resource utilization.  Consider passing AI generated information to customers regarding product attributes or industry research that has errors.  The potential damage to credibility and brand strength can be significant.

On the Agent side, risks are no less significant.  Consider fully automating processes for financial operations (e.g customer portfolio reports).  There must be a safeguard (skilled personnel and governance procedures) to address potential disruptions in autonomous applications which can occur from small changes in process activities or adverse events in routine processes.  The risk can be magnified in operational areas that are comprised of integrated Agents.

We are in the age of AI, which on balance provides value to many facets in the realm of commerce.  However, the technology is not perfect and requires a skilled, knowledge rich base to manage its use. When estimating ROI, organizations must consider no only cost reductions in time and personnel but the full allocation of resources that are required to implement them.

 

References:

1. Aditya Challapally, A. Pease, C. Raskar, R & Chari, P. The Gen AI Divide: State of AI in Business 2025, July, MIT Media.

Tag/s:Artificial Intelligence, Automation, Business Transformation, Finance, ROI,