How to assess the ROI of your AI projects when the recession hits

As businesses scramble to protect themselves from the economic downturn, all sorts of projects are being hit. And applied artificial intelligence (AI) is no exception.

Before the recession, the AI ​​industry was in a gold rush, with companies pouring heavily into machine learning (ML) talent, research, and projects. While these efforts have paid off and can be seen in applications we use every day, much of this investment was driven by an unwarranted hype around AI.

As organizations adapt their AI initiatives to the new market conditions, you can expect the following.

Measure ROI for AI projects

“Even before the recession, we talked about ROI in AI projects,” Anand Rao, Global Artificial Intelligence Lead at PwC, told VentureBeat. “While ROI is a concern with any technology adoption, the difference with AI, unlike other technologies like cloud, is that you’re talking about prediction.”

How do you measure the value of prediction? Most companies use under supervision machine learning models, meaning they train their models on examples labeled by human experts. The model’s percentage accuracy is then measured by comparing the predictions to the ground truth specified by human annotators. However, not all accuracy measurements are created equal.

“In many organizations, it’s not the best person doing the labeling,” Rao said. For example, when a financial institution creates an ML model for underwriting decisions, it will produce a better result if the best underwriters label the training samples than if an intern does the labeling in their spare time. This is critical because a model with (say) 95% accuracy is more valuable than a model with a lower accuracy percentage.

“There’s also a complication when you don’t measure human performance as rigorously as you measure AI performance,” Rao said. “You really don’t know if all your insurers match the accuracy of your AI system. If not, your AI is much better than you initially thought.”

And finally, organizations must also consider the costs of mispredictions, Rao said, which depend on the application, environment, customers and many other factors.

“The challenge of measuring the ROI of AI/ML algorithms has always been there,” says Rao. “We need stricter measures. Now, with the recession, it becomes even more important that we understand the ROI of ML/AI algorithms.”

With a clearer picture of the profitability of their AI projects, organizations are better able to decide whether to continue or stop.

The AI ​​portfolio approach

AI will continue to be important for maintaining a competitive edge in many industries, even during the recession. But companies need to adapt their AI strategies to economic conditions. And this can start with a change in how a company looks at AI projects.

“Executives like to look at every project and wonder, what is the ROI for this recommendation engine or this NLP technology?” Rao said. “Measuring ROI at that level, project by project, is not the right approach. You’re going to say, ‘This project had no ROI, so let’s stop doing that kind of work in the future.’”

Rao recommends what he calls a “portfolio approach.” Rather than measuring the success of AI projects on a project-by-project basis, companies should view their AI initiative as a portfolio that includes a variety of AI projects.

Some projects will be based on ML models that have been tested and proven to work by competitors. These are the low-hanging fruits of applied AI. They have a high chance of success and are easy to adopt. Rao calls them “ROI-generating AI projects”.

Other projects will focus on experimenting with state-of-the-art AI such as large language models, exploring new technology and keeping your data scientists motivated to push the boundaries. These types of projects have a lower chance of success, but can generate a higher return if they succeed.

“You need a portfolio where some projects are new, some are just maintenance types, some are things others have done,” Rao said. “You conduct many experiments, and perhaps three out of ten will succeed. And they will yield much more than the whole 10 together.”

Executives should also consider the risk-return tradeoff of their AI projects. This means that instead of selecting models based on accuracy, AI managers should look at a wide range of attributes, including fairness, explainability, robustness, and safety. Facial recognition technology, for example, poses privacy and ethical risks that must be weighed against the benefits of the technology.

“I think the portfolio approach will take hold, especially in the recession where people are asking questions about the value of AI,” said Rao. “We are almost maturing from a hyped ‘cool’ technology to meeting reality and becoming more entrenched in traditional technology and gaining the rigor needed to be widely accepted.”

The tech talent bubble

In recent years there has been a large influx of data scientists and machine learning engineers in various industries. The growing demand for AI talent has created a bubble in which technology companies offer huge salaries. As businesses grapple with the recession, there will be an adjustment.

“People were paid a lot for their AI talent, not only from the outside but also from within the tech industry. They went from one company to another and back, constantly changing with bigger offers. Salaries and allowances were constantly rising,” said Rao. “In the past year, the pressure on the technology industry has been enormous. In addition to job cuts, there is a freezing of technical talent. We see the tech talent bubble burst.”

With the economy slowing down, many organizations are beginning to question whether they are getting the desired returns from the massive investments they are making to attract and retain AI talent. Do they get a measurable increase in sales? Would their revenue drop by half if they kept only half of their AI/ML engineers?

“The question being asked is: what exactly is the value they add?” Rao said. “There is a lot of focus on ROI and the productivity of AI/ML people in relation to their salary.”

As senior executives begin to question the productivity of AI/ML engineering, there will be a slowdown in hiring, Rao believes. At the same time, companies need to go back to the drawing board and figure out ways to measure the ROI of their AI projects and determine how much of their revenue is a result of AI/ML.

The bright side of the bursting tech talent bubble is that the AI ​​talent pool will become much more accessible to other industries.

“Previously, being a product manager at a large technology company was a dream job for someone with a CS or MBA background. Now they are looking past tech companies because there is not much inflow from technology companies,” Rao said. “The brain drain from other sectors to tech is turning. In a way, it’s good to have that correction. We were previously in an inflationary bubble. Now it becomes a more rational compensation model across the board.”