The long game: Businesses investing in AI infrastructure need to push to the finish line

Businesses are spending big on AI infrastructure, yet many aren’t seeing results. According to a new study from the IBM Institute for Business Value (IBV), organizations attempting to scale AI are hitting a roadblock upon deployment: a fractured foundation built with traditional methods. Which raises a question: If succeeding with AI means redirecting your enterprise infrastructure, will the gains outweigh the costs?

The study, which surveyed 1,200 senior executives with responsibility over AI infrastructure strategy, found that 62% of executives said they’ll be able to use AI across their organization within three years. But just 8% said their current infrastructure meets all their AI needs, suggesting that implementing AI throughout an organization is more than just a technical challenge.

“We’re seeing organizations treat AI infrastructure like a typical IT refresh, but that’s where they get stuck,” said Robert Zabel, Associate Partner and Research Lead at the IBV and co-author of the study, in an interview with IBM Think.

When an organization attempts to scale AI production within the parameters of a traditional planning model, a number of problems can arise, including data gaps, governance risks and the infrastructure’s inability to handle AI workloads. Organizations can’t predict when a model’s processing and resource needs will suddenly increase, and data requirements can shift overnight, illuminating the need for a hybrid infrastructure for AI.

AI innovation is moving at a breakneck pace, and governance and security frameworks are struggling to keep up. To help them meet the moment, the IBV report research suggests organizations use a “trust-by-design” approach, which embeds governance into an enterprise’s AI infrastructure from the very start.

“The governance gap is real,” said Zabel. “Executives keep telling us how critical it is, but when we look under the hood, most organizations are flying blind on AI risk management.”

While 83% of executives say that AI governance is essential, citing ethical considerations and privacy and data security ranking as their most pressing concerns, just 8% say they have a framework for managing AI-related risk, according to the report.

The report points to Banco do Brasil as an example of a company that used a holistic governance strategy to build its own trust-enabled infrastructure, automating governance across the AI lifecycle with the help of accounting firm Ernst & Young (EY) and IBM. The bank built a unified AI governance system with security at its core, using the IBM® watsonx.governance® toolkit as a technical foundation.

“You can’t treat governance like an afterthought and expect to scale responsibly,” Zabel said. “It has to be embedded from the ground up, or you’ll find yourself scrambling to rebuild trust when something goes wrong.”

Zabel believes the relationship between organizations and outside vendors is key to closing the AI skills divide and coming together is an accelerator for AI readiness. Gone are the days of enterprises taking a purely transactional approach, with 99% of executives citing at least one AI ecosystem partner and 68% reporting that they are actively exploring or developing one.

“The biggest surprise in our research was how much partnership execution matters,” Zabel said.

While the researchers found that most organizations had a relationship with at least one vendor, just 5% have embedded these relationships within their operations. Additionally, over a third of them still treat partners as part of a vendor pool, rather than as part of an alliance.

“Organizations are failing, not because they picked the wrong technology, but because they’re still treating vendors like order-takers instead of strategic allies who understand their business challenges.”

So where do organizations go from here? According to Zabel, it’s about building a strong base—including establishing a firm AI foundation and a solid platform for all components to work together. “The organizations succeeding are the ones building infrastructure that adapts in real-time,” Zabel said, “not ones trying to plan their way out of uncertainty.”