The future of AI on the mainframe: 4 paths for mainframe-driven AI innovation

When most people hear the term "mainframe," they might not realize its vital contribution to innovation in enterprise business. Amid the excitement surrounding generative AI, it’s easy to assume that legacy IT infrastructure, such as the mainframe, has not been invited to the party.

The truth is that the mainframe is poised to play a crucial role in driving innovation by integrating next-generation AI acceleration technologies with an optimized ecosystem to enhance business and technology capabilities.

When we take a look back, it’s clear that the mainframe has actually been driving continuous innovation for over 60 years. According to a recent IBM Institute for Business Value report, 43 of the world's top 50 banks and 8 of the top 10 payment companies leverage the mainframe as their core platform.

For industries that handle massive amounts of data—think finance, healthcare and government—mainframes have been gaining more relevance for artificial intelligence (AI) strategies. In fact, that same IBV report showed that 79% of IT executives believe mainframes are essential for enabling AI-driven innovation and value creation.

At the same time, these business leaders are also looking to integrate AI into existing mainframe platforms to tap into valuable insights, automate tasks and enhance efficiency while preserving their strategic infrastructure. This approach maximizes the value of legacy systems while introducing new capabilities through AI-driven analytics and automation.

Many of today's AI deployments require organizations to move their data to the cloud. However, for industries that rely on high-speed data processing to handle highly sensitive data, keeping AI capabilities closer to where the data resides delivers substantial business advantages.

"It's about bringing AI to where the music plays, and this is happening across 2 main areas," says Khadija Souissi, Principal Solution Architect, AI on IBM Z and LinuxONE and Distinguished Technical Specialist at IBM.

"We're infusing AI into transactional workloads to obtain real-time insights into business applications for faster decision-making and allow clients to sustainably create intelligent applications that embrace generative AI solutions while safeguarding sensitive data. Additionally, we're building intelligent infrastructure, using AI for mainframe operating systems and subsystems. This could help predict upcoming workloads to proactively prepare required resources and also detect system operation anomalies that could assist in predicting outages and extracting  actionable insights related to system performance."

Mainframes handle almost 70% of the world's production IT workloads and are relied upon for their stability, high security and scalability. Today, on-chip AI accelerators can scale and process millions of inference requests per second at very low latency rates. This capability allows organizations to use data and transactional gravity by strategically co-locating large datasets, AI and critical business applications. In the future, next-gen accelerators will open up new opportunities to expand AI capabilities and use cases as an organization’s needs grow.

Traditional AI and generative AI (gen AI) models are helping define and shape the modern enterprise. Mission-critical and transactional use cases require machine learning, deep learning and gen AI capabilities, sometimes working together in an ensemble architecture to achieve better, improved business outcomes.

On the mainframe, ensemble AI uses the strengths of each of these diverse models, allowing for more robust, precise and flexible decision-making.

Looking first at traditional AI, these models generally apply pre-defined rules to analyze data and make decisions based on specific patterns. Examples include demand forecasting for inventory management or credit decisions based on a customer's historical data.

The evolution of AI has been marked by new concepts like large language models (LLMs)—advanced AI models trained on vast amounts of data that can understand natural language, generate human-like text, translate languages, answer questions, write code and even hold conversations.

LLMs that generate content are usually referred to as decoder models and are used in generative AI. This capability can be seen in chatbots that provide personalized customer service replies based on user inquiries and context. 

Encoder models are another type of LLM that excels at understanding natural language and processing unstructured text, focusing on extracting key information. Decoder models share this capability but also excel at generating new content.

Ensemble AI is a hybrid concept that integrates different AI technologies, such as traditional AI and LLM encoder models, to deliver faster, more accurate results than any single model can accomplish alone, tapping into the mainframe's massive processing power and data storage capabilities.

In insurance claim processing, for example, traditional AI can conduct an initial analysis of structured data, while encoder LLMs can handle more complex unstructured data to gain more detailed insights and address the claim accordingly.

Take an automobile accident claim. Traditional AI provides automated processing of structured data (for example, police reports, driver's licenses and registration information). The gen AI goes a step further to extract insights from unstructured data (for example, text and images related to injuries and vehicle damage) to help prioritize and meet the urgency of those claims.

Recently, IBM announced the upcoming IBM Telum® II Processor and IBM Spyre™ Accelerator. These technologies are designed to help businesses scale processing capacity across IBM Z® systems and accelerate the use of traditional AI models, large language AI models and ensemble AI.

The Spyre Accelerator card will allow IBM Z and LinuxOne systems to perform AI inferencing for LLMs and gen AI at an even greater scale than previously available.

Here are 4 examples of use cases that demonstrate how organizations could use these AI technologies to drive business innovation, improve operations and accelerate generative AI workloads.

Financial losses from fraudulent credit card transactions cause financial and reputation damage. According to the Nilson Report, credit card losses worldwide are expected to reach $43 billion by 2026.¹ 

An internal IBM case study showed that a large North American bank had developed an AI-powered credit-scoring model and deployed it on an on-premises cloud platform to help fight fraud. However, only 20% of credit card transactions could be scored in real-time. The bank decided to move the complex fraud-detecting tools to its mainframe.

After the mainframe implementation, the bank began scoring 100% of credit card transactions in real-time, with 15,000 transactions per second, providing significant fraud detection.

Moreover, each transaction used to take 80 milliseconds to score. With the reduced latency provided by the mainframe, response times now occur in 2 milliseconds or less. This move to the mainframe has also saved the bank over USD 20 million in annual fraud prevention spend without impacting service-level agreements.

The mainframe is vital in credit card transactions, handling 90% of transactions worldwide.² Now, financial organizations can continue counting on the mainframe and simultaneously integrate AI to detect fraud before a transaction closes, relying on the large amounts of transaction data already stored there instead of moving it to a cloud setting.

According to Forbes, the average cost of IT downtime can run as high as USD 9,000 per minute for large organizations and over USD 5 million an hour for higher-risk enterprises.³

Luckily, organizations can now use AI to proactively prevent or even predict an outage caused by equipment failure. By applying AI mechanisms, organizations can detect anomalies at the transaction, application, subsystem and system levels. For instance, sensors can analyze data from mainframe components to predict potential hardware failures and enable preventative maintenance.

Integrating the mainframe with new technologies, such as cloud and distributed systems, can create complexity for IT infrastructure and application teams. Organizations are increasingly turning to artificial intelligence for IT operations (AIOps)—the application of AI capabilities to automate, streamline and optimize IT infrastructure and operational workflows. AIOps enables IT operations teams to respond quickly to slowdowns and outages, providing better visibility and context.

Data privacy is paramount for organizations that handle sensitive data. This is a primary reason why industries, such as healthcare, continue to rely on the mainframe’s robust security features, including workload isolation, advanced data encryption and secure communication protocols.

Processing documents on the mainframe helps streamline and deliver accurate data extraction in a highly secure setting. Organizations can use gen AI to summarize financial documents and business reports, extract key data points (for example, financial metrics and performance indicators) and identify essential information for compliance processes (for example, financial audits).

As another example, governments can use gen AI to improve customs screening for suspicious cargo through sophisticated image processing techniques and analysis of textual descriptions associated with each shipment.

One of the biggest challenges on the mainframe has been migrating legacy applications written in COBOL into more modern programming languages. Why? This is primarily due to the generational shift in the tech workforce, where newer developers have gained skills in languages such as Java and Python during their education, while many of the seasoned professionals are still well-versed in older technologies.

COBOL is not going away anytime soon though—it still powers many critical business systems in sectors such as banking and government. According to Reuters, 43% of banking systems are built on COBOL, and 220 billion lines of COBOL are in use today.⁴

Virtual assistants on the mainframe are helping to bridge the developer skill gap. Tools, such as IBM® watsonx Code Assistant™ for Z, use generative AI to analyze, understand and modernize existing COBOL applications. This capability allows developers to translate COBOL code into languages like Java. It also accelerates application modernization while preserving legacy COBOL systems' functionality.

Watsonx Code Assistant for Z features include code explanation, automated refactoring and code optimization advice, making it easier for developers to maintain and update old COBOL applications.

For decades, the mainframe has been evolving, and it continues to evolve in lockstep to greet the opportunities presented by next-generation AI. The introduction of the Telum II processor and the Spyre Accelerator card presents an opportunity for organizations to unlock business value and create new competitive advantages.

"Today, the mainframe is not only a transactional platform but an evolving AI platform that will deliver meaningful value as businesses embark on their AI journeys," says Khadija Souissi.