October 24, 2023 By Sunil Joshi 5 min read

As enterprises invest their time and money into digitally transforming their business operations, and move more of their workloads to cloud platforms, their overall systems organically become largely hybrid by design. A hybrid cloud architecture also means too many moving parts and multiple service providers, therefore posing a much bigger challenge when it comes to maintaining highly resilient hybrid cloud systems.

The business impact of system outages

Let’s look at some data points regarding system resiliency over the last few years. Several studies and client conversations reveal that major system outages over the last 4-5 years have either remained flat or have increased slightly, year over year. Over the same timeframe, the revenue impact of the same outages has gone up significantly.

There are several factors contributing to this increase in business impact from outages.

Increased rate of change

One of the very reasons to invest in digital transformation is to have the ability to make frequent changes to the system to meet business demand. It is also to be noted that 60-80% of all outages are usually attributed to a system change, be it functional, configuration or both. While accelerated changes are a must-have for business agility, this has also caused outages to be a lot more impactful to revenue.

New ways of working

The human element is mostly under rated when to comes to digital transformation. The skills needed with Site Reliability Engineering (SRE) and hybrid cloud management are quite different from a traditional system administration. Most enterprises have invested heavily in technology transformation but not so much on talent transformation. Therefore, there is a glaring lack of skills needed to keep systems highly resilient in a hybrid cloud ecosystem.

Over-loaded network and other infrastructure components

With highly distributed architecture comes the challenges of capacity management, especially network. A large portion of hybrid cloud architecture usually includes multiple public cloud providers, which means payloads traversing from on-premises to public cloud and back and forth. This can add disproportionate burden on network capacity, especially if not properly designed leading to either a complete breakdown or unhealthy responses for transactions. The impact of unreliable systems can be felt at all levels. For end users, downtime could mean slight irritation to significant inconvenience (for banking, medical services etc.). For IT Operations team, downtime is a nightmare when it comes to annual metrics (SLA/SLO/MTTR/RPO/RTO, etc.). Poor Key Performance Indicators (KPIs) for IT operations mean lower morale and higher degrees of stress, which can lead to human errors with resolutions. Recent studies have described the average cost of IT outages to be in the range of $6000 to $15,000 per minute. Cost of outages is usually proportionate to the number of people depending on the IT systems, meaning large organization will have a much higher cost per outage impact as compared to medium or small businesses.

AI solutions for hybrid cloud system resiliency

Now let’s look at some potential mitigating solutions for outages in hybrid cloud systems. Generative AI, when combined with traditional AI and other automation techniques can be very effective in not only containing some of the outages, but also mitigating the overall impact of outages when they do occur.

Release management

As stated earlier, rapid releases are a must-have these days. One of the challenges with rapid releases is tracking the specific changes, who did them, and what impact they have on other sub-systems. Especially in large teams of 25+ developers, getting a good handle of changes through change logs is a herculean task, mostly manual and prone to error. Generative AI can help here by looking at bulk change logs and summarizing specifically what changed and who made the change, as well as connecting them to specific work items or user stories associated with the change. This capability is even more relevant when there is a need to rollback a subset of changes because of something being negatively impacted due to the release.

Toil elimination

In many enterprises, the process to take workloads from lower environments to production is very cumbersome, and usually has several manual interventions. During outages, while there are “emergency” protocols and process for rapid deployment of fixes, there are still several hoops to go through. Generative AI, along with other automation, can help greatly speed up phase gate decision-making (e.g., reviews, approvals, deployment artifacts, etc.), so deployments can go through faster, while still maintaining the quality and integrity of the deployment process.

Virtual agent assist

IT Operations personnel, SREs and other roles can greatly benefit by engaging with virtual agent assist, usually powered by generative AI, to get answers for commonly occurring incidents, historical issue resolution and summarization of knowledge management systems. This often means issues can be resolved faster. Empirical evidence suggests a 30-40% productivity gain by using generative AI powered virtual agent assist for operations related tasks.


As an extension to the virtual agent assist concept, generative AI infused AIOps can help with better MTTRs by creating executable runbooks for faster issue resolution. By leveraging historical incidents and resolutions and looking at current health of infrastructure and applications (apps), generative AI can also help prescriptively inform SREs of any potential issues that may be brewing. In essence, generative AI can take operations from being reactive to predictive and get ahead of incidents.

Challenges with generative AI implementation

While there are strong use cases for implementing generative AI to improve IT Operations, it would be remiss if some of the challenges weren’t discussed. It is not always easy to figure out what Large Language Model (LLM) would be the most appropriate for the specific use case being solved. This area is still evolving rapidly, with newer LLMs becoming available almost daily.

Data lineage is another issue with LLMs. There needs to be total transparency on how models were trained so there can be enough trust in the decisions the model will recommend.

Finally, there are additional skill requirements for using generative AI for operations. SREs and other automation engineering will need to be trained on prompt engineering, parameter tuning and other generative AI concepts for them to be successful.

Next steps for generative AI and hybrid cloud systems

In conclusion, generative AI can bring in significant productivity gains when augmented with traditional AI and automation for many of the IT Operations tasks. This will help hybrid cloud systems to be more resilient and, in due course, help mitigate outages that are impacting business operations.

Discover more about the impact of generative AI on business Learn more about site reliability engineering

More from Business transformation

Types of enterprise resource planning (ERP) systems

6 min read - The fast-paced business world we live in today requires smart tools to manage a business’s operations and everyday needs. Enterprise resource planning (ERP) is a business management software built to do just that. This relatively new system offers a centralized platform with applications to manage all aspects of your business from supply chain management to inventory management to financial management. The benefits of ERP include built-in business intelligence and access to real-time data, along with integrated machine learning capabilities unlike…

How to build a successful employee experience strategy

4 min read - Ever since the pandemic changed the corporate world, organizations have rededicated themselves to excelling at employee experience strategy. A successful employee experience strategy (EX strategy) is the best way to recruit and retain top talent, as employees increasingly make decisions on where to work based on how they respond to employee needs. Organizations can prioritize overall employee experience by being thoughtful about how to serve their workers during all stages of the employee journey, from the hiring process to the…

How to build a successful talent acquisition strategy

11 min read - The success of any organization relies on its ability to attract, retain and develop top talent. Talent acquisition refers to the ongoing strategy and process an organization and its HR department uses to source, attract, evaluate, hire and retain the highly-qualified new employees it needs to grow. A well-crafted talent acquisition strategy has become a critical component for organizations seeking to secure a competitive edge. Beyond simply filling open roles, a comprehensive talent acquisition strategy encompasses a holistic approach to…

Innovation and authentic demand

2 min read - Arvind Krishna recently spoke with former IBMer and author Danny Sabbah about a new book he co-authored, The Heart of Innovation: A Field Guide for Navigating to Authentic Demand. In their conversation, Arvind and Danny discussed the secret to determining authentic demand and building innovative products that achieve runaway success. Here are five facts about innovation, taken from the book and their conversation, that may surprise you: While major technology breakthroughs dominate our perception of innovation—think the Xerox machine, the…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters