October 24, 2023 By Sunil Joshi 5 min read

As enterprises invest their time and money into digitally transforming their business operations, and move more of their workloads to cloud platforms, their overall systems organically become largely hybrid by design. A hybrid cloud architecture also means too many moving parts and multiple service providers, therefore posing a much bigger challenge when it comes to maintaining highly resilient hybrid cloud systems.

The business impact of system outages

Let’s look at some data points regarding system resiliency over the last few years. Several studies and client conversations reveal that major system outages over the last 4-5 years have either remained flat or have increased slightly, year over year. Over the same timeframe, the revenue impact of the same outages has gone up significantly.

There are several factors contributing to this increase in business impact from outages.

Increased rate of change

One of the very reasons to invest in digital transformation is to have the ability to make frequent changes to the system to meet business demand. It is also to be noted that 60-80% of all outages are usually attributed to a system change, be it functional, configuration or both. While accelerated changes are a must-have for business agility, this has also caused outages to be a lot more impactful to revenue.

New ways of working

The human element is mostly under rated when to comes to digital transformation. The skills needed with Site Reliability Engineering (SRE) and hybrid cloud management are quite different from a traditional system administration. Most enterprises have invested heavily in technology transformation but not so much on talent transformation. Therefore, there is a glaring lack of skills needed to keep systems highly resilient in a hybrid cloud ecosystem.

Over-loaded network and other infrastructure components

With highly distributed architecture comes the challenges of capacity management, especially network. A large portion of hybrid cloud architecture usually includes multiple public cloud providers, which means payloads traversing from on-premises to public cloud and back and forth. This can add disproportionate burden on network capacity, especially if not properly designed leading to either a complete breakdown or unhealthy responses for transactions. The impact of unreliable systems can be felt at all levels. For end users, downtime could mean slight irritation to significant inconvenience (for banking, medical services etc.). For IT Operations team, downtime is a nightmare when it comes to annual metrics (SLA/SLO/MTTR/RPO/RTO, etc.). Poor Key Performance Indicators (KPIs) for IT operations mean lower morale and higher degrees of stress, which can lead to human errors with resolutions. Recent studies have described the average cost of IT outages to be in the range of $6000 to $15,000 per minute. Cost of outages is usually proportionate to the number of people depending on the IT systems, meaning large organization will have a much higher cost per outage impact as compared to medium or small businesses.

AI solutions for hybrid cloud system resiliency

Now let’s look at some potential mitigating solutions for outages in hybrid cloud systems. Generative AI, when combined with traditional AI and other automation techniques can be very effective in not only containing some of the outages, but also mitigating the overall impact of outages when they do occur.

Release management

As stated earlier, rapid releases are a must-have these days. One of the challenges with rapid releases is tracking the specific changes, who did them, and what impact they have on other sub-systems. Especially in large teams of 25+ developers, getting a good handle of changes through change logs is a herculean task, mostly manual and prone to error. Generative AI can help here by looking at bulk change logs and summarizing specifically what changed and who made the change, as well as connecting them to specific work items or user stories associated with the change. This capability is even more relevant when there is a need to rollback a subset of changes because of something being negatively impacted due to the release.

Toil elimination

In many enterprises, the process to take workloads from lower environments to production is very cumbersome, and usually has several manual interventions. During outages, while there are “emergency” protocols and process for rapid deployment of fixes, there are still several hoops to go through. Generative AI, along with other automation, can help greatly speed up phase gate decision-making (e.g., reviews, approvals, deployment artifacts, etc.), so deployments can go through faster, while still maintaining the quality and integrity of the deployment process.

Virtual agent assist

IT Operations personnel, SREs and other roles can greatly benefit by engaging with virtual agent assist, usually powered by generative AI, to get answers for commonly occurring incidents, historical issue resolution and summarization of knowledge management systems. This often means issues can be resolved faster. Empirical evidence suggests a 30-40% productivity gain by using generative AI powered virtual agent assist for operations related tasks.


As an extension to the virtual agent assist concept, generative AI infused AIOps can help with better MTTRs by creating executable runbooks for faster issue resolution. By leveraging historical incidents and resolutions and looking at current health of infrastructure and applications (apps), generative AI can also help prescriptively inform SREs of any potential issues that may be brewing. In essence, generative AI can take operations from being reactive to predictive and get ahead of incidents.

Challenges with generative AI implementation

While there are strong use cases for implementing generative AI to improve IT Operations, it would be remiss if some of the challenges weren’t discussed. It is not always easy to figure out what Large Language Model (LLM) would be the most appropriate for the specific use case being solved. This area is still evolving rapidly, with newer LLMs becoming available almost daily.

Data lineage is another issue with LLMs. There needs to be total transparency on how models were trained so there can be enough trust in the decisions the model will recommend.

Finally, there are additional skill requirements for using generative AI for operations. SREs and other automation engineering will need to be trained on prompt engineering, parameter tuning and other generative AI concepts for them to be successful.

Next steps for generative AI and hybrid cloud systems

In conclusion, generative AI can bring in significant productivity gains when augmented with traditional AI and automation for many of the IT Operations tasks. This will help hybrid cloud systems to be more resilient and, in due course, help mitigate outages that are impacting business operations.

Discover more about the impact of generative AI on business Learn more about site reliability engineering
Was this article helpful?

More from Business transformation

Attention new clients: exciting financial incentives for VMware Cloud Foundation on IBM Cloud

4 min read - New client specials: Get up to 50% off when you commit to a 1- or 3-year term contract on new VCF-as-a-Service offerings, plus an additional value of up to USD 200K in credits through 30 June 2025 when you migrate your VMware workloads to IBM Cloud®.1 Low starting prices: On-demand VCF-as-a-Service deployments begin under USD 200 per month.2 The IBM Cloud benefit: See the potential for a 201%3 return on investment (ROI) over 3 years with reduced downtime, cost and…

IBM Partner Plus Day: Celebrating our collaborative journey with the IBM Ecosystem

4 min read - Today we kicked off IBM Think 2024 in Boston with Partner Plus Day, a full day dedicated to our partners who are driving client innovation, business growth and scaling the adoption of transformative technology such as generative AI. It is also the perfect opportunity to reflect on the progress and success of the IBM Ecosystem, more than a year after we introduced Partner Plus, the program that embodies our new era of partner-first collaboration. Our partners have generated remarkable business…

How will quantum impact the biotech industry?

5 min read - The physics of atoms and the technology behind treating disease might sound like disparate fields. However, in the past few decades, advances in artificial intelligence, sensing, simulation and more have driven enormous impacts within the biotech industry. Quantum computing provides an opportunity to extend these advancements with computational speedups and/or accuracy in each of those areas. Now is the time for enterprises, commercial organizations and research institutions to begin exploring how to use quantum to solve problems in their respective domains. As a Partner in IBM’s Quantum practice, I've had the pleasure of working…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters