What is incident management?

Incidents can cause a host of problems for organizations, from temporary downtime to data loss. When done well, incident management can provide an efficient and effective way to fix all kinds of incidents with little disruption and leave organizations more prepared for future incidents.

With roots in the IT service desk, incident management has long served as the primary interface between IT operations (ITOps) and the end user. As technology has advanced and become more complex, so has the way organizations view incident identification and incident response. The practice has expanded far beyond helping users fix problems to become a process for maintaining constant app uptime and accelerating continuous improvement efforts.

Incident management within a company’s IT operations, often referred to as ITIL incident management, addresses a wide range of issues that can impact service and business operations, from a laptop crashing or a printer error to wifi connectivity issues and network downtime.

Incident management, under the framework of ITSM (IT service management), functions as one aspect of the ITSM service model. Rather than focusing on creating systems and technology, incident management for IT is more user focused. It aims to keep IT infrastructure operating properly, whether it be an app or an endpoint, such as a sensor or desktop computer.

Within ITSM, the IT department has various roles, including addressing issues as they arise. The severity of these issues is what differentiates an incident from a service request.

A service request, simply put, is when a user is asking for something to be provided, such as advice or equipment. Services can include requesting assistance with a password reset or getting additional memory for a desktop computer.

An incident, on the other hand, is more urgent and indicates an underlying error that needs addressing.

An incident is a single, unplanned event that causes a disruption in service, while a problem is the root cause of a disruption in service, which can be a single incident or a series of cascading incidents.

The difference plays out in remediation and how responders approach fixing the issue. Incident response is reactive. Incident management teams get an alarm and address the incident. However, when addressing a problem, IT teams identify the root cause and then fix it. Problem management takes a proactive approach, looking at various types of incidents and patterns that emerge to understand how future incidents can be prevented.

DevOps teams are focused on finding more efficient ways to build, test, and deploy software, which in part, requires addressing incidents quickly. Like ITIL incident management, DevOps incident management aims to fix issues without disrupting operations. For example, DevOps teams might monitor for poor mean time between failures (MTBF) metrics, which can indicate that there’s an underlying issue that needs to be investigated.

Because DevOps is rooted in continuous improvement, there is a significant focus on post-mortem analysis and a blame-free culture of transparency. The goal is to optimize the overall system performance, streamline and accelerate incident resolution, and prevent future incidents from occurring.

Like today’s IT teams, DevOps teams often use automated provisioning, incident prioritization and artificial intelligence (AI)-enabled root-cause analysis tools to ensure uptime, address the most pressing incidents first, and learn how to fix future problems more quickly. (Or prevent them in the first place.)

Organizations typically create an incident management process that documents the sequence of events the response team should take. All stakeholders should know which staff are responsible for handling incidents, the time it should take to solve the issue, when to escalate the incident to the next level, and how to document the incident and the way it was resolved.

Once the process is defined, the incident management workflow typically goes as follows:

1. Identify the incident: Whether it’s an end user submitting a ticket to the help desk or an automated alert system notifying the team of an issue, the response team needs a way to receive reports of problems within the system.
   
   
2. Log and classify the incident: This includes entering the incident report into an incident logging system and assigning prioritization, including which level of staff should handle it. For example, Level 1 incidents are usually handled by newer, less experienced staff while Level 2 and Level 3 incidents are increasingly challenging to solve and require the most experienced responders.
   
   
3. Contain the issue: If it is a security incident, response teams must act quickly to contain the issue, whether it’s a DDoS attack or a data breach. In all cases, teams must ensure that the incident doesn’t spread and further impact the system.
   
   
4. Diagnose the incident: This is where the troubleshooting comes in. Response teams might use a knowledge base or ChatOps tool to suggest possible causes and save time.
   
   
5. Resolve the incident: Once the cause has been identified, teams get to work addressing the incident, whether it’s provisioning additional memory or addressing a network outage.
   
   
6. Close and review the incident: Post-mortem reviews are an important aspect of improving reliability and availability in today’s digital environments. This data not only increases the organization’s institutional knowledge, but it can also be used in machine learning and AI-enabled tools to help identify incidents more quickly and even create notifications when incidents are likely to happen. Thorough reviews help organizations implement more effective incident remediation procedures.

All organizations need to fix problems and resolve incidents. It’s how they keep the business running. But there are also clear benefits to having effective incident resolution tools—and teams—that can react quickly without major disruption to the business. Those benefits include the following:

The growing complexity of IT operations, which is driven in part by the many applications organizations rely upon in day-to-day business operations, has made incident response tools and automation more important than ever.

Some of the most common incident management tools include:

• Monitoring tools: These tools identify outages, trigger alerts, and diagnose incidents. Monitoring tools also reduce costs by freeing DevOps teams to better manage the software lifecycle.
  
  
• Service desks: This is a place for users to submit tickets, chat with the service desk team, monitor the progress of their tickets and perform some self-service tasks. Typically, the service desk is run through a management system that enables key incident management tasks, such as prioritization and categorization.
  
  
• AlOps platforms: Using logs and historic data, AIOps can provide context for better decision-making, smarter resource allocation and faster incident response.
  
  
• VDocumentation: These are scripts that automatically document changes to an environment, making it easier to record incidents for postmortem analysis. For example, teams can set up the PowerCLI scripts to run on a monthly schedule to record incidents for deeper analysis.