Day4 - Surviving Hurricane Katrina

Body

Marshall Lancaster from [United Stationers Technology Services]presented how their [Lagasse] subsidiary successfully survived [Hurricane Katrina in New Orleans] in 2005. I feel this was one of the bestpresentations of the week, here at the [Data Center Conference]!

Lagasse, Inc. sells janitorial supplies, such as mops, cleaning chemicals, waste receptacles, and garbage can liners. Of the 1000 employees of Lagasse nationwide, about 200 associates were located in New Orleans at their main Headquarters, primary customer care center, and primary IT computing center.

Amazingly, Lagasse did not have a formally documented BCP (Business Continuity Plan) but more of aBCI (Business Continuity Idea). They chose to take a ["donut tire"] approach, putting older previous-generation equipment at their DR site. They knew that in the event of a disaster,they would not be processing as many transactions per second. That was a business trade-offthey could accept.

Evaluating all the different threat scenarios for impact and likelihood, and focused on hurricanes and floods.They had experienced previous hurricanes, learning from each,with the most recent being 2004 Hurricane Ivan and 2005 Hurricane Dennis. From this, they wereable to categorize three levels of DR recovery:

• Tier 1 - The most mission-critical, which for them related to picking, packing and shipping products.
• Tier 2 - The next most important, focused on maintaining good customer service
• Tier 3 - Everything else, including reporting and administrative functions

The time-line of events went as follows:

August 25

The US Government issues warning that a hurricane may hit New Orleans

August 27 - 7pm

Lagasse declares a disaster, starts recovery procedures to an existing IT facility in Chicago, owned by their parent company. A temporary "Southeast" Headquarters were set up in Atlanta.Remote call centers were identified in Dallas, Atlanta, San Antonio, and Miami.

August 28 - just after midnight

In just five hours, they recovered their "Tier 1" applications.

August 28 - 7:30pm

In just over 24 hours, they recovered their "Tier 2" applications.

August 29 - 6am

The Hurricane hits land. With 73 levees breached, the city of New Orleans was flooded.

The following week

Lagasse was fully operational, and recorded their second and third best sales days ever.

I was quite impressed with their company's policy for how they treat their employees during a disaster. For many companies, people during a disaster prioritize on their families, not their jobs.If any associate was asked to work during a disaster, the company would take care of:

• The safety of their family
• The safety of their pets. (In the weeks following this hurricane, I sponsored people in Tucson to go to New Orleans to attend to lost and stray dogs and cats, many of which were left behind when rescuers picked up people from their rooftops.)
• Any emergency repairs to secure the home they leave behind

Marshall felt that if you don't know the names of the spouse and kids of your key employees, you are not emotionally-invested enough to be successful during a disaster.

For communications, cell phones were useless. They could call out on them, but anyone with acell phone with 504 area code had difficulty receiving calls, as the calls had to be processedthrough New Orleans. Instead, they used Voice over IP (VoIP) to redirect calls to whichever remote call center each associate went to. Laptops, Citrix, VPN and email were considered powerful tools during this process. They did not have Instant Messaging (IM) at the time.

While the disk and tapes needed to recover Tiers 1 and 2 were already in Chicago, the tapes for Tier 3 were stored locally by a third-party provider. When Lagasse asked for thier DR tapes back, the third-party refused, based on their [force majeure] clause. Force majeure is a common clause in many business contracts to free parties from liabilityduring major disasters.Marshall advised everyone to strike out any "force majeure" clauses out of any future third-party DR protection contracts.

Hurricane Katrina hit the US hard, killing over 1400 people, and America still has not fully recovered. The recovery of thecity of New Orleans has been slow. Massive relocations has caused a deficit of talent inthe area, not just IT talent, but also in the areas of medicine, education and other professions. The result has been degraded social services, encouraging others to relocate as well. Some have called it the "liberation effect", a major event that causespeople to move to a new location or take on a new career in a different field.

On a personal note, I was in New Orleans for a conference the week prior to landfall, and helped clients with their recoveries the weeks after. For more on how IBM Business Continuity Recovery Services (BCRS) helped clients during Hurricane Katrina, see the following [media coverage].

technorati tags: Marshall Lancaster, United Stationers, Lagasse, BCP, Hurricane Katrina, Business Continuity, Disaster Recovery, donut tire, New Orleans, VoIP, Citrix, VPN, liberation effect, Chicago, force majeure, BCRS