ATLANTA – In the hours after the Delta Air Lines computer outage last month, the airline resorted to an archaic form of passenger processing: Writing boarding passes by hand.
Even after systems came back up, some were still slow, and gate agents in some cases had to manually enter codes from boarding passes.
The episode resulted in a huge black eye for Delta, with more than 2,000 flight cancellations over several days affecting people all over the world. The manual processes slowed check-in and boarding, delaying flights that did operate.
The Delta outage also sent a warning to any company that's heavily dependent on technology for daily operations: Have a Plan B ready if things should go haywire.
Such companies include UPS, which also operates an airline in its massive shipping business and whose employees use hand-held readers to log package movements and deliveries.
It could also include hospitals, financial technology firms and a wide range of other businesses.
UPS, based in Sandy Springs, Ga., said it regularly reviews business continuity plans and conducts exercises.
"UPS has built redundancies throughout our global network to ensure continual operations for our customers," the company said. UPS also said it has technical experts available "to respond rapidly and minimize any impacts on service to our customers."
Yet Delta's outage — one of several that have afflicted airlines — showed how challenging it is for a technology-dependent company to reach back to old practices that are unfamiliar to employees, time-consuming and not designed for the size or speed of the current operation.
"Some of the efficiencies gained by technology have made it a little more difficult for airlines to go more manual," said Joe Leader, CEO of the Airline Passenger Experience Association.
"With electronic flight bags and dispatch, operational control, routing for aircraft, boarding passes and other elements required for travel, there are so many variables in play that you can't conduct airline operations without a baseline of technology."
It also highlights the difficulties of coming up with backup procedures when computer systems are down, or at least procedures that allow for continued operations while complying with security and safety rules.
Paul Jacobson, Delta's chief financial officer, said the airline is making investments over the next couple of years "to minimize the probability that that ever happens again."
Manual processes kick in
Brian Myers, a traveler who lives in Atlanta and flew out on the first day of Delta's outage, said boarding of his flight took nearly two hours because some passengers' boarding passes wouldn't scan correctly and an agent had to manually enter a series of codes from a spiral notepad.
It was "at least a 10-minute process," Myers said. "It's crazy how dependent they are on all the electronics working."
Even when handwritten boarding passes and other methods appear to be manual, there is often still a collection of digital data used to check passenger records and flight schedules, for example, according to Leader.
Delta spokesman Morgan Durrant said the airline has "manual airport procedures in the event of a rare loss of power or network connectivity at an airport gate to safely and securely keep aircraft, bags and customers moving."
Each airline has an outage procedure approved by the Transportation Security Administration that covers validity of boarding passes, TSA spokesman Mark Howell said.
Delta also turned to some unconventional backup procedures, including using its Delta Private Jets subsidiary to fly about 40 passengers from Atlanta to their final destinations.
Those backups, to be sure, were a last resort when Delta's entire computer system crashed after a power control module failed at Delta's technology command center on Aug. 7. The company discovered that about 300 servers were not connected to backup power, which meant the system could not be smoothly restarted.
Re-evaluating best practices
Companies all over that are dependent on technology took note, said Tony Cooper, spokesman for the Technology Association of Georgia.
"I think that everyone's taking notice and re-evaluating their best practices and procedures to avoid this problem," he said.
Southwest has "outage kits" at every airport where it operates, said Biljana Obrenic, customer service manager of regulatory compliance.
"With any outage, we get so used to the ability to use our systems [that] when they go down, you kind of hyperventilate and you panic for a moment," said David Woodard, Southwest's director of ground operations standards.
"Then you have to take a quick breath and say, OK, we have a procedure."