University technicians worked around the clock last week to fix the university’s e-mail system after it suffered a major hardware failure.
The incident, which the university’s chief information officer (CIO) labeled “very unusual and very severe,” occurred a week before opening week, going on into the weekend before being repaired early Sunday morning.
‘”This is severe because we’ve lost so much,” said Martyne Hallgren, the CIO. “It’s very unusual to lose this much.”
The entire system was taken off-line on Friday, due to fears of corrupting the rest of the database.
“We’ve had a major hardware failure,” said Hallgren. “When we have a failure like this, we go into 24-hour mode.”
Computing Services’ IT staff, in 24-hour operations mode since Thursday, had members on thirty-six hours without any sleep.
Normally an eight to five staff composed of 99 people, the IT staff had six working on the hardware failure and were joined by a senior Dell technician flown in from Cincinnati, and one of the company’s Northeast regional representatives.
“Our guys will be here until it’s done,” said Hallgren, in her office in Computing Services. As will Dell’s, she added. “This campus has a crew of people who are incredibly dedicated to it,” she said.
The problems, affecting anyone with e-mail addresses starting A through H, started September 1st. IT staffers replaced the damaged hardware and got the system up and running by late afternoon. The hardware failure happened again at 11 p.m. On Thursday, working with Dell staffers, IT started the 24-hour operations mode. Giving up a key hardware part for lost, a new piece was ordered from Dell, that flew it in on a private jet from Austin, Texas. The rest of the alphabet on the e-mail system was taken down on Friday, due to fears of system corruption. Hallgren kept communications between the campus community going through the main web page, the television sets across campus, voicemail broadcast messages, the IT status website, and a help desk audio message.
In a memo to the university community, soon after the system was back up on Sunday, Hallgren wrote, “While we believe we have successfully restored the system, because of the intermittent performance on Thursday and Friday morning and the length of the downtime, the potential exists for some data to be lost or for e-mail to have been returned to sender. We will continue to investigate reports of problems however our primary concern over the next 24-48 hours is to assure system stability.”
Hallgren also thanked various staffers for their help during the crisis. “As a team, they’ve worked ’round the clock since Thursday night, canceling or delaying personal commitments with their family and friends, to make sure that this system would be fixed as soon as possible,” she wrote. “For the rest of the IT staff that provided them support in many ways, from the Operations staff who stayed Friday night to receive the Dell hardware flown into Boston on a private jet, the helpdesk that started calling departments on Thursday to the administration staff who proofed my communication memos to you: thank you.”
Hallgren said it was fortunate the problems occurred on a Thursday and Friday of a long Labor Day weekend. Think of what could have happened had the e-mail system gone down during opening week, she said.
The situation was one of the worst she’s experienced, due to the length of time it took to fix it. “I wouldn’t say the worst,” she said.
Hallgren, who has worked at Cornell University, arrived at UMass Boston little over a year ago, just as the campus and many other universities were hit with the Sobig.F and MS Blaster worms, felling many of the university’s desktop computers. “It hit us pretty bad,” she said.
The university immediately took infected servers and desktops off the network, reconnecting only after measures showed they were safe.
Problems with e-mail last surfaced this year sometime around Father’s Day, where e-mail was lost for 24 hours. It’s currently unknown if it is related to last week’s incident, says Computing Services.