ITS outage resolved, posed serious threat to data storage space

By Nico Zulli

A rare and more serious ITS outage was detected Wednesday night when storage space in Baylor’s data center was experiencing technical issues.
Bob Hartland, associate vice-president for IT Infrastructure, said the reason this incident was more complicated than usual outages is because it impacted several on-campus systems, including Blackboard, the Baylor identification system, Astra scheduling system and Baylor Web pages.

Despite the severity of the outage, all campus applications are back up and running with all data safe and intact.

“We were told by one of our vendors we needed to update storage within the data center, because there was an issue that came about that was not expected,” Hartland said.

He said the first step in the ITS recovery process is to assess what happened.

“In this case, we already knew that the storage space in the data center essentially had a bug,” Hartland said. “But the investigative part was figuring out what had been affected.”

He said most of Thursday morning was dedicated to getting the affected applications back on track.

“The assessment and recovery procedure is a collaborative effort,” Hartland said. “For instance, [Thursday] morning I walked in to a discussion between the web administrator and server administrator on how each sector had been affected and potential solutions to the issue.”

After it was determined that Blackboard, the Baylor ID system, and the Baylor Web pages had been affected, the next step was to alert other system administrators of the details and develop a collective plan to get everything back on track.

Hartland said after a common thread is found and a collective plan of action established, every department is responsible for carrying out the protocol necessary to implement the recovery solution.

Hartland also said he believes the protocol was carried out successfully on all fronts, prior to and after this outage incident.

“The Web page and Blackboard were fully functional again around 8:30 a.m., and everything was back up and running by 11:30 a.m.,” Hartland said.
Hartland said the people responsible for the other campus applications also followed procedure in order to get things back on track in a timely and effective manner. The focus of the recovery began with what ITS determined were the most important and heavily accessed applications — Baylor Web pages and Blackboard.

“After my review, the technician involved in this situation did everything right and there is nothing procedurally that needs to be changed on the ITS end,” Harland said. “Obviously we regret any outage. It was just an unfortunate issue, but it has been fully resolved.”