At the big data center where this pilot fish works, they're always pulling new cables under the floor -- and the space is so stuffed that if someone pulls up a tile to access a plug or a cable, getting the tile back in place can mean jumping up and down on it to get it to re-seat.
"It was the week I was on call, and at 5 a.m. I got a call that we had lost power to half of our system drive," fish says.
"The system was both hardware and software redundant, so we didn't lose any processing, but we needed to have someone come out and check it out."
So at 8 a.m. fish, the vendor's repair tech, the computer room supervisor and the facilities manager are all in the computer room to figure out what happened.
Vendor tech has already determined that the drive isn't bad, but it isn't getting power. Facilities manager pulls up the appropriate tiles to get access to the power strip -- and discovers that the twist-lock plug for the drive isn't plugged in.
It can't have just fallen out, facilities manager says. To demonstrate how solid the twist-lock connections are, he grabs the other plug on the strip and begins to shake it up and down.
At which point, that plug comes out of the strip too -- and the entire system goes down.
"Turns out that during the night they were running some new cables," sighs fish. "In the process, a cable apparently got wrapped around that plug, causing it to pop out of the socket when it was pulled.
"After that, two changes were made concerning running new cables through existing system areas: Power-supply strips could no longer provide both the primary and backup power on the same strip -- and facility managers were not to touch anything under the floor."
Special note from Sharky: Want bonus Shark Tank tales? Check out a selection of Sharky's favorite stories from the Sharkives -- part of CW@50, Computerworld's year-long 50th-anniversary celebration. And remember...
Sharky is powered by true tales of IT life. So send me your story at sharky@computerworld.com. You'll score a sharp Shark shirt if I use it. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.