Database admin pilot fish is working for this city government on a contract basis, telecommuting to do most of his work overseeing the SQL Server databases, with just occasional trips in to the office.
"I had an encounter with the storage area network admin regarding backups for the ERP database," says fish. "SQL Server DBAs tend to be paranoid and protective, and most of us prefer not to use third-party backup solutions and to stick with Microsoft's tool -- it's as close to 100 percent reliable as you can get, and we all know how to work with it."
But the SAN admin wants to use Microsoft's enterprise backup tool, and he won't give fish access to the tool so fish can run tests and become confident in the tool's reliability for doing restores.
Push comes to shove, the IT director sides with the SAN admin, and fish disables the SQL Server backup job so the Microsoft enterprise tool can take control.
Eventually a permanent full-time DBA is hired, though fish is kept on contract -- good thing, too, because the permanent hire turns out to be not so permanent after all and leaves in short order.
"Another permanent hire was made, and my contract was not renewed," fish says. "That hire also turned out to be not so permanent, but by then I was long gone.
"Then a decision was made that clearly the city did not need an in-house database administrator. After all, they'd been without one for so long, what could possibly go wrong?"
Not long after, a new database is spun up on the SAN for billing traffic violations -- and the database admin duties are handed off to the vendor that sold the city the billing software.
Unfortunately, no one on the vendor's team notices when the backups start failing. Then no one at the vendor notices that the database is showing signs of corruption.
And then the SAN starts glitching, and eventually a glitch causes a drive to drop out of the SAN -- a drive that contains a portion of the traffic violation database.
Result? "Instant database crash," sighs fish. "And no recent backups. And corrupt backups that couldn't restore.
"Eventually they got the database working again -- after losing about two months of billable violations.
"And the vendor? They would continue monitoring the databases. Surely they would do a better job now, right?"
Sharky is paranoid and protective about your true tales of IT life -- I always file off the identifying marks. So send me your stories at sharky@computerworld.com. You'll snag a snazzy Shark shirt every time I use one. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.
Get Sharky's outtakes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.