This retail energy bundler has a huge data warehouse that contains all sorts of promotions and aggregated billing data, according to a recently hired pilot fish who works on it.
"I was tasked with modifying one of the packages that generated daily reports for our business analysts," fish says. "These analysts used the reports to provide summaries for our wholesale energy producers.
"So I went to our source code control system, checked out the package I needed to modify and compiled it on our development database. I made the desired changes, promoted it to our QA database and had the analysts test the results."
Once the modified module is approved, fish schedules the package to be promoted to production status during the next evening maintenance window -- all pretty much standard procedure.
But the following morning, he gets a call from a business analyst. "The monthly sales report for one of the customers is incorrect," analyst tells fish. "It appears that the change we requested six months ago is now gone! What did you do?!?"
Fish has only been with the company for four months, so he knows nothing about a six-month-old change. But he immediately scans the module's history in the source control system. The last change he sees is from more than eight months ago.
Analyst insists there was a change made more recently than that -- and obviously something has changed overnight that testing didn't turn up. Fish goes hunting for what it might be.
And he finds it. Turns out the developer who made the change is a guy who left the company right before fish came on board. Instead of checking the module out and back into the source control system, he just downloaded it directly from the production database, made the change and then compiled it in Production.
"In fact, he tested it in Production," groans fish. "I finally found his modified version on another test database that had been cloned from Production the previous month. I was able to download the correct version, update source control with it, then apply my changes so we had the proper change history.
"My manager gave me kudos for the quick resolution to the problem, and told me it was poor software development practices like this that had prompted my predecessor's termination."
Got a problem predecessor? Tell Sharky about it. Send me your true tales of IT life at sharky@computerworld.com. You'll score a sharp Shark shirt every time I use one. Comment on today's tale at Sharky's Google+ community, and read thousands of great old tales in the Sharkives.
Get your daily dose of out-takes from the IT Theater of the Absurd delivered directly to your Inbox. Subscribe now to the Daily Shark Newsletter.