Blue bombs: death by frozen fecal material (an unlikely odessy) .

Today was a work day in hell, and not just your garden variety hell fire and brimstone mind you, but a day in the seventh level of Dante's inferno. As some of you know, I am a web site developer and Internet consultant. Basically I build websites for companies. I am good at it and I make a good chunk of change hosting websites as well. However, I do not own my own server. Heaven forbid it, I wouldn't want to. I lease servers in their individual climate controlled, security monitored, and hyper connected buildings in places far away for a few hundo a month. So one of my big servers is located outside Philadelphia Pennsylvania. I have a nice deal with an honest hosting company, a good man owns it and he is far from "rinky dink" with `10,000 servers under his control, overall a good deal on fairly high end service. I pay a monthly lease payment and then resell the space and bandwidth to my clients.
The last two weeks I have busted my butt to build, produce and deliver a high profile set of three websites for a national brand of insurance. I was contracted through a marketing firm that has in turn contracted with the big brand. They are fairly clueless, regarding the process of building a website and I was more than happy to host the site for them. Hosting, you see, is easy money most of the time and easy to sell as an addition to the website construction. The machines keep the websites visible to the public and I sit around and collect fees for watching paint dry. Once in awhile the paint doesn't dry correctly and all hell breaks loose. This is what happened overnight and today (Tues Jan 30).
Last night I was peacefully dozing in my bed, almost asleep as I worked till midnight and had only recently gave up for the day. The phone started ringing, at first I didn't want to acknowledge the fact someone was trying to reach me, and I let it ring. Then in rang again, this incessant noise was demanding my attention. I got up and groggily stumbled to the phone in the dark, and missed the call actually. Upon reviewing the call history, I saw that it was one of my programmers and business partners. I immediately called back and asked what was so important she had to call at 12:20 AM.
I learn at this point that the server had to be moved and I had to make a decision on how to handle the situation right that moment. Folks, I was sleepy, grumpy and in no way qualified to make any big decisions at that moment. My first reaction was "fuck that" lets deal with it in the morning after a good nights sleep and some coffee. As she droned on about the problem it became apparent that something had to be done immediately, as the large project I had bled for the last two weeks was due to be proof edited by the large insurance company at 9am that morning. It was indeed crunch time. Within a few moments I got a call from the owner of the server farm and was informed of the situation. The situation was beyond crazy, it was ludicrous in the extreme, my poor brain could hardly wrap itself around it. What happened, to make my life a living hell for the next 20 hours was this. A large chunk of ice fell from a passing aircraft. It fell to earth with enough velocity to hit one of the air handling units on the roof of my data center. The unit promptly crashed through the ceiling upon impact an was hanging on the rafters. Now keep in mind this is a serious building, one cannot get into the server room without thumb print recognition and all the appropriate credentials. Very secure.
The owner calls the local building inspector and the insurance adjuster to take a look. The building inspector immediately orders an evacuation of the building as it is deemed unsafe with a two ton air conditioner hanging on by some unseen force through a hole in the buildings roof. A tarp is placed over the hole and the insurance adjuster is then charged with the investigation of what actually happened. It seems a chunk of ice, roughly three pounds after entry, had fallen from a height of 35,00o feet to demolish the roof. This ice was "blue ice" from an airline lavatory. Apparently it still had turds and toilet paper ensconced within it's frozen form. In layman's terms, a giant hunk of ass water just fell from the sky and wreaked havoc on the parade. All of several hundred servers had to be moved to the new data center about 60 miles away. The server company had been in the process of moving all the servers to the new location at the time, mine was one of the unfortunate ones left in the old building.
So to make a very long story shorter, the owner personally took my physical box to the new building, put it in a rack and plugged it in. Needless to say nothing worked at first and I have been all day making phone calls, placating clients, testing connections and otherwise being miserable. There is nothing so bad as having websites down all over the country because of a server outage.
My team and I eventually got everything going again, and it became a waiting game, waiting for all the connections to solidify and propagate throughout the world wide web. I went to my local pub and ate a late dinner and many beers. The hard part is over, and I have happy customers again, but that errant bomb of turds really fucked with my day and the day of many others. I dont know if it is legal to jettison human waste over a populated area, but this load of shit did a number on us terrestrial peons this day. Should the heavens not open up and deposit anything more than rain for the rest of my life, I will be happy.
slf 01-30-07