Saturday, February 28, 2009

GBL@&%*RGHLE

I have a ton of stuff I want to write, that's pounding from the inside of my head out; but I feel too crappy to write it.

I worked Wednesday through Friday (from home of course), after taking Monday and Tuesday off for illness. I wish I'd taken the whole week; not because I was too sick to work, just because it was a stupidly irritating week.

I spent the whole day today getting punted around trying to figure out final requirements and numbers for an absolutely business critical project, to add critical security software updates to EVERY SINGLE ONE of the 24,000 servers in our organization (covering all 15 operating systems and revisions, and the ridiculous number - hundreds, spanning 8 years - of hardware platforms we support), many of which will need to be installed manually, because either there is no automated or scripted process, or because the systems are not connected to any kind of automated management server.

Of course being business critical, and being so big, no-one was willing to commit to any numbers in any meaningful way.

We had what we thought were final numbers by yesterday (thursday), but ended up finding 5000 or so servers that hadn't been accounted for, outside of datacenters (in stores, remote offices, etc...) that would have to be covered by local support, or by vendor contract (local support only covers windows and linux, and we have to bring in Solaris, AIX, and HP-UX support on contract).

Even better, I think 2500 of the ones on my list are actually either already decommissioned, or they SHOULD have been (a lot of stuff just gets left turned on and plugged into a wire, because someone forgot to turn it off, and didn't document that it should have been); representing half of the unaccounted for boxes. Unfortunately, no-one could get me a straight answer, as to who could authoritatively state these boxes could be struck off, so I had to include them.

Note, I didn't say no-one could give me an authoritative answers; I said no-one could even tell me who COULD give me an authoritative answer; so next week I'm going to have to escalate that one issue to the enterprise CIO, who will have to devolve it down to the CIO covering whatever division supports those boxrs... but at the moment we can't even figure out which division that is.

So, I did what I always do, and make the least stupid guess possible, and put in all the caveats and clauses I could think of.

All told, it'll probably end up costing us about $5 million in labor, and we'll end up having to completely rebuild from the ground up about 2000 of those boxes when they blow up horribly. Security software is like that: Once you've got the process and package down (a chore in and of itself), either it works immediately; or it fails completely, killing everything within sight.

Ahhh the glamorous world of large enterprise (technically, Fortune 20 at this point; though officially fortune 50) information technology and services.

Haloscan died for a full day, finally coming back early this morning, making things even more interesting.

And of course, recovery from valley fever proceeds apace. Actually, today was my first day in a week without any kind of elevated temperature, which was good; and my breathing is probably 75%back to normal. The sinus, coughing, sore throat, and myalgia continue; though nowhere as bad as the first few days.... and unfortunately the rash did come back (it's the biggest visible difference between valley fever and the flu).

Now it's 4am the next day, I've been up since 8am yesterday, and I can't bloody sleep to save my life (mostly because of aforesaid illness; but also my biorythyms are completely thrown off these last few weeks, from the illness and other things). I've been alternating between reading, watching late night TV, and web surfing (I'm writing this right now because it's popped over into all informaercial land).