• toynbee@lemmy.world
    link
    fedilink
    arrow-up
    26
    ·
    9 hours ago

    I once worked in an IT environment where several very expensive servers and their accoutrements were prepped for installation, then put in an empty, locked office for a long holiday weekend, after which the servers were to be installed in our data center.

    After the servers were put in temporary storage and my team had gone home for the day, there were some construction guys doing some kind of construction work in the area directly above the empty office. Whatever they were doing, they in some way unsealed a radiator unit or maybe cut it away and didn’t plug the tubing (I know a lot less about radiators and plumbing than I do about IT). They, presumably, didn’t realize this and went home for their holiday weekend as well.

    After the long break was over, my boss - whose office was adjacent to the storage office - came in and observed that it was unusual for his carpeting to be so squishy. Turns out, as I’m sure you guessed, water from the compromised radiators had dripped onto the servers throughout the entire vacation, then leaked into the carpet and spread out to his office through the shared door between them.

    As soon as my team got in, we were all immediately removed from our duties for the day and tasked with helping him to move the servers over some of the forced air sections of the data center to work on blowing the water out of them. These efforts were bolstered by some of the biggest mobile fans I’d ever seen - I have no idea where we even got them.

    The servers were there for at least a week, but I think actually weeks. In the end, I think only one of the servers failed to boot and the rest stayed running for the rest of my tenure there. Still, I’m sure it was scary for a while for my boss - the one who authorized the original payment and who decided where to store the servers.