cross-posted from: https://feddit.online/c/[email protected]/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
Subject: Fire.
Dear Sir/Madam,
I am writing to inform you of a fire that has broken out on the premises of 123 Cavendon Road.Fire! Fire! Help me!
123 Cavendon Road. Looking forward to hearing from you.
Yours truly,
Maurice MossA fire? At a sea parks?!
I know he tried calling 0118 999 881 999 119 725 3 first, but had he not ended up with another country on the line it would have been a much quicker response.

GTFO of the cloud, you idiot humans

Why the fuck would you open the pool without getting a mail saying it is safe?
The email is that there is a dangerous condition, not that it is safe.
I’m thinking maybe that should be the other way around.
Germans gassing people
2026
did not expect that on my bingo card.
Those who do not study history are doomed to repeat it.
What the fuck?
This isn’t MS fault, what happened to Normal Alarming System
Industrial automaton engineer here, I’ve worked on a bunch of safety critical systems and audited systems after incidents.
Best practice in situations were you are sending a remote start command, is for the stop command to be local to the pump controller.
I audited a system where the main control system sheet a start command to the dosing shed (which stored the concentrated chemicals), unfortunately the stop command was over the same link that got interrupted. The start was received but the stop was never was. Radio is not a reliable link.
For chemical dosing, I’d expect at least two types of command signals, such as concentration and Max allowed time; you could also measure the weight loss of the concentrate as a third check. Loss of communications to the main controller isn’t necessity an instant stop, but I would expect a prolonged loss to trigger a stop. As long as the local controller has stop control, dosing can usually continue.
The technical standards for Email would allow for 3 days of delivery failure before finally giving up. That’s the amount of leeway email has (in theory - practically we assume that everything longer than 3 minutes means something has gone wrong). Sending alerts by email is fine for non-critical applications; if one of my docker containers crash, an email is fine because the amount of damage is nil.
This is not a Microsoft problem. This is a negligence problem on Phantasialands side, using inappropiate tools for a system that can hurt people on failure, and should be followed up by lawsuits.
From: [email protected]
RE: Carbon Dioxide leak detected
Message: To whom it may concern, an elevated carbon dioxide level was measured in the walk-in fridge in the bar kitchen. Probably one of the keg lines is leaking pressurised gas. Gas PPM is classified as alert level 5 - Immediately harmful to life. Please evacuate staff from the area and ventilate the fridge thoroughly before attempting entry with ELSRD hood donned to shut down gas flow.
Folder: Spam
Yeah
“Why do all our alerts go to spam”
Oh, well everyone just ignores them, then Jeff who isn’t even on the IT team but was on the mailing list was getting them. So Jeff marked them as spam instead of asking to be removed from the mail list, and now our corporate email provider just black holes them.
Yeaaah not MS fault here, no sane person would rely solely on email for something like this
What else should they rely on?
Chlorine level monitors that staff check before opening the pool? That doesn’t require internet and would only be a simple embedded device. You might be able to have an electricity-free option too, not a chemistry expert but I imagine there are tests for chlorine levels that don’t require electricity?
There are a ton of on-call systems available that have hard contractual uptime reliability requirements.
Every time I’ve been on a team with on call we had a required app that pushed critical notifications to your phone, plus SNS, plus email, plus slack, plus an actual phone call system.
And I’m not even an SRE or anything like that.
We used to have these things called blinking lights and alarm sirens that required only a few cheap components and no internet at all.
Very rigorous maritime engineering standards.
Good engineering practices for starters.
Auto-shutoff on the pumps side, like it’s standard for even sludge pumps.
Phones? Pagers? Klaxons?
Idk, any self-hosted system, there are so many stuff one can do…even having displays or alarms…
MS wants businesses to sign up for their cloud services, promises the world, and says don’t worry. Businesses rely on this nd when it goes down it’s not MS’s fault even tho MS is the service that went down? Come on.
It is their fault for not having a fallback system when lives are on the line. And the whole strategy of “outsource something so we can blame that when it stops working” is transparent, lazy, and negligent (criminally in this case IMO). Downtime has always been a thing and it’s not like MS has a reputation for reliability (I would have said the opposite based on using their software over the last few years).
If you haven’t realized by now that marketers will say whatever they think will get them the sale, that doesn’t say great things about your intelligence, even if there’s a lot of people falling for that BS.
It’s not MS fault if you don’t also have a fallback system or any system that can work even offline/dipendently on your network. Regardless of microsoft being down or not, emails for such stuff is a VERY bad idea and should be changed with a better system that cannot be accidentally ignored.
Gross negligence is a thing. What’s the point of moving away from MS if they can just point the finger at them and yell “daddy told me I could?”.
An actual automation system. Ideally, a self-hosted one.
I wouldn’t advise self-hosting unless you can guarantee the resources to do it right, and with human lives on the line plus negligent management, that’s a lot to ask.
And for modern SMS alerts you probably need someone else’s platform anyways, so you’re halfway to paying for this.
I mean, the pump is self-hosted. The water is self-hosted. Smoke detectors are self-hosted.
Making an exceedingly simple system that sounds an alarm and/or flashy light when a threshold is crossed is hardly new. It doesn’t take that much and if you’re really worried, set up a duplicate system to run in parallel.
Using email for something important is dumb. This is why 911 doesn’t have an email address.
This guy gets it.
I wouldn’t advise self-hosting unless you can guarantee the resources to do it right,
Bit of a pickle. If you can’t trust the biggest name in the industry to get it right…
Parallel: Pagerduty, Slack, NTFY.
you can self host NTFY then you have 1/3 under your express control.
and it would alert everyone how?
Imagine being so slop brained that you forget that physical alarms and local system alerts are a thing.
This is going to get worse as more and more people turn their brains to mush with chatgpt
A buzzer, a bell, a klaxon… Audio saying “caution: chlorine levels too high”
Oh, I thought you were joking
Notifications on the automation portal, a push notification if you have an app, an sms message, an email, a physical alarm or all of the above. Hell, you could even have a physical alarm at the actual pool to alert customers in case someone from maintenance missed the alarm. I don’t know how this is an actual question for you xd
fuck microslop but that one is 100% on Phantasialand
Damn, Microsoft hurting people they didn’t even mean to (in addition to the ones they mean to).
Wow. Why wire up a blinking lite and speaker in the managers office to the system when you can just rely on fucking email. Pretty sure whoever made it possible to use email intended it to be a backup solution in case no one in charge was on site when the alarm went off.
We were all exposed to this problem: on Friday there was an outage that crippled the US software company Microsoft, including the email program Outlook.
Phantasialand was affected as well. Because of the disruption, no emails arrived, the company told WDR on Saturday. So nobody noticed that the maintenance safety system for the sauna area had long since raised the alarm by email. The automatic dosing system had pumped too much chlorine into the pool.
we we all exposed to this problem
Does seeing memes count? Otherwise, not really. I wouldn’t even have known it was down.













