cross-posted from: https://feddit.online/c/[email protected]/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
cross-posted from: https://feddit.online/c/[email protected]/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
What else should they rely on?
Chlorine level monitors that staff check before opening the pool? That doesn’t require internet and would only be a simple embedded device. You might be able to have an electricity-free option too, not a chemistry expert but I imagine there are tests for chlorine levels that don’t require electricity?
There are a ton of on-call systems available that have hard contractual uptime reliability requirements.
Every time I’ve been on a team with on call we had a required app that pushed critical notifications to your phone, plus SNS, plus email, plus slack, plus an actual phone call system.
And I’m not even an SRE or anything like that.
We used to have these things called blinking lights and alarm sirens that required only a few cheap components and no internet at all.
Very rigorous maritime engineering standards.
Good engineering practices for starters.
Auto-shutoff on the pumps side, like it’s standard for even sludge pumps.
Phones? Pagers? Klaxons?
Idk, any self-hosted system, there are so many stuff one can do…even having displays or alarms…
MS wants businesses to sign up for their cloud services, promises the world, and says don’t worry. Businesses rely on this nd when it goes down it’s not MS’s fault even tho MS is the service that went down? Come on.
It is their fault for not having a fallback system when lives are on the line. And the whole strategy of “outsource something so we can blame that when it stops working” is transparent, lazy, and negligent (criminally in this case IMO). Downtime has always been a thing and it’s not like MS has a reputation for reliability (I would have said the opposite based on using their software over the last few years).
If you haven’t realized by now that marketers will say whatever they think will get them the sale, that doesn’t say great things about your intelligence, even if there’s a lot of people falling for that BS.
It’s not MS fault if you don’t also have a fallback system or any system that can work even offline/dipendently on your network. Regardless of microsoft being down or not, emails for such stuff is a VERY bad idea and should be changed with a better system that cannot be accidentally ignored.
Gross negligence is a thing. What’s the point of moving away from MS if they can just point the finger at them and yell “daddy told me I could?”.
An actual automation system. Ideally, a self-hosted one.
I wouldn’t advise self-hosting unless you can guarantee the resources to do it right, and with human lives on the line plus negligent management, that’s a lot to ask.
And for modern SMS alerts you probably need someone else’s platform anyways, so you’re halfway to paying for this.
I mean, the pump is self-hosted. The water is self-hosted. Smoke detectors are self-hosted.
Making an exceedingly simple system that sounds an alarm and/or flashy light when a threshold is crossed is hardly new. It doesn’t take that much and if you’re really worried, set up a duplicate system to run in parallel.
Using email for something important is dumb. This is why 911 doesn’t have an email address.
Bit of a pickle. If you can’t trust the biggest name in the industry to get it right…
Parallel: Pagerduty, Slack, NTFY.
you can self host NTFY then you have 1/3 under your express control.
and it would alert everyone how?
Imagine being so slop brained that you forget that physical alarms and local system alerts are a thing.
This is going to get worse as more and more people turn their brains to mush with chatgpt
A buzzer, a bell, a klaxon… Audio saying “caution: chlorine levels too high”
Oh, I thought you were joking
Notifications on the automation portal, a push notification if you have an app, an sms message, an email, a physical alarm or all of the above. Hell, you could even have a physical alarm at the actual pool to alert customers in case someone from maintenance missed the alarm. I don’t know how this is an actual question for you xd