This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault.
- The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault.
- Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault.
- The regulatory system that allowed this design is at fault.
- The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
Pinning a fault to a single entity has great cultural tradition. The number of times I witness people being just completely unable to process the idea of shared responsibility, or the difference between having an ability to affect something and having culpability to affect that thing, is staggering.
People just panic and do ridiculous things like evacuating the apartment building at 2 am in their pajamas in the rain, blocking the road. Meanwhile 9 times out of 10 it's a false alarm. We should instead email the fire department while the residents keep sleeping. /s
Good. When they're outside in their PJs standing in the rain and blocking the road like complete dullards, then: They're not in the building, and they are neither on fire nor breathing chlorine gas.
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
> It's the fault of the company that designed the automatic alarm. Not the hotel.
The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.
> But it is accurate to say that the cause of the issue was the outage
The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.
Agreed, I'm not sure how it's acceptable to say "Email is exclusive option you can have" vs "We can email you when alarm is going off but light/siren will flash and it must be visible to those using the facility"
We can assign fault, or we can ask which levers we can pull to make it better:
1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.
2. Could Microsoft have not had an outage? Hard to fix.
3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.
Generally, for sensitive safety systems like this, Light and Siren are visible to those using the facility. So if problem developed, all patrons would be going "Umm, you know your sauna has flashing red light and siren going off?" after they all left the facility.
Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.
It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.
EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.
Things do break, so all alarms need periodic testing, whether they consist of email and/or something else.
Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.
And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.
The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
Seems like if the system can send an email it should be able to, like, illuminate a red light somewhere or something, if what it’s doing can create an actual safety risk.
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
Everything can fail even down to individual capacitors on the board but that doesn’t make the failure probabilities or failure mode equivalent to something like email that is best effort only and can be actively blocked by things like spam filters that aren’t even under your control
I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.
There's no details in the article so you didn't miss anything. TLDR
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
Maybe the information is being lost in translation, but I'm trying to understand where the "chlorine gas" is being talked about. The articles I found just say "too much chlorine", which is hazardous but not the same as chlorine gas. Can anyone shed some light here?
Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.
Are humans born with innate knowledge that some things are deadly to smell? Or are we born with an instinct to smell things to figure out if it’s good or bad?
its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.
Microsoft bullies enterprises into their products and targets tech workers who are against it. This is 100% Microsoft's fault and the amount of corporate goons on YCombinator is ridiculous. I don't think this will change at all when the designated response is gaslight and deny.
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
OK you design a new out of band notification system that somehow has better reliability than email but is also used frequently enough that people will notice when it stops working.
This one’s really simple. The current system is a bad one because if the email password changes or internet connectivity gets interrupted or the system sending the email dies (eg: power supply failure), the system fails because on any lack of signal it’s assumed that everything’s good. That’s a terrible design. Here’s a better one:
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.
Apparently the alarm system sent its alarms via email.
"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."
We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.
No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
Email as the sole notification source for a safety critical situation is criminal, in the literal “the responsible engineer or coder masquerading as an engineer should be in jail” sense of the term.
We all know that services can fail, and consequently will fail at some moment. It is therefore primarily the responsibility of the designers of this system.
> This was because the automatic dosing system had pumped too much chlorine into the pool.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
yeah, these companies need to be told to stop relying on cloud services for everything.
its wild to me how many critical things that absolutely do not need the internet are relying on the internet.
much blame to go around here:
- whoever designed the stupid alert system.
- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”
- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”
- accountant types and their “we can save pennie’s if we cut all of these corners.” and the stupid corners they choose to cut.
- etc…
on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.
People ignored it.
A message to the fire department would have been the better idea
The hotel picked a monitoring solution which relies exclusively on email, or configured it like that. If anything they are absolutely both to blame but for different things. The company for designing a safety related system like this, the hotel for selecting an inadequate solution/configuration thus allowing their guests to be harmed.
> But it is accurate to say that the cause of the issue was the outage
The cause of the issue is the more immediate one - the configuration of the monitoring system. This avoids going up the causality chain to far - shitty coffee caused engineer to take down email system, and such.
Using the unqualified "cause" is misleading click bait and in some places it could get you sued
1. Can there not just be a light / siren right next to the gas sensors? Easy to fix, but it requires hardware changes on-site.
2. Could Microsoft have not had an outage? Hard to fix.
3. Could they use some better notifications systems like idk self-hosted ntfy.sh with a warning if the server goes down? Yes but nobody wants to pay for that.
And you have to check those lights and sirens that they still work.
Everybody knew email was down.
The real problem is nobody thought of the consequences regarding the alarm system
Yes, they still have to work but first off, both of those are extremely reliable systems, alarm circuits have extremely low risk of failure. Second, testing them takes 5 minutes a week/month, press the button, look for flashing light and listen to wailing noise.
It's inexcusable to think that email/slack or whatever is acceptable as anything other than notification system so people not in facility can be notified that alarms are going off.
EDIT: You are defending this setup and I'm not sure why. This seems email only setup seem indefensible.
Testing requires staff and you can be sure those parks are understaffed.
Best thing is to avoid them, next best thing is a cheap solution with low maintenance by the staff to notify as much people as possible.
Email wins.
Not a good solution but the best you will get for the price they are willing to pay
Whether it breaks because a relay has corroded due to years of service in a chlorine-enriched environment, or because O365 is down, or because O365 tightened up on DMARC requirements: It needs tested.
And it's not a binary choice. This isn't the World Cup, the Super Bowl, or Highlander. There can be more than one.
The alarm system can wear a belt, and suspenders. It can send emails, and it can flash a bright light, and it can also sound an audible alarm.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?
Title: 5 nines of uptime
A grand piano is suspended from the ceiling with an electromagnet.
The electromagnet works as long as there is power.
No special battery backups, anything.
Guests are invited to stand underneath it, after signing a waiver.
That’s it. That’s the art.
I know the S in IoT means safety/security, but there must be things that can be done for urgent notification.
You know what they all have? An email account
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
In short, all parties suck here (MS and hotel)
Go invent a transport layer that will never fail, and make yourself rich.
Because right now, there is no such thing.
Email is not.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
I’m so sorry.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
Like, what leads a teenager to do such things?
https://how.complexsystems.fail/
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
- External system polls the monitor and records levels.
- Local system reaches out to external system and fetches status.
- Someone has the responsibility to do rounds and record the status indicated by the local system.
External system down or indicates unacceptable levels, monitor unreachable, etc.? Local system shows a failure. Investigate. Local system down? Investigate.
You can even add redundancy by having a manager double check that the responsible employee is checking off the status plus the manager can do a daily sweep as a fallback. Throw something like PagerDuty into the mix and have multi-channel notifications on any failure that escalate over time.
This seems like a solved problem, e.g. for smoke detectors.
The alarms and lights are easy to test, too. Make pressing the self-test button part of the nightly or weekly rounds.
"As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool."
We're in an era where alarms go out via email, major policy changes are announced on Truth Social, and war is declared on X/Twitter.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.