When stuff like this happens, there's a lot of compassion within the DNS, DDoS and L7 defence community. We all know, it could've been any of us and their bad day could easily be our bad day tomorrow.
We've internally briefed everyone, "Don't be an ambulance chaser". Instead the tone is to help our customers, and their customers, and others, to get back online and stable.
Yesterday was interesting though, in revealing the massively overlapping dependencies against a few DNS providers. For example, chunks of AWS failed because instead of using Route 53 it appears that Amazon were using Dyn. So customers on Cloudflare failed as they had CNAMEd their domain to their AWS address, which failed because Dyn was down.
LFGSS was cool yesterday because I'm fairly obsessive about self-hosting, minimising third party dependencies, etc. If we go down, it can only be one thing.
But yesterday, so many sites went down that were not even a Dyn customer. i.e. Reddit has been cited as having gone down (they did, but they don't use Dyn). It comes down to JavaScript from other places, advertising networks, backends that map to AWS, etc.
It's all crazy stuff. Everyone who works in the space knows it's all fragile, but it's also simple and the simplicity makes it easy to fight this stuff... until the day it doesn't.
When stuff like this happens, there's a lot of compassion within the DNS, DDoS and L7 defence community. We all know, it could've been any of us and their bad day could easily be our bad day tomorrow.
We've internally briefed everyone, "Don't be an ambulance chaser". Instead the tone is to help our customers, and their customers, and others, to get back online and stable.
Yesterday was interesting though, in revealing the massively overlapping dependencies against a few DNS providers. For example, chunks of AWS failed because instead of using Route 53 it appears that Amazon were using Dyn. So customers on Cloudflare failed as they had CNAMEd their domain to their AWS address, which failed because Dyn was down.
LFGSS was cool yesterday because I'm fairly obsessive about self-hosting, minimising third party dependencies, etc. If we go down, it can only be one thing.
But yesterday, so many sites went down that were not even a Dyn customer. i.e. Reddit has been cited as having gone down (they did, but they don't use Dyn). It comes down to JavaScript from other places, advertising networks, backends that map to AWS, etc.
It's all crazy stuff. Everyone who works in the space knows it's all fragile, but it's also simple and the simplicity makes it easy to fight this stuff... until the day it doesn't.