In the news

And, as always, The Register provides a more thorough article on it;

http://www.theregister.co.uk/2016/10/21/dns_devastation_as_dyn_dies_under_denialofservice_attack/

The short version that I'm getting from Twitter is that someone enslaved a massive botnet of cheap Chinese made web enabled baby monitors and fridges to attack Twitter and Spotify.

If that's true then I'm going to spend the remaining time available on the internet to google how to make a primitive stone axe and milk feral cats.

Oh and while I can't work out a correlative link between the two, global maps of outages seem to show high density around early voting and closing registration states in the US. Currently pegged as coincidental but relevant factors suggest these two being related can't be ruled out.

Nobody told me I was in a sleeper cell!

The first rule of sleeper cells....

It's all speculation. Dyn haven't shared anything.

Though it's likely a botnet, and likely Mirai. The worryingg thing is the size. If it is Mirai then it is a larger botnet than anyone thought.

Defence against a DoS attack is simple...

1. Have a bigger pipe than the attacker
   
2. Be able to keep up with the incoming traffic
   
3. Be able to throw the bad traffic away
   
   

But for Dyn to go down means one of those three things failed. I hope it's the latter two things, computation stuff rather than size of pipe, as Cloudflare and others should stand a chance. But if it's volume of traffic, I'm not sure there is any company that could stands in the way of that botnet.

Was the flood of traffic direct from the botnet to Dyn's systems, or a reflection attack using (say) Verisign/Neustar/etc servers to amplify?

In the first wave the chatter is that there were a few attacks, attempts at DNS amplification against Twitter (which all look like they failed), and attacks against Twitter's peering points, whilst also hitting Dyn directly.

The first wave of attacks yesterday were probing, to find out where the defences were and what were the weakest links.

The second wave of attacks were huge and mostly hit Dyn directly.

The attacks were global, and @The_Seldom_Killer there is no correlation between US voting distribution. The correlation is the number of nodes in the botnet and where they are located, which is a wealth and population map of the world, suggesting a lot of devices being used.

When people say "cheap Chinese devices", this isn't strictly true. The Mirai source code has bits to do with Western Digital NAS devices. Amusingly it's got a bit to do with Cloudflare, it's flattering we're large enough to be singled out, and also helped us spot a weak point (a weak point we hadn't identified internally, but has now been shut down).

If it is Mirai, it means the size of the botnet is really huge.

We're semi-comfortable (I wouldn't say totally comfortable) that we'd be good with an attack that made us do computation (handling such traffic volume, and throwing away the bad) as we invested very heavily in the hardware and skills to fight that. But volume of traffic is worrying, the typical defence (which Dyn would have tried) is to rely on anycast (traffic goes to nearest server) and to drop a PoP (datacentre) off the map so that the attack data suddenly gets spread out to a lot of PoPs equal distance from the epicentre. But... if the volume is such that it exceeds your total network capacity, and if the attack can deploy it all at once and sustain it, then it will knock those other PoPs off the map too and carry on doing so like dominos until it's taken the whole network offline.

This is why, if the volume is truly huge (above 10Tbps) then few companies will be able to stand in the way of that botnet, and I'm not even sure Cloudflare could, or even some of the big players like Google (depends on their net and computation topology).

I interviewed with Dyn, they were nice people- didn't take the job but seriously considered it.

When stuff like this happens, there's a lot of compassion within the DNS, DDoS and L7 defence community. We all know, it could've been any of us and their bad day could easily be our bad day tomorrow.

We've internally briefed everyone, "Don't be an ambulance chaser". Instead the tone is to help our customers, and their customers, and others, to get back online and stable.

Yesterday was interesting though, in revealing the massively overlapping dependencies against a few DNS providers. For example, chunks of AWS failed because instead of using Route 53 it appears that Amazon were using Dyn. So customers on Cloudflare failed as they had CNAMEd their domain to their AWS address, which failed because Dyn was down.

LFGSS was cool yesterday because I'm fairly obsessive about self-hosting, minimising third party dependencies, etc. If we go down, it can only be one thing.

But yesterday, so many sites went down that were not even a Dyn customer. i.e. Reddit has been cited as having gone down (they did, but they don't use Dyn). It comes down to JavaScript from other places, advertising networks, backends that map to AWS, etc.

It's all crazy stuff. Everyone who works in the space knows it's all fragile, but it's also simple and the simplicity makes it easy to fight this stuff... until the day it doesn't.

If anyone wants to do their bit: Change your username and passwords on ALL of your devices.

The way Mirai replicates is via default passwords: https://github.com/0x27/linux.mirai/blob/6d5a3e2760852444de9d39b082b06cb7176cd2c1/mirai/bot/scanner.c#L138

You may be part of the botnet soon (if not already) if you haven't changed your usernames and passwords.

Apparently the attack was 1.2Tbps.

http://hosted.ap.org/dynamic/stories/D/DISRUPTIVE_CYBERATTACK?SITE=AP&SECTION=HOME&TEMPLATE=DEFAULT&CTIME=2016-10-21-17-26-28

Cloudflare and others should be good with that.

Change them from defaults or change them from what they were yesterday?(basically everything else you said may aswell be in foreign so just trying to clarify that bit that I might understand)

Just change them from defaults.

If you already have, you are good.

There's going to be so many devices that will stay on defaults though. Things like DVRs and webcams etc.

Also, how will all those infected devices get cleaned?

how will all those infected devices get cleaned?

Also, how will all those infected devices get cleaned?

They won't.

We know where those devices are (L7 attacks use TCP so IP addresses are revealed), we know how to access them (Mirai source code has all the logins and did not close the door behind itself), but it's against the law to do so.

The only people who could stop this are major ISPs to block the traffic from such devices. Or the manufacturers, but there is no economic incentive to do so and when it does come to Chinese devices... there's no legal incentive that could be made either.

So there's still loads of zombie devices out there that can keep doing this?

Put a scanner on the router, block traffic from any device of it fails scans?

That could be your home. How do you know that you're not pay off the botnet?

This is the problem, it's so prevalent that to block those IP addresses is to block so many users that the effect is a negative economic DDoS on online companies and those who run their businesses online.

No one can legally remotely deal with this, not can they be blocked. Except that this specify type of traffic (telnet and SSH) could be blocked by ISPs to prevent further compromise and to disrupt command and control. Anyone who needs that access will know how to change the port number and type around default blocks. A lot of spam was stopped when port 25 was closed, and this is the same thing.

I watched the first season of Mr Robot last night so now everything....and....

Yep that may be a good and simple way to deal with it: Port blocking.

Just was wondering if dealing with it at the isp router end may work. Instead of just accepting any iot traffic it blocks it if there is a default password etc.

Or basic analysis to identify surges in traffic which can indicate a ddos and block.

So rather than block the router, the router is pickier.

Right now all traffic going out of your home is trusted by default..

I do not understand any of this.
Expert
Vote Leave

I think it's great that a few determined people can duff up bug business for a bit.

Sounds like some top exec did a presentation boasting about how awesome his company is at the internet and someone decided to prove him wrong.

It's not always a case of the underdog against big business unfortunately.

These guys also go after journalist (Brian Krebs), steal creditcard data, and all this can lead to security measures by government that mess with our privacy...

is Donald Trump on course to be U.S Jimmy Sav?