Incident: DDoS Attack on 2024-05-24

Attached are two images, a normal day (looks like a hill) and yesterday (looks like needles).

On a normal day we do about 500K requests... and yesterday we received multiple floods of 150-190K requests.

More frustrating, the 150K requests would arrive in about 1-3 seconds, a well-tuned server can typically only do about 20K requests per second... and so LFGSS and all of the other sites fell over, they were totally overloaded and the sites went offline.

Each time I stood the site up, within a minute another flood would arrive... knocking it offline again.

It was... frustrating. More so as I'm on holiday and that's always the way of things... attacks, vulnerabilities, incidents... they tend to happen when one can least spend time on them, and so it was yesterday too.

I finally sat down this morning and was able to really go through all of the log files to find deeper commonalities between the requests so that I could get something more effective to block, and it's 7:30 am in Crete and I finally found a HTTP header spelling mistake within the attack traffic that I can use.

It should be over now, but I'll check periodically.

2 Attachments

oh, and 2.5K IP addresses were involved, blocking IPs became ineffectual very quickly.

and no I do not know who did it, nor why. I suspect it's just a booter and that some disgruntled person on one of the forums was so offended by something someone else said that they just did this... the attack wasn't specifically to LFGSS, it hit 3 sites at once and so seemed targeted more at the platform.

it's typically not constructive to try and be a detective and figure out who and why, it's far more constructive to improve the defenses and prevent it happening again... so that's what I've been doing instead.

also... I very proudly had 99.99% uptime for months on end.. this month is already at only 98.1% uptime... kinda sucks.

Great work! Sorry your holiday was interrupted....
I actually had to do some work yesterday for a change .

Sorry your holiday got disturbed. With no forum to annoy people on I had to pick up the phone and bother @hippy that way instead, it was strangely pleasant.

I dont know what you can learn from it but i’m sure we can all say we are grateful for your hard work.

The early morning lufguss junkies checking in with the man.

I learned python really doesn't like receiving so many requests so quickly 😂

Nginx and the microcosm backend (the API, which is written in Go), these were both fine.

But python/Django just fell over so easily and it was really hard to get that stable again.

Eventually I gave up trying to do that, I was scaling them horizontally and still they fell over. Hence focusing instead on how to block the flood

I wasted way too much time yesterday trying to build a capacity greater than the flood... At cloudflare I saw that it's far easier to deal with these things if you can just absorb them... But that starts to require time and money...

Yesterday I spent almost as much money as a whole month's hosting on tens of additional servers to horizontally scale the python layer... But everytime I opened the door to traffic again, it would still fall over.

So yes, the lfgss kitty is now depleted, and it took even longer to recover the site.

I've also massively improved the logging, and all of the firewall, such that I can see more, and that an attack must come through the front door and cannot evade being fully seen.

Fuck me a day without this place seems way more strange than I could have thought it would, donation incoming. Enjoy the rest of your holiday @Velocio.

I think I've learned that I am addicted to this site.

Well done on getting it back up and running and hope you enjoy the rest of your holiday.

I actually had to do some work yesterday for a change

lol, me too. It was awful.

There was a moment later in the evening where I was like “what if it never came back?”. And I was afraid.

Sorry about your holiday, and thanks for your efforts. Sending a little money to the kitty this morning.

life without luffguss doesn't bear thinking about!

thanks for all the hard work @Velocio 🙏

So glad it's back. Thanks for the efforts.

@Velocio thanks for the hard work and hope you enjoy the rest of the holiday

Aren't we hiding behind CloudFlare? Aren't they supposed to prevent things like this?

Kudos for single-handedly beating them off, as it were

So yes, the lfgss kitty is now depleted

You heard the man. Get the one-off donations in. (Or set up monthly DD if you haven’t already)

Interesting to learn about the ‘fuck it - just scale to absorb it’ thing.

Sorry you burned a bit of your holiday and the kitty dealing with it.

We are behind Cloudflare, but to not reveal too much about the way their system works... it didn't work to protect us.

The difficulty is in the question "Is this a DDoS? Or is it just regular users?", and the answer is subjective depending on what normal looks like.

For a given IP address or website:
1rps = not a DDoS to anyone
10rps = not a DDoS to most, but probably is to a single server WordPress without caching
100rps = a DDoS to some, but probably not to sites designed for decent traffic
1K rps = a DDoS to most, but probably not to sites designed for decent traffic
10K rps = a DDoS to virtually everyone, but not to seriously scalable sites
100K rps = a DDoS to all but a handful of top sites by tech companies

Cloudflare detect and kick in about the 10k rps per website, per PoP (Cloudflare has about 300 points of presence, i.e. data centers, and as ANYCAST networking is used, a DoS attack usually appears within a single PoP and would be detected)... but this attack hit several websites, and was globally distributed... meaning no single PoP detected > 10K rps to a single site.

Even then... by the time detection would've happened... the Python backend was already dead, the spikes were too quick.

So yes we have Cloudflare in front of http://www.lfgss.com but it only would help if the attack were more centralised, or larger, and we'd cached a fraction more.

microcosm.app does not use Cloudflare, and actually wasn't really at risk yesterday... the API was up throughout the attack, it was only the frontend that died.

This all does suggest I really should get back to work on porting the Python to Go, as none of this would've been an issue if I'd had that done already.

the key to an effective DDoS on Cloudflare:

1. actually make it a DDoS
   
2. stay below the 10k rps threshold per PoP
   
3. stay below a global 100k rps threshold
   
4. do it very very fast, so the servers hurt
   
   

which is roughly what all DDoS looks like nowadays, as Cloudflare made everyone change how they do attacks

Thank you for all the work @Velocio - hope you can enjoy the rest of your holiday! ☀️

Also, thank you, and donated

Thanks and small donation from me too. Enjoy the rest of your holiday.