Incident: DDoS Attack on 2024-05-24

Curious about how you've deployed Python/Django and if you have any recommendations given what you've just been through? Considering using it for a fairly high traffic site but haven't quite reached the point of doing a proof of concept to test how well our proposed setup will handle various multiples of traffic. Will be a donation on the way when I get home later.

I was going to ask about Cloudflare, interesting stuff.

So if threat actors have learnt how to circumvent Cloudflare protections then what are Cloudflare doing to improve their system?

Well done sorting it out yourself.

Cloudflare are still great, they've rendered most cheap attacks like TCP ACK floods and UDP reflections useless... So attacks moved to the application level which are more expensive and difficult to execute, which means few really have impact and there's a natural deterrent involved.

Attacks like these are very very rare, so cloudflare did achieve a lot of good stuff changing things.

Some attacks will always exist though, and we've not seen any for a long long time

Thanks for sorting (on holiday no less!)
Just fed the kitty.

Feels like there’s been a lot of attacks atm. This… the bing / DDG outage… my local bus information screens.

For a while there I thought the peado chat had killed the forum

Curious about how you've deployed Python/Django and if you have any recommendations given what you've just been through?

Nothing we have is comparable to anything anyone else will have.

We're still on python 2.7, a very old Django, a small server (which yesterday I cloned to be a fleet of about 100 servers).

It's so old I cannot even deploy it 🤷 hence the cloning.

The problem here is probably the memory and limits of that small server, it expects to serve many small requests quickly, but some of the attack requests has 1Mb payloads, so it seemed to be a resource exhaustion.

All the natural guidance kicks in: use limits liberally, so that you break everything down to the smallest possible thing, as then you can control things.

We effectively use Django statelessly... No database! It just calls the API server, and the state only really exists in the database. If recommend always running like this even though this didn't help yesterday.

Thankyou for your service and donated

Also donated, cheers all

Small extra donation sent. Enjoy your hols!

I don’t know what any of this means but having deleted all social media six months ago, I did miss the forum when it was gone.

1 Attachment

which is roughly what all DDoS looks like nowadays, as Cloudflare made everyone change how they do attacks

That's super interesting. I'm not involved in that area of tech at all. Really interesting to see how it's evolved.

LFGSS is one of my favourite places. Thanks for all you do for us Velocio. Hope you can enjoy the rest of your holiday.

Great job, thanks for all your work.

Purely out of curiosity, rather than identifying an actual need (because, except for an occasional large sale, nothing here is time sensitive): if LFGSS were taken down for multiple days and Velocio wasn’t around or able to stop it, is there a backup location identified where evacuees can go? A Reddit forum for example? Additionally, is there an external comms channel that parishioners can refer to for authoritative updates?

I posted this in partial jest, but realised that a party halfway thru a significant sale (e.g., someone has paid and shipping or collection is pending) might be quite inconvenienced if they didn’t exchange contact details beyond this website. Lesson and mitigation then is to always exchange contact details on sales where you’d be bothered if the website became unavailable.

I comunicati to a few people via Signal groups, and also email.

But a fallback hasn't been put in place because this is the first time in a very very long time that there's been any real outage.

If I suspected it was going to be down for longer than yesterday afternoon / evening then I would've put a static page in place redirecting people to an alternative

great work as ever boss man. get back to that holidaying!

Great work as always @Velocio. Forum donation incoming.

(Also, randomly came across these "beauts" while trying to find out if the site was down for everyone or just me: https://www.redbubble.com/i/t-shirt/LFGSS-No-Breaks-Love-Bikes-Singlespeed-Fixed-Gear-by-K0tK0tu/116220091.NL9AC)

Amazing to fix this on your holibobs with a sea view. Strange to think it was probably done out of spite. I was half expecting a ransom demand.

Bearing in mind the number of members of the site how much would each person have to donate per month to keep the lights on?

Not much, but I haven't crunched the numbers recently and nor have I been bothered when we dipped into my savings over winter. It ebbs and flows, but I've turned off all affiliate and advert stuff, so we're 100% reliant on donations. Costs vary, about £500 per month on average, and then a couple of annual costs that add another £500.

Most people donate in the £3-8 range, some in the £10-20 range, and a few do random donations of £50-100.

The real risk is typically just the slow expiration of card details, or changing banks, or someone not having money that month and the payment failing and then it stops the subscription... It's the constant small erosion of income that eventually bites, as it's not like I push for money or do funding drives.

So as long as a pot is rattled occasionally this are fine and it's not really an issue.

Obviously sucks that yesterday we'd already paid this month's bills and then I basically spent most of that figure again in temporary servers... Running with 1 month spare cash in the bank was quite good considering 6 months ago we were in the red - though I didn't say anything because it's seasonal and just recovers usually.

Thanks for the massive amount of effort + time keeping the forums going.

I've been waiting to donate until more stuff sold but no need to put it off... Donating now

Are we having another attack? I noticed yesterday or the day before that some avatars were broken. I put it down to the shitty holiday WiFi.

Just noticed some avatars are broken again.

Yes but the more normal kind which is just background noise on the internet. Ignorable and doesn't affect us.

I'm not sure why some avatars are returning a 404 when all other attachments and images are working.

Some onion forums have a ddos protection puzzle, could you have something similar that is normally switched off but it can be turned on when attacked?
Would mean we would have to solve the puzzle for each session but would immediately render the ddos useless.

Interesting, thanks! Have thrown a few ££ in the pot.

Ugh, sorry it interrupted the holiday
Donated a bit on top of the monthly