Forum Slow?

Is the forum uber slow for anyone else?

r i d i c u l o u s l y
*
(i need my fix)*

me too; and i couldn't PM

There seemed to be a bit of a blip this afternoon, but it seems back to normal now.

yeh, it's working ok for me now.

Thats a relief.

It all went a bit E9 earlier, but we're back to N16.

It all went a bit E9 earlier, but we're back to N16.

N16 - that's a nice road...

Something happened a few days ago.

I think basically the database has gone past some magic point and suddenly some of the queries are slow. We've got around 3 million rows in the database, which is a lot, but then not really a huge amount. And I think that at some point the query optimiser has optimised itself for X rows, and now we're above X some queries are going slower.

The problem is peak times.

We have over 10,000 unique visits a day, and averaged out we're serving a page every second throughout the year. But it's not averaged... at peak times we're serving 5 dynamic pages a second.

What this means is that if the queries are slow at peak times, things get queued up. And because the traffic is constant, the queue never gets a chance to be cleared. And the longer the queue gets, the more connections are tied up and the slow the forum becomes.

The server can cope, but the queue just gets too long so everything goes slow. So whilst it can cope it's not ideal.

What I need to do is look at how to scale the forum further. Possibly through adding a second server, replicating the MySql, and then load balancing the front end (reduce query time plus wipe out the queueing problem).

But this is money that we don't have, and it's also time and energy which I don't have. So unless something changes we'll just chug along as we are for a while.

I've asked the guys at Jelsoft for recommendations on server optimisation given the current setup, and I'll see whether that makes a difference.

a lot of forums I use dont allow attatching images to posts, you have to host them yourself on photobucket etc... this may help to streamline the forum

That's a space issue rather than database or queuing thing.

Space we have lots of. And apache web server serves static files blindingly fast without any issues.

It's always the dynamic pages (every page) and the SQL queries that kill websites.

I turned off a few small things to help though. I bet no-one noticed. Things like the tag cloud on the advanced search page. That was taking 5 seconds to generate yesterday and is pretty much useless... so it's gone.

The problem is peak times.

We have over 10,000 unique visits a day, and averaged out we're serving a page every second throughout the year. But it's not averaged... at peak times we're serving 5 dynamic pages a second.

What this means is that if the queries are slow at peak times, things get queued up. And because the traffic is constant, the queue never gets a chance to be cleared. And the longer the queue gets, the more connections are tied up and the slow the forum becomes.

The server can cope, but the queue just gets too long so everything goes slow. So whilst it can cope it's not ideal.

Just like road traffic. :)

Hmm, I'm getting an odd thing today where the Names and Faces thread seems to want to automamamatically reload itself all the time. Is it just me?

not everyone is a rocket scientist or a brain surgeon you know

Hmm, I'm getting an odd thing today where the Names and Faces thread seems to want to automamamatically reload itself all the time. Is it just me?

I can't seem to reproduce that.

But I have been stopping and restarting the server in the last hour as I've been moving some of the configuration into memory and enabling output buffering.

^^^^ this means nothing to me (like Vienna really), I'll see whether it happens on a different server later on today.

Hmm, I'm getting an odd thing today where the Names and Faces thread seems to want to automamamatically reload itself all the time. Is it just me?

Nothing to worry about, it's just that there are more and more pretty shifty characters joining the forum.

If you noticed it was just slow, then congrats as that was very observant (only affected 20% of all requests!).

It appears that my TFL API for the boris bike data is popular. Like... holy shit sweet mother of pearl... popular.

I've stopped it from updating right now, and will instead bring online another server tonight just to do the TFL stuff.

This is the opposite of what TFL do, if they have a service that is popular (such as tube times) they take the API offline. What I'll do as clearly some people have made iPhone apps or the like which point to my API is that I'll just scale my API up to take the load.

Fun times.

I'll work to make sure it doesn't affect the server is the key point.

I might also write an assassin script. If some dumb schmuck of a programmer doesn't know how to cache data I'll kill their app by sending them some digital sludge.

What is your TFL API Ken Bike* application thingy?

*please everyone, stop calling them Boris Bikes. Ken started the programme, credit where it's due and all that.

You mean Ken was the MOL when someone proposed the idea to him and he was the public face.

What is your TFL API Ken Bike* application thingy?

*please everyone, stop calling them Boris Bikes. Ken started the programme, credit where it's due and all that.

Yeah.

I was wondering where the load was coming from, and specifically why it was affecting just one server all the time. But only one server gets the updates, and it was that one which was having trouble.

At peak it was receiving 90 requests per second. That's a bit much.

It's still doing that now, and I have architected it to take the load (it's all from memory)... so it's not the traffic that is the problem. The problem is refreshing the cache... I hadn't really put a lot of thought into that as I didn't think it was going to be this busy. I'm re-thinking it now, will drop in a new solution this evening.

http://www.lfgss.com/api/tfl.php

That's the page.

Still damn fast... it is from RAM... but now not receiving updates whilst I figure out how to safely do the updates without locking the threads on the server (which is what was happening and why it eventually would drain resources and screw with the server).

Note to self: Stop writing shit in C.

I use an Android hire app, is it possible someone has written one thats using your feed?