Server load issues/lag

issues with pictures

more at home than here and when I click on the links for thumbnails i just get the web address but no picture, so sadly could not see emilie's self portrait last night

That was last night... I was still working on things.

Let me know if it's been a problem since about 1am this morning when I finished that bit.

Oh, and steff... if you know about using nginx as a reverse proxy cache based on file patterns, and how to use that to sit in front of Apache and pass_through the html and php requests... then do let me know.

nginx I've used very little indeed. If I get time tonight/tomorrow I might try setting up some Virtualbox images and have a play though.

I am about to make a post about my yard sale-

Happy to donate monies or a % of sales etc to the server fund

A full on jumble, it seems, would take some organising and be a risk. Unless we squat some where. The Nags Head in Walthamstow has potential as a site or the Orford rd square..

nginx I've used very little indeed. If I get time tonight/tomorrow I might try setting up some Virtualbox images and have a play though.

That sounds great.

Basically we have two servers: Neptune and Uranus.

Neptune used to be the sole server, and Uranus is new.

Now that I've set up Uranus, that is doing the database and Neptune is still the Apache2 web server.

I'm using Apache2 because I also use mod_rewrite (for the pretty URLs), XCache (for PHP op-code compiling) and mod_deflate (gzip).

What I'd like is for Uranus to do the stuff that Neptune shouldn't have to care about. Basically images, css, scripts.

So I've installed nginx on Uranus and set it up as a Reverse Proxy Cache for Neptune.

The problem I'm seeing though is several-fold:
1) The gzip on Neptune results in Uranus caching the gzip output and nginx is failing to send the header for this... meaning the end client receives gibberish. I'd like to add a conditional to Apache2 to say that if the request comes from Uranus to skip mod_deflate, or just to tell nginx not to gzip a second time but to still add the header, or to only gzip the .html and .php on Neptune and let Uranus gzip everything else.
2) Uranus is caching .html and .php. It shouldn't. Ideally it wouldn't even pass_through, but would 302 redirect .html and .php requests to Neptune.
3) nginx is occasionally caching the request URL as a string for the destination rather than the destination itself. I think this is a bug, it's weird.

I'm using the nginx from the Debian repository, one version behind latest I believe. Happy to compile if need be, but I generally like to avoid the software update headache of manually updating packages if I can.

Btw, the Uranus joke never gets old.

so sadly could not see emilie's self portrait last night
I can email it to you if you learn how to spell my name correctly.

Btw, the Uranus joke never gets old.

Especially if you say: 'Uranus is caching ...'

Oh dear, it's catching.

I am still wondering what happened to cause all of the extra server load... but these stats tell a story:

Basically in the last 2 and half weeks, traffic to the site has shot up.

I'd say it was fair weather perhaps... but in prior years the site has been reasonably constant as we lot usually ride all year around. It preceded the coverage in the free sheets too, so it's not as if The London Paper can take credit.

I'm not sure what caused it, but it hasn't yet translated into more advert revenue, but I am deeply grateful for the £200+ of donations in the last 2 days.

What are the referring sites, VB? Or is just plain old Google that sends the visits?

Yeah, Google mostly:
http://chartbeat.com/dashboard/?url=londonfgss.com&k=a2bda2cfba3b183f467df38e54bfa6ad

Look at the bottom left.

From google but where physically? UK or US? Humans or crawlers?
Maybe it is the weather here but you'd think that'd cause people to be outside and not online talking shite?

hippy, your own post rate has dropped considerably since you reached your milestone on New Year's Eve. This is a concern to those of us who only come on here to fill the time between your blog entries. Perhaps the increased traffic is caused by frustrated lurkers all searching in vain for a new hippy post?

I'm looking to the T-Shirt competition to save our arses and to physically pay for a server or two rather than renting one.

Velocio, whats the annual damage and costs that you incur? That way, we have a direct target, and can try to raise that money a la Wikipeida.

The DONATE button is good, but also, what about buying the forum servers a "drink", at gatherings? Somebody trusted (BDW, Scott, Corny, Clefty, or others) could take cash off we punters, and redirect it to you (via Paypal or cash hand-over). I think that could be an option. Although I do have a Paypal account, not everyone does, or wishes it to use it for all payments.

We'll now be paying somewhere between (exchange rate dependent) £220 and £240 per month in hosting costs. And each year we have additional costs like software (£60), SSL certificates (£100 per 2 years), remote backup (£30 per month), domains (£40 per year).

In all that equates to currently costing: (230 * 12) + 60 + (100 / 2) + (30 * 12) + 40 = £3,270 per year total. To date I've only been charging back hosting, the extra £510 in other stuff I've just covered out of my pocket, and in the early days when there was zero revenue I covered the other stuff too.

Anyhow, the annual goal should be around £3,300. And the monthly goal is therefore £275.

BTW, That's the first time I've written it all down, it's the monthly hosting that gives me headaches, not the other stuff that I generally regard as just donations from me throughout the year.

sounds like a lot of money..

with 5,000 odd 'active members' though that's less than £1 donation per person per year. Obviously not that many 'active' members will donate but with perhaps 200-300 regulars donating that's still only £1 per month per person.. That's not counting ad revenue or profits from shirts/stickers etc

Not too shabby

the annual goal should be around £3,300. And the monthly goal is therefore £275.

Goal(s) set.

Now we now what to head for. What chance of monthly updates as we go along?

It would be good to know if we're ahead of schedule, or when running a bit behind.

All for one, and one for all, ............and all that good stuff.

That sounds great.

Basically we have two servers: Neptune and Uranus.

Neptune used to be the sole server, and Uranus is new.

Roger that. I have a laptop to rescue and Xubuntify for a lovely ex, but I shall have a hack at it shortly thereafter.

I'll certainly happily donate something, but I don't use PayPal so I'll get it to you some other way--a drinks night sounds good.

Velocio: Is the board a LAMP stack app? if yes, have you profiled where most of the load occurs? If you PM me I am happy to have a look at certain aspects of the app for you. Might be able to customize certain aspects.

Also, things that come to my mind when you're talking performance:

• Proper DB indexing
  
• Mysql query caching
  
• Looking into the use of ETags for static files
  
• Looking into etags for cached output that hasn't changed
  

hippy, your own post rate has dropped considerably since you reached your milestone on New Year's Eve. This is a concern to those of us who only come on here to fill the time between your blog entries. Perhaps the increased traffic is caused by frustrated lurkers all searching in vain for a new hippy post?

Work has picked up and my interest in this place has waned. Since I can no longer read EVERY post and I now have to filter the huge amounts of boring covered-already-1000-times shite it's all a bit meh.

I now come along to post random insults whilst waiting for stuff to build, files to copy, servers to crash.. that kind of thing. Oh and I do still keep an eye out for rides and such.

There's still the other forum and the other other forum holding my attention..

just saw this on a website and thought it was a great, to remind people, and make people aware of donations.

We have the hideous flashing donate button. Looks a lot less like we're trying to repair a church roof.

Velocio: Is the board a LAMP stack app? if yes, have you profiled where most of the load occurs? If you PM me I am happy to have a look at certain aspects of the app for you. Might be able to customize certain aspects.

Also, things that come to my mind when you're talking performance:

• Proper DB indexing
  
• Mysql query caching
  
• Looking into the use of ETags for static files
  
• Looking into etags for cached output that hasn't changed
  
  

Etags don't work with CDN's. And if you examine the site using Firebug you'll see that caching is pretty rock solid.

The DB is indexed thoroughly and correctly. Only slow indexes are the fulltext search. We're using MyISAM for those tables (posts, threads, pms) and InnoDB everywhere else.

MySQL Query cache is enabled, and common queries have been normalised to use a DAL to ensure that the same query can use the cache. We get about 60% hits on those, it does fine.

PHP5 is op-code compiled using XCache.

All common and heavily re-used fragments of data (such as styles, templates, configuration) are stored in XCache.

XCache is running entirely from RAM.

We now have separate DB and Web servers, and the DB server doesn't break a sweat, it's all on the web server, and it's all about Apache2 with XCache (opcode and key/value pairs).

Oh, and I've unloaded all unncessary and unused PHP modules so that only what is needed it loaded by Apache.

The gobbledygook in this thread is simply exquisite. Best read on the forum. ;)

Yeah, have you tried reverting the flux compensator matrix?

I would also check that the ODN relays are running on full speed up to the Bussard collectors.