1-2-3 and you're back in the room...

Database failed over to the other server at 12:16am. The security on the web servers hadn't been updated though, so the iptables rules didn't allow them to talk to the database. Everything was fine, it just wasn't connected up once the database failed over.

Looking at why the database failed over, and it seems that the swap space had run out and the first database server had just gotten a bit too busy for comfort with queries taking almost half a second to run. That's what triggered the failover and is when the site then become disconnected from the database and went off-line.

well, that's cleared that up then