• Purging the HTML cache now, it will take a moment and there may be a short outage again.

    All links will be fixed after this.

  • awesome work 👊
    much appreciated

  • My OH is Italian and a trained translator, so anything you need done on that front, please shout…

  • Thanks, gafferman. Truly rapid work.
    DAS! (Do a surgery)

  • Top work VB.

  • Impressive response to a P1 critical situation, many plaudits

  • Good work fella

  • Great work V, thank you.

  • Still some problems with icons on the homepage:


    1 Attachment

    • Screenshot from 2023-05-16 16-13-24.png
  • Also ... Donated!

  • Impressive response to a P1 critical situation, many plaudits

    Yeah, I realise already that I've never seen any business handle something like a suspended primary domain so rapidly. The ad-hoc "I've got this" business continuity plan is pretty incredible.

  • Still some problems with icons on the homepage

    Will fix.

    1. Fix all forum logos on all sites, as they typically are assets on microco.sm
    2. Replace all microco.sm with microcosm.app in all comments

    That last one was only 20K comments out of 17M.

    1. Purge nginx file cache for the API.
  • That's a lot of comments. Did you do it in one query? What was your rollback strategy?

  • 👀

    Yes

    UPDATE revisions
       SET raw = REPLACE(raw, 'microco.sm', 'microcosm.app')
          ,html = NULL
     WHERE raw LIKE ('%microco.sm%');
    
  • In the past 12 hours I would say... there wasn't, and still isn't, a rollback strategy.

    It's more a "fail forward" strategy.

    The adrenalin helps provide focus.

  • How long did that take?

  • Love it. Proper cowboy stuff. No snapshots or anything?

  • Hell, no transactions.

    It took about 7 seconds.

    1. Fix Sendgrid DKIM and SPF, fix the HSTS for the pervasive link tracking that I can't remove
  • As you well know from my ... cough cough ... contributions the focus part isn't so good for my work when I fly on auto pilot like that. I really rely on stuff like backups and rollbacks and staging environments to make absolutely sure I don't fuck up.

  • I mean... if I wanted to be professional about this it would be down for a month and cost £20K in consultants to restore.

    But just doing what needs to be done it was only down for 3h and has cost about £300 (£250 for the TLS cert and just shy of £50 for some email campaigns telling some of the sites that they have a new URL).

  • An no dreaded VACUUM

  • I really rely on stuff like backups and rollbacks and staging environments to make absolutely sure I don't fuck up.

    I looked up the syntax for LIKE before I ran it... but that's only because the first time I ran it, it proudly said "0 rows updated" so I knew I got something wrong (I missed the % chars).

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Emergency maintenance: microco.sm domain suspended and moving to microcosm.app

Posted by Avatar for Velocio @Velocio

Actions