Emergency maintenance: microco.sm domain suspended and moving to microcosm.app

Bravo you tech legend.

Also donated.

Thanks David, from the ICC side the speed this was fixed and level of communication was amazing.

🙏

Single person incident response team here 😂

Tsk, I should've used Grafana Incident... a public view of that would've been incredible. I effectively did that in the first post of this thread though.

Thanks @Velocio really appreciate it!

So.. San Marino cycle holidays soon? 😂🤭🤭

1. Network traffic now I've moved microcosm.app off of Cloudflare is more than double, verified that we have a fast enough set of SSDs and also several 4Gbps links and should be able to keep up. But, I shall prep a load balanced second cache machine to bear the load if needed.
   
   

Just watching graphs whilst in meetings... it turns out the Cloudflare Cache Everything Page Rule was doing a lot of heavy lifting.

I'm going to guesstimate that we do around 12TB of traffic per month, but most usage is diurnal and so there is a daytime peak and it's quiet at night.

What's the cost implication of that?

On a bandwidth basis... no cost impact as the Linode account includes 20TB of bandwidth because of the number of machines we're using.

On a server basis, about an extra ~$30 per month for a cheap Linode 4GB... which I'll load balance via a DNS round robin now I'm not on Cloudflare.

So the cost implication per year is about $350 extra.

I'll prep the script, observe it over the next day and then do it if needed.

A few changes you can likely make if you are inclined to do so.

SPF record for microsm.app contains the Sendgrid include. This is not required so is potentially over permissive as the SPF check on the emails (that I receive anyway) is carried out against the subdomain em8141.microcosm.app, which has the required record. Sendgrid rarely uses your org domain for SPF unless you have some niche Sendgrid apps.

The DMARC record for microcosm.app asks for strict SPF alignment, but the SPF (Mail From) domain and Header From domain in the emails are not strictly aligned. So you are only passing DMARC on DKIM alone. Should there be a temperror (or fuck up on Sendgrid's side) with DKIM, the messages will fail DMARC and be blocked by the p=reject policy in the DMARC record. removing the aspf=s or changing it to aspf=r (the implied default) will give you double passes. And so long as you are not delegating subdomains to random folks, should be no less secure.

Thanks,

I use Fastmail to be able to reply authoritatively from the domain.

I use SendGrid to send email from the websites and also from Auth0.

SendGrid is configured with a custom IP, which is 192.254.117.114 as seen below.

This is what I've currently got:

; Fastmail (receive and send email)
microcosm.app. 1 IN MX 20 in2-smtp.messagingengine.com.
microcosm.app. 1 IN MX 10 in1-smtp.messagingengine.com.
fm1._domainkey.microcosm.app. 1 IN CNAME fm1.microcosm.app.dkim.fmhosted.com.
fm2._domainkey.microcosm.app. 1 IN CNAME fm2.microcosm.app.dkim.fmhosted.com.
fm3._domainkey.microcosm.app. 1 IN CNAME fm3.microcosm.app.dkim.fmhosted.com.

; SendGrid (send email)
151537.microcosm.app. 1 IN CNAME sendgrid.net.
em8141.microcosm.app. 1 IN CNAME u151537.wl038.sendgrid.net.
o1.out.microcosm.app. 1 IN A 192.254.117.114
out.microcosm.app. 1 IN A 192.254.117.114
s1._domainkey.microcosm.app. 1 IN CNAME s1.domainkey.u151537.wl038.sendgrid.net.
s2._domainkey.microcosm.app. 1 IN CNAME s2.domainkey.u151537.wl038.sendgrid.net.
sendgrid.microcosm.app. 1 IN CNAME u151537.wl038.sendgrid.net.
url3929.microcosm.app. 1 IN CNAME sendgrid.net.

; Both (control spam)
microcosm.app. 1 IN TXT "v=spf1 include:sendgrid.net include:spf.messagingengine.com -all"
_dmarc.microcosm.app. 1 IN TXT "v=DMARC1; p=reject; sp=reject; pct=100; adkim=s; aspf=s"

I would like strict alignment in SPF, and when achieved do want DMARC enforcement. I pay for nothing that gives me insight into DMARC so once semi-confident I chose to fail fast and hard so that someone just screams if something were amiss.

Given all that... what do I need to change? Just the aspf=s to aspf=r?

Hey, the rep system still seems to be down...

I looked for the rep system but couldn't find it.

Given all that... what do I need to change? Just the aspf=s to aspf=r?

Exactly that, or remove the aspf tag entirely as the implied default is aspf=r so we don't even bother telling people to use the tag at all (same with adkim, but your DKIM is strictly aligned so fine for that to stay).

em8141.microcosm.app. 1 IN CNAME u151537.wl038.sendgrid.net. is the record that is covering your SPF for the dedicated IP currently.

Authentication-Results: mx.google.com;
dkim=pass header.i=@microcosm.app header.s=s1 header.b=s5EMKk8t;
dkim=pass header.i=@sendgrid.info header.s=smtpapi header.b=XTesXENi;
spf=pass (google.com: domain of bounces+151537-6093-ams.stevenson=googlemail.com@em8141.microcosm.app designates 192.254.117.114 as permitted sender) smtp.mailfrom="bounces+151537-6093-ams.stevenson=googlemail.com@em8141.microcosm.app";
dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=microcosm.app

I would ideally need to see DMARC reporting to be fully confident to shit-can the "permit ALL of Sendgrid" records from 151537.microcosm.app. and the org domain. They won't be breaking anything, just being over permissive. 99.9% of the time, Sendgrid will use a subdomain for the Mail From so they can specifiy themselves in the MX (by way of the above CNAME) so they can handle bounce processing.

Gandi are not useful... glad I failed over to a different domain name.

Thank you for your message and for your interest in Gandi Corporate.

Your domain name is not expiring. We are checking with the registry why the domain name status is SUSPENDED, we will get back to you once we have more details.

Don't hesitate to inform us should you have any inquiry or require any additional assistance.

1. The admin panel on https://microcosm.app login was broken, re-wrote all references in the admin app.
   
   

btw, I do all this in vim... really inspires confidence, right?

Thank You! Velocio.

Did we almost die back there?

Well the forum platform almost did.

When it was obvious what the issue was, it wasn't obvious how to avoid what was then inevitable.

https://microco.sm is basically offline now... so in that respect yes, we dodged a bullet.

Gandi got bought out recently by some investment company, didn't it? Loads of people I know have dumped them in the last few years and lack of tech support seemed to be the biggest motivator 😬

Anyway, nice work, hope you're planning an early night after all this excitement

Gandi doing some non-violent disruption?

I can also now rate tech support for different Cloud providers:

• Cloudflare = 2.5h response for an Enterprise account and a "Urgent" level ticket, response was personal and actionable.
  
• Gandi = 17h response for a Gandi Corporate account and a "P1" level ticket for a "live site down" incident, response was generic, not actionable.
  
• OpenDNS = No response yet for a Consumer account.
  
• DNS Made Easy = Doesn't work in Firefox, can't open a Support ticket, had to use a different laptop to add a domain name... am a bit worried this is not the long-term home for the new domain.
  
• Akamai Linode = 1h response for a self-serve account and a generic ticket, response was personal and actionable.
  
  

I'm still looking for a permanent home for the DNS for microcosm.app, but it may not be Cloudflare as Cloudflare prohibit the cross-user CNAME (which is how any SaaS platform works, standard DNS stuff).

I've not been able to find any pricing info on NS1 (now owned by IBM), and this is probably because hosting a single domain just isn't their game.

I don't use AWS at all, so signing up to use just a single hosted domain feels... just kinda dumb.

May use Linode, but need to verify they're not using Cloudflare behind the scenes for it.

What a lot of work! Major respect!

• Akamai Linode - 45m for a second Support ticket about DNS CNAMEs.
  
  

Linode are winning the support game.

@Velocio as always, thank you.

1. Changed nameservers to Linode as DNS Made Easy simply does not work in Firefox 🤷 will get the pro-rated refund and see how Linode fares. Linode Support confirmed that they're not white-labelling Cloudflare, so the cross-user CNAME issues should not arise.