Re-writing microcosm in Go

I'm working on rewriting the front-end (which is Django + Python + HTML + JS) in Go (making it Go + HTML + JS).

The reasons for doing so:

• Django changes too often and we're on an old version, upgrading is horrible and 3rd party modules are likely broken
  
• Django has had too many security issues (they have been patched but the regularity that they've occurred is a concern)
  
• Django was originally chosen as it was hoped it would encourage people to contribute code, but they have not
  
• Python isn't my primary language and I am wary of making changes
  
• Go on the Microcosm API has been rock solid, performed brilliantly and has been pain free to keep constantly up to date
  
• Go is easier to monitor
  
• Go is my primary language and it is easier for me to make changes
  
  

What I'd like to do is simply replace the Django/Python bit with Go. Whilst leaving the HTML and JS alone right now, but obviously as the template language changes the HTML templates will need converting.

Eventually I'd like to merge in the API stuff... to give a single code base for everything, and yet it could still be deployed as separate front-end and back-end if needed.

To this end, there's a new repo:
https://github.com/buro9/microcosm

I've got a working home page already done, laid out the skeleton of the project, and it's possible to spin it up and login to a site, view the home page as logged in and then log out.

I'm seeing some hugely impressive numbers too.

For a guest viewing the home page (generation time of HTML and sending over the wire to a local browser):

• Django = ~450ms
  
• Go = ~12ms
  
  

That's basically because I can control caching internally far better.

For a logged in user viewing the home page:

• Django = ~600ms
  
• Go = ~250ms
  
  

I should be able to get that faster... perhaps down to ~150ms... by running requests concurrently.

This is an FYI post... I'm going to update this thread with progress. Once converted I'll go back to adding features.

Also... if anyone else is serious about wanting to work on this, then I will add Let's Encrypt stuff sooner rather than later. This will allow you to spin it up locally but pointing at an existing site API and still be able to do things that require HTTPS like signing in.

Nice, following. We do a huge amount of Python and Go (and AWS) at work.

This is a nice challenge too.

I've not yet seen web apps written in Go, only web services and systems.

There are some things that don't seem to exist, so I'm having to create those. i.e. the Django template tags, Go has an equivalent capability called template funcs but there are no open source generic ones I've found so I've had to create that https://github.com/buro9/funcs

This is then included in the Go microcosm thing.

Then there are other things that seem less mature in the Go templating. For example the idea of having multiple dependencies... that lots of templates may share some common template, or derive from the same parent template. So I'm going to have to find a smart way to declare a DAG (directed acyclic graph) of dependencies so that each compiled template isn't some sprawlingly huge thing.

Would this be a suitable project for an absolute Python beginner who's done some Codecademy courses?

Sounds interesting. Have only dabbled a little with go, but I have been impressed with its speed and size.

Would this be a suitable project for an absolute Python beginner who's done some Codecademy courses?

This is to migrate away from Django, and requires virtually no knowledge of Django or Python.

The new system is going to be in Go, and requires knowledge of strongly typed languages and a vague idea of how MVC (model-view-controller) systems works.

If you want to learn Go, you can dabble... or you can review pull requests and ask me why I am doing things the way I am (if you're going to do this I'll start making pull requests rather than just pushing directly to Master).

This is to migrate away from Django, and requires virtually no knowledge of Django or Python.

...which I would know if I had read your OP properly! Sorry.

Just curious, have you considered ditching the django server-side templating completely and have the browser talk directly to the API? ie move the django logic into browser javascript instead of Go.

The goals of this project:

1. Make the non-API code more stable and require less maintenance (over a period of many years)
   
2. Make the non-API code more secure
   
3. Make the non-API code more performant
   
4. Achieve those things with least effort
   
5. Make it easier to add new features
   
   

And on #2 and #3... when new features emerge in browsers and infrastructure, add support as soon as possible.

I do not believe that going the JS route with a rich client would deliver on #1, #2, #3, #4. I do believe it would do #5, but that over time maintenance of that code would slow down adding further features.

Instead I'm aiming for:

• Render pages in less than 2ms on the server
  
• Reduce API calls and cache where possible (so total serve time is < 10ms on a cached page, and is the shortest possible time for non-cached)
  
• During the render of templates, fully hydrate the HTML (i.e. the "6 hours ago" is currently JavaScript... everything will be done server-side) so that JS is not required to draw and all JS can be deferred and the draw time reduced
  
• Use local storage as a cache and add service workers to populate the cache. Things like the web fonts are not going to be loaded on page load, I'm going to draw the page without them and then populate the cache in the background and then they will be there for the next and all subsequent requests
  
• Isolate features in the templates to speed up addition of new features
  
  

I'm building this quite an old school way, because ultimately it will be very maintainable for a seriously long period of time.

I'm aiming for long-term neglect as a design feature... we have a bus factor of 1, and if I die I would like the hundreds of sites that exist on Microcosm to be able to exist for a very long period of time with the only maintenance of the system required is my wife making sure that every few years the payment details are kept up to date and that the PayPal keeps working and donations keep covering costs.

The JS approach may be snazzy but it doesn't deliver on most of my goals. I could've stayed on Django and started layering in the JS there.

Oh, and I want to gain freedom of hosting choice.

The UI portion will be merged with the API into an homogeneous executable that can do all code paths, and these will be placed in a container and run without the load balancer (I'll be using cloud load balancing).

I'm aiming to move the database and hosting to Google Cloud Platform, which is now on par with Linode for cost, and yet is a lot more stable, secure and trusted.

Fair :) Javascript frameworks have a short half-life.

I might try to get it running locally in Docker. Is there any more setup required beyond what's in the README? Is the PgSQL db automatically populated on first run, etc?

The readme is idealistic ;)

And LFGSS makes running it hard because it's running on a custom domain with HSTS enabled and this is in the Chrome and Firefox HSTS preload list.

What I've done:

1. Create a self-signed SSL cert for *.microcosm.app
   
2. Checkout the repo
   
3. make && bin/microcosm-web --help
   
4. Fill in the MICROCOSM_WEB_CERT_FILE and MICROCOSM_WEB_KEY_FILE with the self-signed cert created in step #1
   
5. Note that the static files are expected to be at /srv/microcosm-web and for me that is a symlink to where I have checked out my repo, and then the path to the static files: microcosm-web -> /home/buro9/Dev/src/github.com/buro9/microcosm/web/files/ because my $GOPATH is /home/buro9/Dev
   
6. I'm practising on Brixton Cycles ;) I'll stop doing this now and will create a forum somewhere for dev work... but whilst I'm read-only (doing the read path before the write path) I have an /etc/hosts that is 127.0.0.1 forum.brixtoncycles.cc
   
7. make && bin/microcosm-web should now work
   
   

Then I hit the home path of forum.brixtoncycles.cc which is now served by my local dev copy of Microcosm.

The aim is to have 3 binaries from 1 repo:

• microcosm-web = just the front-end (for those who want to bespoke that but not run the API or database)
  
• microcosm-api = just the back-end and PostgreSQL (so I can scale this independent of other things)
  
• microcosm = all the things (you don't need the others as it includes front-end, back-end and PostgreSQL, scale by just having more instances)
  
  

I'm working exclusively on microcosm-web right now, just the front-end... which is talking to a live API :)

When done with the front-end, I'll be copying the existing backend code into this repo... and at that point, yes it will create the PostgreSQL tables, etc.

Today: Just the front-end, and you need some certs as all dev work is HTTPS.

The readme is idealistic ;)

Ah hah! Thought I was missing something. Well I'm out of side-projects for the time being so cautiously happy to help out. I'll check out the code and see about getting some containers set up for now.

Oh, and I'm using make with hellogopher https://github.com/cloudflare/hellogopher

This means all of the dependencies are vendored, you shouldn't need to worry about fetching them.

Just run make and it will build within the repo path /bin.

How did you get on setting up Docker files?

@Velocio

Are you accepting pull-requests to the backend project? I don't have anything to contribute at the moment, but I'm just curious.

To support a decent mobile app you'd probably want to add some push capabilities. Is that something that's on the road map?

Hoo boy, it's been a while. I'm seeing some commits go by though - some kind of bot? What's the latest @Velocio ?

Oops, just found the repo at https://github.com/buro9/microcosm. I was looking at https://github.com/microcosm-cc/microcosm

Boom. and I just found your docker files. I'll check them out in a bit.