Grand rewrite: Rewriting the web client from Django to Go

Old repo: https://github.com/microcosm-cc/microweb
New repo: https://github.com/buro9/microcosm

Motivation

The architecture is a web client (this website) talking to an API that talks to a database. The web client is Django+Python, the API is Go, the database PostgreSQL.

The web client is a very old Django and is no longer supported (neither the Django version, nor the Python version, nor any of the requirements/dependencies). We can no longer deploy it... and are flying on a wing and a prayer. It's also messy to install, fairly fragile (depends on a mess of file system things), and needs an Nginx to front it as well as some other things (like memcached).

The API behind the web client is here https://github.com/microcosm-cc/microcosm and has proven to be very stable in terms of maintainership over a long period of time. Trivial to build against newer versions of Go, highly performance, dependencies have aged really well. It also is trivial to install, and only needs memcached. It does not need an Nginx fronting it but as we have one for the web client we do front it.

Goal

Fully implement the web client in Go, deploy it on the same server, uninstall Nginx.

Stretch Goal

If the web client and API are both in Go, and both use the same core models... they could be in the same repo and allow building as a single binary, or as separate binaries for the API and web client. The single binary (big server) mode would work for 95% of anyone who ever runs Microcosm, but the separate API and web client would allow horizontal scaling across the different layers and allow third parties to modify the web client without having to go into the depths of the API.

In the single binary mode, we would skip all JSON serialisation and deserialisation to improve performance.

Method and Dev Phases (milestones)

We're basically working in the new repo, it's buildable... and we're going through phases:

1. Get everything read-only
   
2. Get the user experience read / write
   
3. Get the moderator / admin experience read / write
   
   

There are instructions in the readme of the new repo.... go read it.

Nitty Gritty

The old web client is a Django server... it is interesting in that it doesn't have a database at all. There are just controllers (views) that talk to the API over the internet. Nothing that the web site does is special... it's all just API calls.

For example... to view a single conversation in the old server:

1. Route to the conversations controller https://github.com/microcosm-cc/microweb/blob/main/microweb/urls.py#L14
   
2. Route to the view https://github.com/microcosm-cc/microweb/blob/main/conversations/urls.py#L8
   
3. Define what the view needs https://github.com/microcosm-cc/microweb/blob/main/conversations/views.py#L41
   
4. Build the API calls https://github.com/microcosm-cc/microweb/blob/main/conversations/views.py#L49-L68
   
5. Make the API calls https://github.com/microcosm-cc/microweb/blob/main/core/api/resources.py#L1341-L1436
   
6. Build the view data from the API response and render the template https://github.com/microcosm-cc/microweb/blob/main/conversations/views.py#L70-L80
   
7. The template https://github.com/microcosm-cc/microweb/blob/main/core/templates/conversation.html
   
   

The new server is kinda similar...

1. Route to the conversations controller https://github.com/buro9/microcosm/blob/main/web/server/server.go#L40
   
2. Route to the view and define what the view needs and build and make the API calls https://github.com/buro9/microcosm/blob/main/web/controllers/conversation.go#L13-L58
   
3. Build the API calls https://github.com/microcosm-cc/microweb/blob/main/conversations/views.py#L49-L68
   
4. The template https://github.com/buro9/microcosm/blob/main/web/templates/templates/pages/conversation.gohtml
   
   

A lot of the work is:

• Compare controllers and views and make sure they exist in both and behave the same.
  
• Compare templates and make sure they exist in both and behave the same.
  
• Create / convert template tags as necessary.
  
• Test the UI constantly by walking through various scenarios... basically use the new web client and find issues and fix them.
  
  

But there is other work:

• Remove JQuery by converting everything to native JS... should be very readable and maintainable.
  
• Add Let's Encrypt into the Go server to remove the need for certs.
  
• Add better security features like correct CORS and subresource integrity.
  
• Add dark/light mode to the CSS (all of the work above doesn't touch the CSS, that is a can of worms in itself).
  
• Remove as many 3rd party dependencies of JS as possible.
  
• Replace auth0 for passwordless login.
  
• Add an LRU file system cache for the attachments.
  
• Add support for an integrated firewall, i.e. to be able to block specific user agents, TLS ciphers, paths, cookies, etc... all from a rule file. This is needed to protect the sites when they're not run behind a heavy weight CDN like Fastly, Akamai or Cloudflare.
  

Some comment on why and some details is in this thread: https://www.lfgss.com/conversations/131364/?offset=7275#16659856

@pascalo we now have a public place to chat rather than random threads and DMs.

And for those interested in helping... wild times, we're using a domain I no longer own as the test domain.

gfora.com

Simply /etc/hosts it to your dev box and issue a self-signed cert and you will have almost enough to spin up the frontend.

When you get that far DM me for a client secret that will make it work against the production API server.

What we're doing is making it so that dev work is against a production site... this site! Which provides all of the real testing data... you can literally place the old and new side by side in browser windows and compare to see if it's spot on.

Ok, I merged PR 135 .

This means that the base.js file now doesn't use any moment or jquery anymore.
One down, quite a few to go.

I've added a couple more issues, but am going to jump back into the Profile page today... specifically this issue: https://github.com/buro9/microcosm/issues/42

Winding down for today, but tomorrow I might have a look at the reply form template, simply so that I can get the attachment JS stuff going as well

That's a big one... the comment box?

It's basically a textarea, with some controls overlaid... and then an entirely separate file upload system.

File upload is going to need me to explain things... because the files are loaded async from the form being submitted. Attachments are uploaded, and the references are kept in the form and submitted. The server cleans up orphaned attachments (in case the comment is never submitted).

But the big bit will initially be getting the Markdown controls overlaid onto a textarea in native JS that allows the neat functionality of it:

• Selection of text and then hitting a control for Bold, Italics, Link, Image, Quote and Code blocks... as well as preview of the Markdown in the client.
  
• Selection of text in another comment and hitting "Reply" with the selected text inserted as quoted text.
  

Also... 🙏thank you for all the help.

So with these changes (not a clue what they are) we'll get our yachts after all?

lol... no.

But the forum, and any other forums, may survive years/decades rather than likely months.

Better than yachts

Sail on forums :)

Getting search working was hard... so much minor cruft and so many complex templates. That bit was definitely easier in Django but it works now.

So... end of week summary:

I have (done frontend work):

• Got a dev env working (WSLv2 Ubuntu inside of Windows, with Sublime Text 4 running in the Linux machine with the UI in Windows 🤯)
  
• Implemented /huddles and /huddles/$id
  
• Implemented /profiles and /profiles/$id
  
• Implemented /search
  
• Fixed the authentication so it works
  
• Fixed most of the templates that already existed
  
• Fixed /today which was most broken
  
• Added a few new template tags
  
  

Pascalopitz has (done a mix of systems and frontend work):

• Got a dev env working
  
• Inlined the static and template files (embedding them) to simplify the dev environment but also to produce a single binary install
  
• Added static file caching
  
• Got to grips with the JS and started ripping out JQuery (already replacing moment.js)
  
  

Where it stands... most of the site is surprisingly usable... but there are ommissions. Currently it's only usable by following known good paths.

I've identified some milestones and shuffled all tasks into them... the Github Issues are reasonably high level but it's useful to see progress:
https://github.com/buro9/microcosm/milestones?direction=asc&sort=title&state=open

I feel good about what was done this week. It's kick-started playing with it, and it's overcome the "I haven't touched this in ages and it was partially done and I've no idea how to dive in again"... as work has recommenced, and I've found my way around it. Go templates aren't as bad as I remembered, but damn it takes time to compare HTML and see that things are just right.

Can I ask what ‘cruft’ is?
Nothing else on this thread makes any sense to me at all but I do like the sound of that word

Tiny differences in HTML, or behaviour that isn't quite the same.

The templates are being manually converted, and there are differences between Python and Go templating that emerge in subtle ways.

For example, python might have a 'not' function that evals some kind of false value. Whereas in Go it's really just doing a nil check. So the template may look right, but will actually behave differently.

Additionally I'm subtly changing the underlying models. In Django the field names and visibility suited the front-end, but in this I'm making the models be precisely the same as the ones inside the API to allow those to be shared in future. But it means that the models are just slightly different.

Another subtle thing is the length of truncation on text has come out different. The same values are in the old and new system, and you'd only notice with the old and new side by side... But I notice.

The HTML should be the same of course, but I started this 6 years ago and tweaked the Django one since, so they diverged and all needing checking.

I have zero idea what any of this is, but somehow still find it fascinating to read about. Thanks for a cool site!

So cruft is a kind of reference to ‘craft’ (what was intended to happen) but where the programmatic (instruction?) intention varies due to the instruction specified is subtlety different between the languages?
Apologies, didn’t mean to derail the thread but am fascinated by the term

You can probably guess in a fraction of a moment I have absolutely no grasp of all this stuff

Oh... You meant the word cruft rather than specificity on what the cruft is

1 Attachment

Ha! Thank you!

I just noticed that the new Go UI as been in development since 2016...

Are you doing some load-balancer magic and serving the UI from both projects at the same time? Slowly migrating endpoints across?

I just noticed that the new Go UI as been in development since 2016...

Yup... that's when I had already identified the risk from Python and Django being unsupported and realised I needed to get started. I'm glad I did, the vast majority of the foundation was laid back then... i.e. everything about templating, the API, the authentication, the caching... all the big things I poured time into. Then I got demotivated by the sheer amount of work remaining, and my full-time work became ridiculously demanding and going into burn-out zone.

Are you doing some load-balancer magic and serving the UI from both projects at the same time? Slowly migrating endpoints across?

Nope... I intend to get the majority of the user path done and then to switch a single website (this one) to use the new frontend and to find issues and fix them. When it's basically there, then I intend to switch all of the others over.

I could do a complex thing... i.e. "read only version of that page goes to the Go backend, all other versions go to the Django backend"... but it would be very hard. Mostly because I've shifted the new stuff to use a cryptographically secure cookie and that isn't implemented in the Django side. Same for CSRF tokens, etc... the read only and write parts of the site have to be served by the same app. So I have no intention of maintaining that in any way that would support a mixed-platform migration.

Is there a guide to getting django/microweb up and running? I think I'm pretty much there, but the custom middleware seems to firing a request at the api: (e.g. GET http://dev1.microcosm.app:8080/api/v1/hosts/my.dev.forum), which responds with a 404, and then Django raises an exception about not being able to resolve a CNAME.

Thing is, I tried firing similar requests at the lufguss API, which also responded with a 404.

You're trying to run the Django?! 🤯

I'd be surprised if you can get it running.

I only run the Go now... and just look at the source code for the templates and controllers in the Django.

What I'm doing:

1. Issue a self-signed cert for http://www.gfora.com
   
2. Edit /etc/hosts or your local equivalent, and map your local IP to http://www.gfora.com
   
3. DM @Velocio for the client secret
   
4. Clone the Go repo github.com/buro9/microcosm
   
5. Run the Go from the repo: make && ./microcosm-web
   
6. Open a browser and go to https://www.gfora.com/
   
   

You would now be running the new version of the frontend, against LFGSS backend (there's a hack in the API code that recognises http://www.gfora.com as LFGSS).

Then I look at the Django templates and controllers... and am manually creating the equivalent controllers and templates in the Go repo. The first post in this thread kinda describes the flow in the Nitty Gritty section.

You also may be able to get the Django running, also with the http://www.gfora.com alias, and the same client secret.

But I'm not sure of the value in that... if you want to see what the Django would render, just open LFGSS.

I have begun porting some of the comment reply code:

https://github.com/pascalopitz/microcosm/tree/reply-container-template

That said, it's mainly so I can refactor the JS scripts that hang off of it, i.e. attachments.js

@Velocio at some point we should have a little thread that explains how conceptually we'll deal with the form submissions.

I was basically wondering if I could get the entire microcosm estate running locally in docker.

I've had the API running fine for a while now. Set that up years ago when I decided to have a go at building a mobile app.

I wasn't planning on getting either of the front ends up and running, but when you mentioned the new secure cookies & csrf tokens, I wondered how hard it would be to port it to Django. Not that anything would come of it - I was just curious.

It's actually been really easy to get Django up and running using the python:2.7 docker image... apart from the weird CNAME resolution issue.