Subtle changes, bugs and feedback

No, the forum was down for most of yesterday for me too.

Here's the reason why:

https://www.lfgss.com/conversations/396550/

When replying to a PM, I am getting the following error. The post does actually get sent (which caught me out, so sent several!)

1 Attachment

try now, I was testing a thing (see the end of the DDoS Incident thread) and I broke redirects after posting replies.

I detected it by triggering it myself, and have just fixed it... your reply would've been sent, only the redirect after success was broken.

All good thanks (once again)

Avatar not working for this profile, only noticed it since the DDoS incident?

https://www.lfgss.com/profiles/120414/

404 on the image url https://lfgss.microcosm.app/api/v1/files/379e77c45a2510a6a456cd1957b28a0fcc5c6ee7.jpg

EDIT: another one https://www.lfgss.com/profiles/60198/ & https://lfgss.microcosm.app/api/v1/files/a478afa938c1a83777eae28f3d84ecf1d0923aec.jpg

1 Attachment

I've seen a couple of those recently, and before the DDoS thing.

Not important but: if a comment contains a link, the link gets replaced with a shortened link that looks something like: https://microcosm.app/out/abc123, but if I request that same comment via the API, it looks like: http://microcosm.app/out/abc123. I can't figure out where the difference is coming from. Is the django client switching HTTP for HTTPS? I've had a look but I can't see anything.

this one is odd.

it's a 404, and I don't know where it's coming from.

the images are there, just remove the .jpg from the end and it works... but one of the requests caches a 404, and then it doesn't work for all subsequent requests.

I've just cleared the caches at 9:06 UTC and am waiting for a 404 to appear again... the first one will tell me what the criteria is

it's the mobile app...

body_bytes_sent	19
filename	/var/log/nginx/https.wildcard.microcosm.app.access.log
geoip_country_code	AU
host	http://www.lfgss.com
http_user_agent	Dart/3.4 (dart:io)
job	lb.microcosm.cc
msec	1717233343.914
remote_addr	REDACTED
request	GET /api/v1/files/677fd6380f7e4b1f9428035dfa7e3dbcb227cfbd.png HTTP/1.1
request_length	372
request_method	GET
request_time	0.003
request_uri	/api/v1/files/677fd6380f7e4b1f9428035dfa7e3dbcb227cfbd.png
server_protocol	HTTP/1.1
service_name	lb.microcosm.cc
ssl_cipher	TLSv1.3/TLS_AES_256_GCM_SHA384
ssl_protocol	TLSv1.3/TLSv1.3
status	404
time_local	2024-06-01T09:15:43+00:00
uri	/api/v1/files/677fd6380f7e4b1f9428035dfa7e3dbcb227cfbd.png

if the first request to a file is via Dart, it results in a 404, and that is subsequently cached which results in all future requests to that file being the cached 404.

I still don't understand why... but after I cleared the caches and restart, there were no errors... until the app was used and all of the errors had the same UA... and then all future requests for those files were now 404s.

lol... your error is the same as my error... it's the http: which causes the 404.

I'm a bit baffled as to where it comes from, as the server only knows the API domain is microcosm.app and the load balancer is just a wildcard virtual host, meaning lfgss.com simply doesn't exist in the code, it's echoed back to the caller.

and http doesn't really exist either... but I did change the server about a month or two ago, upgraded the load balancer, and the old config may have had a redirect from http to https... and perhaps this revealed an old bug? as the redirect would have covered up the issue by correcting it.

could you share an API request and response so that I can see what you're calling and how, and then I can either just think about how the hell it produced the output, or I'll try and recreate it myself.

Quick question: I can't attach images to posts. The text posts, but I get an error.

Android 14, Chrome.

Is this not supported and I should just install the app?
Or does it indicate a bug and I should bughunt and report back?

Thank you.

Interesting... I'll send some curl commands over when I get on the computer.

Could CloudFlare be rewriting links?

I've done some more digging and come up with more questions than answers.

Firstly on the API, most assets are served with relative links: So that raises the unpleasant prospect that the mobile app has been defaulting to HTTP and requesting assets over insecure connections. But I've been through the codebase, and wherever a scheme and domain is missing, the app is hardcoded to assume HTTPS. E.g. attachments, profile avatars.

The app should never request anything over HTTP, and indeed, I just ran the app in dev mode, logging all image requests and there were no requests for LFGSS assets over HTTP. So I really can't explain how you're seeing a request from dart, over an insecure connections. Do you know why the logs mention an SSL cipher and protocol, despite the request being over HTTP? Incidentally, I've never seen a Host header include the scheme. That's pretty unusual.

The problem could have been masked by HSTS - (the website serving http:// links, but chrome silently swapping them with https://), but it's not that. The website is serving attachments with absolute URLs:

curl https://www.lfgss.com/comments/17405497/ | grep "9dbf.jpg"

<img src="https://lfgss.microcosm.app/api/v1/files/a1192d75683ebe6f1104c13f03b4f71443919dbf.jpg" alt="cactus.jpg" title="cactus.jpg" />

On the link shortener:

The api seems to be serving HTTP links:

curl -v -H "Authorization: Bearer $LFGSS_TOKEN" https://lfgss.microcosm.app/api/v1/comments/17405497 | jq ".data.html"

"<p>A link: <a href=\"http://microcosm.app/out/LgqWi\" rel=\"nofollow noopener\" target=\"_blank\">https://theuselessweb.com/</a></p>\n"

But the same comment via the web is magically HTTPS:

curl https://www.lfgss.com/comments/17405497/ | grep "A link:"

<p>A link: <a href="https://microcosm.app/out/LgqWi" rel="nofollow noopener" target="_blank">https://theuselessweb.com/</a></p>

Also, couldn't post the above from the web just then. Was getting an error message from CloudFlare 🤷

1 Attachment

now you mention it, weird choice of log filename too - if the request is http

filename	/var/log/nginx/https.wildcard.microcosm.­app.access.log

Sigh... fixed all URL rewriting, by removing it, which has been an oft-requested feature.

But it's broken embedding of videos, etc.

I've just done three things to try and help here:

1. I've removed URL rewriting... it no longer happens, but I do strip most common site tracking on third party links.
   
2. I've left in all code to make old /out links work, but I've purged all caches to get rid of all redirects that were implicitly in all old comments.
   
3. I've intercepted the port 80 requests to /out and will now continue to redirect them... for the really old cached content out on the web.
   

I have broken video embeds though... https://www.youtube.com/watch?v=hnRdIMklPow
this no longer embeds.

I'm unsure whether this is a good or bad thing... I appreciate that none of the embeds can add cookies, track things, etc... but dislike the fact that you cannot get a preview of what you're seeing, or choose not to leave LFGSS when all you want is to see a quick video.

Oh... I see what I've done.

The new "do not redirect links" code doesn't store state in the database... with the old /out code I needed to know which links were in a comment and map shortLink ==> fullUrl.

That information was stored in the revision_links database table.

The EmbedAllMedia code used the state that was created, it could look through the saved state and see which links were in a revision, not by processing the HTML, but by just checking the state in the database.

It was a quick and easy way to know whether I had anything to do, without having to process the HTML of a post again.

But... it doesn't work now, as I save no state.

Meaning I have a choice between these two things:

• Save state again.
  
• Spend a little more CPU on processing HTML to find the links again.
  
  

I think I prefer the latter, as the result of this is cached.

This is where the lookup is done (for my reference, as I likely won't fix it immediately) https://git.dee.kitchen/buro9/microcosm/src/branch/main/models/link_embedmedia.go#L38-L68

Were you able to figure out what was going on with the profile pictures?

Is there anything else I can do to help from my end?

I modified the nginx pattern yesterday so that it matches more, but am waiting to see if the metrics suggest it fixed things.

I have fixed the embeds.

https://git.dee.kitchen/buro9/microcosm/src/branch/main/models/embedly.go

And am reminded that I should not be allowed to write code.

https://www.youtube.com/watch?v=euEY7lkRRaE

I've no effing idea what that video is, but wanted one that expressed that the code in the preceding link hurts the head.

Also... that has a test case !

No other code in microcosm has a single test :)