Using a desktop web browser…
It doesn’t seem like a timeout, it is always fast to respond with the error and also fast on successful loading pages. It seems to me there is some kind of resource/parameter starvation in nginx or the nginx bridges to the NodeJS app…
I’m also seeing problems with static content loading, images and css files not loading and getting a mangled half-generated page or missing icons for upvote/downvote that I have to refresh the browser with. This happens at least 1 in 8 refreshes of the page.
Same for me.
In the most recent hour, it has gotten better. I’d say 500 errors from nginx are only coming up more like 1 in 25 web page refreshes.
I spent the past 20 minutes reading over on Beehaw.org instance, and the same problem is happening there. Pages always load or fail fast, but nginx 500 errors come in batches - where for 20 seconds I instantly get broken page/error page. And then the site resumes normal operation.
I’ve seen similar behavior (fail fast,) when there are outbound networking issues from any of the service containers. In one instance, lemmy backend was not able to make any outbound connections to any external services. It wild fail in milliseconds and show a timeout error in the logs.
One hypothesis here could be there are sporadic network issues on the lemmy.ml servers. 🤷♂️
Lemmy.ml is just now throwing me more nginx 500 errors.
Reports on Beehaw have been posted by end-users: https://beehaw.org/post/574558
I’ve spent over 60 hours in the past 11 or 12 days on this problem, and it’s been brushed under the rug. There are MAJOR problems being logged on that server, and the server logs are not being shared.
ok, the past 5 minutes it was nearly constant. Mangled web pages half-loaded, images missing, nginx 500 errors. It is very fast to load each time, but failing to generate the content. Like only 2 out of 15 web page refreshes worked in that 5 minutes.
