Exploring The Lemmy Archipelago 🏝️🍾

  • 1 Post
  • 28 Comments
Joined 1 year ago
cake
Cake day: June 26th, 2023

help-circle




  • I think I found what eats the memory. DB iops isn’t the cause - looks like the server doesn’t reply before all the database operations are done. The problem is the unbounded queue in the activitypub_federation crate, spawned when creating the ActivityQueue struct. The point is, this queue holds all the “activities” - events to be sent to federated instances. If, for whatever reason, the events aren’t delivered to all the federated servers, they are retried with an exponential backoff for up to 2.5 days. If even a single federated instance is unreachable, all events remain in memory. For a large instance, this will eat up the memory for every upvote/downvote, post or comment.

    Lemmy needs to figure out a scalable eventual consistency algorithm. Most importantly, to store the messages in the DB, not in memory.