With forewarning about a huge influx of users, you know Lemmy.ml will go down. Even if people go to https://join-lemmy.org/instances and disperse among the great instances there, the servers will go down.

Ruqqus had this issue too. Every time there was a mass exodus from Reddit, Ruqqus would go down, and hardly reap the rewards.

Even if it’s not sustainable, just for one month, I’d like to see Lemmy.ml drastically boost their server power. If we can raise money as a community, what kind of server could we get for 100$? 500$? 1,000$?

  • RoundSparrow@lemmy.ml
    34·
    2 years ago

    Right now it looks to me like Lemmy is built all around live real-time data queries of the SQL database. This may work when there are 100 postings a day and an active posting gets 80 comments… but it likely doesn’t scale very well. You tend to have to evolve to a queue system where things like comments and votes are merged into the main database in more of a batch process (Reddit does this, you will see on their status page that comments and votes have different uptime tracking than the main website).

    On the output side, it seems ideal to have all data live and up to the very instant, but it can fall over under load surges (which may be a popular topic, not just an influx from the decline of Twitter or Reddit). To scale, you tend to have to make some compromises and reuse output. Some kind of intermediate layer such as every 10 seconds only regenerate the output page if there has been a new write (vote or comment change).

    don’t necessarily see the complexity (not saying it’s not there

    It’s the lack of complexity that’s kind of the problem. Doing direct SQL queries gets you the latest data, but it becomes a big bottleneck. Again, what might have seemed to work fine when there were only 5000 postings and 100,000 total comments in the database can start to seriously fall over when you have reached the point of accumulating 1000 times that.

      • poVoq@slrpnk.net
        8·
        2 years ago

        The advantage kbin has is that it is build on a pretty well known and tested php Symphony stack. In theory Lemmy is faster due to being built in Rust, but it is much more home-grown and not as optimized yet.

        That said, kbin is also still a pretty new project that hasn’t seen much actual load, so likely some dragons linger in its codebase as well.

        • sam_uk@slrpnk.net
          2·
          2 years ago

          I think it’s probably undesirable to end up with big instances. I think the best situation might be one instance that’s designed to scale. This could be lemmy.ml or another one. It can absorb these waves of new users.

          However it’s also designed to expire accounts after six months.

          After three months it sends users a email explaining it’s time to choose a server, it nags them to do so for a further three months. After that their ability to post is removed. They remain able to migrate their account to a new server.

          After 12 months of not logging in the account is purged.

          • andrewA
            2·
            2 years ago

            Thought on this a bit more, and I’m thinking encouraging users not to silo (and make it easy to discover instances and new communities) will probably be the best bet for scaling the network long-term.

            “Smart” rate limiting of new accounts and activity per-instance might help with this organically. If a user is told that the instance they’re posting on is too active to create a new post, or too active to accept new users, and then is given alternatives, they might not outright leave.

              • andrewA
                2·
                2 years ago

                If the instance in question has email support, I don’t see why the instance couldn’t notify them directly - but I think providing alternative instances first (with the option to get notified if this instance opens up) would be more reasonable

                • sam_uk@slrpnk.net
                  2·
                  2 years ago

                  The idea would be to retain the ability to collect email addresses, beyond the point that the main app can’t keep up. So you’d want something lightweight just for capturing the emails.