Doesn’t the fact that every Lemmy server has a copy of every federated post mean that if Lemmy takes off, only a few people with strong donation feeds can afford to survive?

Yes I’ve seen something like that on mastodon already. Though the caching is scaled by time so you can just say to cache only last 24 hours (or less) which will scale down storage requirements.

Doesn’t the fact that every Lemmy server has a copy of every federated post mean that if Lemmy takes off, only a few people with strong donation feeds can afford to survive?

It’s not precisely true that every Lemmy instance stores every post. A given Lemmy instance will store a given post if and only if:

• A user on the storing instance is “interested” in that post. Being interested has a somewhat complicated definition that I myself don’t fully understand, but two examples of being interested include being subscribed to the community BEFORE the post was made (importantly, the storing instance doesn’t fetch the forever backlog of posts just because someone subscribes… rather… it asks to receive FUTURE posts), or a user searches a post by url.
• Subject to some caching policy. I just learned this today from another comment, and maybe it doesn’t apply to Lemmy, or doesn’t apply yet… but we already know that the storing instance doesn’t fetch historical posts going back forever. It could also decide to forget posts older than a certain time.

The first of these is most important though, because it means that posts and comments that no one is interested in don’t get shipped around the federated network. And this leads to the property that the size/cost of a Lemmy instance is going to depend on the size of the “active” usage. A single user Lemmy instance subscribing to a handful of communities will always be small and cheap, because it doesn’t subscribe to much content. A bigger Lemmy instance need not scale to the entirely of content in the lemmyverse, but rather to the “active set” of posts and comments its users interact with this month. That could get big, but what the Lemmy devs are saying (sorry no link, I’ve read too many posts lately to remember all my sources) is that user-traffic browsing the local DB of the Lemmy instance is dwarfing the replication load, which is great news because user browsing is much easier to optimize than federated replication.

If there’s an active forum (sub-lemmy?) on a server that has to spin down, the history stays on the remaining active ones, but I assume the only option is forking?

(FYI, the thing you subscribe to is called a community in Lemmy. Some folks say sublemmy, but this is a redditism that isn’t used in the code or official docs. It’s a “community”, which is why the url for a community is hxxp://my.lemmy.social/c/mycommunity. The “c” in the middle stands for community.)

Well, we’ve already talked about caching and expiry. It’s not clear to me than any Lemmy instance other than the one that hosts the community is required to keep the ENTIRE post/comment history (though yeah the active/recent ones will be all over the federated network).

I haven’t lived through a major instance shutdown, maybe an old-timer can weigh in here. Speculating, I’d think there would be 2 options:

• BEFORE the old instance shuts down (or using a third big community like !lemmy@lemmy.ml to coordinate), make a new community on a different server and the mods post telling everyone to subscribe there. The new community would be… well… new. It wouldn’t have the old posts, it would be made from scratch. The only things that would bind it to the old community are the mods that come over, the users that follow them, and the culture.
• Optionally, using database tricks, or a migration tool (that I don’t think currently exists, but could almost certainly be created)… after doing all of the above… someone direct db access on both instances (aka an admin) could copy the the old posts to the new community. This might be a terrible idea for federation reasons, or it might be prohibitively complicated because of the db schema… but it feels to me like it COULD work. I’m not aware of it having been done before, or of any tooling that makes it “easier” for someone who isn’t pretty strong at doing complex data migrations in Postgres.

There’s definitely scope for this.

I know I would never pay for facebook / twitter / reddit. Not 1 cent, but I’m happily contributing to fosstodon on patreon each month - and I don’t think I’m alone.

I think there’s a not-insignificant segment of the population that are weary enough of the advertising revenue model that we’re happy to pay to avoid it. The expectation that everything on the net should be free has ruined the net IMO.

What do you mean by “my home instance on lemmy.world”?

I mean that my account, with the username PriorProject is hosted on the instance called lemmy.world. So that instance is “home” for my account. When commenting on a community like !asklemmy@lemmy.ml, the home instance for that community is lemmy.ml, and I’m relying on federation between those two instances to shuttle posts/comments back and forth so I can see them and people on other instances can see my comments.

If you were confused because we were discussing “home labs”, I confusingly used “home” in a different way. I don’t mean that lemmy.world is hosted out of my house (it’s hosted on a cloud provider, and I have nothing to do with running/hosting lemmy world… I just happened to sign up for an account there), I mean that instance is the home for my account. I probably could have found a clearer way to say that initially, but the ambiguity didn’t occur to me till you just responded.

The best way would be for everyone to host their instance (a small one should be fine for individual/small groups) alongside the bigger servers.

It’s worth noting that it’s possible to overtune a federated network on single-user instances as well. The advantage of a shared instance is that servers hosting communities can replicate posts/comments from that community to a server, and that server can handle browse traffic for hundreds or thousands of users. If each one of those users instead ran their own single-user instance, the server hosting the community would have to work much harder to deliver hundreds or thousands of copies of those posts/comments… and if some users went inactive but left their lemmy server running… maybe nobody is even reading those posts.

It definitely IS possible to overtune a federated network both for too few servers that are each too big, and too many servers that are each too small. There is a goldilocks zone of a medium number of servers that are medium in size. But Lemmy is a long way from too many single-user instances (the devs have commented that replication traffic is not a bottleneck), and there’s also a lot of tuning they can do to make large instances run smoothly. So Lemmy has room to both host bigger servers, and more of them.

We’re all learning. And I want to stress – and maybe should have in my original reply – that I wasn’t really trying to criticize. My reply was just going to be so far down, I figured it’d only be noticed by a handful of people.

The way you intuited things is the way a lot of people have intuited things on the Fediverse. There were big blow-ups on Mastodon about this 6 months ago, as users spoke out about instance admins defederating from other Mastodon instances who hosted accounts that chronically broke local rules.

It became clear that people interpreted these defederations as denying new local users’ access to “parts of Mastodon”, as if there were was some central object called “Mastodon” and local admins were being nannies about letting some of their users wander over to some parts of it, when in reality it was a case of the local admins refusing to host content from other websites that gave shelter to individuals who were abusive or otherwise breaking rules that existed on the local site.

If that makes sense.

If you can grok that we’re basically all on different, independent web forums, and there’s just an implicit agreement between the forums to cross-post and share content, you can better grok why somethings that will happen here happen.

It’ll be interesting to see how similar communities from different instances will work out. I’ve noticed a lot of Technology instances from beehaw.org, lemmy.ml, lemmy.ca, and others.

How will the fediverse as a whole determine which Technology instance is the “main” one? I assume it’ll be whichever one has the largest subscriber count accumulated over time.

Thank you for the response. EDIT: I read another comment (also from you) about the way federation actually works and I was totally wrong in my assumptions so I deleted them. I suppose I should be leaving these questions up to the people building the platform, rather than a layperson such as myself.

Can’t help but ask though, is the federation somewhat comparable to a blockchain? In a blockchain each member has a full copy of the chain, and in the federation each instance has a copy of all parts of the chain it cares about. If one peer goes down, the other peers can continue to function. Cool.

This comment should be stickied or included in some sort of tutorial though.

https://lemmy.world/comment/20357

I found another thread discussing this topic and most people seem to expect that there will be natural consolidation and fragmentation of communities over time. There was also the suggestion that individual communities could federate such that posts from two different meme communities in different instances could be merged into the same feed on the user end, which sounds like a pretty good compromise.

https://lemmy.ml/post/1163258