Fun with Federation: Lemmy edition

<p>It all started <a href="https://community.nodebb.org/topic/18824/did-4.4.1-break-federation-with-lemmy-groups">with a report about federation breaking between Lemmy and NodeBB</a>. I was subconsciously aware that something was going on, but had chalked it up to network issues.</p> <p>Observed behaviour showed that some remote categories would be receiving content in spurts, with long gaps in between.</p> <p>I spent the next 3-4 days looking into it, but came up empty. Whatever was happening wasn't throwing any obvious errors, and along the way, I found what I <em>thought</em> was related (it was), but I wasn't sure why: against some Lemmy servers, the "follow"/"unfollow" mechanic would simply stop working, and this would often coincide with gaps in content. In some egregious cases, the flow of content stopped completely!</p> <p>Unable to make headway, I had to <a href="https://github.com/LemmyNet/lemmy/issues/5722" rel="nofollow ugc">reach out to the folks at Lemmy</a> to figure out what the issue was. NodeBB occasionally sends non-200 level responses depending on the activity. Specifically, the following scenarios:</p> <ul> <li>A remote user upvoting more than 20 posts in a single day (a spam prevention tactic) causing NodeBB to throw an error, which was caught and returned an <code>HTTP 500 Internal Server Error</code>.</li> <li>A <code>Dislike</code> activity, which is not currently handled by NodeBB. In these cases, NodeBB would send an <code>HTTP 501 Not Implemented</code></li> </ul> <p>When encountering either of these responses, Lemmy would return the activity back to the queue for later delivery and <strong>mark a delivery failure</strong>. If enough of these (~40) happened within 24 hours, Lemmy would give the instance a time-out and <em>pause delivery completely</em>.</p> <p>That was it — a quick pair of code updates later, and we started working through Lemmy's backlog of 4.1M activities.</p> <p>As of 4am this morning, <code>community.nodebb.org</code> is no longer behind <code>lemmy.world</code>.</p> <p><img src="https://community.nodebb.org/assets/uploads/files/1748873636999-d2b3dc1c-01f3-4203-b281-2406e949667d-image.png" alt="d2b3dc1c-01f3-4203-b281-2406e949667d-image.png" /></p> <p>Fun week. Let's not do that again LOL.</p> <p>v4.4.2 of NodeBB contains the updated logic for smoother Lemmy federation.</p>