You think Mastodon.social getting denial of service attacked is something? I just denial of service attacked myself again.
If you see this, it’ll mean the Sidekiq queue has likely cleared. The last post I made three hours ago hasn’t gone through yet.
Something tells me there should be a different queue for posts you make, no?
¯\_(ツ)_/¯
(If you’re wondering what I’m talking about, read this, about the last time it happened: https://ar.al/2022/11/09/is-the-fediverse-about-to-get-fryed-or-why-every-toot-is-also-a-potential-denial-of-service-attack/)
“Something tells me there should be a different queue for posts you make, no?”
There sort of is, on a single user instance: the push
queue will deal only with your posts.
You can split this out into its own worker, and then new posts you write will be pushed to other servers separately from other stuff.
I mentioned this in my blog post under the heading ‘Sidekiq opitimsation’ here https://blog.thms.uk/2023/01/setting-up-mastodon
(Unless of course that isn’t what you meant?)
@michael Ooh, this is exactly what I meant, yes. @mastohost Hugo, can we implement this? It would solve my problem completely.
Basically, I wouldn’t be prevented from my new posts being seen until all the replies, etc., have been delivered.
@aral I do split Sidekiq automatically for plans that are above 30 Sidekiq threads.
So, everyone bellow the 30 threads uses `
sidekiq -c NUMBER_OF_THREADS`
and that is/should be equivalent to
`sidekiq -c NUMBER_OF_THREADS -q default,8 -q push,6 -q ingress,4 -q mailers,2 -q pull`
meaning the push queue should have priority over other queues.
@aral So, I'm not entirely sure the description you made "the last post I made three hours ago hasn’t gone through yet" is accurate. Were the queued jobs from the push queue? Did the post not reach other instances?
I'm just saying because if you post something and that posts starts to get engagement, creating a Sidekiq queue. What is queued may not be the push jobs.
If that is the case, splitting or having a dedicated worker for push jobs would make no difference.
@aral In Sidekiq you can go to /sidekiq/queues and it will show you the totals per queue.
And sorry @michael to keep mentioning you on my replies but if you have any suggestions/thoughts I would appreciate it.
From my testing, with plenty of CPU or RAM available, having a single worker performs equivalently to having multiple workers if you have less than 25 threads. But I made these tests several years ago and a lot has changed in Mastodon code, since.
"And sorry @michael to keep mentioning you on my replies but if you have any suggestions/thoughts I would appreciate it."
No need to apologise! I find it interesting.
I am far from an expert in Mastodon, so I don't think I'll have a particularly qualified opinion here, but for what it's worth, I think you are mostly right:
Firstly, many actions in mastodon appear to need processing by multiple queues, so the bottleneck might not be a single queue.
I also have the impression that in most cases, splitting up the queues into multiple workers may not actually speed up overall processing. (Though I haven't done any actual measurements.) What I think it does change, though, is that one really busy queue with higher priority can no longer completely block the processing of another (potentially shorter) queue with lower priority. And that can make the whole thing appear quicker from the end users' perspective, because everything keeps moving, not just the higher priority queues.
E.g. in your setup a busy ingress queue (and ingress seems to be the busiest queue on my server) can completely bring the mailers and pull queues to a halt, until its completely cleared, which isn't great.
So, as a totally hypothetical example, if you had 1,000 items in default
, and 1 in push
, those 1,001 items would be processed in roughly the same time overall, regardless of your queue setup (assuming the same number of threads overall, obviously). But if you had split it out, the 1 item in push
would be processed in parallel with the 1st of the 1,000 in default
, rather than after all 1,000 default
items are completed, which may or may not be advantageous.
Sorry, a lot of waffling, but I hope that makes sense ...
@michael Yep, I agree.
It's always a tradeoff. Using your example (1001 jobs) if you have 5 threads on a 'default' worker and 5 thread on the 'push' worker that will be slower overall to process than having a single Sidekiq worker with 10 threads. But yes, the single push will go faster.
It depends on what you want to optimise for and what your bottleneck is at a given moment. If there is a surge in federation, you need to optimise differently than a surge in engagement.
"if you have 5 threads on a 'default' worker and 5 thread on the 'push' worker that will be slower overall to process than having a single Sidekiq worker with 10 threads. But yes, the single push will go faster."
Yes, but if you had 5 threads doing -q default -q push
and 5 threads doing -q push -q default
it would be processed in the same time as a single 10-worker thread, and the push would still go faster.
Best of both worlds, unless I'm missing something, which is totally possible of course!
@michael It's true but one needs to consider the CPU/RAM usage. More threads maxes out your CPU first, more workers maxes out your RAM first.
So, if you have 5x2 threads in terms of RAM you are using more than 10x1 and would probably be using the same amount of RAM with 12x1.
Again, it depends if you want to optimise to process the max amount of jobs per second or to have less wait on a specific queue.