Worker queue in admin area get's to full? Please check worker_queues
I have observed here on my Friendica node, that the worker queue gets fuller and fuller over time and I was really wondering why is that so. I checked the queue
table and found no real issue with it.
Then I checked throught the PHP code and also saw no logic mistakes (related to this, e.g. in Friendica\Core\Worker
class).
Here are some typical lines at the start of my investigation:
2021-02-20T23:24:16Z worker [NOTICE]: Load: 4.34/50 - processes: 409/4/115055 - jpm: 6/4/8 (0:1, 20:0/6, 30:0/6, 40:3/115452) - maximum: 4/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"343740","process_id":17119}
2021-02-20T23:24:17Z daemon [NOTICE]: Load: 4.34/50 - processes: 409/4/115056 - jpm: 6/4/8 (0:1, 20:0/7, 30:0/6, 40:3/115452) - maximum: 5/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"163d4b","process_id":17116}
2021-02-20T23:24:18Z worker [NOTICE]: Load: 4.34/50 - processes: 409/4/115056 - jpm: 6/4/8 (0:1, 20:0/7, 30:0/6, 40:3/115452) - maximum: 5/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"343740","process_id":17119}
2021-02-20T23:24:19Z worker [NOTICE]: Load: 4.36/50 - processes: 409/5/115057 - jpm: 6/4/8 (0:2, 20:0/7, 30:0/7, 40:3/115452) - maximum: 6/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"3f65a2","process_id":17123}
2021-02-20T23:24:19Z worker [NOTICE]: Load: 4.36/50 - processes: 409/5/115057 - jpm: 6/4/8 (0:2, 20:0/7, 30:0/7, 40:3/115452) - maximum: 6/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"343740","process_id":17119}
2021-02-20T23:24:20Z daemon [NOTICE]: Load: 4.36/50 - processes: 409/5/115057 - jpm: 6/4/8 (0:1, 20:1/7, 30:0/7, 40:3/115452) - maximum: 4/5 [] - {
"file":"Worker.php","line":782,"function":"tooMuchWorkers","uid":"163d4b","process_id":17116}
As you can see maximum: 4/5
and even maximum: 6/5
is present. So I checked at the code where the 5
is coming from and I found it here:
$queues = DI::config()->get("system", "worker_queues", 10);
$maxqueues = $queues;
Line 681 in
src/Core/Worker.php
So I checked that configuration entry in config
table and ... Bingo! There it is: 5
. I then disabled debug log again (not to have the hard drive flooded with debug log) and restarted it.
Since then the daemon is working happily on the queue. So as it seems to me, there are indeed more queues (spawned PHP processes) needed to work on a public Friendica instance. And this one here has only a few local users while one (me) is most of the time active.
Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •renice 19 `pidof php`
the PHP processes of the daemon, so I added it to my check-daemon.sh script.Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •queue
table, now it is down at 109k+ records. So within 7 hours 6k records have been worked on and only a minor of (12) have been defered.Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •430 - 91596
Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •645 - 74036
It seem to remain there at 74k?Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •971 - 53872
Roland Häder🇩🇪
in reply to Roland Häder🇩🇪 • •2308 - 78382