[SR-Users] async workers causing unexplained high load average

[Alex Balashov]

Another thing I have found is that having a certain amount of async
workers running, even if they are not doing anything, appears to cause
unexplained CPU load, even if the Kamailio instance is completely idle
and not processing any calls.

Here is the baseline load with no async workers:

[***@centosity6 ~]# uptime
14:06:28 up 2 min, 2 users, load average: 0.01, 0.00, 0.00

If I start Kamailio with 1 async worker...

modparam("async", "workers", 1)
async_workers=1

14:07:59 up 3 min, 2 users, load average: 0.00, 0.00, 0.00

Still nothing.

Now I try 2:

14:08:54 up 4 min, 2 users, load average: 0.45, 0.11, 0.04
14:08:59 up 4 min, 2 users, load average: 0.49, 0.12, 0.04
14:09:04 up 4 min, 2 users, load average: 0.53, 0.14, 0.04
14:09:09 up 4 min, 2 users, load average: 0.57, 0.15, 0.05
14:09:14 up 5 min, 2 users, load average: 0.60, 0.17, 0.05
14:09:19 up 5 min, 2 users, load average: 0.63, 0.18, 0.06
14:09:31 up 5 min, 2 users, load average: 0.69, 0.21, 0.07
14:09:37 up 5 min, 2 users, load average: 0.71, 0.22, 0.07
14:10:35 up 6 min, 2 users, load average: 0.89, 0.36, 0.13
14:11:07 up 6 min, 2 users, load average: 0.93, 0.42, 0.16
...

I don't have time right now to see if it will rise to infinity, but it
just keeps going up.

There are no SIP messages being received at all, and there is absolutely
nothing going on.

The ascent of the load average is much higher with >= 4 threads. I have
8 "CPUs" available from /proc/cpuinfo (quad-core processor with
HyperThreading).

When I attach strace to the PID of the Async Worker, there's nothing
going on except the normal recvfrom() blocking wait:

[***@centosity6 kamailio]# kamctl ps | grep Async
which: no greadlink in
(/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/)
Process:: ID=11 PID=1559 Type=Async Task Worker
Process:: ID=12 PID=1560 Type=Async Task Worker

[***@centosity6 kamailio]# strace -p 1559
Process 1559 attached - interrupt to quit
recvfrom(9,

[***@centosity6 ~]# strace -p 1560
Process 1560 attached - interrupt to quit
recvfrom(9,

Nor do the other Kamailio threads seem to be doing anything unusual. 'ps
aux' and 'top' show 0.0% CPU load on the Kamailio processes themselves,
and 'iowait' shows the system as 98-99% idle.

I suppose it's possible that this is a peculiarity of the testing
environment: I am running this on CentOS 6.5 inside VirtualBox, which I
think is KVM-based. I have not had a chance to duplicate this experiment
in a different kind of execution environment yet. It's possible that the
load average is simply being improperly indicated because of the VM
environment.

[Alex Balashov]

I just tested the same thing on a Rackspace VPS (Xen I think) and am
seeing the same climbing load average with 2 async workers. It seems to
top out at 1.05.

Post by Alex Balashov
Another thing I have found is that having a certain amount of async
workers running, even if they are not doing anything, appears to cause
unexplained CPU load, even if the Kamailio instance is completely idle
and not processing any calls.
14:06:28 up 2 min, 2 users, load average: 0.01, 0.00, 0.00
If I start Kamailio with 1 async worker...
modparam("async", "workers", 1)
async_workers=1
14:07:59 up 3 min, 2 users, load average: 0.00, 0.00, 0.00
Still nothing.
14:08:54 up 4 min, 2 users, load average: 0.45, 0.11, 0.04
14:08:59 up 4 min, 2 users, load average: 0.49, 0.12, 0.04
14:09:04 up 4 min, 2 users, load average: 0.53, 0.14, 0.04
14:09:09 up 4 min, 2 users, load average: 0.57, 0.15, 0.05
14:09:14 up 5 min, 2 users, load average: 0.60, 0.17, 0.05
14:09:19 up 5 min, 2 users, load average: 0.63, 0.18, 0.06
14:09:31 up 5 min, 2 users, load average: 0.69, 0.21, 0.07
14:09:37 up 5 min, 2 users, load average: 0.71, 0.22, 0.07
14:10:35 up 6 min, 2 users, load average: 0.89, 0.36, 0.13
14:11:07 up 6 min, 2 users, load average: 0.93, 0.42, 0.16
...
I don't have time right now to see if it will rise to infinity, but it
just keeps going up.
There are no SIP messages being received at all, and there is absolutely
nothing going on.
The ascent of the load average is much higher with >= 4 threads. I have
8 "CPUs" available from /proc/cpuinfo (quad-core processor with
HyperThreading).
When I attach strace to the PID of the Async Worker, there's nothing
which: no greadlink in
(/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/)
Process:: ID=11 PID=1559 Type=Async Task Worker
Process:: ID=12 PID=1560 Type=Async Task Worker
Process 1559 attached - interrupt to quit
recvfrom(9,
Process 1560 attached - interrupt to quit
recvfrom(9,
Nor do the other Kamailio threads seem to be doing anything unusual. 'ps
aux' and 'top' show 0.0% CPU load on the Kamailio processes themselves,
and 'iowait' shows the system as 98-99% idle.
I suppose it's possible that this is a peculiarity of the testing
environment: I am running this on CentOS 6.5 inside VirtualBox, which I
think is KVM-based. I have not had a chance to duplicate this experiment
in a different kind of execution environment yet. It's possible that the
load average is simply being improperly indicated because of the VM
environment.