Were you taking the memory logs after the errors, before restarting?
I didn't get the chance to look over the previous email, I will do it soon.
Post by Heenan, Timothy StevenAfter restarting Kamailio on September 3rd, the problem has again
gotten to the point of no call processing.
Here are the memory statistics; additionally, I can see the virtual
memory has ballooned up to 400mb again.
shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792
-Tim
Timothy Steven
*Sent:* Wednesday, September 03, 2014 3:28 PM
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
I have about 460 destination sets. In each set I could have from 1 address up to 6.
When I took that initial memory log, Kamailio had been restarted
shortly beforehand. I just took another one after the system had
decayed so Kamailio was responding to Invites with SIP/2.0 500 No
error (2/SL). (exhibiting the issue after a few days) and the log is
about 400Mb! Ill attach a small snippet.
Seems to be going on and on about the dispatcher module.
-Tim
*Sent:* Thursday, August 28, 2014 5:03 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Looking at the logs, there are 1336 URIs, which seems ok given your
numbers, because the module keeps also the previous set of records
(needed because at reload time there can be kamailio worker processes
using the records -- perhaps we can improve a bit here, I will review
the current reload code a bit, since it was a contribution from
another developer quite long time ago).
- how many destination sets (groups) do you have? Can you estimate the
minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was
kamailio running? Did you wait enough to notice the steady increase of
memory usage?
Cheers,
Daniel
Thank you for the help.
For dispatcher, Im using a database that contains about 700 records.
kamctl dispatcher reload
Thanks,
-Tim
*Sent:* Wednesday, August 27, 2014 5:22 PM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
The logs suggest some leaks in dispatcher module -- it allocates
shared memory only for caching routing records. Few more questions
I need to get the answers in order to troubleshoot further.
How many records do you have in dispatcher table? Is it in
database or the text file?
How ofter do you reload? What is the command you use for reload?
Cheers,
Daniel
I have attached is the output of kamcmd cfg.set_now_int core
mem_dump_shm 1 to this email. I had to restart Kamailio to
load ctl and ctl_rpc, so let me know if I need to do another
one after some time passes and calls are processed.
Also, here is the output of kamctl mi get_statistics shmem:
shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768
Thanks!
-Tim
*Sent:* Tuesday, August 26, 2014 10:46 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
It is not needed to see the memory operations logs. Just to
dump the chunks in memory.
You would need ctl and ctl_rpc modules (iirc -- they should be
kamcmd cfg.set_now_int core mem_dump_shm 1
Extract the logs of the dump from the shared memory and send them over.
Cheers,
Daniel
Thank you.
I have recompiled with MEMDBG, however I am not observing
any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.
Here are my log settings from Kamailio.cfg
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")
*Sent:* Tuesday, August 26, 2014 3:54 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
I quickly looked over the modules that use shared memory
and I couldn't spot a commit related to any leak.
Have you had the chance to compile with MEMDBG?
Cheers,
Daniel
Do you think any of these modules possibly causing
this issue?
-Tim
Behalf Of *Heenan, Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
Were setting the memory allocation to 256 ( -m 256 M
64 ).
Ill see about recompiling with MEMDBG=1.
Regards,
- Tim
Behalf Of *Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much
memory did you allocate (-m parameter value)?
It will narrow down searches to see if there was
anything similar fixed since 4.0.4.
To troubleshoot easier, would be good to recompile
with MEMDBG=1, then the details of chunks in memory
can be dumped and analysed.
Cheers,
Daniel
Greetings,
Im running into a slow memory leak on my kamailio
4.0.4 SIP proxies. Im observing a steady increase
in the memory consumption until there is no more
add_dest2list(): no more memory.
What would be the best way to debug this kind of a
memory leak? The proxy does not handle any
registrations but does route a fair amount of calls.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of
the original message and any attachments.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA