Discussion:
[SR-Users] Kamailio 4.0.4 slow memory leak
(too old to reply)
Heenan, Timothy Steven
2014-08-11 23:40:59 UTC
Permalink
Raw Message
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIP Wholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116


----------------------------------------------------------------------
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.
Daniel-Constantin Mierla
2014-08-12 09:17:02 UTC
Permalink
Raw Message
Hello,

can you give the list of loaded modules? How much memory did you
allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed
since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then
the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
Post by Heenan, Timothy Steven
Greetings,
I’m running into a slow memory leak on my kamailio 4.0.4 SIP proxies.
I’m observing a steady increase in the memory consumption until there
ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.
What would be the best way to debug this kind of a memory leak? The
proxy does not handle any registrations but does route a fair amount
of calls.
Thank you in advance,
*Tim Heenan*
*Engineer I - VoIP**Wholesale | Windstream*
windstreambusiness.com <http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the
original message and any attachments.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-08-12 19:37:21 UTC
Permalink
Raw Message
Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org [mailto:sr-users-***@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIPWholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116


________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.




_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-08-19 17:14:13 UTC
Permalink
Raw Message
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org [mailto:sr-users-***@lists.sip-router.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIPWholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116


________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.



_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Daniel-Constantin Mierla
2014-08-26 08:54:05 UTC
Permalink
Raw Message
Hello,

I quickly looked over the modules that use shared memory and I couldn't
spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
Post by Heenan, Timothy Steven
Do you think any of these modules possibly causing this issue?
-Tim
Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
We’re setting the memory allocation to 256 ( -m 256 –M 64 ).
I’ll see about recompiling with MEMDBG=1.
Regards,
- Tim
*Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much memory did you
allocate (-m parameter value)?
It will narrow down searches to see if there was anything similar fixed since 4.0.4.
To troubleshoot easier, would be good to recompile with MEMDBG=1, then
the details of chunks in memory can be dumped and analysed.
Cheers,
Daniel
Greetings,
I’m running into a slow memory leak on my kamailio 4.0.4 SIP
proxies. I’m observing a steady increase in the memory consumption
until there is no more left. Kamailio then starts repeating this
ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.
What would be the best way to debug this kind of a memory leak?
The proxy does not handle any registrations but does route a fair
amount of calls.
Thank you in advance,
*Tim Heenan*
*Engineer I - VoIPWholesale | Windstream*
windstreambusiness.com <http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of
the original message and any attachments.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-08-26 15:34:23 UTC
Permalink
Raw Message
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIPWholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116


________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.




_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Daniel-Constantin Mierla
2014-08-26 15:45:40 UTC
Permalink
Raw Message
It is not needed to see the memory operations logs. Just to dump the
chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the
default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
Post by Heenan, Timothy Steven
Thank you.
I have recompiled with MEMDBG, however I am not observing any
additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.
Here are my log settings from Kamailio.cfg
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
…
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")
*Sent:* Tuesday, August 26, 2014 3:54 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
I quickly looked over the modules that use shared memory and I
couldn't spot a commit related to any leak.
Have you had the chance to compile with MEMDBG?
Cheers,
Daniel
Do you think any of these modules possibly causing this issue?
-Tim
*Heenan, Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
(SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
We’re setting the memory allocation to 256 ( -m 256 –M 64 ).
I’ll see about recompiling with MEMDBG=1.
Regards,
- Tim
*Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much memory did you
allocate (-m parameter value)?
It will narrow down searches to see if there was anything similar
fixed since 4.0.4.
To troubleshoot easier, would be good to recompile with MEMDBG=1,
then the details of chunks in memory can be dumped and analysed.
Cheers,
Daniel
Greetings,
I’m running into a slow memory leak on my kamailio 4.0.4 SIP
proxies. I’m observing a steady increase in the memory
consumption until there is no more left. Kamailio then starts
ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.
What would be the best way to debug this kind of a memory
leak? The proxy does not handle any registrations but does
route a fair amount of calls.
Thank you in advance,
*Tim Heenan*
*Engineer I - VoIPWholesale | Windstream*
windstreambusiness.com <http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116
------------------------------------------------------------------------
This email message and any attachments are for the sole use of
the intended recipient(s). Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email
and destroy all copies of the original message and any
attachments.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-08-26 20:42:56 UTC
Permalink
Raw Message
I have attached is the output of "kamcmd cfg.set_now_int core mem_dump_shm 1" to this email. I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed.

Also, here is the output of "kamctl mi get_statistics shmem:"

shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768

Thanks!
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 10:46 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

It is not needed to see the memory operations logs. Just to dump the chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
On 26/08/14 17:34, Heenan, Timothy Steven wrote:
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.

Thank you in advance,
Tim Heenan
Engineer I - VoIPWholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116


________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.





_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users


--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA




--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Daniel-Constantin Mierla
2014-08-27 22:21:48 UTC
Permalink
Raw Message
The logs suggest some leaks in dispatcher module -- it allocates shared
memory only for caching routing records. Few more questions I need to
get the answers in order to troubleshoot further.

How many records do you have in dispatcher table? Is it in database or
the text file?

How ofter do you reload? What is the command you use for reload?

Cheers,
Daniel
I have attached is the output of “kamcmd cfg.set_now_int core
mem_dump_shm 1” to this email. I had to restart Kamailio to load ctl
and ctl_rpc, so let me know if I need to do another one after some
time passes and calls are processed.
Also, here is the output of “kamctl mi get_statistics shmem:”
shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768
Thanks!
-Tim
*Sent:* Tuesday, August 26, 2014 10:46 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
It is not needed to see the memory operations logs. Just to dump the chunks in memory.
You would need ctl and ctl_rpc modules (iirc -- they should be in the
kamcmd cfg.set_now_int core mem_dump_shm 1
Extract the logs of the dump from the shared memory and send them over.
Cheers,
Daniel
Thank you.
I have recompiled with MEMDBG, however I am not observing any
additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.
Here are my log settings from Kamailio.cfg
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
…
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")
*Sent:* Tuesday, August 26, 2014 3:54 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
I quickly looked over the modules that use shared memory and I
couldn't spot a commit related to any leak.
Have you had the chance to compile with MEMDBG?
Cheers,
Daniel
Do you think any of these modules possibly causing this issue?
-Tim
*Heenan, Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
(SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
We’re setting the memory allocation to 256 ( -m 256 –M 64 ).
I’ll see about recompiling with MEMDBG=1.
Regards,
- Tim
*Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much memory did
you allocate (-m parameter value)?
It will narrow down searches to see if there was anything
similar fixed since 4.0.4.
To troubleshoot easier, would be good to recompile with
MEMDBG=1, then the details of chunks in memory can be dumped
and analysed.
Cheers,
Daniel
Greetings,
I’m running into a slow memory leak on my kamailio 4.0.4
SIP proxies. I’m observing a steady increase in the memory
consumption until there is no more left. Kamailio then
ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no
more memory.
What would be the best way to debug this kind of a memory
leak? The proxy does not handle any registrations but does
route a fair amount of calls.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-08-27 23:56:15 UTC
Permalink
Raw Message
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Wednesday, August 27, 2014 5:22 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

The logs suggest some leaks in dispatcher module -- it allocates shared memory only for caching routing records. Few more questions I need to get the answers in order to troubleshoot further.

How many records do you have in dispatcher table? Is it in database or the text file?

How ofter do you reload? What is the command you use for reload?

Cheers,
Daniel
On 26/08/14 22:42, Heenan, Timothy Steven wrote:
I have attached is the output of "kamcmd cfg.set_now_int core mem_dump_shm 1" to this email. I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed.

Also, here is the output of "kamctl mi get_statistics shmem:"

shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768

Thanks!
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 10:46 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

It is not needed to see the memory operations logs. Just to dump the chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
On 26/08/14 17:34, Heenan, Timothy Steven wrote:
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

----------------------------------------------------------------------
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.
Daniel-Constantin Mierla
2014-08-28 10:02:50 UTC
Permalink
Raw Message
Looking at the logs, there are 1336 URIs, which seems ok given your
numbers, because the module keeps also the previous set of records
(needed because at reload time there can be kamailio worker processes
using the records -- perhaps we can improve a bit here, I will review
the current reload code a bit, since it was a contribution from another
developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the
minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was
kamailio running? Did you wait enough to notice the steady increase of
memory usage?

Cheers,
Daniel
Post by Heenan, Timothy Steven
Thank you for the help.
For dispatcher, I’m using a database that contains about 700 records.
kamctl dispatcher reload
Thanks,
-Tim
*Sent:* Wednesday, August 27, 2014 5:22 PM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
The logs suggest some leaks in dispatcher module -- it allocates
shared memory only for caching routing records. Few more questions I
need to get the answers in order to troubleshoot further.
How many records do you have in dispatcher table? Is it in database or the text file?
How ofter do you reload? What is the command you use for reload?
Cheers,
Daniel
I have attached is the output of “kamcmd cfg.set_now_int core
mem_dump_shm 1” to this email. I had to restart Kamailio to load
ctl and ctl_rpc, so let me know if I need to do another one after
some time passes and calls are processed.
Also, here is the output of “kamctl mi get_statistics shmem:”
shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768
Thanks!
-Tim
*Sent:* Tuesday, August 26, 2014 10:46 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
It is not needed to see the memory operations logs. Just to dump
the chunks in memory.
You would need ctl and ctl_rpc modules (iirc -- they should be in
kamcmd cfg.set_now_int core mem_dump_shm 1
Extract the logs of the dump from the shared memory and send them over.
Cheers,
Daniel
Thank you.
I have recompiled with MEMDBG, however I am not observing any
additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.
Here are my log settings from Kamailio.cfg
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
…
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")
*Sent:* Tuesday, August 26, 2014 3:54 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
I quickly looked over the modules that use shared memory and I
couldn't spot a commit related to any leak.
Have you had the chance to compile with MEMDBG?
Cheers,
Daniel
Do you think any of these modules possibly causing this issue?
-Tim
Of *Heenan, Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
We’re setting the memory allocation to 256 ( -m 256 –M 64 ).
I’ll see about recompiling with MEMDBG=1.
Regards,
- Tim
Of *Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much memory
did you allocate (-m parameter value)?
It will narrow down searches to see if there was anything
similar fixed since 4.0.4.
To troubleshoot easier, would be good to recompile with
MEMDBG=1, then the details of chunks in memory can be
dumped and analysed.
Cheers,
Daniel
Greetings,
I’m running into a slow memory leak on my kamailio
4.0.4 SIP proxies. I’m observing a steady increase in
the memory consumption until there is no more left.
no more memory.
What would be the best way to debug this kind of a
memory leak? The proxy does not handle any
registrations but does route a fair amount of calls.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the
original message and any attachments.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-09-03 20:27:34 UTC
Permalink
Raw Message
I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Wednesday, August 27, 2014 5:22 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

The logs suggest some leaks in dispatcher module -- it allocates shared memory only for caching routing records. Few more questions I need to get the answers in order to troubleshoot further.

How many records do you have in dispatcher table? Is it in database or the text file?

How ofter do you reload? What is the command you use for reload?

Cheers,
Daniel
On 26/08/14 22:42, Heenan, Timothy Steven wrote:
I have attached is the output of "kamcmd cfg.set_now_int core mem_dump_shm 1" to this email. I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed.

Also, here is the output of "kamctl mi get_statistics shmem:"

shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768

Thanks!
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 10:46 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

It is not needed to see the memory operations logs. Just to dump the chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
On 26/08/14 17:34, Heenan, Timothy Steven wrote:
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.




--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Heenan, Timothy Steven
2014-09-08 15:48:04 UTC
Permalink
Raw Message
After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing.

Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again.

shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org [mailto:sr-users-***@lists.sip-router.org] On Behalf Of Heenan, Timothy Steven
Sent: Wednesday, September 03, 2014 3:28 PM
To: 'miconda-***@public.gmane.org'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Wednesday, August 27, 2014 5:22 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

The logs suggest some leaks in dispatcher module -- it allocates shared memory only for caching routing records. Few more questions I need to get the answers in order to troubleshoot further.

How many records do you have in dispatcher table? Is it in database or the text file?

How ofter do you reload? What is the command you use for reload?

Cheers,
Daniel
On 26/08/14 22:42, Heenan, Timothy Steven wrote:
I have attached is the output of "kamcmd cfg.set_now_int core mem_dump_shm 1" to this email. I had to restart Kamailio to load ctl and ctl_rpc, so let me know if I need to do another one after some time passes and calls are processed.

Also, here is the output of "kamctl mi get_statistics shmem:"

shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768

Thanks!
-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 10:46 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

It is not needed to see the memory operations logs. Just to dump the chunks in memory.

You would need ctl and ctl_rpc modules (iirc -- they should be in the default config file), then run:

kamcmd cfg.set_now_int core mem_dump_shm 1

Extract the logs of the dump from the shared memory and send them over.

Send also the output for:

kamctl mi get_statistics shmem:

Cheers,
Daniel
On 26/08/14 17:34, Heenan, Timothy Steven wrote:
Thank you.

I have recompiled with MEMDBG, however I am not observing any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.

Here are my log settings from Kamailio.cfg

debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
...
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Tuesday, August 26, 2014 3:54 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

I quickly looked over the modules that use shared memory and I couldn't spot a commit related to any leak.

Have you had the chance to compile with MEMDBG?

Cheers,
Daniel
On 19/08/14 19:14, Heenan, Timothy Steven wrote:
Do you think any of these modules possibly causing this issue?

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Tuesday, August 12, 2014 2:37 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hi Daniel,

Here's a list of the modules I'm running:

mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace

We're setting the memory allocation to 256 ( -m 256 -M 64 ).

I'll see about recompiling with MEMDBG=1.

Regards,
- Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Daniel-Constantin Mierla
Sent: Tuesday, August 12, 2014 4:17 AM
To: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Hello,

can you give the list of loaded modules? How much memory did you allocate (-m parameter value)?

It will narrow down searches to see if there was anything similar fixed since 4.0.4.

To troubleshoot easier, would be good to recompile with MEMDBG=1, then the details of chunks in memory can be dumped and analysed.

Cheers,
Daniel
On 12/08/14 01:40, Heenan, Timothy Steven wrote:
Greetings,

I'm running into a slow memory leak on my kamailio 4.0.4 SIP proxies. I'm observing a steady increase in the memory consumption until there is no more left. Kamailio then starts repeating this in the logs:

ERROR: dispatcher [dispatch.c:279]: add_dest2list(): no more memory.

What would be the best way to debug this kind of a memory leak? The proxy does not handle any registrations but does route a fair amount of calls.



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.


--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
Daniel-Constantin Mierla
2014-09-08 16:11:20 UTC
Permalink
Raw Message
Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel
Post by Heenan, Timothy Steven
After restarting Kamailio on September 3rd, the problem has again
gotten to the point of no call processing.
Here are the memory statistics; additionally, I can see the virtual
memory has ballooned up to 400mb again.
shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792
-Tim
Timothy Steven
*Sent:* Wednesday, September 03, 2014 3:28 PM
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
I have about 460 destination sets. In each set I could have from 1 address up to 6.
When I took that initial memory log, Kamailio had been restarted
shortly beforehand. I just took another one after the system had
decayed so Kamailio was responding to Invites with “SIP/2.0 500 No
error (2/SL).” (exhibiting the issue after a few days) and the log is
about 400Mb! I’ll attach a small snippet.
Seems to be going on and on about the dispatcher module.
-Tim
*Sent:* Thursday, August 28, 2014 5:03 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Looking at the logs, there are 1336 URIs, which seems ok given your
numbers, because the module keeps also the previous set of records
(needed because at reload time there can be kamailio worker processes
using the records -- perhaps we can improve a bit here, I will review
the current reload code a bit, since it was a contribution from
another developer quite long time ago).
- how many destination sets (groups) do you have? Can you estimate the
minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was
kamailio running? Did you wait enough to notice the steady increase of
memory usage?
Cheers,
Daniel
Thank you for the help.
For dispatcher, I’m using a database that contains about 700 records.
kamctl dispatcher reload
Thanks,
-Tim
*Sent:* Wednesday, August 27, 2014 5:22 PM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
The logs suggest some leaks in dispatcher module -- it allocates
shared memory only for caching routing records. Few more questions
I need to get the answers in order to troubleshoot further.
How many records do you have in dispatcher table? Is it in
database or the text file?
How ofter do you reload? What is the command you use for reload?
Cheers,
Daniel
I have attached is the output of “kamcmd cfg.set_now_int core
mem_dump_shm 1” to this email. I had to restart Kamailio to
load ctl and ctl_rpc, so let me know if I need to do another
one after some time passes and calls are processed.
Also, here is the output of “kamctl mi get_statistics shmem:”
shmem:fragments = 398
shmem:free_size = 259170720
shmem:max_used_size = 9265184
shmem:real_used_size = 9264736
shmem:total_size = 268435456
shmem:used_size = 8559768
Thanks!
-Tim
*Sent:* Tuesday, August 26, 2014 10:46 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
It is not needed to see the memory operations logs. Just to
dump the chunks in memory.
You would need ctl and ctl_rpc modules (iirc -- they should be
kamcmd cfg.set_now_int core mem_dump_shm 1
Extract the logs of the dump from the shared memory and send them over.
Cheers,
Daniel
Thank you.
I have recompiled with MEMDBG, however I am not observing
any additional output in my log files.
I think I have something set incorrectly in kamailio.cfg.
Here are my log settings from Kamailio.cfg
debug=3
log_stderror=no
log_facility=LOG_LOCAL0
memdbg=2
memlog=2
…
modparam("debugger", "cfgtrace", 1)
modparam("debugger", "log_level", 2)
modparam("debugger", "cfgpkgcheck", 1)
modparam("debugger", "mod_level", "core=2")
modparam("debugger", "mod_level", "tm=2")
modparam("debugger", "mod_level", "dispatcher=2")
modparam("debugger", "mod_level", "siptrace=2")
*Sent:* Tuesday, August 26, 2014 3:54 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
I quickly looked over the modules that use shared memory
and I couldn't spot a commit related to any leak.
Have you had the chance to compile with MEMDBG?
Cheers,
Daniel
Do you think any of these modules possibly causing
this issue?
-Tim
Behalf Of *Heenan, Timothy Steven
*Sent:* Tuesday, August 12, 2014 2:37 PM
Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hi Daniel,
mi_fifo
db_mysql
sl
kex
tm
tmx
rr
xlog
maxfwd
usrloc
registrar
textops
pv
acc
permissions
siputils
auth
lcr
dispatcher
sanity
debugger
siptrace
We’re setting the memory allocation to 256 ( -m 256 –M
64 ).
I’ll see about recompiling with MEMDBG=1.
Regards,
- Tim
Behalf Of *Daniel-Constantin Mierla
*Sent:* Tuesday, August 12, 2014 4:17 AM
*To:* Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Hello,
can you give the list of loaded modules? How much
memory did you allocate (-m parameter value)?
It will narrow down searches to see if there was
anything similar fixed since 4.0.4.
To troubleshoot easier, would be good to recompile
with MEMDBG=1, then the details of chunks in memory
can be dumped and analysed.
Cheers,
Daniel
Greetings,
I’m running into a slow memory leak on my kamailio
4.0.4 SIP proxies. I’m observing a steady increase
in the memory consumption until there is no more
add_dest2list(): no more memory.
What would be the best way to debug this kind of a
memory leak? The proxy does not handle any
registrations but does route a fair amount of calls.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of
the original message and any attachments.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany
Daniel-Constantin Mierla
2014-09-11 14:59:11 UTC
Permalink
Raw Message
Short note to say that last log you sent had more leads, and apparently
there is a leak at least in 4.0.

I need a bit more time to look at it and check if still on master and
4.1 (hopefully event later today) - I will come back with an update
about it soon.

Daniel
Post by Daniel-Constantin Mierla
Were you taking the memory logs after the errors, before restarting?
I didn't get the chance to look over the previous email, I will do it soon.
Daniel
Post by Heenan, Timothy Steven
After restarting Kamailio on September 3rd, the problem has again
gotten to the point of no call processing.
Here are the memory statistics; additionally, I can see the virtual
memory has ballooned up to 400mb again.
shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792
-Tim
Timothy Steven
*Sent:* Wednesday, September 03, 2014 3:28 PM
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
I have about 460 destination sets. In each set I could have from 1 address up to 6.
When I took that initial memory log, Kamailio had been restarted
shortly beforehand. I just took another one after the system had
decayed so Kamailio was responding to Invites with “SIP/2.0 500 No
error (2/SL).” (exhibiting the issue after a few days) and the log is
about 400Mb! I’ll attach a small snippet.
Seems to be going on and on about the dispatcher module.
-Tim
*Sent:* Thursday, August 28, 2014 5:03 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Looking at the logs, there are 1336 URIs, which seems ok given your
numbers, because the module keeps also the previous set of records
(needed because at reload time there can be kamailio worker processes
using the records -- perhaps we can improve a bit here, I will review
the current reload code a bit, since it was a contribution from
another developer quite long time ago).
- how many destination sets (groups) do you have? Can you estimate
the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was
kamailio running? Did you wait enough to notice the steady increase
of memory usage?
Cheers,
Daniel
Thank you for the help.
For dispatcher, I’m using a database that contains about 700 records.
kamctl dispatcher reload
Thanks,
-Tim
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany
Heenan, Timothy Steven
2014-09-11 16:03:53 UTC
Permalink
Raw Message
Thanks for your help Daniel!

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, September 11, 2014 9:59 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Short note to say that last log you sent had more leads, and apparently there is a leak at least in 4.0.

I need a bit more time to look at it and check if still on master and 4.1 (hopefully event later today) - I will come back with an update about it soon.

Daniel
On 08/09/14 18:11, Daniel-Constantin Mierla wrote:
Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel
On 08/09/14 17:48, Heenan, Timothy Steven wrote:
After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing.

Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again.

shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Wednesday, September 03, 2014 3:28 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany

----------------------------------------------------------------------
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.
Daniel-Constantin Mierla
2014-09-11 20:44:57 UTC
Permalink
Raw Message
I pushed a patch to master branch, hopefully fixing the leak:

-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88c1d4aeb50d1e637697132ab0994dcdb28

Before backporting, I need to have it tested and be sure is not having
side effects.

For 4.0.x, if you installed from sources, either try to cherry pick the
commit via git or edit the file:

modules/dispatcher/dispatch.c

and replace existing function destroy_list(...) with:

void destroy_list(int list_id)
{
ds_set_t *sp = NULL;
ds_set_t *sp1 = NULL;
ds_dest_t *dest = NULL;

sp = ds_lists[list_id];

while(sp)
{
sp1 = sp->next;
for(dest = sp->dlist; dest!= NULL; dest=dest->next)
{
if(dest->uri.s!=NULL)
{
shm_free(dest->uri.s);
dest->uri.s = NULL;
}
}
if (sp->dlist != NULL)
shm_free(sp->dlist);
shm_free(sp);
sp = sp1;
}

ds_lists[list_id] = NULL;
}

Let me know the results and if all ok, I will backport.

Cheers,
Daniel
Post by Heenan, Timothy Steven
Thanks for your help Daniel!
-Tim
*Sent:* Thursday, September 11, 2014 9:59 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Short note to say that last log you sent had more leads, and
apparently there is a leak at least in 4.0.
I need a bit more time to look at it and check if still on master and
4.1 (hopefully event later today) - I will come back with an update
about it soon.
Daniel
Were you taking the memory logs after the errors, before restarting?
I didn't get the chance to look over the previous email, I will do it soon.
Daniel
After restarting Kamailio on September 3rd, the problem has
again gotten to the point of no call processing.
Here are the memory statistics; additionally, I can see the
virtual memory has ballooned up to 400mb again.
shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792
-Tim
*Heenan, Timothy Steven
*Sent:* Wednesday, September 03, 2014 3:28 PM
(SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
I have about 460 destination sets. In each set I could have
from 1 address up to 6.
When I took that initial memory log, Kamailio had been
restarted shortly beforehand. I just took another one after
the system had decayed so Kamailio was responding to Invites
with “SIP/2.0 500 No error (2/SL).” (exhibiting the issue
after a few days) and the log is about 400Mb! I’ll attach a
small snippet.
Seems to be going on and on about the dispatcher module.
-Tim
*Sent:* Thursday, August 28, 2014 5:03 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Looking at the logs, there are 1336 URIs, which seems ok given
your numbers, because the module keeps also the previous set
of records (needed because at reload time there can be
kamailio worker processes using the records -- perhaps we can
improve a bit here, I will review the current reload code a
bit, since it was a contribution from another developer quite
long time ago).
- how many destination sets (groups) do you have? Can you
estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how
long was kamailio running? Did you wait enough to notice the
steady increase of memory usage?
Cheers,
Daniel
Thank you for the help.
For dispatcher, I’m using a database that contains about
700 records.
A reload is performed via cronjob every 5 minutes. The
kamctl dispatcher reload
Thanks,
-Tim
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the
original message and any attachments.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany
Heenan, Timothy Steven
2014-09-15 15:49:55 UTC
Permalink
Raw Message
Thank you!

I will test this on my 4.0.4 setup and report back how it works.

Thank you,

Tim Heenan
Engineer I - VoIP Wholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116



From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, September 11, 2014 3:45 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I pushed a patch to master branch, hopefully fixing the leak:

- http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88c1d4aeb50d1e637697132ab0994dcdb28

Before backporting, I need to have it tested and be sure is not having side effects.

For 4.0.x, if you installed from sources, either try to cherry pick the commit via git or edit the file:

modules/dispatcher/dispatch.c

and replace existing function destroy_list(...) with:

void destroy_list(int list_id)
{
ds_set_t *sp = NULL;
ds_set_t *sp1 = NULL;
ds_dest_t *dest = NULL;

sp = ds_lists[list_id];

while(sp)
{
sp1 = sp->next;
for(dest = sp->dlist; dest!= NULL; dest=dest->next)
{
if(dest->uri.s!=NULL)
{
shm_free(dest->uri.s);
dest->uri.s = NULL;
}
}
if (sp->dlist != NULL)
shm_free(sp->dlist);
shm_free(sp);
sp = sp1;
}

ds_lists[list_id] = NULL;
}

Let me know the results and if all ok, I will backport.

Cheers,
Daniel
On 11/09/14 18:03, Heenan, Timothy Steven wrote:
Thanks for your help Daniel!

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, September 11, 2014 9:59 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Short note to say that last log you sent had more leads, and apparently there is a leak at least in 4.0.

I need a bit more time to look at it and check if still on master and 4.1 (hopefully event later today) - I will come back with an update about it soon.

Daniel
On 08/09/14 18:11, Daniel-Constantin Mierla wrote:
Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel
On 08/09/14 17:48, Heenan, Timothy Steven wrote:
After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing.

Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again.

shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Wednesday, September 03, 2014 3:28 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim




--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany
Heenan, Timothy Steven
2014-10-14 20:43:42 UTC
Permalink
Raw Message
I've tested this patch pretty thoroughly. Afterwards, I've pushed it to our production environment and have not seen the problem recur.
This has fixed the problem!

Thanks for the patch Daniel, I would definitely support its inclusion in a backport for Kamailio 4.0.X .

Thanks!
-Tim


From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org [mailto:sr-users-***@lists.sip-router.org] On Behalf Of Heenan, Timothy Steven
Sent: Monday, September 15, 2014 10:50 AM
To: 'miconda-***@public.gmane.org'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Thank you!

I will test this on my 4.0.4 setup and report back how it works.

Thank you,

Tim Heenan
Engineer I - VoIP Wholesale | Windstream
timothy.heenan-***@public.gmane.org<mailto:email-***@public.gmane.org> | windstreambusiness.com<http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116



From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, September 11, 2014 3:45 PM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I pushed a patch to master branch, hopefully fixing the leak:

- http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88c1d4aeb50d1e637697132ab0994dcdb28

Before backporting, I need to have it tested and be sure is not having side effects.

For 4.0.x, if you installed from sources, either try to cherry pick the commit via git or edit the file:

modules/dispatcher/dispatch.c

and replace existing function destroy_list(...) with:

void destroy_list(int list_id)
{
ds_set_t *sp = NULL;
ds_set_t *sp1 = NULL;
ds_dest_t *dest = NULL;

sp = ds_lists[list_id];

while(sp)
{
sp1 = sp->next;
for(dest = sp->dlist; dest!= NULL; dest=dest->next)
{
if(dest->uri.s!=NULL)
{
shm_free(dest->uri.s);
dest->uri.s = NULL;
}
}
if (sp->dlist != NULL)
shm_free(sp->dlist);
shm_free(sp);
sp = sp1;
}

ds_lists[list_id] = NULL;
}

Let me know the results and if all ok, I will backport.

Cheers,
Daniel
On 11/09/14 18:03, Heenan, Timothy Steven wrote:
Thanks for your help Daniel!

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, September 11, 2014 9:59 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Short note to say that last log you sent had more leads, and apparently there is a leak at least in 4.0.

I need a bit more time to look at it and check if still on master and 4.1 (hopefully event later today) - I will come back with an update about it soon.

Daniel
On 08/09/14 18:11, Daniel-Constantin Mierla wrote:
Were you taking the memory logs after the errors, before restarting?

I didn't get the chance to look over the previous email, I will do it soon.

Daniel
On 08/09/14 17:48, Heenan, Timothy Steven wrote:
After restarting Kamailio on September 3rd, the problem has again gotten to the point of no call processing.

Here are the memory statistics; additionally, I can see the virtual memory has ballooned up to 400mb again.

shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792

-Tim

From: sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org<mailto:sr-users-***@lists.sip-router.org> [mailto:sr-users-bounces-cR8azDVoa3IcDhw6gZKtMWD2FQJk+8+***@public.gmane.org] On Behalf Of Heenan, Timothy Steven
Sent: Wednesday, September 03, 2014 3:28 PM
To: 'miconda-***@public.gmane.org<mailto:miconda-***@public.gmane.org>'; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

I have about 460 destination sets. In each set I could have from 1 address up to 6.

When I took that initial memory log, Kamailio had been restarted shortly beforehand. I just took another one after the system had decayed so Kamailio was responding to Invites with "SIP/2.0 500 No error (2/SL)." (exhibiting the issue after a few days) and the log is about 400Mb! I'll attach a small snippet.
Seems to be going on and on about the dispatcher module.

-Tim

From: Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
Sent: Thursday, August 28, 2014 5:03 AM
To: Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] Kamailio 4.0.4 slow memory leak

Looking at the logs, there are 1336 URIs, which seems ok given your numbers, because the module keeps also the previous set of records (needed because at reload time there can be kamailio worker processes using the records -- perhaps we can improve a bit here, I will review the current reload code a bit, since it was a contribution from another developer quite long time ago).

Some more questions:
- how many destination sets (groups) do you have? Can you estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for how long was kamailio running? Did you wait enough to notice the steady increase of memory usage?

Cheers,
Daniel
On 28/08/14 01:56, Heenan, Timothy Steven wrote:
Thank you for the help.

For dispatcher, I'm using a database that contains about 700 records.

A reload is performed via cronjob every 5 minutes. The command being used is:

kamctl dispatcher reload

Thanks,
-Tim



--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany

________________________________
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.


--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Next Kamailio Advanced Trainings 2014 - http://www.asipto.com

Sep 22-25, Berlin, Germany
Daniel-Constantin Mierla
2014-10-15 10:00:50 UTC
Permalink
Raw Message
Thanks for reporting back. After 4.2.0 release I will try to get the
time to release another 4.0.x version, to include this fix as well.

Cheers,
Daniel
I’ve tested this patch pretty thoroughly. Afterwards, I’ve pushed it
to our production environment and have not seen the problem recur.
*This has fixed the problem! *
Thanks for the patch Daniel, I would definitely support its inclusion
in a backport for Kamailio 4.0.X .
Thanks!
-Tim
Timothy Steven
*Sent:* Monday, September 15, 2014 10:50 AM
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Thank you!
I will test this on my 4.0.4 setup and report back how it works.
Thank you,
*Tim Heenan*
*Engineer I - VoIP**Wholesale | Windstream*
windstreambusiness.com <http://www.windstreambusiness.com/>
o: 847.348.1338 | f: 847.963.0116
*Sent:* Thursday, September 11, 2014 3:45 PM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=7fb8c88c1d4aeb50d1e637697132ab0994dcdb28
Before backporting, I need to have it tested and be sure is not having side effects.
For 4.0.x, if you installed from sources, either try to cherry pick
modules/dispatcher/dispatch.c
void destroy_list(int list_id)
{
ds_set_t *sp = NULL;
ds_set_t *sp1 = NULL;
ds_dest_t *dest = NULL;
sp = ds_lists[list_id];
while(sp)
{
sp1 = sp->next;
for(dest = sp->dlist; dest!= NULL; dest=dest->next)
{
if(dest->uri.s!=NULL)
{
shm_free(dest->uri.s);
dest->uri.s = NULL;
}
}
if (sp->dlist != NULL)
shm_free(sp->dlist);
shm_free(sp);
sp = sp1;
}
ds_lists[list_id] = NULL;
}
Let me know the results and if all ok, I will backport.
Cheers,
Daniel
Thanks for your help Daniel!
-Tim
*Sent:* Thursday, September 11, 2014 9:59 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Short note to say that last log you sent had more leads, and
apparently there is a leak at least in 4.0.
I need a bit more time to look at it and check if still on master
and 4.1 (hopefully event later today) - I will come back with an
update about it soon.
Daniel
Were you taking the memory logs after the errors, before restarting?
I didn't get the chance to look over the previous email, I will do it soon.
Daniel
After restarting Kamailio on September 3rd, the problem
has again gotten to the point of no call processing.
Here are the memory statistics; additionally, I can see
the virtual memory has ballooned up to 400mb again.
shmem:fragments = 2378
shmem:free_size = 209840
shmem:max_used_size = 268393216
shmem:real_used_size = 268225616
shmem:total_size = 268435456
shmem:used_size = 220534792
-Tim
Of *Heenan, Timothy Steven
*Sent:* Wednesday, September 03, 2014 3:28 PM
Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
I have about 460 destination sets. In each set I could
have from 1 address up to 6.
When I took that initial memory log, Kamailio had been
restarted shortly beforehand. I just took another one
after the system had decayed so Kamailio was responding to
Invites with “SIP/2.0 500 No error (2/SL).” (exhibiting
the issue after a few days) and the log is about 400Mb!
I’ll attach a small snippet.
Seems to be going on and on about the dispatcher module.
-Tim
*Sent:* Thursday, August 28, 2014 5:03 AM
*To:* Heenan, Timothy Steven; Kamailio (SER) - Users Mailing List
*Subject:* Re: [SR-Users] Kamailio 4.0.4 slow memory leak
Looking at the logs, there are 1336 URIs, which seems ok
given your numbers, because the module keeps also the
previous set of records (needed because at reload time
there can be kamailio worker processes using the records
-- perhaps we can improve a bit here, I will review the
current reload code a bit, since it was a contribution
from another developer quite long time ago).
- how many destination sets (groups) do you have? Can you
estimate the minimum and maximum addresses in a set?
- when did you take the memory log? In other words, for
how long was kamailio running? Did you wait enough to
notice the steady increase of memory usage?
Cheers,
Daniel
Thank you for the help.
For dispatcher, I’m using a database that contains
about 700 records.
A reload is performed via cronjob every 5 minutes. The
kamctl dispatcher reload
Thanks,
-Tim
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany
------------------------------------------------------------------------
This email message and any attachments are for the sole use of the
intended recipient(s). Any unauthorized review, use, disclosure or
distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of
the original message and any attachments.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Loading...