[SR-Users] Crash Kamailio 4.1.4

Discussion:

Igor Potjevlesch

2014-06-25 12:49:29 UTC

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

Kamailio crashed again this afternoon.

here an extract from /var/log/messages :
Jun 25 13:49:01 /usr/local/sbin/kamailio[20259]: WARNING: <script>:
time=[Wed Jun 25 13:49:01 2014] call
id=[***@A.B.C.D] call seq=[929405] contact
header=[<sip:***@A.B.C.D:2057;transport=UDP>] from
uri=[sip:***@domain;user=phone] from tag=[16632949] request's
method=[INVITE] request's uri=[sip:***@domain;user=phone] to
uri=[sip:***@domain;user=phone] to tag=[<null>] sip message
id=[46275] process id=[20259] ip source=[A.B.C.D] flags=[2]
ua=[(innovaphone IP6010/9.00 hotfix24 [9.061271/9061271/300])], INVITE from
'untrusted' host
Jun 25 13:49:01 /usr/local/sbin/kamailio[20259]: WARNING: <script>:
time=[Wed Jun 25 13:49:01 2014] call
id=[***@A.B.C.D] call seq=[929405] contact
header=[<sip:***@A.B.C.D:2057;transport=UDP>] from
uri=[sip:***@domain;user=phone] from tag=[16632949] request's
method=[INVITE] request's uri=[sip:***@domain;user=phone] to
uri=[sip:***@domain;user=phone] to tag=[<null>] sip message
id=[46275] process id=[20259] ip source=[A.B.C.D] flags=[2], INVITE from an
authorized SIP trunk
Jun 25 13:49:01 /usr/local/sbin/kamailio[20259]: WARNING: <script>:
time=[Wed Jun 25 13:49:01 2014] call
id=[***@A.B.C.D] call seq=[929405] contact
header=[<sip:***@A.B.C.D:2057;transport=UDP>] from
uri=[sip:***@domain;user=phone] from tag=[16632949] request's
method=[INVITE] request's uri=[sip:***@domain;user=phone] to
uri=[sip:***@domain;user=phone] to tag=[<null>] sip message
id=[46275] process id=[20259] ip source=[A.B.C.D] flags=[2], INVITE from an
authorized SDA for current SIP trunk
Jun 25 13:49:01 /usr/local/sbin/kamailio[20259]: INFO: carrierroute
[cr_func.c:710]: cr_do_route(): uri 0123456789 was rewritten to
sip:***@GW, carrier 1, domain 1
Jun 25 13:49:02 /usr/local/sbin/kamailio[20259]: : <core>
[mem/q_malloc.c:140]: qm_debug_frag(): BUG: qm_*: fragm. 0x7f12803cb450
(address 0x7f12803cb480) beginning overwritten(abcdefed)!

And this is the btfull :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at
forward.c:777
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
#16 0x000000000046d42b in main_loop () at main.c:1617
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

It seems to be the same problem but in a different source. Can you help me?

Regards,

Igor.

Hello,
we didn't set the early media parameter . its '0' by default, isn't it?
regards,
Igor
Hello,

if you get a record for 180 response, then you have also the early_media
parameter set for acc module, isn't it?
In the morning I pushed a patch that should fix this issue. Use latest
release 4.1.4 and see if works fine. Report the results to know that it is
gone or not.
Cheers,
Daniel
Hello,
We don't use $ai in xlog nor in other process. only in ACC.
modparam("acc", "db_flag", FLT_ACC)
modparam("acc", "db_missed_flag", 3)
modparam("acc", "db_url", DBURLWO)
modparam("acc", "db_extra",
"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$ai;"
"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")
For the 3781-4b1-572014182635-OGNAJ-1-A.B.C.D there is a code 180
ringing in the INVITE in ACC table.
regards,
Igor

Few more things...
Are you recording 1xx events? Can you check to see if there is another
record in acc table for the same call? You can search by call-id
3781-4b1-572014182635-OGNAJ-1-A.B.C.D
Eventually, send also the parameters you set for acc module.
Cheers,
Daniel
Hello,
so you don't print $ai in xlog() statements nor use it in any
assignments or other functions besides acc parameter?
Cheers,
Daniel
Hello,
We do not access to the P-asserted-identity in our configuration but
we added the field PAI in the db base ACC ( for INVITE, ACK and BYE) .
I dont know if it's in request_route, failure_route or branch_route .
(gdb) p mem_block
$3 = (struct qm_block *) 0x7f6a6bef1010
(gdb) p shm_block
$4 = (struct qm_block *) 0x7f6a5666a000
Regards,
Igor

Hello,
cloning to shm for tm seems ok. Can you tell where you access
P-Asserted-Identity header, via variables? Does it happen in request_route,
failure_route or branch_route?
p mem_block
p shm_block
I want to see if parsed filed point to shm or pkg memory.
Cheers,
Daniel
Hello,
at least I narrowed it down a bit. It is empty also in the clone stored
in transaction, so it happens either during cloning or before. I will have
to check these parts.
Cheers,
Daniel
Hello,
(gdb) p *t->uas.request->pai
$1 = {type = HDR_PAI_T, name = {
s = 0x7f6a60cd34b8 "P-Asserted-Identity: \"0987654321\"
INVITE, BYE, REGISTER, ACK, OPTIONS, CANCEL, SUBSCRIBE, NOTIFY, INFO,
REFER, UPD"..., len = 19}, body = {
path,"..., len = 43}, len = 66, parsed = 0x7f6a6d81da88, next =
0x7f6a60cd3f10}
(gdb) p *((p_id_body_t*)(t->uas.request->pai->parsed))
$2 = {id = 0x0, num_ids = 0, next = 0x0}
*Did *you find one thing in common between the 2 occurrences? Do you
have any ideas about what is the cause of this pai reset?
Regards,
Igor

Hello,
p *t->uas.request->pai
p *((p_id_body_t*)(t->uas.request->pai->parsed))
Cheers,
Daniel
Hello,
(gdb) frame 5
#5 0x00007f6a687e9b43 in acc_onreply (t=0x7f6a60d16ff8,
req=0x7f6a60cd2c10, reply=0x7f6a6c119aa8, code=200) at acc_logic.c:471
471 acc_db_request(req);
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-06-25 16:26:05 UTC

Permalink

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-06-26 10:58:34 UTC

Permalink

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity:
<sip:***@domain;user=phone>\r\nP-Sig-Options:
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
s = 0x7f1274c3b6e2 "<sip:***@domain;user=phone>\r\nP-Sig-Options:
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
new_uri_bk = {s = 0x7f1274b53cdf "sip:***@GW
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia:
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
***@A.B.C.D\r\nFrom:
<sip:***@domain;user=phone>;epid=00903"...,
len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
***@A.B.C.D\r\nFrom: <sip:***@domain
;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len
= 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at
forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia:
SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env
= {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816,
12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID:
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0,
0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8
= '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about
km_val.c : db_mysql_val2str

Post by Igor Potjevlesch
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-06-26 12:06:28 UTC

Permalink

Hello,

here is the new bt full Kamailio 's crash but about km_val.c :
db_mysql_val2str

#0 0x00000030f2230f30 in escape_string_for_mysql () from
/usr/lib64/mysql/libmysqlclient.so.16
No symbol table info available.
#1 0x00000030f22269c1 in mysql_real_escape_string () from
/usr/lib64/mysql/libmysqlclient.so.16
No symbol table info available.
#2 0x00007fde5dde8089 in db_mysql_val2str (_c=0x7fde5e210050,
_v=0x7fde5aafef60, _s=0x2808bef
"'A.B.C.D','','0987654321','0987654321','A.B.C.D')", _len=0x7fff488532e4)
at km_val.c:79
l = 10
tmp = 1
old_s = 0x2808bee
"''A.B.C.D','','0987654321','0987654321','A.B.C.D')"
__FUNCTION__ = "db_mysql_val2str"
#3 0x00007fde5d9be137 in db_print_values (_c=0x7fde5e210050,
_b=0x2808b0c
"'INVITE','as687ade58','a94c095b773be1dd6e8d668a785a9c848e3dcf3f','***@domain','200','OK','2014-06-26
10:00:27','0123456789','***@domain','domain"..., _l=65379,
_v=0x7fde5aafee00, _n=15,
val2str=0x7fde5dde7c88 <db_mysql_val2str>) at db_ut.c:318
i = 11
l = 65153
len = 226
__FUNCTION__ = "db_print_values"
#4 0x00007fde5d9b7028 in db_do_insert_cmd (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15, val2str=0x7fde5dde7c88
<db_mysql_val2str>, submit_query=0x7fde5dddf092 <db_mysql_submit_query>,
mode=0) at db_query.c:224
off = 156
ret = 10
__FUNCTION__ = "db_do_insert_cmd"
#5 0x00007fde5d9b73b9 in db_do_insert (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15, val2str=0x7fde5dde7c88
<db_mysql_val2str>, submit_query=0x7fde5dddf092 <db_mysql_submit_query>) at
db_query.c:249
No locals.
#6 0x00007fde5dde1419 in db_mysql_insert (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15) at km_dbase.c:415
No locals.
#7 0x00007fde5a8e8783 in acc_db_request (rq=0x7fde529d7148) at acc.c:492
m = 15
n = 1386049864
i = 15
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#8 0x00007fde5a8f2bc8 in acc_onreply (t=0x7fde52905780,
req=0x7fde529d7148, reply=0x7fde5e22bd70, code=200) at acc_logic.c:471
new_uri_bk = {s = 0x7fde529fa5c7 "sip:***@GW
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK853d.4ce5cf98739172086bed0d898d48aa38.0\r\nVia:
SIP/2.0/UDP A.B.C.D:5060;branch=z9hG4bK3d"..., len = 19}
br = 0
hdr = 0x7fde5a8f9c20
__FUNCTION__ = "acc_onreply"
#9 0x00007fde5a8f330a in tmcb_func (t=0x7fde52905780, type=512,
ps=0x7fff48853650) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#10 0x00007fde5caed478 in run_trans_callbacks_internal
(cb_lst=0x7fde529057f0, type=512, trans=0x7fde52905780,
params=0x7fff48853650) at t_hooks.c:290
cbp = 0x7fde51d50f70
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#11 0x00007fde5caed68a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7fde52905840, req=0x7fde529d7148, repl=0x7fde5e22bd70, flags=200) at
t_hooks.c:336
params = {req = 0x7fde529d7148, rpl = 0x7fde5e22bd70, param =
0x7fde51d50f80, code = 200, flags = 200, branch = 0, t_rbuf =
0x7fde52905840, dst = 0x7fde52905890, send_buf = {
s = 0x7fde4fc34b28 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D:5060;branch=z9hG4bK3dbe6573;rport=5060\r\nCall-ID:
***@domain\r\nFrom: \"0123456789\"
<sip:***@domain>;ta"..., len = 924}}
trans = 0x7fde52905780
#12 0x00007fde5cb1fc06 in relay_reply (t=0x7fde52905780,
p_msg=0x7fde5e22bd70, branch=0, msg_status=200, cancel_data=0x7fff488539b0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7fde5e2350d0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D:5060;branch=z9hG4bK3dbe6573;rport=5060\r\nCall-ID:
***@domain\r\nFrom: \"0123456789\"
<sip:***@domain>;ta"...
res_len = 924
relayed_code = 200
relayed_msg = 0x7fde5e22bd70
reply_bak = 0x7fff48853800
bm = {to_tag_val = {s = 0x7fde52906d18 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7fde52905840
to_tag = 0x0
reason = {s = 0x1488537b0 <Address 0x1488537b0 out of bounds>, len
= 1}
onsend_params = {req = 0x200924a03, rpl = 0x7fde5cb3ff90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7fde48591828, dst =
0x7fde5e22bf68, send_buf = {s = 0x488537d0 <Address 0x488537d0 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"qui
#13 0x00007fde5cb220b7 in reply_received (p_msg=0x7fde5e22bd70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = 1579339872
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7fde529058e8
t = 0x7fde52905780
lack_dst = {send_sock = 0x7fde5e234bf8, to = {s = {sa_family =
40976, sa_data = "\377]\336\177\000\000\310##\000\000\000\000"}, sin =
{sin_family = 40976, sin_port = 24063, sin_addr = {s_addr = 32734},
sin_zero = "\310##\000\000\000\000"}, sin6 = {
sin6_family = 40976, sin6_port = 24063, sin6_flowinfo =
32734, sin6_addr = {__in6_u = {__u6_addr8 =
"\310##\000\000\000\000\000\bP\026^\336\177\000", __u6_addr16 = {9160, 35,
0, 0, 20488, 24086, 32734, 0}, __u6_addr32 = {2302920, 0, 1578520584,
32734}}},
sin6_scope_id = 1577039928}}, id = 32734, proto = -24 '\350',
send_flags = {f = 59 ';', blst_imask = 35 '#'}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1216690800
blst_503_timeout = 32767
hf = 0x7fde5e22bd90
onsend_params = {req = 0x7fff48853a30, rpl = 0x550b94, param =
0x232288, code = 0, flags = 3, branch = 0, t_rbuf = 0x7fde5dffa380, dst =
0x7fde5e233bb8, send_buf = {s = 0x7fff48853a30 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {140593038802792, -2282156308276021403, 4279488,
140734410080352, 0, 0, -2282156308296992923, 2281837801139107685},
__mask_was_saved = 0, __saved_mask = {__val = {140593037933136,
9586941, 9586263, 140734410078528, 140593038807136,
9587203, 9586197, 361695345073193192, 9586295, 9586274, 1579339824,
140593038807136, 140593037965344, 140593038802792, 4279488,
140734410080352}}}}}
__FUNCTION__ = "reply_received"
#14 0x000000000045d837 in do_forward_reply (msg=0x7fde5e22bd70, mode=0) at
forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32734
r = 1
s = 0x4035e22bd78 <Address 0x4035e22bd78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#15 0x000000000045e0f8 in forward_reply (msg=0x7fde5e22bd70) at
forward.c:860
No locals.
#16 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia:
SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK853d.4ce5cf98739172086bed0d898d48aa38.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:5060;branch=z9hG4bK3dbe6573;rport=5060\r\nCall-ID:
077"...,
len=1027, rcv_info=0x7fff48853d30) at receive.c:273
msg = 0x7fde5e22bd70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env
= {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169080759064, 9586112},
__mask_was_saved = 1216691512, __saved_mask = {__val = {140593036756896,
12884901898, 140593036756896, 4279488,
140734410080352, 140734410079408, 5477954, 0,
140592673265024, 50195, 168717522304, 9586112, 140734410079536,
140734410079456, 5474789, 4279488}}}}}
ret = 32734
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK853d.4ce5cf98739172086bed0d898d48aa38.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:5060;branch=z9hG4bK3dbe6573;rport=5060\r\nCall-ID:
077"..., len = 1027}
__FUNCTION__ = "receive_msg"
#17 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1027
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
A.B.C.D;branch=z9hG4bK853d.4ce5cf98739172086bed0d898d48aa38.0;received=A.B.C.D\r\nVia:
SIP/2.0/UDP A.B.C.D:5060;branch=z9hG4bK3dbe6573;rport=5060\r\nCall-ID:
0777"...
tmp = 0x9245c0 "10.143.1.2"
from = 0x7fde5e168f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
140593036756896}, addr32 = {403182777, 0, 1577289632, 32734}, addr16 =
{5305, 6152, 0, 0, 34720, 24067, 32734, 0}, addr =
"\271\024\b\030\000\000\000\000\240\207\003^\336\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0,
0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8
= '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7fde5e038638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#18 0x000000000046d42b in main_loop () at main.c:1617
i = 5
pid = 0
si = 0x7fde5e038638
si_desc = "udp receiver child=5
sock=A.B.C.D:5060\000\035^\336\177\000\000\000\326\v^\336\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000i\223k_\000\000\000\000\300LA\000\000\000\000\000`@\205H\377\177",
'\000' <repeats 18 times>"\240,

\205H\377\177\000\000\364\244K\000\000\000\000"

nrprocs = 15
#19 0x0000000000470533 in main (argc=7, argv=0x7fff48854068) at main.c:2545
cfg_stream = 0x27f9010
c = -1
r = 0
tmp = 0x7fff48854f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3406612612
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

Regards,

Igor.

Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
len = 1021}}
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len
= 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at
forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env
= {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816,
12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0,
0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but about
km_val.c : db_mysql_val2str
Hello,

Post by Daniel-Constantin Mierla
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-06-30 09:37:36 UTC

Permalink

Hello,

The second problem occured again today. Do you have any idea about the
cause of this crash?
the problem seems to be caused by quote before values in an DB
insertion like ''A.B.C.D' for Ip(on thursday) or 123456' for Id(today)

Regards,

Igor.

Post by Igor Potjevlesch
Hello,
db_mysql_val2str
#0 0x00000030f2230f30 in escape_string_for_mysql () from
/usr/lib64/mysql/libmysqlclient.so.16
No symbol table info available.
#1 0x00000030f22269c1 in mysql_real_escape_string () from
/usr/lib64/mysql/libmysqlclient.so.16
No symbol table info available.
#2 0x00007fde5dde8089 in db_mysql_val2str (_c=0x7fde5e210050,
_v=0x7fde5aafef60, _s=0x2808bef
"'A.B.C.D','','0987654321','0987654321','A.B.C.D')", _len=0x7fff488532e4)
at km_val.c:79
l = 10
tmp = 1
old_s = 0x2808bee
"''A.B.C.D','','0987654321','0987654321','A.B.C.D')"
__FUNCTION__ = "db_mysql_val2str"
#3 0x00007fde5d9be137 in db_print_values (_c=0x7fde5e210050,
_b=0x2808b0c
_v=0x7fde5aafee00, _n=15,
val2str=0x7fde5dde7c88 <db_mysql_val2str>) at db_ut.c:318
i = 11
l = 65153
len = 226
__FUNCTION__ = "db_print_values"
#4 0x00007fde5d9b7028 in db_do_insert_cmd (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15, val2str=0x7fde5dde7c88
<db_mysql_val2str>, submit_query=0x7fde5dddf092 <db_mysql_submit_query>,
mode=0) at db_query.c:224
off = 156
ret = 10
__FUNCTION__ = "db_do_insert_cmd"
#5 0x00007fde5d9b73b9 in db_do_insert (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15, val2str=0x7fde5dde7c88
<db_mysql_val2str>, submit_query=0x7fde5dddf092 <db_mysql_submit_query>) at
db_query.c:249
No locals.
#6 0x00007fde5dde1419 in db_mysql_insert (_h=0x7fde5e210050,
_k=0x7fde5aafeb20, _v=0x7fde5aafee00, _n=15) at km_dbase.c:415
No locals.
#7 0x00007fde5a8e8783 in acc_db_request (rq=0x7fde529d7148) at acc.c:492
m = 15
n = 1386049864
i = 15
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#8 0x00007fde5a8f2bc8 in acc_onreply (t=0x7fde52905780,
req=0x7fde529d7148, reply=0x7fde5e22bd70, code=200) at acc_logic.c:471
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:5060;branch=z9hG4bK3d"..., len = 19}
br = 0
hdr = 0x7fde5a8f9c20
__FUNCTION__ = "acc_onreply"
#9 0x00007fde5a8f330a in tmcb_func (t=0x7fde52905780, type=512,
ps=0x7fff48853650) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#10 0x00007fde5caed478 in run_trans_callbacks_internal
(cb_lst=0x7fde529057f0, type=512, trans=0x7fde52905780,
params=0x7fff48853650) at t_hooks.c:290
cbp = 0x7fde51d50f70
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#11 0x00007fde5caed68a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7fde52905840, req=0x7fde529d7148, repl=0x7fde5e22bd70, flags=200) at
t_hooks.c:336
params = {req = 0x7fde529d7148, rpl = 0x7fde5e22bd70, param =
0x7fde51d50f80, code = 200, flags = 200, branch = 0, t_rbuf =
0x7fde52905840, dst = 0x7fde52905890, send_buf = {
s = 0x7fde4fc34b28 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7fde52905780
#12 0x00007fde5cb1fc06 in relay_reply (t=0x7fde52905780,
p_msg=0x7fde5e22bd70, branch=0, msg_status=200, cancel_data=0x7fff488539b0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7fde5e2350d0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 924
relayed_code = 200
relayed_msg = 0x7fde5e22bd70
reply_bak = 0x7fff48853800
bm = {to_tag_val = {s = 0x7fde52906d18 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7fde52905840
to_tag = 0x0
reason = {s = 0x1488537b0 <Address 0x1488537b0 out of bounds>, len
= 1}
onsend_params = {req = 0x200924a03, rpl = 0x7fde5cb3ff90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7fde48591828, dst =
0x7fde5e22bf68, send_buf = {s = 0x488537d0 <Address 0x488537d0 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"qui
#13 0x00007fde5cb220b7 in reply_received (p_msg=0x7fde5e22bd70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = 1579339872
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7fde529058e8
t = 0x7fde52905780
lack_dst = {send_sock = 0x7fde5e234bf8, to = {s = {sa_family =
40976, sa_data = "\377]\336\177\000\000\310##\000\000\000\000"}, sin =
{sin_family = 40976, sin_port = 24063, sin_addr = {s_addr = 32734},
sin_zero = "\310##\000\000\000\000"}, sin6 = {
sin6_family = 40976, sin6_port = 24063, sin6_flowinfo =
32734, sin6_addr = {__in6_u = {__u6_addr8 =
"\310##\000\000\000\000\000\bP\026^\336\177\000", __u6_addr16 = {9160, 35,
0, 0, 20488, 24086, 32734, 0}, __u6_addr32 = {2302920, 0, 1578520584,
32734}}},
sin6_scope_id = 1577039928}}, id = 32734, proto = -24
'\350', send_flags = {f = 59 ';', blst_imask = 35 '#'}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1216690800
blst_503_timeout = 32767
hf = 0x7fde5e22bd90
onsend_params = {req = 0x7fff48853a30, rpl = 0x550b94, param =
0x232288, code = 0, flags = 3, branch = 0, t_rbuf = 0x7fde5dffa380, dst =
0x7fde5e233bb8, send_buf = {s = 0x7fff48853a30 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {140593038802792, -2282156308276021403, 4279488,
140734410080352, 0, 0, -2282156308296992923, 2281837801139107685},
__mask_was_saved = 0, __saved_mask = {__val = {140593037933136,
9586941, 9586263, 140734410078528, 140593038807136,
9587203, 9586197, 361695345073193192, 9586295, 9586274, 1579339824,
140593038807136, 140593037965344, 140593038802792, 4279488,
140734410080352}}}}}
__FUNCTION__ = "reply_received"
#14 0x000000000045d837 in do_forward_reply (msg=0x7fde5e22bd70, mode=0) at
forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32734
r = 1
s = 0x4035e22bd78 <Address 0x4035e22bd78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#15 0x000000000045e0f8 in forward_reply (msg=0x7fde5e22bd70) at
forward.c:860
No locals.
#16 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
077"...,
len=1027, rcv_info=0x7fff48853d30) at receive.c:273
msg = 0x7fde5e22bd70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env
= {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169080759064, 9586112},
__mask_was_saved = 1216691512, __saved_mask = {__val = {140593036756896,
12884901898, 140593036756896, 4279488,
140734410080352, 140734410079408, 5477954, 0,
140592673265024, 50195, 168717522304, 9586112, 140734410079536,
140734410079456, 5474789, 4279488}}}}}
ret = 32734
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
077"..., len = 1027}
__FUNCTION__ = "receive_msg"
#17 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1027
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
0777"...
tmp = 0x9245c0 "10.143.1.2"
from = 0x7fde5e168f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
140593036756896}, addr32 = {403182777, 0, 1577289632, 32734}, addr16 =
{5305, 6152, 0, 0, 34720, 24067, 32734, 0}, addr =
"\271\024\b\030\000\000\000\000\240\207\003^\336\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0,
0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7fde5e038638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#18 0x000000000046d42b in main_loop () at main.c:1617
i = 5
pid = 0
si = 0x7fde5e038638
si_desc = "udp receiver child=5
'\000' <repeats 18 times>"\240,

\205H\377\177\000\000\364\244K\000\000\000\000"

(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
len = 1021}}
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>,
len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u =
{text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0,
len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0)
at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
'\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368,
9586112}, __mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0,
0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but about
km_val.c : db_mysql_val2str
Hello,

Post by Daniel-Constantin Mierla
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-07-01 14:40:00 UTC

Permalink

Hello,

can you give it a try with the patch from next commit?

-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4
A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4
A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0,
next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010,
p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c",
func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70,
flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200,
cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>,
len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90,
param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf =
0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700
<Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u =
{text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s =
0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"},
sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr =
32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16
= {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id =
32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380,
dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len =
5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471,
139717437685488, 9587300, 9586197, 361695345073193192, 9586295,
9586274, 2151546560, 139717437685488, 139717437615640,
139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70,
mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
'\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0,
0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags =
{f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368,
9586112}, __mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,
len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip =
{af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457,
0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr =
"\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family =
2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but
about km_val.c : db_mysql_val2str
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-07-07 10:40:11 UTC

Permalink

Hello,

Can you explain the modification and the impact on our plateform?
Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for
Kamailio too?

Regards,

Igor

Post by Igor Potjevlesch
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len
= 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text
= {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at
forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000'
<repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env
= {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816,
12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0,
0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but
about km_val.c : db_mysql_val2str

Post by Igor Potjevlesch
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-07-17 15:18:29 UTC

Permalink

Hello,

I don't understand the patch you sent me. Is it for the PAI problem?

Regards,

Igor

Post by Daniel-Constantin Mierla
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>,
len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param =
0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u =
{text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0,
len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0)
at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
'\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368,
9586112}, __mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0,
0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but
about km_val.c : db_mysql_val2str

Post by Igor Potjevlesch
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-07-17 15:50:47 UTC

Permalink

Hello,

the patch is adding a lock to protect against races executing the acc
callback -- acc was parsing the header pointing to private memory, but
linking it in the shared memory structure. Now it is cleaned up and such
cases don't overlap anymore.

Hope is clear enough. That could have been the cause for the other
crash, as the pointer might have been invalidated by the raced execution.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
I don't understand the patch you sent me. Is it for the PAI problem?
Regards,
Igor
2014-07-07 12:40 GMT+02:00 Igor Potjevlesch
Hello,
Can you explain the modification and the impact on our plateform?
Is it for the pai problem?
Do you have explanation for the km_val.c problem wich cause crash
for Kamailio too?
Regards,
Igor
2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44},
len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010,
parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body",
line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238)
at parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0,
type=512, ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf
(type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
repl=0x7f12804a6d70, flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70,
param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0,
t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
len = 1021}}
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200,
cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of
bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl =
0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch
= 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf
= {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received
(p_msg=0x7f12804a6d70) at t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause =
200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0,
packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s =
{sa_family = 20496, sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr =
32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807,
sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0},
__u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}},
id = 32530, proto = 72 'H', send_flags = {f = 228 '\344',
blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl =
0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0,
t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s
= 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665,
4279488, 140733378027408, 0, 0, 3644308075281583049,
-3644194520509117495}, __mask_was_saved = 0, __saved_mask =
{__val = {9586395,
1065161476041, 124554051613, 9586471,
139717437685488, 9587300, 9586197, 361695345073193192,
9586295, 9586274, 2151546560, 139717437685488,
139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f12804a6d70, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0,
sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0,
sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0,
proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0
'\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70)
at forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0
200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode
= 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0,
169653785368, 9586112}, __mask_was_saved = 184638568,
__saved_mask = {__val = {139717436454816, 12884901899,
139717436454816, 4279488,
140733378027408, 140733378026464, 5477954,
0, 139717072962944, 50195, 169290548608, 9586112,
140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
SIP/2.0/UDP
SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{403182777, 139717436454816}, addr32 = {403182777, 0,
2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811,
32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0},
addr = "\271\024\b\004", '\000' <repeats 11 times>}},
src_port = 5060, dst_port = 5060, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6
= {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u
= {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1
'\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98)
at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's
crash but about km_val.c : db_mysql_val2str
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4 with your
patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-07-17 16:41:12 UTC

Permalink

Hello,

When this patch will be add in a new release? I can't try without
validation of new release.

when is scheduled the next release?

Regards,

Igor.

Post by Igor Potjevlesch
Hello,
the patch is adding a lock to protect against races executing the acc
callback -- acc was parsing the header pointing to private memory, but
linking it in the shared memory structure. Now it is cleaned up and such
cases don't overlap anymore.
Hope is clear enough. That could have been the cause for the other crash,
as the pointer might have been invalidated by the raced execution.
Cheers,
Daniel
Hello,
I don't understand the patch you sent me. Is it for the PAI problem?
Regards,
Igor

Post by Daniel-Constantin Mierla
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body
= {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0
0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at
parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at
t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param =
0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>,
len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param
= 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at
t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u =
{text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0,
len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family =
20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530},
sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo =
32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 =
{7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0,
2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530,
proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param =
0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488,
9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0)
at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
'\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr =
{s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0,
0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0
'\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368,
9586112}, __mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 =
{5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af =
2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0,
0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060,
proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2,
sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0,
0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash but
about km_val.c : db_mysql_val2str

Post by Igor Potjevlesch
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-07-17 22:06:16 UTC

Permalink

Hello,

Post by Igor Potjevlesch
Hello,
When this patch will be add in a new release? I can't try without
validation of new release.
when is scheduled the next release?

you were the only one reporting this issue. The patch will be backported
if you can test and confirm it is working. You don't need to install
master for it, you can cherry-pick it on your local clone.

Otherwise, it will get in 4.2 which should be out later in the autumn.

Cheers,
Daniel

Post by Igor Potjevlesch
Regards,
Igor.
Hello,
the patch is adding a lock to protect against races executing the
acc callback -- acc was parsing the header pointing to private
memory, but linking it in the shared memory structure. Now it is
cleaned up and such cases don't overlap anymore.
Hope is clear enough. That could have been the cause for the other
crash, as the pointer might have been invalidated by the raced
execution.
Cheers,
Daniel

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-08-01 11:00:45 UTC

Permalink

Hello,

To try on a lab, i need to understand the behaviour of your patch. How your
patch avoid segmentation fault?
I can't download the patch with your link, i can only see the diff. can you
send me the patch again?

Regards,

Igor

Post by Igor Potjevlesch
Hello,
Hello,
When this patch will be add in a new release? I can't try without
validation of new release.
when is scheduled the next release?
you were the only one reporting this issue. The patch will be backported
if you can test and confirm it is working. You don't need to install master
for it, you can cherry-pick it on your local clone.
Otherwise, it will get in 4.2 which should be out later in the autumn.
Cheers,
Daniel
Regards,
Igor.

--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-08-04 14:23:57 UTC

Permalink

Hello,

the problem was that a structure in shared memory (the request cloned in
tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in
setting the header pointer to a private memory. The other process could
overtake in processing, using the same cloned request, and this time the
pai pointer is set, but to another private memory zone. I added the
locks for calling the callbacks, so the process that parse the PAI is
the one cleaning it.

Performances should not be impacted that much, the transaction lock is
used and will add sequential processing when there are two replies at
the same time, which is not the common.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
Can you explain the modification and the impact on our plateform?
Is it for the pai problem?
Do you have explanation for the km_val.c problem wich cause crash for
Kamailio too?
Regards,
Igor
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel

Post by Igor Potjevlesch
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN
IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN
IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed
= 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010,
f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010,
p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c",
func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at
parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at
acc_logic.c:501
SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512,
ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70,
flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70,
param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0,
t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200,
cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of
bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90,
param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf =
0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700
<Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70)
at t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200,
u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0,
packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s =
{sa_family = 20496, sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr =
32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807,
sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0},
__u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id =
32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask =
61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94,
param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf =
0x7f1280275380, dst = 0x7f12803de418, send_buf = {s =
0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665,
4279488, 140733378027408, 0, 0, 3644308075281583049,
-3644194520509117495}, __mask_was_saved = 0, __saved_mask =
{__val = {9586395,
1065161476041, 124554051613, 9586471,
139717437685488, 9587300, 9586197, 361695345073193192, 9586295,
9586274, 2151546560, 139717437685488, 139717437615640,
139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70,
mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data
= '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port =
0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto
= 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at
forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode =
0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0,
169653785368, 9586112}, __mask_was_saved = 184638568,
__saved_mask = {__val = {139717436454816, 12884901899,
139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0,
139717072962944, 50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139717436454816}, addr32 = {403182777, 0, 2150315936, 32530},
addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0},
addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port =
5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0,
src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id =
0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's crash
but about km_val.c : db_mysql_val2str
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4 with your patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-08-06 14:07:01 UTC

Permalink

Hello Daniel,

Thank you for this exhaustive feedback.

Do you include the patch to 4.1.5?

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : lundi 4 aoÃ»t 2014 16:24
Ã : Igor Potjevlesch
Cc : Kamailio $SER$ - Users Mailing List
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

the problem was that a structure in shared memory (the request cloned in tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in setting the header pointer to a private memory. The other process could overtake in processing, using the same cloned request, and this time the pai pointer is set, but to another private memory zone. I added the locks for calling the callbacks, so the process that parse the PAI is the one cleaning it.

Performances should not be impacted that much, the transaction lock is used and will add sequential processing when there are two replies at the same time, which is not the common.

Cheers,
Daniel

On 07/07/14 12:40, Igor Potjevlesch wrote:

Hello,

Can you explain the modification and the impact on our plateform?

Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?

Regards,

Igor

2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give it a try with the patch from next commit?

- http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

On 26/06/14 12:58, Igor Potjevlesch wrote:

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501

501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity: <sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2 "<sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102

102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142

__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464

f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at parser/hf.c:126

h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501

new_uri_bk = {s = 0x7f1274b53cdf "sip:***@GW SIP/2.0\r\nRecord-Route: <sip:***@GW%20SIP/2.0\r\nRecord-Route:%20%3csip:A.B.C.D;lr=on%3e\r\nVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12> <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0, params=0x7fff0b015580) at t_hooks.c:290

cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: ***@A.B.C.D\r\nFrom <mailto:***@A.B.C.D%5Cr%5CnFrom> : <sip:***@domain;user=phone>;epid=00903"..., len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001

relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: ***@A.B.C.D\r\nFrom <mailto:***@A.B.C.D%5Cr%5CnFrom> : <sip:***@domain;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at t_reply.c:2499

msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family = 20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 = "\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665, 4279488, 140733378027408, 0, 0, 3644308075281583049, -3644194520509117495}, __mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488, 9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560, 139717437685488, 139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at forward.c:777

new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at forward.c:860

No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia: SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,

len=1124, rcv_info=0x7fff0b015c60) at receive.c:273

msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112}, __mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0, 139717072962944, 50195, 169290548608, 9586112, 140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536

len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr = "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617

i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1 sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177", '\000' <repeats 18 times>"\320, ]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about km_val.c : db_mysql_val2str

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

On 25/06/14 14:49, Igor Potjevlesch wrote:

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-08-06 14:38:38 UTC

Permalink

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on
testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello Daniel,
Thank you for this exhaustive feedback.
Do you include the patch to 4.1.5?
Regards,
Igor.
*EnvoyÃ© :* lundi 4 aoÃ»t 2014 16:24
*Ã :* Igor Potjevlesch
*Cc :* Kamailio $SER$ - Users Mailing List
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
the problem was that a structure in shared memory (the request cloned
in tm) could have been used in parallel by different kamailio processes.
If there were two processes at the same time, parsing PAI resulted in
setting the header pointer to a private memory. The other process
could overtake in processing, using the same cloned request, and this
time the pai pointer is set, but to another private memory zone. I
added the locks for calling the callbacks, so the process that parse
the PAI is the one cleaning it.
Performances should not be impacted that much, the transaction lock is
used and will add sequential processing when there are two replies at
the same time, which is not the common.
Cheers,
Daniel
Hello,
Can you explain the modification and the impact on our plateform?
Is it for the pai problem?
Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?
Regards,
Igor
2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len =
44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag
(qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010,
parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field
(hf=0x7f1274c3c238) at parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0,
req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at
acc_logic.c:501
new_uri_bk = {s = 0x7f1274b53cdf
<sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12
len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0,
type=512, ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0,
params=0x7fff0b015580) at t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf
(type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
repl=0x7f12804a6d70, flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl =
0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags
= 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst =
0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0,
p_msg=0x7f12804a6d70, branch=0, msg_status=200,
cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at
t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out
of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl =
0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0,
branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68,
send_buf = {s = 0xb015700 <Address 0xb015700 out of
bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received
(p_msg=0x7f12804a6d70) at t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause
= 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel =
0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s =
{sa_family = 20496, sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"}, sin =
{sin_family = 20496, sin_port = 32807, sin_addr = {s_addr
= 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807,
sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0},
__u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id =
2150060928}}, id = 32530, proto = 72 'H', send_flags = {f
= 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl =
0x550b94, param = 0x231dc8, code = 0, flags = 3, branch =
0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf
= {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode =
0, jmp_env = {{__jmpbuf = {139717438500712,
3644308075193502665, 4279488, 140733378027408, 0, 0,
3644308075281583049, -3644194520509117495},
__mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471,
139717437685488, 9587300, 9586197, 361695345073193192,
9586295, 9586274, 2151546560, 139717437685488,
139717437615640, 139717438500712, 4279488,
140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f12804a6d70, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0,
sa_data = '\000' <repeats 13 times>}, sin = {sin_family =
0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id
= 0, proto = 0 '\000', send_flags = {f = 0 '\000',
blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply
(msg=0x7f12804a6d70) at forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600
"SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464,
5477954, 0, 139717072962944, 50195, 169290548608, 9586112,
140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,
len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{403182777, 139717436454816}, addr32 = {403182777, 0,
2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240,
32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0,
0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}},
src_port = 5060, dst_port = 5060, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
403182777}, sin_zero = "\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7,
argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio 's
crash but about km_val.c : db_mysql_val2str
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
Also, it would be good to have full trace for other
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4 with
your patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

Igor Potjevlesch

2014-09-23 17:16:47 UTC

Permalink

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 "sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 6 aoÃ»t 2014 16:39
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:

Hello Daniel,

Thank you for this exhaustive feedback.

Do you include the patch to 4.1.5?

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : lundi 4 aoÃ»t 2014 16:24
Ã : Igor Potjevlesch
Cc : Kamailio $SER$ - Users Mailing List
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

the problem was that a structure in shared memory (the request cloned in tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in setting the header pointer to a private memory. The other process could overtake in processing, using the same cloned request, and this time the pai pointer is set, but to another private memory zone. I added the locks for calling the callbacks, so the process that parse the PAI is the one cleaning it.

Performances should not be impacted that much, the transaction lock is used and will add sequential processing when there are two replies at the same time, which is not the common.

Cheers,
Daniel

On 07/07/14 12:40, Igor Potjevlesch wrote:

Hello,

Can you explain the modification and the impact on our plateform?

Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?

Regards,

Igor

2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give it a try with the patch from next commit?

- <http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf> http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

On 26/06/14 12:58, Igor Potjevlesch wrote:

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501

501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity: < <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2 "< <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102

102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142

__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464

f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at parser/hf.c:126

h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501

new_uri_bk = {s = 0x7f1274b53cdf " <sip:***@GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12> sip:***@GW SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0, params=0x7fff0b015580) at t_hooks.c:290

cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"..., len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001

relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at t_reply.c:2499

msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = - <tel:2143420688> 2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family = 20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 = "\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665, 4279488, 140733378027408, 0, 0, 3644308075281583049, -3644194520509117495}, __mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488, 9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560, 139717437685488, 139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at forward.c:777

new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at forward.c:860

No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia: SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,

len=1124, rcv_info=0x7fff0b015c60) at receive.c:273

msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112}, __mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0, 139717072962944, 50195, 169290548608, 9586112, 140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536

len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr = "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617

i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1 sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177", '\000' <repeats 18 times>"\320, ]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about km_val.c : db_mysql_val2str

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

On 25/06/14 14:49, Igor Potjevlesch wrote:

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla
<http://twitter.com/#!/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

Daniel-Constantin Mierla

2014-09-24 07:00:15 UTC

Permalink

Hello,

so it is still related to PAI header -- I will have more time to look at
it by end of the week.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello Daniel,
Patching has been done in the same time than the upgrade to 4.1.5.
A new crash occurred in pvapi.c (in addition of my other recent post
âCrash Kamailio 4.1.5â).
#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521
No locals.
#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026
idxf = 0
idx = 0
pai_body = 0x7f14fb679d38
pai_uri = 0xa
i = 0
cur_id = 0
__FUNCTION__ = "pv_get_pai"
#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8,
sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266
ret = 0
__FUNCTION__ = "pv_get_spec_value"
#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030,
rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c,
type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261
value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}
n = 4
r = 0
__FUNCTION__ = "extra2strar"
#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474
m = 7
n = -270381400
i = 6
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471
br = 0
hdr = 0x7f14f7d3dc20
__FUNCTION__ = "acc_onreply"
#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512,
ps=0x7fff6c6b76b0) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#7 0x00007f14f9f3146c in run_trans_callbacks_internal
(cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
params=0x7fff6c6b76b0) at t_hooks.c:290
cbp = 0x7f14ee4c81b0
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48,
flags=183) at t_hooks.c:336
params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param =
0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf =
0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {
s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f14efe525b8
#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
p_msg=0x7f14fb670c48, branch=0, msg_status=183,
cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
SIP/2.0/UDP
res_len = 777
relayed_code = 183
relayed_msg = 0x7f14fb670c48
reply_bak = 0x414cc0
bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}
totag_retr = 0
reply_status = RPS_PROVISIONAL
uas_rb = 0x7f14efe52678
to_tag = 0x0
reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}
onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90,
param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf =
0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830
"`xkl\377\177", len = -101469275}}
__FUNCTION__ = "relay_reply"
#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499
msg_status = 183
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -77092928
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u =
{text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s =
0x0, len = 9586205}}}}
uac = 0x7f14efe52720
t = 0x7f14efe525b8
lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family =
57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin =
{sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532},
sin_zero = "\000F#\000\000\000\000"}, sin6 = {
sin6_family = 57360, sin6_port = 64323, sin6_flowinfo =
32532, sin6_addr = {__in6_u = {__u6_addr8 =
"\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 =
{17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0,
4215529744,
32532}}}, sin6_scope_id = 4215529744}}, id =
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1818983120
blst_503_timeout = 32767
hf = 0x7f14fb670c68
onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param =
0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380,
dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len =
5538065}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env =
{{__jmpbuf = {139728093908544, 1898006658898931560, 4279488,
140735012372672, 0, 0, 1898006658936680296, -1897762211976106136},
__mask_was_saved = 0, __saved_mask = {__val = {9586373,
1365809186688, 124554051613, 9586450,
139728093947840, 9587056, 9586211, 361695345073193192, 9586309,
9586288, 4217874320, 139728093947840, 139728093942016,
139728093908544, 4279488, 140735012372672}}}}}
__FUNCTION__ = "reply_received"
#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
'\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0,
0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags =
{f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32532
r = 1
s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860
No locals.
#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183
Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,
len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273
msg = 0x7f14fb670c48
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808,
9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val =
{139728091862768, 12884901899, 139728091862768, 4279488,
140735012372672, 140735012371728, 5477982, 0,
139727728366976, 50195, 171356018048, 9586112, 140735012371856,
140735012371776, 5474817, 4279488}}}}}
ret = 32532
SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}
__FUNCTION__ = "receive_msg"
#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536
len = 880
buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f14fb5add70
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 =
{5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr =
"\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip =
{af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457,
0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr =
"\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family =
2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port =
50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0,
0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f14fb47d588, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#15 0x000000000046d447 in main_loop () at main.c:1617
i = 13
pid = 0
si = 0x7f14fb47d588
si_desc = "udp receiver child=13
sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
'\000' <repeats 19 times>,
"\177kl\377\177\000\000\020\245K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545
cfg_stream = 0xf42010
c = -1
r = 0
tmp = 0x7fff6c6b8f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3572644655
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"
__FUNCTION__ = "main"
Let me know if you need further information.
Regards,
Igor.
*EnvoyÃ© :* mercredi 6 aoÃ»t 2014 16:39
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
it is not in my plans for 4.1.5, because I didn't get any feedback on
testing and its results, whether it fixes or not the issue.
Cheers,
Daniel
Hello Daniel,
Thank you for this exhaustive feedback.
Do you include the patch to 4.1.5?
Regards,
Igor.
*EnvoyÃ© :* lundi 4 aoÃ»t 2014 16:24
*Ã :* Igor Potjevlesch
*Cc :* Kamailio $SER$ - Users Mailing List
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
the problem was that a structure in shared memory (the request
cloned in tm) could have been used in parallel by different
kamailio processes.
If there were two processes at the same time, parsing PAI resulted
in setting the header pointer to a private memory. The other
process could overtake in processing, using the same cloned
request, and this time the pai pointer is set, but to another
private memory zone. I added the locks for calling the callbacks,
so the process that parse the PAI is the one cleaning it.
Performances should not be impacted that much, the transaction
lock is used and will add sequential processing when there are two
replies at the same time, which is not the common.
Cheers,
Daniel
Hello,
Can you explain the modification and the impact on our plateform?
Is it for the pai problem?
Do you have explanation for the km_val.c problem wich cause
crash for Kamailio too?
Regards,
Igor
2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101
13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len =
44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag
(qm=0x7f1280275010, f=0x7f12803cb450) at
mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010,
parser/parse_ppi_pai.c", func=0x627a00
"free_pai_ppi_body", line=102) at mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field
(hf=0x7f1274c3c238) at parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12
len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0,
type=512, ps=0x7fff0b015580) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512,
trans=0x7f1274c157f0, params=0x7fff0b015580) at
t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in run_trans_callbacks_with_buf
(type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
repl=0x7f12804a6d70, flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70,
param = 0x7f1274ac0ea0, code = 200, flags = 200,
branch = 0, t_rbuf = 0x7f1274c158b0, dst =
0x7f1274c15900, send_buf = {
SIP/2.0/UDP
= 1021}}
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply
(t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0,
msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len
= 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0
out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl =
0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0,
branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address
0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received
(p_msg=0x7f12804a6d70) at t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason =
{cause = 200, u = {text = {s = 0x0, len = 9586191},
e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len =
9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to =
{s = {sa_family = 20496, sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"}, sin
= {sin_family = 20496, sin_port = 32807, sin_addr =
{s_addr = 32530}, sin_zero =
"\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807,
sin6_flowinfo = 32530, sin6_addr = {__in6_u =
{__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530,
0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id =
2150060928}}, id = 32530, proto = 72 'H', send_flags =
{f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl =
0x550b94, param = 0x231dc8, code = 0, flags = 3,
branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960
"`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf =
{139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049,
-3644194520509117495}, __mask_was_saved = 0,
__saved_mask = {__val = {9586395,
1065161476041, 124554051613,
9586471, 139717437685488, 9587300, 9586197,
361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712,
4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f12804a6d70, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family =
0, sa_data = '\000' <repeats 13 times>}, sin =
{sin_family = 0, sin_port = 0, sin_addr = {s_addr =
0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 =
'\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0,
0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, id = 0, proto = 0 '\000',
send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out
of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply
(msg=0x7f12804a6d70) at forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600
"SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask = {__val =
{139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464,
5477954, 0, 139717072962944, 50195, 169290548608,
9586112, 140733378026592, 140733378026512, 5474789,
4279488}}}}}
ret = 32530
SIP/2.0/UDP
SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at
udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{403182777, 139717436454816}, addr32 = {403182777, 0,
2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240,
32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0},
addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0,
0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000'
<repeats 11 times>}}, src_port = 5060, dst_port =
5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su
= {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr
= 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family =
2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7,
argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of Kamailio
's crash but about km_val.c : db_mysql_val2str
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
Also, it would be good to have full trace for
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4 with
your patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - http://www.asipto.com
Sep 22-25, Berlin, Germany

Igor Potjevlesch

2014-10-02 07:50:58 UTC

Permalink

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 24 septembre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 " <sip:ABCDEFGHIJ@> sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE <sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" < <sip:KLMNOPQRST-2F/Cjg+***@public.gmane.org> sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" < <sip:***@tru> sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 6 aoÃ»t 2014 16:39
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:

Hello Daniel,

Thank you for this exhaustive feedback.

Do you include the patch to 4.1.5?

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : lundi 4 aoÃ»t 2014 16:24
Ã : Igor Potjevlesch
Cc : Kamailio $SER$ - Users Mailing List
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

the problem was that a structure in shared memory (the request cloned in tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in setting the header pointer to a private memory. The other process could overtake in processing, using the same cloned request, and this time the pai pointer is set, but to another private memory zone. I added the locks for calling the callbacks, so the process that parse the PAI is the one cleaning it.

Performances should not be impacted that much, the transaction lock is used and will add sequential processing when there are two replies at the same time, which is not the common.

Cheers,
Daniel

On 07/07/14 12:40, Igor Potjevlesch wrote:

Hello,

Can you explain the modification and the impact on our plateform?

Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?

Regards,

Igor

2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give it a try with the patch from next commit?

- <http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf> http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

On 26/06/14 12:58, Igor Potjevlesch wrote:

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501

501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity: < <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2 "< <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102

102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142

__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464

f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at parser/hf.c:126

h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501

new_uri_bk = {s = 0x7f1274b53cdf " <sip:***@GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12> sip:***@GW SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0, params=0x7fff0b015580) at t_hooks.c:290

cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"..., len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001

relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at t_reply.c:2499

msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = - <tel:2143420688> 2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family = 20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 = "\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665, 4279488, 140733378027408, 0, 0, 3644308075281583049, -3644194520509117495}, __mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488, 9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560, 139717437685488, 139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at forward.c:777

new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at forward.c:860

No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia: SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,

len=1124, rcv_info=0x7fff0b015c60) at receive.c:273

msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112}, __mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0, 139717072962944, 50195, 169290548608, 9586112, 140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536

len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr = "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617

i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1 sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177", '\000' <repeats 18 times>"\320, ]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about km_val.c : db_mysql_val2str

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

On 25/06/14 14:49, Igor Potjevlesch wrote:

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
<http://twitter.com/#%21/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA

Daniel-Constantin Mierla

2014-10-02 08:16:33 UTC

Permalink

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of
it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that
you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in
all cases), better do it on the part doing parallel access out of tm.
Acc is doing some particular processing -- it uses a callback for a
reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside
request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello Daniel,
Have you had the opportunity to look at this?
Thank you for your feedback.
Regards,
Igor.
*EnvoyÃ© :* mercredi 24 septembre 2014 09:00
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
so it is still related to PAI header -- I will have more time to look
at it by end of the week.
Cheers,
Daniel
Hello Daniel,
Patching has been done in the same time than the upgrade to 4.1.5.
A new crash occurred in pvapi.c (in addition of my other recent
post âCrash Kamailio 4.1.5â).
#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521
No locals.
#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026
idxf = 0
idx = 0
pai_body = 0x7f14fb679d38
pai_uri = 0xa
i = 0
cur_id = 0
__FUNCTION__ = "pv_get_pai"
#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8,
sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266
ret = 0
__FUNCTION__ = "pv_get_spec_value"
#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030,
rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c,
type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261
value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}
n = 4
r = 0
__FUNCTION__ = "extra2strar"
#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474
m = 7
n = -270381400
i = 6
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471
new_uri_bk = {s = 0x7f14efe25590
br = 0
hdr = 0x7f14f7d3dc20
__FUNCTION__ = "acc_onreply"
#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512,
ps=0x7fff6c6b76b0) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#7 0x00007f14f9f3146c in run_trans_callbacks_internal
(cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
params=0x7fff6c6b76b0) at t_hooks.c:290
cbp = 0x7f14ee4c81b0
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512,
rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48,
flags=183) at t_hooks.c:336
params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48,
param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0,
t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {
s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
len = 980}}
trans = 0x7f14efe525b8
#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
p_msg=0x7f14fb670c48, branch=0, msg_status=183,
cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
SIP/2.0/UDP
res_len = 777
relayed_code = 183
relayed_msg = 0x7f14fb670c48
reply_bak = 0x414cc0
bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}
totag_retr = 0
reply_status = RPS_PROVISIONAL
uas_rb = 0x7f14efe52678
to_tag = 0x0
reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}
onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90,
param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf
= 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830
"`xkl\377\177", len = -101469275}}
__FUNCTION__ = "relay_reply"
#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499
msg_status = 183
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -77092928
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u =
{text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs =
{s = 0x0, len = 9586205}}}}
uac = 0x7f14efe52720
t = 0x7f14efe525b8
lack_dst = {send_sock = 0x7f14fb584420, to = {s =
{sa_family = 57360, sa_data =
"C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family
= 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero =
"\000F#\000\000\000\000"}, sin6 = {
sin6_family = 57360, sin6_port = 64323,
sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 =
"\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16
= {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 =
{2311680, 0, 4215529744,
32532}}}, sin6_scope_id = 4215529744}}, id =
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1818983120
blst_503_timeout = 32767
hf = 0x7f14fb670c68
onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0,
param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf =
0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s =
0x7fff6c6b7a90 "`G\223", len = 5538065}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560,
4279488, 140735012372672, 0, 0, 1898006658936680296,
-1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val
= {9586373,
1365809186688, 124554051613, 9586450,
139728093947840, 9587056, 9586211, 361695345073193192, 9586309,
9586288, 4217874320, 139728093947840, 139728093942016,
139728093908544, 4279488, 140735012372672}}}}}
__FUNCTION__ = "reply_received"
#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48,
mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data
= '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats
15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 =
{0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000',
send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32532
r = 1
s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860
No locals.
#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183
Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,
len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273
msg = 0x7f14fb670c48
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808,
9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val =
{139728091862768, 12884901899, 139728091862768, 4279488,
140735012372672, 140735012371728, 5477982, 0,
139727728366976, 50195, 171356018048, 9586112, 140735012371856,
140735012371776, 5474817, 4279488}}}}}
ret = 32532
SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}
__FUNCTION__ = "receive_msg"
#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536
len = 880
buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f14fb5add70
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777,
139728091862768}, addr32 = {403182777, 0, 4215789296, 32532},
addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr =
"\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr
= "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family
= 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2,
sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u =
{__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f14fb47d588, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#15 0x000000000046d447 in main_loop () at main.c:1617
i = 13
pid = 0
si = 0x7f14fb47d588
si_desc = "udp receiver child=13
sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
'\000' <repeats 19 times>,
"\177kl\377\177\000\000\020\245K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545
cfg_stream = 0xf42010
c = -1
r = 0
tmp = 0x7fff6c6b8f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a58
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3572644655
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"
__FUNCTION__ = "main"
Let me know if you need further information.
Regards,
Igor.
*EnvoyÃ© :* mercredi 6 aoÃ»t 2014 16:39
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
it is not in my plans for 4.1.5, because I didn't get any feedback
on testing and its results, whether it fixes or not the issue.
Cheers,
Daniel
Hello Daniel,
Thank you for this exhaustive feedback.
Do you include the patch to 4.1.5?
Regards,
Igor.
*EnvoyÃ© :* lundi 4 aoÃ»t 2014 16:24
*Ã :* Igor Potjevlesch
*Cc :* Kamailio $SER$ - Users Mailing List
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
the problem was that a structure in shared memory (the request
cloned in tm) could have been used in parallel by different
kamailio processes.
If there were two processes at the same time, parsing PAI
resulted in setting the header pointer to a private memory.
The other process could overtake in processing, using the same
cloned request, and this time the pai pointer is set, but to
another private memory zone. I added the locks for calling the
callbacks, so the process that parse the PAI is the one
cleaning it.
Performances should not be impacted that much, the transaction
lock is used and will add sequential processing when there are
two replies at the same time, which is not the common.
Cheers,
Daniel
Hello,
Can you explain the modification and the impact on our plateform?
Is it for the pai problem?
Do you have explanation for the km_val.c problem wich
cause crash for Kamailio too?
Regards,
Igor
2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla
Hello,
can you give it a try with the patch from next commit?
-
http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf
If all goes fine while testing, I will backport.
Cheers,
Daniel
Hello,
(gdb) frame 6
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8
101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2
Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4
A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8
101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101
tele"..., len = 44}, len = 67, parsed = 0x0, next
= 0x7f1274c3c278}
(gdb) frame 4
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102
102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}
#0 0x0000003d6f6328a5 in raise () from
/lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from
/lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag
(qm=0x7f1280275010, f=0x7f12803cb450) at
mem/q_malloc.c:142
__FUNCTION__ = "qm_debug_frag"
#3 0x0000000000548b26 in qm_free
(qm=0x7f1280275010, p=0x7f12803cb480,
file=0x6276a0 "<core>: parser/parse_ppi_pai.c",
func=0x627a00 "free_pai_ppi_body", line=102) at
mem/q_malloc.c:464
f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"
#4 0x000000000056e5e6 in free_pai_ppi_body
(pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102
__FUNCTION__ = "free_pai_ppi_body"
#5 0x000000000054fee0 in clean_hdr_field
(hf=0x7f1274c3c238) at parser/hf.c:126
h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"
#6 0x00007f127cb6dde6 in acc_onreply
(t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501
new_uri_bk = {s = 0x7f1274b53cdf
<sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12
len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"
#7 0x00007f127cb6e30a in tmcb_func
(t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at
acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#8 0x00007f127ed68478 in
run_trans_callbacks_internal
(cb_lst=0x7f1274c15860, type=512,
trans=0x7f1274c157f0, params=0x7fff0b015580) at
t_hooks.c:290
cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#9 0x00007f127ed6868a in
run_trans_callbacks_with_buf (type=512,
rbuf=0x7f1274c158b0, req=0x7f1274c3ac08,
repl=0x7f12804a6d70, flags=200) at t_hooks.c:336
params = {req = 0x7f1274c3ac08, rpl =
0x7f12804a6d70, param = 0x7f1274ac0ea0, code =
200, flags = 200, branch = 0, t_rbuf =
0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
len = 1021}}
trans = 0x7f1274c157f0
#10 0x00007f127ed9ac06 in relay_reply
(t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0,
msg_status=200, cancel_data=0x7fff0b0158e0,
do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "",
len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address
0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl =
0x7f127edbaf90, param = 0x414cc0, code = 1, flags
= 0, branch = 0, t_rbuf = 0x7f126a80c828, dst =
0x7f12804a6f68, send_buf = {s = 0xb015700 <Address
0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"
#11 0x00007f127ed9d0b7 in reply_received
(p_msg=0x7f12804a6d70) at t_reply.c:2499
msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -2143420688 <tel:2143420688>
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason =
{cause = 200, u = {text = {s = 0x0, len =
9586191}, e2e_cancel = 0x0, packed_hdrs = {s =
0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to
= {s = {sa_family = 20496, sa_data =
"'\200\022\177\000\000\310\036#\000\000\000\000"},
sin = {sin_family = 20496, sin_port = 32807,
sin_addr = {s_addr = 32530}, sin_zero =
"\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port =
32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u
= {__u6_addr8 =
"\310\036#\000\000\000\000\000\360\247=\200\022\177\000",
__u6_addr16 = {7880, 35, 0, 0, 42992, 32829,
32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id =
2150060928}}, id = 32530, proto = 72 'H',
send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl
= 0x550b94, param = 0x231dc8, code = 0, flags = 3,
branch = 0, t_rbuf = 0x7f1280275380, dst =
0x7f12803de418, send_buf = {s = 0x7fff0b015960
"`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf =
{139717438500712, 3644308075193502665, 4279488,
140733378027408, 0, 0, 3644308075281583049,
-3644194520509117495}, __mask_was_saved = 0,
__saved_mask = {__val = {9586395,
1065161476041, 124554051613,
9586471, 139717437685488, 9587300, 9586197,
361695345073193192, 9586295, 9586274, 2151546560,
139717437685488, 139717437615640, 139717438500712,
4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"
#12 0x000000000045d837 in do_forward_reply
(msg=0x7f12804a6d70, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s =
{sa_family = 0, sa_data = '\000' <repeats 13
times>}, sin = {sin_family = 0, sin_port = 0,
sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 =
'\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0,
0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, id = 0, proto = 0 '\000',
send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78
out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#13 0x000000000045e0f8 in forward_reply
(msg=0x7f12804a6d70) at forward.c:860
No locals.
#14 0x00000000004a58e7 in receive_msg
(buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...,
len=1124, rcv_info=0x7fff0b015c60) at
receive.c:273
msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
263853236176, 1, 0, 169653785368, 9586112},
__mask_was_saved = 184638568, __saved_mask =
{__val = {139717436454816, 12884901899,
139717436454816, 4279488,
140733378027408,
140733378026464, 5477954, 0, 139717072962944,
50195, 169290548608, 9586112, 140733378026592,
140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200
OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"
#15 0x000000000053c9a8 in udp_rcv_loop () at
udp_server.c:536
len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u =
{addrl = {403182777, 139717436454816}, addr32 =
{403182777, 0, 2150315936, 32530}, addr16 = {5305,
6152, 0, 0, 14240, 32811, 32530, 0}, addr =
"\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0},
addr32 = {67638457, 0, 0, 0}, addr16 = {5305,
1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004",
'\000' <repeats 11 times>}}, src_port = 5060,
dst_port = 5060, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"},
sin = {sin_family = 2, sin_port = 50195, sin_addr
= {s_addr = 403182777}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr
= {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}},
bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#16 0x000000000046d42b in main_loop () at main.c:1617
i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1
sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177",
'\000' <repeats 18 times>"\320,
]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#17 0x0000000000470533 in main (argc=7,
argv=0x7fff0b015f98) at main.c:2545
cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80
"H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"
In a next mail you will find a new bt full of
db_mysql_val2str
2014-06-25 18:26 GMT+02:00 Daniel-Constantin
2014-06-25 18:26 GMT+02:00 Daniel-Constantin
Hello,
frame 6
print hdr
print *hdr
frame 4
print *pid_b
Also, it would be good to have full trace for
bt full
Cheers,
Daniel
Hello,
We updated this morning Kamailio in 4.1.4
with your patch.
[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 -http://www.asipto.com
Sep 22-25, Berlin, Germany

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-10-02 16:42:54 UTC

Permalink

Hello,

Thank you for your time on this issue.

I understand that I can define âdb_extraâ like:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$avp(s:pai);"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

Instead of:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$ai;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

And, in the request_route

request_route {

#Store PAI in AVP for accounting

$avp(s:pai)=$ai;

# per request initial checks

route(REQINIT);

[âŠ]

}

I can put this fix while waiting for 4.2.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 10:17
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers,
Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 24 septembre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 " <sip:ABCDEFGHIJ@> sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE <sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" < <sip:KLMNOPQRST-2F/Cjg+***@public.gmane.org> sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" < <sip:***@tru> sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 6 aoÃ»t 2014 16:39
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:

Hello Daniel,

Thank you for this exhaustive feedback.

Do you include the patch to 4.1.5?

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : lundi 4 aoÃ»t 2014 16:24
Ã : Igor Potjevlesch
Cc : Kamailio $SER$ - Users Mailing List
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

the problem was that a structure in shared memory (the request cloned in tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in setting the header pointer to a private memory. The other process could overtake in processing, using the same cloned request, and this time the pai pointer is set, but to another private memory zone. I added the locks for calling the callbacks, so the process that parse the PAI is the one cleaning it.

Performances should not be impacted that much, the transaction lock is used and will add sequential processing when there are two replies at the same time, which is not the common.

Cheers,
Daniel

On 07/07/14 12:40, Igor Potjevlesch wrote:

Hello,

Can you explain the modification and the impact on our plateform?

Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?

Regards,

Igor

2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give it a try with the patch from next commit?

- <http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf> http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

On 26/06/14 12:58, Igor Potjevlesch wrote:

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501

501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity: < <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2 "< <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102

102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142

__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464

f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at parser/hf.c:126

h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501

new_uri_bk = {s = 0x7f1274b53cdf " <sip:***@GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12> sip:***@GW SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0, params=0x7fff0b015580) at t_hooks.c:290

cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"..., len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001

relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at t_reply.c:2499

msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = - <tel:2143420688> 2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family = 20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 = "\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665, 4279488, 140733378027408, 0, 0, 3644308075281583049, -3644194520509117495}, __mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488, 9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560, 139717437685488, 139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at forward.c:777

new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at forward.c:860

No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia: SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,

len=1124, rcv_info=0x7fff0b015c60) at receive.c:273

msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112}, __mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0, 139717072962944, 50195, 169290548608, 9586112, 140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536

len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr = "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617

i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1 sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177", '\000' <repeats 18 times>"\320, ]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about km_val.c : db_mysql_val2str

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

On 25/06/14 14:49, Igor Potjevlesch wrote:

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
<http://twitter.com/#%21/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
<http://twitter.com/#%21/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-10-03 15:00:02 UTC

Permalink

Hello Daniel,

Iâm just seeing that 4.2 is scheduled for 15th October.

I can wait until this date.

Regards,

Igor.

De : Igor Potjevlesch [mailto:igor.potjevlesch-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 18:43
Ã : miconda-***@public.gmane.org
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : RE: [SR-Users] Crash Kamailio 4.1.4

Hello,

Thank you for your time on this issue.

I understand that I can define âdb_extraâ like:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$avp(s:pai);"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

Instead of:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$ai;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

And, in the request_route

request_route {

#Store PAI in AVP for accounting

$avp(s:pai)=$ai;

# per request initial checks

route(REQINIT);

[âŠ]

}

I can put this fix while waiting for 4.2.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 10:17
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers,
Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 24 septembre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 " <sip:ABCDEFGHIJ@> sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE <sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" < <sip:KLMNOPQRST-2F/Cjg+***@public.gmane.org> sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" < <sip:***@tru> sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 6 aoÃ»t 2014 16:39
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

it is not in my plans for 4.1.5, because I didn't get any feedback on testing and its results, whether it fixes or not the issue.

Cheers,
Daniel

On 06/08/14 16:07, Igor Potjevlesch wrote:

Hello Daniel,

Thank you for this exhaustive feedback.

Do you include the patch to 4.1.5?

Regards,

Igor.

De : Daniel-Constantin Mierla [ <mailto:miconda-***@public.gmane.org> mailto:miconda-***@public.gmane.org]
EnvoyÃ© : lundi 4 aoÃ»t 2014 16:24
Ã : Igor Potjevlesch
Cc : Kamailio $SER$ - Users Mailing List
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

the problem was that a structure in shared memory (the request cloned in tm) could have been used in parallel by different kamailio processes.

If there were two processes at the same time, parsing PAI resulted in setting the header pointer to a private memory. The other process could overtake in processing, using the same cloned request, and this time the pai pointer is set, but to another private memory zone. I added the locks for calling the callbacks, so the process that parse the PAI is the one cleaning it.

Performances should not be impacted that much, the transaction lock is used and will add sequential processing when there are two replies at the same time, which is not the common.

Cheers,
Daniel

On 07/07/14 12:40, Igor Potjevlesch wrote:

Hello,

Can you explain the modification and the impact on our plateform?

Is it for the pai problem?

Do you have explanation for the km_val.c problem wich cause crash for Kamailio too?

Regards,

Igor

2014-07-01 16:40 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give it a try with the patch from next commit?

- <http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf> http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=da9d56be28e050dd0cb4aed50efcbda043a3e5cf

If all goes fine while testing, I will backport.

Cheers,
Daniel

On 26/06/14 12:58, Igor Potjevlesch wrote:

Hello,

Here the result :

(gdb) frame 6

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08,
reply=0x7f12804a6d70, code=200) at acc_logic.c:501

501 clean_hdr_field(hdr);
(gdb) print hdr
$1 = (hdr_field_t *) 0x7f1274c3c238
(gdb) print *hdr
$2 = {type = HDR_PAI_T, name = {
s = 0x7f1274c3b6cd "P-Asserted-Identity: < <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C"..., len = 19}, body = {
s = 0x7f1274c3b6e2 "< <sip:***@domain;user=phone> sip:***@domain;user=phone>\r\nP-Sig-Options: Sending-Complete\r\n\r\nv=0\r\no=- 111851 1 IN IP4 A.B.C.D\r\ns=-\r\nt=0 0\r\nm=audio 21336 RTP/AVP 8 101 13\r\nc=IN IP4 A.B.C.D\r\na=rtpmap:101 tele"..., len = 44}, len = 67, parsed = 0x0, next = 0x7f1274c3c278}

(gdb) frame 4

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480)
at parser/parse_ppi_pai.c:102

102 pkg_free(pid_b);
(gdb) print *pid_b
$3 = {id = 0x0, num_ids = 0, next = 0x1d0}

This is the bt full :

#0 0x0000003d6f6328a5 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003d6f634085 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x0000000000546d3c in qm_debug_frag (qm=0x7f1280275010, f=0x7f12803cb450) at mem/q_malloc.c:142

__FUNCTION__ = "qm_debug_frag"

#3 0x0000000000548b26 in qm_free (qm=0x7f1280275010, p=0x7f12803cb480, file=0x6276a0 "<core>: parser/parse_ppi_pai.c", func=0x627a00 "free_pai_ppi_body", line=102) at mem/q_malloc.c:464

f = 0x7f12803cb450
size = 139717434027144
next = 0xf00000000
prev = 0x7f127cd79e00
__FUNCTION__ = "qm_free"

#4 0x000000000056e5e6 in free_pai_ppi_body (pid_b=0x7f12803cb480) at parser/parse_ppi_pai.c:102

__FUNCTION__ = "free_pai_ppi_body"

#5 0x000000000054fee0 in clean_hdr_field (hf=0x7f1274c3c238) at parser/hf.c:126

h_parsed = 0x7f1274c3c268
__FUNCTION__ = "clean_hdr_field"

#6 0x00007f127cb6dde6 in acc_onreply (t=0x7f1274c157f0, req=0x7f1274c3ac08, reply=0x7f12804a6d70, code=200) at acc_logic.c:501

new_uri_bk = {s = 0x7f1274b53cdf " <sip:***@GW%20SIP/2.0%5Cr%5CnRecord-Route:%20%3csip:A.B.C.D;lr=on%3e%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0%5Cr%5CnVia:%20SIP/2.0/UDP%20A.B.C.D:2057;branch=z9hG4bK-12> sip:***@GW SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr=on>\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-12"..., len = 19}
br = 0
hdr = 0x7f1274c3c238
__FUNCTION__ = "acc_onreply"

#7 0x00007f127cb6e30a in tmcb_func (t=0x7f1274c157f0, type=512, ps=0x7fff0b015580) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#8 0x00007f127ed68478 in run_trans_callbacks_internal (cb_lst=0x7f1274c15860, type=512, trans=0x7f1274c157f0, params=0x7fff0b015580) at t_hooks.c:290

cbp = 0x7f1274ac0e90
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"

#9 0x00007f127ed6868a in run_trans_callbacks_with_buf (type=512, rbuf=0x7f1274c158b0, req=0x7f1274c3ac08, repl=0x7f12804a6d70, flags=200) at t_hooks.c:336

params = {req = 0x7f1274c3ac08, rpl = 0x7f12804a6d70, param = 0x7f1274ac0ea0, code = 200, flags = 200, branch = 0, t_rbuf = 0x7f1274c158b0, dst = 0x7f1274c15900, send_buf = {
s = 0x7f1274c27620 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"..., len = 1021}}
trans = 0x7f1274c157f0

#10 0x00007f127ed9ac06 in relay_reply (t=0x7f1274c157f0, p_msg=0x7f12804a6d70, branch=0, msg_status=200, cancel_data=0x7fff0b0158e0, do_put_on_wait=1) at t_reply.c:2001

relay = 0
save_clone = 0
buf = 0x7f12804a7cc0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: <mailto:***@A.B.C.D%5Cr%5CnFrom> ***@A.B.C.D\r\nFrom: < <sip:***@domain;user=phone> sip:***@domain;user=phone>;epid=00903"...
res_len = 1021
relayed_code = 200
relayed_msg = 0x7f12804a6d70
reply_bak = 0x7fff0b015730
bm = {to_tag_val = {s = 0x7f1274c16d88 "", len = 5449343}}
totag_retr = 0
reply_status = RPS_COMPLETED
uas_rb = 0x7f1274c158b0
to_tag = 0x0
reason = {s = 0x10b0156e0 <Address 0x10b0156e0 out of bounds>, len = 1}
onsend_params = {req = 0x200924a64, rpl = 0x7f127edbaf90, param = 0x414cc0, code = 1, flags = 0, branch = 0, t_rbuf = 0x7f126a80c828, dst = 0x7f12804a6f68, send_buf = {s = 0xb015700 <Address 0xb015700 out of bounds>, len = 1024}}
__FUNCTION__ = "relay_reply"

#11 0x00007f127ed9d0b7 in reply_received (p_msg=0x7f12804a6d70) at t_reply.c:2499

msg_status = 200
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = - <tel:2143420688> 2143420688
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 9586191}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586191}}}}
uac = 0x7f1274c15958
t = 0x7f1274c157f0
lack_dst = {send_sock = 0x7f12803e4110, to = {s = {sa_family = 20496, sa_data = "'\200\022\177\000\000\310\036#\000\000\000\000"}, sin = {sin_family = 20496, sin_port = 32807, sin_addr = {s_addr = 32530}, sin_zero = "\310\036#\000\000\000\000"}, sin6 = {
sin6_family = 20496, sin6_port = 32807, sin6_flowinfo = 32530, sin6_addr = {__in6_u = {__u6_addr8 = "\310\036#\000\000\000\000\000\360\247=\200\022\177\000", __u6_addr16 = {7880, 35, 0, 0, 42992, 32829, 32530, 0}, __u6_addr32 = {2301640, 0, 2151524336,
32530}}}, sin6_scope_id = 2150060928}}, id = 32530, proto = 72 'H', send_flags = {f = 228 '\344', blst_imask = 61 '='}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 184637856
blst_503_timeout = 32767
hf = 0x7f12804a6d90
onsend_params = {req = 0x7fff0b015960, rpl = 0x550b94, param = 0x231dc8, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f1280275380, dst = 0x7f12803de418, send_buf = {s = 0x7fff0b015960 "`G\223", len = 5538037}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139717438500712, 3644308075193502665, 4279488, 140733378027408, 0, 0, 3644308075281583049, -3644194520509117495}, __mask_was_saved = 0, __saved_mask = {__val = {9586395,
1065161476041, 124554051613, 9586471, 139717437685488, 9587300, 9586197, 361695345073193192, 9586295, 9586274, 2151546560, 139717437685488, 139717437615640, 139717438500712, 4279488, 140733378027408}}}}}
__FUNCTION__ = "reply_received"

#12 0x000000000045d837 in do_forward_reply (msg=0x7f12804a6d70, mode=0) at forward.c:777

new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
new_len = 32530
r = 1
s = 0x464804a6d78 <Address 0x464804a6d78 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"

#13 0x000000000045e0f8 in forward_reply (msg=0x7f12804a6d70) at forward.c:860

No locals.
#14 0x00000000004a58e7 in receive_msg (buf=0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 185.20.8.4;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=185.20.8.4\r\nVia: SIP/2.0/UDP 10.143.1.2:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...,

len=1124, rcv_info=0x7fff0b015c60) at receive.c:273

msg = 0x7f12804a6d70
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 169653785368, 9586112}, __mask_was_saved = 184638568, __saved_mask = {__val = {139717436454816, 12884901899, 139717436454816, 4279488,
140733378027408, 140733378026464, 5477954, 0, 139717072962944, 50195, 169290548608, 9586112, 140733378026592, 140733378026512, 5474789, 4279488}}}}}
ret = 32530
inb = {s = 0x924600 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"..., len = 1124}
__FUNCTION__ = "receive_msg"

#15 0x000000000053c9a8 in udp_rcv_loop () at udp_server.c:536

len = 1124
buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP A.B.C.D;branch=z9hG4bK512b.82b197888826f6b60c0c63b79801294d.0;received=A.B.C.D\r\nVia: SIP/2.0/UDP A.B.C.D:2057;branch=z9hG4bK-129F259C;rport=2057\r\nCall-ID: cb0"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f12803e3f68
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139717436454816}, addr32 = {403182777, 0, 2150315936, 32530}, addr16 = {5305, 6152, 0, 0, 14240, 32811, 32530, 0}, addr = "\271\024\b\030\000\000\000\000\240\067+\200\022\177\000"}}, dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f12802b3638, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"

#16 0x000000000046d42b in main_loop () at main.c:1617

i = 1
pid = 0
si = 0x7f12802b3638
si_desc = "udp receiver child=1 sock=A.B.C.D:5060\000D\200\022\177\000\000\000\206\063\200\022\177\000\000.\205^\000\000\000\000\000\020w^\000\000\000\000\000\275\005r/\000\000\000\000\300LA\000\000\000\000\000\220_\001\v\377\177", '\000' <repeats 18 times>"\320, ]\001\v\377\177\000\000\364\244K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"

#17 0x0000000000470533 in main (argc=7, argv=0x7fff0b015f98) at main.c:2545

cfg_stream = 0xe20010
c = -1
r = 0
tmp = 0x7fff0b017f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a68 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1972285608
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5cab80 "H\211l$\330L\211d$\340H\215-o\244*"
__FUNCTION__ = "main"

In a next mail you will find a new bt full of Kamailio 's crash but about km_val.c : db_mysql_val2str

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

2014-06-25 18:26 GMT+02:00 Daniel-Constantin Mierla <miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org> >:

Hello,

can you give the output of:

frame 6
print hdr
print *hdr

frame 4
print *pid_b

Also, it would be good to have full trace for other details:

bt full

Cheers,
Daniel

On 25/06/14 14:49, Igor Potjevlesch wrote:

Hello,

We updated this morning Kamailio in 4.1.4 with your patch.

[...]
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
<http://twitter.com/#%21/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany ::: Oct 15-17, San Francisco, USA
--
Daniel-Constantin Mierla
<http://twitter.com/#%21/miconda> http://twitter.com/#!/miconda - <http://www.linkedin.com/in/miconda> http://www.linkedin.com/in/miconda
Next Kamailio Advanced Trainings 2014 - <http://www.asipto.com> http://www.asipto.com
Sep 22-25, Berlin, Germany

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-10-07 06:59:35 UTC

Permalink

Hello,

I pushed a fix to kamailio master branch that should solve it -- last
commits to acc module. If you can give it a try and report results,
would be appreciated.

Cheers,
Daniel

Post by Igor Potjevlesch
Hello Daniel,
Iâm just seeing that 4.2 is scheduled for 15th October.
I can wait until this date.
Regards,
Igor.
*EnvoyÃ© :* jeudi 2 octobre 2014 18:43
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* RE: [SR-Users] Crash Kamailio 4.1.4
Hello,
Thank you for your time on this issue.
modparam("acc", "db_extra",
"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=*$avp(s:pai)*;"
"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")
modparam("acc", "db_extra",
"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;*src_pai=$ai*;"
"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")
And, in the request_route
request_route {
* #Store PAI in AVP for accounting*
* $avp(s:pai)=$ai;*
# per request initial checks
route(REQINIT);
[âŠ]
}
I can put this fix while waiting for 4.2.
Regards,
Igor.
*EnvoyÃ© :* jeudi 2 octobre 2014 10:17
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
not really troubleshooting, as I got traveling, but I thought a bit of
it, also from performances point of view.
As it was related to parallel processing of 1xx/2xx sip responses that
- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)
Instead of using locks or other workarounds in tm (lower performances
in all cases), better do it on the part doing parallel access out of
tm. Acc is doing some particular processing -- it uses a callback for
a reply to access the response, so it is quite specific to it.
If you want to be safe side immediately, store the PAI in an avp
inside request_route and use that avp for accounting.
A solution will be there before 4.2 is out.
Cheers,
Daniel
Hello Daniel,
Have you had the opportunity to look at this?
Thank you for your feedback.
Regards,
Igor.
*EnvoyÃ© :* mercredi 24 septembre 2014 09:00
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
so it is still related to PAI header -- I will have more time to
look at it by end of the week.
Cheers,
Daniel
Hello Daniel,
Patching has been done in the same time than the upgrade to 4.1.5.
A new crash occurred in pvapi.c (in addition of my other
recent post âCrash Kamailio 4.1.5â).
#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521
No locals.
#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026
idxf = 0
idx = 0
pai_body = 0x7f14fb679d38
pai_uri = 0xa
i = 0
cur_id = 0
__FUNCTION__ = "pv_get_pai"
#2 0x0000000000499594 in pv_get_spec_value
(msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430)
at pvapi.c:1266
ret = 0
__FUNCTION__ = "pv_get_spec_value"
#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030,
rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30,
int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7
"\002\002\002\002") at acc_extra.c:261
value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}
n = 4
r = 0
__FUNCTION__ = "extra2strar"
#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474
m = 7
n = -270381400
i = 6
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at
acc_logic.c:471
new_uri_bk = {s = 0x7f14efe25590
= 19}
br = 0
hdr = 0x7f14f7d3dc20
__FUNCTION__ = "acc_onreply"
#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8,
type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#7 0x00007f14f9f3146c in run_trans_callbacks_internal
(cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
params=0x7fff6c6b76b0) at t_hooks.c:290
cbp = 0x7f14ee4c81b0
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf
(type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8,
repl=0x7f14fb670c48, flags=183) at t_hooks.c:336
params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48,
param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0,
t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {
SIP/2.0/UDP
trans = 0x7f14efe525b8
#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
p_msg=0x7f14fb670c48, branch=0, msg_status=183,
cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f14fb67e740 "SIP/2.0 183 Session
Progress\r\nVia: SIP/2.0/UDP
res_len = 777
relayed_code = 183
relayed_msg = 0x7f14fb670c48
reply_bak = 0x414cc0
bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}
totag_retr = 0
reply_status = RPS_PROVISIONAL
uas_rb = 0x7f14efe52678
to_tag = 0x0
reason = {s = 0x800000001 <Address 0x800000001 out of
bounds>, len = 1}
onsend_params = {req = 0x200924970, rpl =
0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags =
3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf =
{s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}
__FUNCTION__ = "relay_reply"
#10 0x00007f14f9f660ab in reply_received
(p_msg=0x7f14fb670c48) at t_reply.c:2499
msg_status = 183
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -77092928
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause = 0,
u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0,
packed_hdrs = {s = 0x0, len = 9586205}}}}
uac = 0x7f14efe52720
t = 0x7f14efe525b8
lack_dst = {send_sock = 0x7f14fb584420, to = {s =
{sa_family = 57360, sa_data =
"C\373\024\177\000\000\000F#\000\000\000\000"}, sin =
{sin_family = 57360, sin_port = 64323, sin_addr = {s_addr =
32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {
sin6_family = 57360, sin6_port = 64323,
sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 =
"\000F#\000\000\000\000\000\020\341C\373\024\177\000",
__u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0},
__u6_addr32 = {2311680, 0, 4215529744,
32532}}}, sin6_scope_id = 4215529744}}, id
= 103 'g'}}
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1818983120
blst_503_timeout = 32767
hf = 0x7f14fb670c68
onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0,
param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf =
0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s =
0x7fff6c6b7a90 "`G\223", len = 5538065}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0,
jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560,
4279488, 140735012372672, 0, 0, 1898006658936680296,
-1897762211976106136}, __mask_was_saved = 0, __saved_mask =
{__val = {9586373,
1365809186688, 124554051613, 9586450,
139728093947840, 9587056, 9586211, 361695345073193192,
9586309, 9586288, 4217874320, 139728093947840,
139728093942016, 139728093908544, 4279488, 140735012372672}}}}}
__FUNCTION__ = "reply_received"
#11 0x000000000045d853 in do_forward_reply
(msg=0x7f14fb670c48, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0,
sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0,
sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0},
__u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0,
proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0
'\000'}}
new_len = 32532
r = 1
s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48)
at forward.c:860
No locals.
#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0
183 Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,
len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273
msg = 0x7f14fb670c48
ctx = {rec_lev = 8868984, run_flags = 0, last_retcode
= 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0,
171719254808, 9586112}, __mask_was_saved = 1818983832,
__saved_mask = {__val = {139728091862768, 12884901899,
139728091862768, 4279488,
140735012372672, 140735012371728, 5477982,
0, 139727728366976, 50195, 171356018048, 9586112,
140735012371856, 140735012371776, 5474817, 4279488}}}}}
ret = 32532
inb = {s = 0x924600 "SIP/2.0 183 Session
Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}
__FUNCTION__ = "receive_msg"
#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536
len = 880
SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f14fb5add70
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{403182777, 139728091862768}, addr32 = {403182777, 0,
4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327,
32532, 0}, addr =
"\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0},
addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port
= 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2
= 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 =
{sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr = {__in6_u
= {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0,
0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1
'\001'}
__FUNCTION__ = "udp_rcv_loop"
#15 0x000000000046d447 in main_loop () at main.c:1617
i = 13
pid = 0
si = 0x7f14fb47d588
si_desc = "udp receiver child=13
sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
'\000' <repeats 19 times>,
"\177kl\377\177\000\000\020\245K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545
cfg_stream = 0xf42010
c = -1
r = 0
tmp = 0x7fff6c6b8f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a58
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3572644655
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"
__FUNCTION__ = "main"
Let me know if you need further information.
Regards,
Igor.

<http://www.linkedin.com/in/miconda>

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-10-09 17:50:57 UTC

Permalink

Hello Daniel,

I donât use the GIT repository. I still use the source from tar.gz.

Can you give me the link to the patch? I will patch acc module and compile it.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mardi 7 octobre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

I pushed a fix to kamailio master branch that should solve it -- last commits to acc module. If you can give it a try and report results, would be appreciated.

Cheers,
Daniel

On 03/10/14 17:00, Igor Potjevlesch wrote:

Hello Daniel,

Iâm just seeing that 4.2 is scheduled for 15th October.

I can wait until this date.

Regards,

Igor.

De : Igor Potjevlesch [mailto:igor.potjevlesch-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 18:43
Ã : miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org>
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : RE: [SR-Users] Crash Kamailio 4.1.4

Hello,

Thank you for your time on this issue.

I understand that I can define âdb_extraâ like:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$avp(s:pai);"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

Instead of:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$ai;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

And, in the request_route

request_route {

#Store PAI in AVP for accounting

$avp(s:pai)=$ai;

# per request initial checks

route(REQINIT);

[âŠ]

}

I can put this fix while waiting for 4.2.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 10:17
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers,
Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 24 septembre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 " <sip:ABCDEFGHIJ@> sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE <sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" < <sip:KLMNOPQRST-2F/Cjg+***@public.gmane.org> sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" < <sip:***@tru> sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Daniel-Constantin Mierla

2014-10-09 20:26:11 UTC

Permalink

Hello,

you can browse the git repository online and get from there the patch or
diff:

-
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=kamailio;a=history;f=modules/acc;hb=HEAD

Or on github:

- https://github.com/kamailio/kamailio/commits/master/modules/acc

Cheers,
Daniel

Post by Igor Potjevlesch
Hello Daniel,
I donât use the GIT repository. I still use the source from tar.gz.
Can you give me the link to the patch? I will patch acc module and
compile it.
Regards,
Igor.
*EnvoyÃ© :* mardi 7 octobre 2014 09:00
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
I pushed a fix to kamailio master branch that should solve it -- last
commits to acc module. If you can give it a try and report results,
would be appreciated.
Cheers,
Daniel
Hello Daniel,
Iâm just seeing that 4.2 is scheduled for 15th October.
I can wait until this date.
Regards,
Igor.
*EnvoyÃ© :* jeudi 2 octobre 2014 18:43
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* RE: [SR-Users] Crash Kamailio 4.1.4
Hello,
Thank you for your time on this issue.
modparam("acc", "db_extra",
"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=*$avp(s:pai)*;"
"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")
modparam("acc", "db_extra",
"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;*src_pai=$ai*;"
"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")
And, in the request_route
request_route {
* #Store PAI in AVP for accounting*
* $avp(s:pai)=$ai;*
# per request initial checks
route(REQINIT);
[âŠ]
}
I can put this fix while waiting for 4.2.
Regards,
Igor.
*EnvoyÃ© :* jeudi 2 octobre 2014 10:17
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
not really troubleshooting, as I got traveling, but I thought a
bit of it, also from performances point of view.
As it was related to parallel processing of 1xx/2xx sip responses
- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)
Instead of using locks or other workarounds in tm (lower
performances in all cases), better do it on the part doing
parallel access out of tm. Acc is doing some particular processing
-- it uses a callback for a reply to access the response, so it is
quite specific to it.
If you want to be safe side immediately, store the PAI in an avp
inside request_route and use that avp for accounting.
A solution will be there before 4.2 is out.
Cheers,
Daniel
Hello Daniel,
Have you had the opportunity to look at this?
Thank you for your feedback.
Regards,
Igor.
*EnvoyÃ© :* mercredi 24 septembre 2014 09:00
*Ã :* Igor Potjevlesch
*Cc :* 'Kamailio $SER$ - Users Mailing List'
*Objet :* Re: [SR-Users] Crash Kamailio 4.1.4
Hello,
so it is still related to PAI header -- I will have more time
to look at it by end of the week.
Cheers,
Daniel
Hello Daniel,
Patching has been done in the same time than the upgrade to 4.1.5.
A new crash occurred in pvapi.c (in addition of my other
recent post âCrash Kamailio 4.1.5â).
#0 0x000000000049580e in pv_get_strval
(msg=0x7f14efe24ea8, param=0x7f14fb65d058,
res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521
No locals.
#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8,
param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026
idxf = 0
idx = 0
pai_body = 0x7f14fb679d38
pai_uri = 0xa
i = 0
cur_id = 0
__FUNCTION__ = "pv_get_pai"
#2 0x0000000000499594 in pv_get_spec_value
(msg=0x7f14efe24ea8, sp=0x7f14fb65d040,
value=0x7fff6c6b7430) at pvapi.c:1266
ret = 0
__FUNCTION__ = "pv_get_spec_value"
#3 0x00007f14f7d3481d in extra2strar
(extra=0x7f14fb65d030, rq=0x7f14efe24ea8,
val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c,
type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261
value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}
n = 4
r = 0
__FUNCTION__ = "extra2strar"
#4 0x00007f14f7d2c3e3 in acc_db_request
(rq=0x7f14efe24ea8) at acc.c:474
m = 7
n = -270381400
i = 6
t = 0x414cc0
__FUNCTION__ = "acc_db_request"
#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8,
req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at
acc_logic.c:471
new_uri_bk = {s = 0x7f14efe25590
len = 19}
br = 0
hdr = 0x7f14f7d3dc20
__FUNCTION__ = "acc_onreply"
#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8,
type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573
__FUNCTION__ = "tmcb_func"
#7 0x00007f14f9f3146c in run_trans_callbacks_internal
(cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8,
params=0x7fff6c6b76b0) at t_hooks.c:290
cbp = 0x7f14ee4c81b0
backup_from = 0x934630
backup_to = 0x934638
backup_dom_from = 0x934640
backup_dom_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
__FUNCTION__ = "run_trans_callbacks_internal"
#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf
(type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8,
repl=0x7f14fb670c48, flags=183) at t_hooks.c:336
params = {req = 0x7f14efe24ea8, rpl =
0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags
= 183, branch = 0, t_rbuf = 0x7f14efe52678, dst =
0x7f14efe526c8, send_buf = {
s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP
trans = 0x7f14efe525b8
#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8,
p_msg=0x7f14fb670c48, branch=0, msg_status=183,
cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at
t_reply.c:2001
relay = 0
save_clone = 0
buf = 0x7f14fb67e740 "SIP/2.0 183 Session
Progress\r\nVia: SIP/2.0/UDP
res_len = 777
relayed_code = 183
relayed_msg = 0x7f14fb670c48
reply_bak = 0x414cc0
bm = {to_tag_val = {s = 0x7f14efe53b50 "", len =
-79437808}}
totag_retr = 0
reply_status = RPS_PROVISIONAL
uas_rb = 0x7f14efe52678
to_tag = 0x0
reason = {s = 0x800000001 <Address 0x800000001 out
of bounds>, len = 1}
onsend_params = {req = 0x200924970, rpl =
0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags
= 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40,
send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len =
-101469275}}
__FUNCTION__ = "relay_reply"
#10 0x00007f14f9f660ab in reply_received
(p_msg=0x7f14fb670c48) at t_reply.c:2499
msg_status = 183
last_uac_status = 183
ack = 0x40 <Address 0x40 out of bounds>
ack_len = 0
branch = 0
reply_status = -77092928
onreply_route = 1
cancel_data = {cancel_bitmap = 0, reason = {cause
= 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel =
0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}
uac = 0x7f14efe52720
t = 0x7f14efe525b8
lack_dst = {send_sock = 0x7f14fb584420, to = {s =
{sa_family = 57360, sa_data =
"C\373\024\177\000\000\000F#\000\000\000\000"}, sin =
{sin_family = 57360, sin_port = 64323, sin_addr = {s_addr
= 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {
sin6_family = 57360, sin6_port = 64323,
sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8
= "\000F#\000\000\000\000\000\020\341C\373\024\177\000",
__u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0},
__u6_addr32 = {2311680, 0, 4215529744,
32532}}}, sin6_scope_id =
4215529744}}, id = 32532, proto = 96 '`', send_flags = {f
backup_user_from = 0x934630
backup_user_to = 0x934638
backup_domain_from = 0x934640
backup_domain_to = 0x934648
backup_uri_from = 0x934620
backup_uri_to = 0x934628
backup_xavps = 0x934760
replies_locked = 1
branch_ret = 0
prev_branch = 1818983120
blst_503_timeout = 32767
hf = 0x7f14fb670c68
onsend_params = {req = 0x7fff6c6b7a90, rpl =
0x550bb0, param = 0x234540, code = 0, flags = 3, branch =
0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf
= {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}
ctx = {rec_lev = 0, run_flags = 0, last_retcode =
0, jmp_env = {{__jmpbuf = {139728093908544,
1898006658898931560, 4279488, 140735012372672, 0, 0,
1898006658936680296, -1897762211976106136},
__mask_was_saved = 0, __saved_mask = {__val = {9586373,
1365809186688, 124554051613, 9586450,
139728093947840, 9587056, 9586211, 361695345073193192,
9586309, 9586288, 4217874320, 139728093947840,
139728093942016, 139728093908544, 4279488,
140735012372672}}}}}
__FUNCTION__ = "reply_received"
#11 0x000000000045d853 in do_forward_reply
(msg=0x7f14fb670c48, mode=0) at forward.c:777
new_buf = 0x0
dst = {send_sock = 0x0, to = {s = {sa_family = 0,
sa_data = '\000' <repeats 13 times>}, sin = {sin_family =
0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0,
sin6_port = 0, sin6_flowinfo = 0,
sin6_addr = {__in6_u = {__u6_addr8 = '\000'
<repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0,
0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id
= 0, proto = 0 '\000', send_flags = {f = 0 '\000',
blst_imask = 0 '\000'}}
new_len = 32532
r = 1
s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>
len = 0
__FUNCTION__ = "do_forward_reply"
#12 0x000000000045e114 in forward_reply
(msg=0x7f14fb670c48) at forward.c:860
No locals.
#13 0x00000000004a5903 in receive_msg (buf=0x924600
"SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,
len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273
msg = 0x7f14fb670c48
ctx = {rec_lev = 8868984, run_flags = 0,
last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0,
263853236176, 1, 0, 171719254808, 9586112},
__mask_was_saved = 1818983832, __saved_mask = {__val =
{139728091862768, 12884901899, 139728091862768, 4279488,
140735012372672, 140735012371728,
5477982, 0, 139727728366976, 50195, 171356018048, 9586112,
140735012371856, 140735012371776, 5474817, 4279488}}}}}
ret = 32532
inb = {s = 0x924600 "SIP/2.0 183 Session
Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP
<IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}
__FUNCTION__ = "receive_msg"
#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536
len = 880
buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP
SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...
tmp = 0x9245c0 "10.143.1.10"
from = 0x7f14fb5add70
fromlen = 16
ri = {src_ip = {af = 2, len = 4, u = {addrl =
{403182777, 139728091862768}, addr32 = {403182777, 0,
4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024,
64327, 32532, 0}, addr =
"\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}},
dst_ip = {af = 2,
len = 4, u = {addrl = {67638457, 0}, addr32 =
{67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0,
0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}},
src_port = 5060, dst_port = 5060, proto_reserved1 = 0,
proto_reserved2 = 0, src_su = {s = {
sa_family = 2, sa_data =
"\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin =
{sin_family = 2, sin_port = 50195, sin_addr = {s_addr =
403182777}, sin_zero = "\000\000\000\000\000\000\000"},
sin6 = {sin6_family = 2, sin6_port = 50195,
sin6_flowinfo = 403182777, sin6_addr =
{__in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
__u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0,
0, 0, 0}}}, sin6_scope_id = 0}}, bind_address =
0x7f14fb47d588, proto = 1 '\001'}
__FUNCTION__ = "udp_rcv_loop"
#15 0x000000000046d447 in main_loop () at main.c:1617
i = 13
pid = 0
si = 0x7f14fb47d588
si_desc = "udp receiver child=13
sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177",
'\000' <repeats 19 times>,
"\177kl\377\177\000\000\020\245K\000\000\000\000"
nrprocs = 15
__FUNCTION__ = "main_loop"
#16 0x000000000047054f in main (argc=7,
argv=0x7fff6c6b80c8) at main.c:2545
cfg_stream = 0xf42010
c = -1
r = 0
tmp = 0x7fff6c6b8f70 ""
tmp_len = 0
port = 0
proto = 0
options = 0x5e0a58
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 3572644655
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x3d6f60fb88
p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"
__FUNCTION__ = "main"
Let me know if you need further information.
Regards,
Igor.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Igor Potjevlesch

2014-10-10 16:01:55 UTC

Permalink

Hello,

Thank you ! Not sure to have the time for testing it before 4.2.

Will it be ported to 4.2 or are you waiting for my feedback?

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : jeudi 9 octobre 2014 22:26
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

you can browse the git repository online and get from there the patch or diff:

- http://git.sip-router.org/cgi-bin/gitweb.cgi?p=kamailio;a=history;f=modules/acc;hb=HEAD

Or on github:

- https://github.com/kamailio/kamailio/commits/master/modules/acc

Cheers,
Daniel

On 09/10/14 19:50, Igor Potjevlesch wrote:

Hello Daniel,

I donât use the GIT repository. I still use the source from tar.gz.

Can you give me the link to the patch? I will patch acc module and compile it.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mardi 7 octobre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

I pushed a fix to kamailio master branch that should solve it -- last commits to acc module. If you can give it a try and report results, would be appreciated.

Cheers,
Daniel

On 03/10/14 17:00, Igor Potjevlesch wrote:

Hello Daniel,

Iâm just seeing that 4.2 is scheduled for 15th October.

I can wait until this date.

Regards,

Igor.

De : Igor Potjevlesch [mailto:igor.potjevlesch-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 18:43
Ã : miconda-***@public.gmane.org <mailto:miconda-***@public.gmane.org>
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : RE: [SR-Users] Crash Kamailio 4.1.4

Hello,

Thank you for your time on this issue.

I understand that I can define âdb_extraâ like:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$avp(s:pai);"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

Instead of:

modparam("acc", "db_extra",

"src_user=$fU;username=$Au;src_domain=$fd;src_ip=$si;src_pai=$ai;"

"dst_ouser=$tU;dst_user=$rU;dst_domain=$rd")

And, in the request_route

request_route {

#Store PAI in AVP for accounting

$avp(s:pai)=$ai;

# per request initial checks

route(REQINIT);

[âŠ]

}

I can put this fix while waiting for 4.2.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : jeudi 2 octobre 2014 10:17
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

not really troubleshooting, as I got traveling, but I thought a bit of it, also from performances point of view.

As it was related to parallel processing of 1xx/2xx sip responses that you are accounting, I think the safest solutions would be:

- clone the message locally to acc if it is the tm (share memory)
- do synchronized access to tm data inside the acc (use locks there)

Instead of using locks or other workarounds in tm (lower performances in all cases), better do it on the part doing parallel access out of tm. Acc is doing some particular processing -- it uses a callback for a reply to access the response, so it is quite specific to it.

If you want to be safe side immediately, store the PAI in an avp inside request_route and use that avp for accounting.

A solution will be there before 4.2 is out.

Cheers,
Daniel

On 02/10/14 09:50, Igor Potjevlesch wrote:

Hello Daniel,

Have you had the opportunity to look at this?

Thank you for your feedback.

Regards,

Igor.

De : Daniel-Constantin Mierla [mailto:miconda-***@public.gmane.org]
EnvoyÃ© : mercredi 24 septembre 2014 09:00
Ã : Igor Potjevlesch
Cc : 'Kamailio $SER$ - Users Mailing List'
Objet : Re: [SR-Users] Crash Kamailio 4.1.4

Hello,

so it is still related to PAI header -- I will have more time to look at it by end of the week.

Cheers,
Daniel

On 23/09/14 19:16, Igor Potjevlesch wrote:

Hello Daniel,

Patching has been done in the same time than the upgrade to 4.1.5.

A new crash occurred in pvapi.c (in addition of my other recent post âCrash Kamailio 4.1.5â).

Here is the result of a âbt fullâ:

#0 0x000000000049580e in pv_get_strval (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430, sval=0x22) at pvapi.c:521

No locals.

#1 0x00007f14f9677f2b in pv_get_pai (msg=0x7f14efe24ea8, param=0x7f14fb65d058, res=0x7fff6c6b7430) at pv_core.c:1026

idxf = 0

idx = 0

pai_body = 0x7f14fb679d38

pai_uri = 0xa

i = 0

cur_id = 0

__FUNCTION__ = "pv_get_pai"

#2 0x0000000000499594 in pv_get_spec_value (msg=0x7f14efe24ea8, sp=0x7f14fb65d040, value=0x7fff6c6b7430) at pvapi.c:1266

ret = 0

__FUNCTION__ = "pv_get_spec_value"

#3 0x00007f14f7d3481d in extra2strar (extra=0x7f14fb65d030, rq=0x7f14efe24ea8, val_arr=0x7f14f7f41e30, int_arr=0x7f14f7f4237c, type_arr=0x7f14f7f424e7 "\002\002\002\002") at acc_extra.c:261

value = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}

n = 4

r = 0

__FUNCTION__ = "extra2strar"

#4 0x00007f14f7d2c3e3 in acc_db_request (rq=0x7f14efe24ea8) at acc.c:474

m = 7

n = -270381400

i = 6

t = 0x414cc0

__FUNCTION__ = "acc_db_request"

#5 0x00007f14f7d36bc8 in acc_onreply (t=0x7f14efe525b8, req=0x7f14efe24ea8, reply=0x7f14fb670c48, code=200) at acc_logic.c:471

new_uri_bk = {s = 0x7f14efe25590 " <sip:ABCDEFGHIJ@> sip:ABCDEFGHIJ@<IP_GW>oTE sINVITE <sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> sip:ABCDEFGHIJ-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org SIP/2.0\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;branch=z9hG4bK5f32deec\r\nMax-Forwards: 69\r\nFrom: \"KLMNOPQRST\" < <sip:KLMNOPQRST-2F/Cjg+***@public.gmane.org> sip:KLMNOPQRST-2F/Cjg+qhPXhvxM+***@public.gmane.org"..., len = 19}

br = 0

hdr = 0x7f14f7d3dc20

__FUNCTION__ = "acc_onreply"

#6 0x00007f14f7d3730a in tmcb_func (t=0x7f14efe525b8, type=512, ps=0x7fff6c6b76b0) at acc_logic.c:573

__FUNCTION__ = "tmcb_func"

#7 0x00007f14f9f3146c in run_trans_callbacks_internal (cb_lst=0x7f14efe52628, type=512, trans=0x7f14efe525b8, params=0x7fff6c6b76b0) at t_hooks.c:290

cbp = 0x7f14ee4c81b0

backup_from = 0x934630

backup_to = 0x934638

backup_dom_from = 0x934640

backup_dom_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

__FUNCTION__ = "run_trans_callbacks_internal"

#8 0x00007f14f9f3167e in run_trans_callbacks_with_buf (type=512, rbuf=0x7f14efe52678, req=0x7f14efe24ea8, repl=0x7f14fb670c48, flags=183) at t_hooks.c:336

params = {req = 0x7f14efe24ea8, rpl = 0x7f14fb670c48, param = 0x7f14ee4c81c0, code = 200, flags = 183, branch = 0, t_rbuf = 0x7f14efe52678, dst = 0x7f14efe526c8, send_buf = {

s = 0x7f14efd7c408 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org> <sip:KLMNOPQRST-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org>"..., len = 980}}

trans = 0x7f14efe525b8

#9 0x00007f14f9f63bfa in relay_reply (t=0x7f14efe525b8, p_msg=0x7f14fb670c48, branch=0, msg_status=183, cancel_data=0x7fff6c6b7a10, do_put_on_wait=1) at t_reply.c:2001

relay = 0

save_clone = 0

buf = 0x7f14fb67e740 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32deec\r\nCall-ID: <mailto:7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org%5Cr%5CnFrom> 7846f7332ce6a7db4484c3d06ce1c387-2F/Cjg+qhPUVpVtQvJkt/***@public.gmane.org\r\nFrom: \"KLMNOPQRST\" < <sip:***@tru> sip:***@tru"...

res_len = 777

relayed_code = 183

relayed_msg = 0x7f14fb670c48

reply_bak = 0x414cc0

bm = {to_tag_val = {s = 0x7f14efe53b50 "", len = -79437808}}

totag_retr = 0

reply_status = RPS_PROVISIONAL

uas_rb = 0x7f14efe52678

to_tag = 0x0

reason = {s = 0x800000001 <Address 0x800000001 out of bounds>, len = 1}

onsend_params = {req = 0x200924970, rpl = 0x7f14f9f83f90, param = 0x414cc0, code = 1818984640, flags = 3, branch = 0, t_rbuf = 0x0, dst = 0x7f14fb670e40, send_buf = {s = 0x7fff6c6b7830 "`xkl\377\177", len = -101469275}}

__FUNCTION__ = "relay_reply"

#10 0x00007f14f9f660ab in reply_received (p_msg=0x7f14fb670c48) at t_reply.c:2499

msg_status = 183

last_uac_status = 183

ack = 0x40 <Address 0x40 out of bounds>

ack_len = 0

branch = 0

reply_status = -77092928

onreply_route = 1

cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 9586205}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 9586205}}}}

uac = 0x7f14efe52720

t = 0x7f14efe525b8

lack_dst = {send_sock = 0x7f14fb584420, to = {s = {sa_family = 57360, sa_data = "C\373\024\177\000\000\000F#\000\000\000\000"}, sin = {sin_family = 57360, sin_port = 64323, sin_addr = {s_addr = 32532}, sin_zero = "\000F#\000\000\000\000"}, sin6 = {

sin6_family = 57360, sin6_port = 64323, sin6_flowinfo = 32532, sin6_addr = {__in6_u = {__u6_addr8 = "\000F#\000\000\000\000\000\020\341C\373\024\177\000", __u6_addr16 = {17920, 35, 0, 0, 57616, 64323, 32532, 0}, __u6_addr32 = {2311680, 0, 4215529744,

32532}}}, sin6_scope_id = 4215529744}}, id = 32532, proto = 96 '`', send_flags = {f = 64 '@', blst_imask = 103 'g'}}

backup_user_from = 0x934630

backup_user_to = 0x934638

backup_domain_from = 0x934640

backup_domain_to = 0x934648

backup_uri_from = 0x934620

backup_uri_to = 0x934628

backup_xavps = 0x934760

replies_locked = 1

branch_ret = 0

prev_branch = 1818983120

blst_503_timeout = 32767

hf = 0x7f14fb670c68

onsend_params = {req = 0x7fff6c6b7a90, rpl = 0x550bb0, param = 0x234540, code = 0, flags = 3, branch = 0, t_rbuf = 0x7f14fb43e380, dst = 0x7f14fb674030, send_buf = {s = 0x7fff6c6b7a90 "`G\223", len = 5538065}}

ctx = {rec_lev = 0, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {139728093908544, 1898006658898931560, 4279488, 140735012372672, 0, 0, 1898006658936680296, -1897762211976106136}, __mask_was_saved = 0, __saved_mask = {__val = {9586373,

1365809186688, 124554051613, 9586450, 139728093947840, 9587056, 9586211, 361695345073193192, 9586309, 9586288, 4217874320, 139728093947840, 139728093942016, 139728093908544, 4279488, 140735012372672}}}}}

__FUNCTION__ = "reply_received"

#11 0x000000000045d853 in do_forward_reply (msg=0x7f14fb670c48, mode=0) at forward.c:777

new_buf = 0x0

dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0,

sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}

new_len = 32532

r = 1

s = 0x370fb670c50 <Address 0x370fb670c50 out of bounds>

len = 0

__FUNCTION__ = "do_forward_reply"

#12 0x000000000045e114 in forward_reply (msg=0x7f14fb670c48) at forward.c:860

No locals.

#13 0x00000000004a5903 in receive_msg (buf=0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...,

len=880, rcv_info=0x7fff6c6b7d90) at receive.c:273

msg = 0x7f14fb670c48

ctx = {rec_lev = 8868984, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {0, 0, 0, 263853236176, 1, 0, 171719254808, 9586112}, __mask_was_saved = 1818983832, __saved_mask = {__val = {139728091862768, 12884901899, 139728091862768, 4279488,

140735012372672, 140735012371728, 5477982, 0, 139727728366976, 50195, 171356018048, 9586112, 140735012371856, 140735012371776, 5474817, 4279488}}}}}

ret = 32532

inb = {s = 0x924600 "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"..., len = 880}

__FUNCTION__ = "receive_msg"

#14 0x000000000053c9c4 in udp_rcv_loop () at udp_server.c:536

len = 880

buf = "SIP/2.0 183 Session Progress\r\nVia: SIP/2.0/UDP <IP_KAMAILIO>;branch=z9hG4bKca48.51c2c569361ea0fedd9c6c70c21b5eed.0;received=<IP_KAMAILIO>\r\nVia: SIP/2.0/UDP <IP_UAC>:5060;rport=5060;branch=z9hG4bK5f32d"...

tmp = 0x9245c0 "10.143.1.10"

from = 0x7f14fb5add70

fromlen = 16

ri = {src_ip = {af = 2, len = 4, u = {addrl = {403182777, 139728091862768}, addr32 = {403182777, 0, 4215789296, 32532}, addr16 = {5305, 6152, 0, 0, 55024, 64327, 32532, 0}, addr = "\271\024\b\030\000\000\000\000\360\326G\373\024\177\000"}}, dst_ip = {af = 2,

len = 4, u = {addrl = {67638457, 0}, addr32 = {67638457, 0, 0, 0}, addr16 = {5305, 1032, 0, 0, 0, 0, 0, 0}, addr = "\271\024\b\004", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {

sa_family = 2, sa_data = "\023ÃÂ¹\024\b\030\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 403182777}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195,

sin6_flowinfo = 403182777, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f14fb47d588, proto = 1 '\001'}

__FUNCTION__ = "udp_rcv_loop"

#15 0x000000000046d447 in main_loop () at main.c:1617

i = 13

pid = 0

si = 0x7f14fb47d588

si_desc = "udp receiver child=13 sock=<IP_KAMAILIO>:5060\000\373\024\177\000\000\b$P\373\024\177\000\000\036\205^\000\000\000\000\000\000w^\000\000\000\000\000\000\244\303v\000\000\000\000\300LA\000\000\000\000\000\300\200kl\377\177", '\000' <repeats 19 times>, "\177kl\377\177\000\000\020\245K\000\000\000\000"

nrprocs = 15

__FUNCTION__ = "main_loop"

#16 0x000000000047054f in main (argc=7, argv=0x7fff6c6b80c8) at main.c:2545

cfg_stream = 0xf42010

c = -1

r = 0

tmp = 0x7fff6c6b8f70 ""

tmp_len = 0

port = 0

proto = 0

options = 0x5e0a58 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

ret = -1

seed = 3572644655

rfd = 4

debug_save = 0

debug_flag = 0

dont_fork_cnt = 0

n_lst = 0x3d6f60fb88

p = 0x5caba0 "H\211l$\330L\211d$\340H\215-O\244*"

__FUNCTION__ = "main"

Let me know if you need further information.

Regards,

Igor.
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda

--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda