Empathy List Archives

AG

Adrian Georgescu

Mon, Sep 13, 2010 10:44 AM

On Sep 13, 2010, at 12:36 PM, Benny Prijono wrote:

We're good, thanks! I'm still looking forward to seeing you again in
one of the next SIPits.

Right, that one. Actually, most pjmedia components are thread
agnostic, hence it can accommodate any threading models that the
designer chooses to achieve high performance. But, there is an
exception to this, i.e. the conference bridge, which need to work in
one thread, which makes sense since it needs to "synchronize" the
timing of all ports connected to it. Having said that, nothing
prevents you from creating multiple conference bridges and make each
of them use different thread.

We are using this model in SIP SIMPLE client SDK and Blink (multiple conference bridges) and we run into random crashes caused by multithreading issues. Hard to debug and impossible to reproduce deterministically but its happens sooner or later after running for awhile. Probably nobody else have done this before in production environments and this scenario was never properly tested, this is why I am mentioning it here.

Adrian

On Sep 13, 2010, at 12:36 PM, Benny Prijono wrote: > We're good, thanks! I'm still looking forward to seeing you again in > one of the next SIPits. > > Right, that one. Actually, most pjmedia components are thread > agnostic, hence it can accommodate any threading models that the > designer chooses to achieve high performance. But, there is an > exception to this, i.e. the conference bridge, which need to work in > one thread, which makes sense since it needs to "synchronize" the > timing of all ports connected to it. Having said that, nothing > prevents you from creating multiple conference bridges and make each > of them use different thread. We are using this model in SIP SIMPLE client SDK and Blink (multiple conference bridges) and we run into random crashes caused by multithreading issues. Hard to debug and impossible to reproduce deterministically but its happens sooner or later after running for awhile. Probably nobody else have done this before in production environments and this scenario was never properly tested, this is why I am mentioning it here. Adrian

BP

Benny Prijono

Mon, Sep 13, 2010 11:02 PM

On Mon, Sep 13, 2010 at 5:44 PM, Adrian Georgescu ag@ag-projects.com wrote:

We are using this model in SIP SIMPLE client SDK and Blink (multiple conference bridges) and we run into random crashes caused by multithreading issues. Hard to debug and impossible to reproduce deterministically but its happens sooner or later after running for awhile. Probably nobody else have done this before in production environments and this scenario was never properly tested, this is why I am mentioning it here.

Hi Adrian,

True I haven't tried this scenario myself, but conferencing is one of
the thing that's central to PJSUA-LIB's design, hence it's been used
by everybody and I think it's fair to say it's stable. Now when you're
supposed to just create multiple instances of the same thing, I would
expect it to stay stable if done properly.

And we've discussed your problem few times off the list. Apart from
this, you also have done major modifications elsewhere to the library,
so it could be that too. And all this time, I never saw a crash dump,
so it's impossible to say what this is.

Best regards
Benny

On Mon, Sep 13, 2010 at 5:44 PM, Adrian Georgescu <ag@ag-projects.com> wrote: > > We are using this model in SIP SIMPLE client SDK and Blink (multiple conference bridges) and we run into random crashes caused by multithreading issues. Hard to debug and impossible to reproduce deterministically but its happens sooner or later after running for awhile. Probably nobody else have done this before in production environments and this scenario was never properly tested, this is why I am mentioning it here. > Hi Adrian, True I haven't tried this scenario myself, but conferencing is one of the thing that's central to PJSUA-LIB's design, hence it's been used by everybody and I think it's fair to say it's stable. Now when you're supposed to just create multiple instances of the same thing, I would expect it to stay stable if done properly. And we've discussed your problem few times off the list. Apart from this, you also have done major modifications elsewhere to the library, so it could be that too. And all this time, I never saw a crash dump, so it's impossible to say what this is. Best regards Benny

AG

Adrian Georgescu

Tue, Sep 14, 2010 7:56 AM

Hi Benny,

It is true that we have done modifications though in parts not related with this and your support was always to the point.

Trying to isolate the possible cause for this illusive problem that I still need to solve, I wish to ask the list if anyone else is using more than one conference bridge at a time in their application and if so if they encountered such a problem?

An excerpt of the problem is here:

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Crashed Thread: 6

Application Specific Information:
Assertion failed: (mutex->owner == pj_thread_this()), function pj_mutex_unlock, file ../src/pj/os_core_unix.c, line 1234.

Thread 6 Crashed:
0 libSystem.B.dylib 0x929b728e __semwait_signal_nocancel + 10
1 libSystem.B.dylib 0x929b7172 nanosleep$NOCANCEL$UNIX2003 + 166
2 libSystem.B.dylib 0x92a3203a usleep$NOCANCEL$UNIX2003 + 61
3 libSystem.B.dylib 0x92a536f0 abort + 105
4 libSystem.B.dylib 0x92a4028c __pthread_markcancel + 0
5 _core.so 0x05b7870b pj_mutex_unlock + 283
6 _core.so 0x05aeb3ec get_frame + 556
7 _core.so 0x05afc767 play_cb + 71
8 _core.so 0x05b0dc70 PaPlayerCallback + 416
9 _core.so 0x05b72a18 AdaptingOutputOnlyProcess + 216
10 _core.so 0x05b734b6 PaUtil_EndBufferProcessing + 550
11 _core.so 0x05b6b05c AudioIOProc + 2652
12 ...pple.audio.units.Components 0x70008c2f 0x70000000 + 35887
13 ...pple.audio.units.Components 0x70009830 0x70000000 + 38960
14 ...pple.audio.units.Components 0x70008d08 0x70000000 + 36104
15 ....audio.toolbox.AudioToolbox 0x92245f5b AudioConverterChain::CallInputProc(unsigned long) + 881
16 ....audio.toolbox.AudioToolbox 0x92245a83 AudioConverterChain::FillBufferFromInputProc(unsigned long*, CABufferList*) + 95
17 ....audio.toolbox.AudioToolbox 0x92245a11 BufferedAudioConverter::GetInputBytes(unsigned long, unsigned long&, CABufferList const*&) + 199
18 ....audio.toolbox.AudioToolbox 0x922458cd CBRConverter::RenderOutput(CABufferList*, unsigned long, unsigned long&, AudioStreamPacketDescription*) + 129
19 ....audio.toolbox.AudioToolbox 0x92245677 BufferedAudioConverter::FillBuffer(unsigned long&, AudioBufferList&, AudioStreamPacketDescription*) + 315
20 ....audio.toolbox.AudioToolbox 0x922457df AudioConverterChain::RenderOutput(CABufferList*, unsigned long, unsigned long&, AudioStreamPacketDescription*) + 157
21 ....audio.toolbox.AudioToolbox 0x92245677 BufferedAudioConverter::FillBuffer(unsigned long&, AudioBufferList&, AudioStreamPacketDescription*) + 315
22 ....audio.toolbox.AudioToolbox 0x922453ec AudioConverterFillComplexBuffer + 268
23 ...pple.audio.units.Components 0x70008df5 0x70000000 + 36341
24 ...pple.audio.units.Components 0x700082b4 0x70000000 + 33460
25 ...pple.audio.units.Components 0x70006b64 0x70000000 + 27492
26 ...pple.audio.units.Components 0x700057c3 0x70000000 + 22467
27 ...pple.audio.units.Components 0x7000b9fb AUGenericOutputEntry + 7172
28 com.apple.audio.CoreAudio 0x9795b454 HP_IOProc::Call(AudioTimeStamp const&, AudioTimeStamp const&, AudioBufferList const*, AudioTimeStamp const&, AudioBufferList*) + 374
29 com.apple.audio.CoreAudio 0x9795b1ba IOA_Device::CallIOProcs(AudioTimeStamp const&, AudioTimeStamp const&, AudioTimeStamp const&) + 370
30 com.apple.audio.CoreAudio 0x9795afb6 HP_IOThread::PerformIO(AudioTimeStamp const&, double) + 620
31 com.apple.audio.CoreAudio 0x9795839a HP_IOThread::WorkLoop() + 2506
32 com.apple.audio.CoreAudio 0x979579cb HP_IOThread::ThreadEntry(HP_IOThread*) + 17
33 com.apple.audio.CoreAudio 0x979578e2 CAPThread::Entry(CAPThread*) + 140
34 libSystem.B.dylib 0x9297781d _pthread_start + 345
35 libSystem.B.dylib 0x929776a2 thread_start + 34

Regards,
Adrian

On Sep 14, 2010, at 1:02 AM, Benny Prijono wrote:

On Mon, Sep 13, 2010 at 5:44 PM, Adrian Georgescu ag@ag-projects.com wrote:

We are using this model in SIP SIMPLE client SDK and Blink (multiple conference bridges) and we run into random crashes caused by multithreading issues. Hard to debug and impossible to reproduce deterministically but its happens sooner or later after running for awhile. Probably nobody else have done this before in production environments and this scenario was never properly tested, this is why I am mentioning it here.

Hi Adrian,

True I haven't tried this scenario myself, but conferencing is one of
the thing that's central to PJSUA-LIB's design, hence it's been used
by everybody and I think it's fair to say it's stable. Now when you're
supposed to just create multiple instances of the same thing, I would
expect it to stay stable if done properly.

And we've discussed your problem few times off the list. Apart from
this, you also have done major modifications elsewhere to the library,
so it could be that too. And all this time, I never saw a crash dump,
so it's impossible to say what this is.

Best regards
Benny

pjsip@lists.pjsip.org

Create a PBX