Video conference issue
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber does
not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the left
side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side of
the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Hello,
The problem was because of wrong decoding direction of call 1 and encoding
direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber does
not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the left
side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side of
the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Hi Branko,
It is interesting to hear any feedback on the new video conference feature.
But I am afraid I don't completely understand the question, could you
elaborate more about the problem, e.g: what settings is in "high settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and encoding
direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber
does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the
left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side of
the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028. My
application use low level id.
As described below I have success 3pty video call only with all three equal
terminals which has the same seettings. With all another there is no video
for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application has
equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before 3pty
user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints
is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port =
pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port =
pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port
== PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d",
pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port =
pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port =
pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port
== PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d",
pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port ->
call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port ->
call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port ->
win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d",
pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green.
But only see video from user1 which is initiator. -
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width,
height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal
settings for H264 encoder/decoder (levelid=42801e) - user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before
3pty user2 is on hold and user3 is connected with video call with user - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is other
manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, //
when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to
all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width,
height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id,
unsigned profile_iop, unsigned level_id, unsigned width, unsigned
height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id,
level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d",
level_info.id, level_info.def_w, level_info.def_h,
level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w *
param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) *
0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate )
? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate )
? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str,
param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen *
sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s",
param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d",
param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status =
pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin nanang@pjsip.org
napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber
does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the
left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side of
the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an issue
in call setup, once the video call is established properly (i.e: video nego
successful in the SDP offer/answer), it should not cause problems in video
conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028. My
application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application has
equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before 3pty
user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints
is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is other
manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin nanang@pjsip.org
napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber
does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the
left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side
of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Nanang,
Thanks for your response and help.
I'm sending you a log with HOLD - test case 2.
I still have to check without hold.
Regards,
Branko
V V sre., 11. sep. 2019 ob 09:55 je oseba Nanang Izzuddin nanang@pjsip.org
napisala:
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an issue
in call setup, once the video call is established properly (i.e: video nego
successful in the SDP offer/answer), it should not cause problems in video
conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028. My
application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application has
equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before 3pty
user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is other
manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber
does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2, EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the
left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side
of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Nanang,
Without HOLD it works. But I need HOLD. Did you found something from log
trace?
If I have only one video call and HOLD/RETRIEVE works. Only with video
conference bridge it doesn't work.
Thanks.
Regards,
Branko
V V čet., 26. sep. 2019 ob 11:44 je oseba Branko Zebec <
branko.zebec@gmail.com> napisala:
Hi Nanang,
Thanks for your response and help.
I'm sending you a log with HOLD - test case 2.
I still have to check without hold.
Regards,
Branko
V V sre., 11. sep. 2019 ob 09:55 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an
issue in call setup, once the video call is established properly (i.e:
video nego successful in the SDP offer/answer), it should not cause
problems in video conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028. My
application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application
has equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before 3pty
user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is
other manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second subscriber
does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2,
EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on the
left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right side
of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Branko,
Cannot see anything suspicious from the log, but I think there is a
possible scenario that may cause it:
- the video conf bridge needs to know the resolution of all ports, so if
any port updates the resolution, it needs to be updated by refreshing the
connections in the vid conf bridge. - unfortunately, the exact resolution of incoming video will only be known
after some successful decodings, while the vid conf connections may have
been established.
I could not reproduce it earlier perhaps because I used pjsua app, where
the port connections were set up manually using 'vid' commands and the
incoming video resolution might had been known when the connection setups
were being done.
The attached patch will allow PJSUA to automatically refresh all vid conf
bridge port connections after an incoming video resolution is known (i.e:
on format changed event).
BR,
nanang
On Tue, Oct 1, 2019 at 7:17 PM Branko Zebec branko.zebec@gmail.com wrote:
Hi Nanang,
Without HOLD it works. But I need HOLD. Did you found something from log
trace?
If I have only one video call and HOLD/RETRIEVE works. Only with video
conference bridge it doesn't work.
Thanks.
Regards,
Branko
V V čet., 26. sep. 2019 ob 11:44 je oseba Branko Zebec <
branko.zebec@gmail.com> napisala:
Hi Nanang,
Thanks for your response and help.
I'm sending you a log with HOLD - test case 2.
I still have to check without hold.
Regards,
Branko
V V sre., 11. sep. 2019 ob 09:55 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an
issue in call setup, once the video call is established properly (i.e:
video nego successful in the SDP offer/answer), it should not cause
problems in video conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028.
My application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application
has equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before 3pty
user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is
other manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second
subscriber does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2,
EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on
the left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right
side of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Hi Nanang,
This patch solves the problem.
I found that it also works with the added delay (2 - 3seconds)
between successful retrieve (reINVITE with sendreceive) and call of
function void SIP::connectSessionConfVideoPoints(int session1, int session2)
which connect video ports with pjsua_vid_conf_connect.
Thanks for your help.
BR
Branko
V V tor., 15. okt. 2019 ob 08:45 je oseba Nanang Izzuddin nanang@pjsip.org
napisala:
Hi Branko,
Cannot see anything suspicious from the log, but I think there is a
possible scenario that may cause it:
- the video conf bridge needs to know the resolution of all ports, so if
any port updates the resolution, it needs to be updated by refreshing the
connections in the vid conf bridge. - unfortunately, the exact resolution of incoming video will only be
known after some successful decodings, while the vid conf connections may
have been established.
I could not reproduce it earlier perhaps because I used pjsua app, where
the port connections were set up manually using 'vid' commands and the
incoming video resolution might had been known when the connection setups
were being done.
The attached patch will allow PJSUA to automatically refresh all vid conf
bridge port connections after an incoming video resolution is known (i.e:
on format changed event).
BR,
nanang
On Tue, Oct 1, 2019 at 7:17 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
Without HOLD it works. But I need HOLD. Did you found something from log
trace?
If I have only one video call and HOLD/RETRIEVE works. Only with video
conference bridge it doesn't work.
Thanks.
Regards,
Branko
V V čet., 26. sep. 2019 ob 11:44 je oseba Branko Zebec <
branko.zebec@gmail.com> napisala:
Hi Nanang,
Thanks for your response and help.
I'm sending you a log with HOLD - test case 2.
I still have to check without hold.
Regards,
Branko
V V sre., 11. sep. 2019 ob 09:55 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an
issue in call setup, once the video call is established properly (i.e:
video nego successful in the SDP offer/answer), it should not cause
problems in video conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028.
My application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application
has equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before
3pty user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is
other manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second
subscriber does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2,
EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on
the left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right
side of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Glad to hear that. Thanks for the confirmation.
Just checked in the patch to SVN trunk with ticket
https://trac.pjsip.org/repos/ticket/2245.
BR,
nanang
On Tue, Oct 15, 2019 at 3:27 PM Branko Zebec branko.zebec@gmail.com wrote:
Hi Nanang,
This patch solves the problem.
I found that it also works with the added delay (2 - 3seconds)
between successful retrieve (reINVITE with sendreceive) and call of
function void SIP::connectSessionConfVideoPoints(int session1, int session2)
which connect video ports with pjsua_vid_conf_connect.
Thanks for your help.
BR
Branko
V V tor., 15. okt. 2019 ob 08:45 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
Cannot see anything suspicious from the log, but I think there is a
possible scenario that may cause it:
- the video conf bridge needs to know the resolution of all ports, so if
any port updates the resolution, it needs to be updated by refreshing the
connections in the vid conf bridge. - unfortunately, the exact resolution of incoming video will only be
known after some successful decodings, while the vid conf connections may
have been established.
I could not reproduce it earlier perhaps because I used pjsua app, where
the port connections were set up manually using 'vid' commands and the
incoming video resolution might had been known when the connection setups
were being done.
The attached patch will allow PJSUA to automatically refresh all vid conf
bridge port connections after an incoming video resolution is known (i.e:
on format changed event).
BR,
nanang
On Tue, Oct 1, 2019 at 7:17 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
Without HOLD it works. But I need HOLD. Did you found something from log
trace?
If I have only one video call and HOLD/RETRIEVE works. Only with video
conference bridge it doesn't work.
Thanks.
Regards,
Branko
V V čet., 26. sep. 2019 ob 11:44 je oseba Branko Zebec <
branko.zebec@gmail.com> napisala:
Hi Nanang,
Thanks for your response and help.
I'm sending you a log with HOLD - test case 2.
I still have to check without hold.
Regards,
Branko
V V sre., 11. sep. 2019 ob 09:55 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
The problem of test case 1 (not working on 15fps, working on 30fps) is
quite strange indeed. Tried to reproduce here, three party video conf, all
endpoints using profile-level-id=428028, 352x288 @15Hz, but still no luck
(i.e: user 3 sees the video mix properly, initiator on left & user 2 on
right).
Regarding test case 2, profile-level-id difference should only be an
issue in call setup, once the video call is established properly (i.e:
video nego successful in the SDP offer/answer), it should not cause
problems in video conference.
As the symptoms seem to be around '"user 3 cannot see user 2 video",
perhaps it was somehow related to call hold. So could you try again, but
never hold the first call. Note that when a call is on hold, the streams
(audio or video) may be deactivated/removed and connecting the call stream
ports in conference bridge can be useless. If the problem persists, could
you also send the PJSIP log file with log level 5 too?
BR,
nanang
On Fri, Sep 6, 2019 at 4:35 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hi Nanang,
I mean with high settings H264 profile, etc. profile-level-id=428028.
My application use low level id.
As described below I have success 3pty video call only with all three
equal terminals which has the same seettings. With all another there is no
video for first one. Very interesting is that only from first user which is
called firstly is no video. This is always the same also with another video
terminals.
Two test cases:
TEST1:
- All three android phones (user1, user2, user3) with my application
has equal settings for H264 encoder/decoder (levelid=42801e). - user1 is initiator and first call user 2 and then user3. Before
3pty user2 is on hold and user3 is connected with video call with user 1. - After 3pty user2 is connected again with user1 and also user3.
Function connectSessionConfVideoPoints is called.
void SIP::connectSessionConfVideoPoints(int session1, int session2)
{
LogInf("session1=%d, session2=%d", session1, session2);
int pjCallId1 = this->convABSTRcallId2PJcallId(session1);
int pjCallId2 = this->convABSTRcallId2PJcallId(session2);
if (PJSUA_INVALID_ID != pjCallId1 && PJSUA_INVALID_ID != pjCallId2)
{
// Get video ports of call 1, there are two ports as in a video call as
// encoding and decoding directions may not use the same frame rate or
// resolution.
pjsua_conf_port_id call_1_dec_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_1_enc_port = pjsua_call_get_vid_conf_port(pjCallId1, PJMEDIA_DIR_ENCODING);
if ((call_1_dec_port == PJSUA_INVALID_ID) || (call_1_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId1=%d", pjCallId1);
return;
}
// Get video ports of call 2
pjsua_conf_port_id call_2_dec_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_DECODING);
pjsua_conf_port_id call_2_enc_port = pjsua_call_get_vid_conf_port(pjCallId2, PJMEDIA_DIR_ENCODING);
if ((call_2_dec_port == PJSUA_INVALID_ID) || (call_2_enc_port == PJSUA_INVALID_ID)) {
LogErr("ERROR pjsua_call_get_vid_conf_port pjCallId2=%d", pjCallId2);
return;
}
// Connect video ports of call 1 and call 2.
// Note that the source is the stream port in decoding direction,
// and the sink is the stream port in encoding direction.
pj_status_t status = PJ_FALSE;
status = pjsua_vid_conf_connect(call_1_dec_port, call_2_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> call_2_enc_port");
}
status = pjsua_vid_conf_connect(call_2_dec_port, call_1_enc_port, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_2_dec_port -> call_1_enc_port");
}
pjsua_vid_win_id wid1, wid2;
pjsua_vid_win_info win2_info;
// Put incoming video stream from call 1 into call 2 window
wid2 = pjsua_call_get_vid_win(pjCallId2);
if (wid2 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId2");
return;
}
pjsua_vid_win_get_info(wid2, &win2_info);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_get_info win2_info");
}
pjsua_vid_conf_connect(call_1_dec_port, win2_info.slot_id, NULL);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_conf_connect call_1_dec_port -> win2_info.slot_id");
}
// Now hide the video window of call 1
wid1 = pjsua_call_get_vid_win(pjCallId1);
if (wid1 == PJSUA_INVALID_ID)
{
LogErr("ERROR pjsua_call_get_vid_win pjCallId1");
return;
}
pjsua_vid_win_set_show(wid1, PJ_FALSE);
if (status != PJ_SUCCESS)
{
LogErr("ERROR pjsua_vid_win_set_show wid1");
}
}
else
{
LogErr("ERROR PJSUA_INVALID_ID pjCallId1=%d, pjCallId2=%d", pjCallId1, pjCallId2);
}
}
-
After that user3 does not see video from user2. Picture is green. But only see video from user1 which is initiator.
-
If I change fps=15 to fps=30 for encoder then user3 see video from user2.
This works:
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
With
int fps = 15;
it does not works.
TEST2:
- Two android phones (user1, user3) with my application has equal settings for H264 encoder/decoder (levelid=42801e)
- user2 is other manufacturer and has (levelid=428028). This can't be changed.
- user1 is initiator and first call user 2 and then user3. Before 3pty user2 is on hold and user3 is connected with video call with user 1.
- After 3pty user2 is connected again with user1 and also user3. Function connectSessionConfVideoPoints is called.
No setting helps here fps, resolution, levelid on user1 and user3.
After 3pty user3 always does not see video from this user2 which is
other manufacturer and with other settings. Picture is green.
I tried a lot of different settings for:
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
And it does not help. Picture is still green.
For test encoder/decoder settings is used function:
/* Declaration of H.264 level info /
typedef struct h264_level_info_t
{
unsigned id; / Level id. /
unsigned max_mbps; / Max macroblocks per second. /
unsigned max_mb; / Max macroblocks. /
unsigned bitrate; / Max bitrate (kbps). /
unsigned def_w; / Default width. /
unsigned def_h; / Default height. /
unsigned def_fps; / Default fps. */
} h264_level_info_t;
/* Declaration of H.264 profile info /
typedef struct h264_profile_info_t
{
unsigned idx; / profile index /
unsigned id; / profile id. /
char name; /* profile name */
} h264_profile_info_t;
static const h264_level_info_t H264_LEVELS_INFO[] =
{
{ 10, 1485, 99, 64, 176, 144, 15 },
{ 9, 1485, 99, 128, 176, 144, 15 }, /*< level 1b */
{ 11, 3000, 396, 192, 320, 240, 10 },
{ 12, 6000, 396, 384, 352, 288, 15 },
{ 13, 11880, 396, 768, 352, 288, 15 },
{ 20, 11880, 396, 2000, 352, 288, 30 },
{ 21, 19800, 792, 4000, 352, 288, 30 },
{ 22, 20250, 1620, 4000, 352, 288, 30 },
{ 30, 40500, 1620, 10000, 720, 480, 30 },
{ 31, 108000, 3600, 14000, 1280, 720, 30 },
{ 32, 216000, 5120, 20000, 1280, 720, 30 },
{ 40, 245760, 8192, 20000, 1920, 1080, 30 },
{ 41, 245760, 8192, 50000, 1920, 1080, 30 },
{ 42, 522240, 8704, 50000, 1920, 1080, 30 },
{ 50, 589824, 22080, 135000, 1920, 1080, 30 },
{ 51, 983040, 36864, 240000, 1920, 1080, 30 },
};
static const h264_profile_info_t h264_PROFILE[] =
{
{ 0, 0, "No" }, // when profile=RCDO and level=0 - "RCDO" - RCDO bitstream MUST obey to all the constraints of the Baseline profile
{ 1, 66, "Baseline" }, // 42 in-hex
{ 2, 77, "Main" }, // 4d in-hex
{ 3, 88, "Extended" }, // 58 in-hex
{ 4, 100, "High" }, // 64 in-hex
{ 5, 110, "High 10" }, // 6e in-hex
{ 6, 122, "High 4:2:2" }, // 7a in-hex
{ 7, 244, "High 4:4:4" }, // f4 in-hex
{ 8, 44, "CAVLC 4:4:4" }, // 2c in-hex
//profiles for SVC - Scalable Video Coding extension to H.264
{ 9, 83, "Scalable Baseline" }, // 53 in-hex
{ 10, 66, "Scalable High" }, // 56 in-hex
//profiles for MVC - Multiview Video Coding extension to H.264
{ 11, 128, "Stereo High" }, // 80 in-hex
{ 12, 118, "Multiview High" }, // 76 in-hex
{ 13, 138, "Multiview Depth High" }, // 8a in-hex
};
// For video H264
//
// profile Baseline 42 hex - 66 dec
// profile Main 4d hex - 77 dec// profile High 64 hex - 100
int h264profileId = h264_PROFILE[1].id;
int h264profileIop = 128;
// https://en.wikipedia.org/wiki/H.264/MPEG-4_AVC#Levels
int h264level = H264_LEVELS_INFO[8].id; // 3.0 from table H264_LEVELS_INFO
int width = 352;
int height = 288;
int fps = 30;
int h264bitrate = 256; // bandwith in (kbps)
setCodecH264Profile(h264profileId, h264profileIop, h264level, width, height, fps, h264bitrate, 0);
void SIPcodecManager::setCodecH264Profile(unsigned profile_id, unsigned profile_iop, unsigned level_id, unsigned width, unsigned height, unsigned fps, unsigned avg_kbps, unsigned max_kbps)
{
pj_status_t status = PJ_ENOTSUP;
#if PJMEDIA_HAS_VIDEO
LogInf("Set H264 profile %d-%d %dx%d@%d %dkbps", profile_id, level_id, width, height, fps, avg_kbps);
pjmedia_vid_codec_param param;
unsigned i;
const pj_str_t codec_id = { "H264", 4 };
const pj_str_t PROFILE_LEVEL_ID = {"profile-level-id", 16};
h264_level_info_t level_info;
int macro_block_size_sec;
char profile_level_id_str[7];
status = PJ_EINVAL;
status = pjsua_vid_codec_get_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
if(level_id == 0 && width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
int idx;
for(idx = 0; idx < PJ_ARRAY_SIZE(H264_LEVELS_INFO); ++idx){
if(H264_LEVELS_INFO[idx].max_mbps <= macro_block_size_sec){
level_id = H264_LEVELS_INFO[idx].id;
}else{
break;
}
}
}
bool ret = getH264LevelInfo( (level_id > 0) ? level_id : 30, &level_info);
if(ret != true) {
return;
}
// Check level regarding width/height parameters
if(width > 0 && height > 0 && fps > 0){
macro_block_size_sec = ((width+15) / 16) * ((height+15) / 16 ) * fps;
if(macro_block_size_sec > level_info.max_mbps){
// Invalid reg selected level
width = height = fps = 0;
}
}else{
// If we have not the 3 params, it's invalid
width = height = fps = 0;
}
LogInf("Found default infos for this level %d %dx%d@%d", level_info.id, level_info.def_w, level_info.def_h, level_info.def_fps);
param.enc_fmt.det.vid.size.w = (width > 0) ? width : level_info.def_w;
param.enc_fmt.det.vid.size.h = (height > 0) ? height : level_info.def_h;
param.enc_fmt.det.vid.fps.num = (fps > 0 ) ? fps : level_info.def_fps;
param.enc_fmt.det.vid.fps.denum = 1;
if(avg_kbps == 0){
/* H264 primer from adobe : w x h x fps x motion rank x 0.07 */
avg_kbps = ((float)(param.enc_fmt.det.vid.size.w * param.enc_fmt.det.vid.size.h * param.enc_fmt.det.vid.fps.num) ) * 0.07;
}
if(max_kbps == 0) {
max_kbps = avg_kbps;
}
param.enc_fmt.det.vid.avg_bps = ( avg_kbps <= level_info.bitrate ) ? avg_kbps * 1000 : level_info.bitrate * 1000;
param.enc_fmt.det.vid.max_bps = ( max_kbps <= level_info.bitrate ) ? max_kbps * 1000 : level_info.bitrate * 1000;
// We expect here to already have fmtp_level_profile_id
for (i = 0; i < param.dec_fmtp.cnt; ++i) {
if (pj_stricmp(¶m.dec_fmtp.param[i].name, &PROFILE_LEVEL_ID) == 0) {
if(param.dec_fmtp.param[i].val.slen == 6) {
// First copy current value
pj_memcpy(profile_level_id_str, param.dec_fmtp.param[i].val.ptr, param.dec_fmtp.param[i].val.slen * sizeof(char));
// Set profile_id
if(profile_id > 0){
pj_val_to_hex_digit(profile_id, (profile_level_id_str));
}
// Set profile_iop
if(profile_iop > 0){
pj_val_to_hex_digit(profile_iop, (profile_level_id_str+2));
}
// Set level_id
if(level_id > 0) {
pj_val_to_hex_digit(level_id, (profile_level_id_str+4));
}
profile_level_id_str[6] = '\0';
param.dec_fmtp.param[i].val = pj_str(profile_level_id_str);
LogInf("Profile is now %.*s", param.dec_fmtp.param[i].val.slen, param.dec_fmtp.param[i].val.ptr);
} else {
LogInf("Impossible to set dec_fmtp %d", param.dec_fmtp.param[i].val.slen);
}
}
}
status = pjsua_vid_codec_set_param(&codec_id, ¶m);
if(status != PJ_SUCCESS) {
return;
}
#endif
}
For Android window I call:
void* window is pointer from java:
static Surface surfaceInVideoWindow;
static Surface surfacePreviewVideoWindow;
void SIP::setVideoWindow(int windowId, void* window) {
pj_status_t status = PJ_SUCCESS;
LogInf("Entry...");
pjmedia_vid_dev_hwnd vhwnd;
vhwnd.type = PJMEDIA_VID_DEV_HWND_TYPE_ANDROID;
vhwnd.info.window = window;
if(windowId == 0) {
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_win_set_win");
}
} else if (windowId == 1) {
pjsua_vid_win_id wid;
pjsua_vid_win_info wi;
pjsua_vid_preview_param pre_param;
if(window == NULL) {
status = pjsua_vid_preview_stop(PJMEDIA_VID_DEFAULT_CAPTURE_DEV);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_stop");
}
} else {
pjsua_vid_preview_param_default(&pre_param);
pre_param.rend_id = PJMEDIA_VID_DEFAULT_RENDER_DEV;
pre_param.show = PJ_TRUE;
pre_param.wnd = vhwnd;
status = pjsua_vid_preview_start(PJMEDIA_VID_DEFAULT_CAPTURE_DEV, &pre_param);
if (status != PJ_SUCCESS) {
LogInf("Error pjsua_vid_preview_start");
}
}
} else if(windowId == 2){
status = pjsua_vid_win_set_win(windowId, &vhwnd);
if(status != PJ_SUCCESS){
LogInf("Error pjsua_vid_win_set_win");
}
}
}
Thanks for the help.
Best regards,
Branko
V V pet., 6. sep. 2019 ob 07:52 je oseba Nanang Izzuddin <
nanang@pjsip.org> napisala:
Hi Branko,
It is interesting to hear any feedback on the new video conference
feature. But I am afraid I don't completely understand the question, could
you elaborate more about the problem, e.g: what settings is in "high
settings"?
If it is about frame rate & resolution, the video conference should
automatically readjust the frame rate (and also resize accordingly) video
data from the source ports.
BR,
nanang
On Wed, Aug 14, 2019 at 3:47 PM Branko Zebec branko.zebec@gmail.com
wrote:
Hello,
The problem was because of wrong decoding direction of call 1 and
encoding direction of call 2 (frame rate, resolution).
But question is how to solve it if we have different terminals with
different settings. My Android terminal can decode with high settings, but
can't encode with high settings.
Thanks.
Regards,
Branko
---------- Forwarded message ---------
Od: Branko Zebec branko.zebec@gmail.com
Date: V tor., 13. avg. 2019 ob 13:11
Subject: Video conference issue
To: pjsip list pjsip@lists.pjsip.org
Hello,
I did 3pty conference (new version of PJSIP 2.9) and second
subscriber does not see right picture from first subscriber.
Picture is in green color.
I have it in Android project with openH264 library and with GLESv2,
EGL.
Test scenario:
- Initiator call subscriber 1
- Between initiator and subscriber 1 is video fine.
- Initiator put subscriber 1 to HOLD
- Initiator call subscriber 2
- Between initiator and subscriber 2 is video fine.
- Initiator create 3pty
- Video from initiator to subsciber 1 and subscriber 2 is fine on
the left side of the window of both subscriber 1 and 2.
Video from subscriber 2 is fine on the right side of the window of
subscriber 1.
Video from subscriber 1 is not ok and is green color on the right
side of the window of subscriber 2. And this is main issue.
I have equal settings on all three (bandwith, codec profile).
What else could i check?
Thanks for helping me.
Regards,
Branko
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
Visit our blog: http://blog.pjsip.org
pjsip mailing list
pjsip@lists.pjsip.org
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org