I have a suggestion for tryserver runs. Suggestion/Proposal On the tryserver, we should run a mochitest and xpcshell test using the ASAN binary at least once a month for C-C, and/or as new nightly appears. For running xpcshell tests, we should add "--serialize" and "--verbose" flag. This is how I run my local memory tests of C-C TB. Otherwise, the error messages may not be visible at all and the error messages get mixed up among different processes and hard to analyze. I would also suggest a monthly valgrind run. If valgrind + C-C TB runs under the linux OS of tryserver. I have not been able to run C-C TB mochitest under valgrind at all since we switched to mochitest from |make mozmill| test suite. Tough. Also, I have no idea why, but valgrind cannot run the big TB binary as ordinary user anymore on my Debian GNU/Linux OS. It seems that Valgrind+TB fails when the stack needs to be extended (i.e., new page needs to be mapped for the stack.). This is my educated guess of where the error occurs by comparing the verbose output of valgrind when it runs C-C TB as a normal user and superuser. Valgrind+C-C TB does run under superuser. However, mochitest has its own problem and I never get past the initial window or anything. I set the time out to more than 30 minutes, but eventually, the timeout is triggered before no tests are performed with mochitest. |make mozmll| ran fine after my tweaking of timeout values... xpcshelltests do run and I could find a few memory-related errors. So at least ASAN version should give us a minimum check. However, if the memory errors are found and the error logs are visible to anybody including blackhats, we are in trouble unless the problems are taken care of immediately. Chiaki

Hi Chiaki, I'm planning for the ASAN builds to run once a day along with the nightly build. I've been finishing up some work in Taskgraph to allow that to happen. Once that is working automated tests can be added to those builds. I've held off on the tests for now to keep our build resource usage down. -Rob On 11/15/20 4:17 AM, ISHIKAWA,chiaki wrote: > I have a suggestion for tryserver runs. > > Suggestion/Proposal > > On the tryserver, we should run a mochitest and xpcshell test using > the ASAN binary at least once a month for C-C, and/or as new nightly > appears. > For running xpcshell tests, we should add "--serialize" and > "--verbose" flag. > This is how I run my local memory tests of C-C TB. > Otherwise, the error messages may not be visible at all and the error > messages get mixed up among different processes and hard to analyze. > > I would also suggest a monthly valgrind run. > If valgrind + C-C TB runs under the linux OS of tryserver. > > I have not been able to run C-C TB mochitest under valgrind at all > since we switched to mochitest from |make mozmill| test suite. Tough. > Also, I have no idea why, but valgrind cannot run the big TB binary as > ordinary user anymore on my Debian GNU/Linux OS. It seems that > Valgrind+TB fails when the stack needs to be extended (i.e., new page > needs to be mapped for the stack.). > This is my educated guess of where the error occurs by comparing the > verbose output of valgrind when it runs C-C TB as a normal user and > superuser. > Valgrind+C-C TB does run under superuser. However, mochitest has its > own problem and I never get past the initial window or anything. I set > the time out to more than 30 minutes, but eventually, the timeout is > triggered before no tests are performed with mochitest. > |make mozmll| ran fine after my tweaking of timeout values... > xpcshelltests do run and I could find a few memory-related errors. > > So at least ASAN version should give us a minimum check. > > However, if the memory errors are found and the error logs are visible > to anybody including blackhats, we are in trouble unless the problems > are taken care of immediately. > > Chiaki > > > > _______________________________________________ > Maildev mailing list > Maildev@lists.thunderbird.net > http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net > -- *Rob Lemley* Thunderbird Release Engineer rob@thunderbird.net - :rjl -*- #thereisonlyxul -*-

On 2020/11/16 7:08, Rob Lemley wrote: > Hi Chiaki, > > I'm planning for the ASAN builds to run once a day along with the > nightly build. I've been finishing up some work in Taskgraph to allow > that to happen. Once that is working automated tests can be added to > those builds. I've held off on the tests for now to keep our build > resource usage down. > > -Rob > Thank you. That would be great. Yeah, ASAN is resource hungry. valgrind run is much more so. Chiaki > > On 11/15/20 4:17 AM, ISHIKAWA,chiaki wrote: >> I have a suggestion for tryserver runs. >> >> Suggestion/Proposal >> >> On the tryserver, we should run a mochitest and xpcshell test using >> the ASAN binary at least once a month for C-C, and/or as new nightly >> appears. >> For running xpcshell tests, we should add "--serialize" and >> "--verbose" flag. >> This is how I run my local memory tests of C-C TB. >> Otherwise, the error messages may not be visible at all and the error >> messages get mixed up among different processes and hard to analyze. >> >> I would also suggest a monthly valgrind run. >> If valgrind + C-C TB runs under the linux OS of tryserver. >> >> I have not been able to run C-C TB mochitest under valgrind at all >> since we switched to mochitest from |make mozmill| test suite. Tough. >> Also, I have no idea why, but valgrind cannot run the big TB binary >> as ordinary user anymore on my Debian GNU/Linux OS. It seems that >> Valgrind+TB fails when the stack needs to be extended (i.e., new page >> needs to be mapped for the stack.). >> This is my educated guess of where the error occurs by comparing the >> verbose output of valgrind when it runs C-C TB as a normal user and >> superuser. >> Valgrind+C-C TB does run under superuser. However, mochitest has its >> own problem and I never get past the initial window or anything. I >> set the time out to more than 30 minutes, but eventually, the timeout >> is triggered before no tests are performed with mochitest. >> |make mozmll| ran fine after my tweaking of timeout values... >> xpcshelltests do run and I could find a few memory-related errors. >> >> So at least ASAN version should give us a minimum check. >> >> However, if the memory errors are found and the error logs are >> visible to anybody including blackhats, we are in trouble unless the >> problems are taken care of immediately. >> >> Chiaki >> >> >> >> _______________________________________________ >> Maildev mailing list >> Maildev@lists.thunderbird.net >> http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net >> > > -- > *Rob Lemley* > Thunderbird Release Engineer > rob@thunderbird.net - :rjl > -*- #thereisonlyxul -*- > > _______________________________________________ > Maildev mailing list > Maildev@lists.thunderbird.net > http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net