We currently use mozmill for much of the UI testing of thunderbird and lightning. I've noticed that is has a number of deficiencies: - The logs are poorly organized. For a particular example, looking at this failure <https://treeherder.mozilla.org/logviewer.html#?job_id=154614341&repo=comm-central&lineNumber=3203> (from bug 1420695 <https://bugzilla.mozilla.org/show_bug.cgi?id=1420695>) the bit of the log that led me to a potential fix is here <https://taskcluster.github.io/unified-logviewer/?url=https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FLga8mPbkRCCoYsMKvIH02A%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log&highlightStart=3246&highlightEnd=3254&lineNumber=3196&wrapLines=false&showLineNumbers=true&jumpToHighlight=false&followLog=false&lineHeight=13&asText=false> (which is some ways after the next test failure) - The treeherder test viewer <https://treeherder.mozilla.org/testview.html?repo=comm-central&revision=27191bf09df89656d92f1daba6d9a6e88f398bba> doesn't show mozmill tests, since it doesn't emit the right kind of structured logs. We currently use a heavily patched version of mozmill 1.5.6, and there is a newer version 2.0.10 <https://github.com/mozilla/mozmill/tree/master/mozmill> but I'm not sure. There has also been suggestion <https://bugzilla.mozilla.org/show_bug.cgi?id=487380#c5> of moving to another test framework. I'm interested in improving the state of our tooling in this area, and was wondering if people think that trying to upgrade mozmill is worthwhile at this point, or if it would be better to just double-down on our fork in the short-term (since mozmill is deprecated upstream anyway) to improve the above issues. -- Tom

----- Pôvodná správa ----- Predmet: [Maildev] Mozmill and testing frameworks Od: Tom Prince <mozilla@hocat.ca> Pre: Thunderbird Dev <maildev@lists.thunderbird.net> Dátum: Mon, 08 Jan 2018 16:55:33 +0000 > We currently use mozmill for much of the UI testing of thunderbird and > lightning. I've noticed that is has a number of deficiencies: > > * The logs are poorly organized. For a particular example, looking at > this failure > <https://treeherder.mozilla.org/logviewer.html#?job_id=154614341&repo=comm-central&lineNumber=3203> > (from bug 1420695 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1420695>) the bit of > the log that led me to a potential fix is here > <https://taskcluster.github.io/unified-logviewer/?url=https%3A%2F%2Fqueue.taskcluster.net%2Fv1%2Ftask%2FLga8mPbkRCCoYsMKvIH02A%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive_backing.log&highlightStart=3246&highlightEnd=3254&lineNumber=3196&wrapLines=false&showLineNumbers=true&jumpToHighlight=false&followLog=false&lineHeight=13&asText=false> > (which is some ways after the next test failure) > * The treeherder test viewer > <https://treeherder.mozilla.org/testview.html?repo=comm-central&revision=27191bf09df89656d92f1daba6d9a6e88f398bba> > doesn't show mozmill tests, since it doesn't emit the right kind of > structured logs. I haven't used any of the log viewers you mention here so I can't comment on their deficiencies. > We currently use a heavily patched version of mozmill 1.5.6, and there > is a newer version 2.0.10 > <https://github.com/mozilla/mozmill/tree/master/mozmill> but I'm not > sure. There has also been suggestion > <https://bugzilla.mozilla.org/show_bug.cgi?id=487380#c5> of moving to > another test framework. Even the newest mozmill version is very old and our modifications also need to be adapted to it. Stuff like top level vars, undefined vars and similar warnings we had to hunt for due to changes in JS engine over the years. But migrating to other test framework is currently our of the question. E.g. there is marionette, but that appears to be in python. Rewriting all of our test suite is not something we have the workforce yet. > I'm interested in improving the state of our tooling in this area, and > was wondering if people think that trying to upgrade mozmill is > worthwhile at this point, or if it would be better to just double-down > on our fork in the short-term (since mozmill is deprecated upstream > anyway) to improve the above issues. Does the newer version improve on the problems you mention? Or do you just ask into which mozmill version to make improvements to solve your problems? That is a hard question. I wanted to try mozmill 2.0 in the past due to all the improvements and some of them could improve the intermittent focus/event problems we observe e.g. on Linux. Apparently our tests should work in mozmill 2.0 without many modifications (if any). But nobody has tried yet and the supposed improvements are not confirmed. If you had the time to try it out, I would be thankful. However the attempts so far were unsuccessful, the differences from stock mozmill are heaping up and even the GSoC proposal to try this was voted down. So it is up to the Council to decide whether you should devote time to this experiment, as it is not free. Would you maybe be able to check the mozmill 2.0 package and asses how much time it could take you to drop it into our tree and make the necessary changes to it at least runs? Maybe you have better tools and experience with this. You do not need to bother to make all tests pass, we can then do the necessary tweaks. aceman

On Mon, Jan 8, 2018 at 1:15 PM ace <acelists@atlas.sk> wrote: > But migrating to other test framework is currently our of the question. > E.g. there is marionette, but that appears to be in python. Rewriting > all of our test suite is not something we have the workforce yet. Certainly rewriting everything at once is a non-starter. But adding a new framework, and pushing new tests to use it, and possibly migrating tests as they change might be feasible > Does the newer version improve on the problems you mention? Or do you > just ask into which mozmill version to make improvements to solve your > problems? There appeared to be enough change to our forked copy of mozmill that just dropping in a newer version seemed non-trivial last time I looked. It looked like (among other things) there a number of places where both we and upstream-mozmill made changes to reflect gecko changes, but in incompatible ways that leak into tests. And I think there are probably change that we have made to keep up with gecko since mozmill upstream was deprecated. Also, I haven't looked at the history of mozilla automation to see whether the logging improvements and changes happened before or after mozmill was deprecated upstream. I do think there is probably a bunch of low-hanging fruit that could be done to improve the state of our mozmill test-suite. But doing so is going to diverge us more from upstream, making any future attempts at upgrading to mozmill 2.0. -- Tom

On 08-01-2018 22:56, Tom Prince wrote: > On Mon, Jan 8, 2018 at 1:15 PM ace <acelists@atlas.sk > <mailto:acelists@atlas.sk>> wrote: > > But migrating to other test framework is currently our of the question. > > E.g. there is marionette, but that appears to be in python. Rewriting > > all of our test suite is not something we have the workforce yet. > > Certainly rewriting everything at once is a non-starter. But adding a > new framework, and pushing new tests to use it, and possibly migrating > tests as they change might be feasible > > > Does the newer version improve on the problems you mention? Or do you > > just ask into which mozmill version to make improvements to solve your > > problems? > > There appeared to be enough change to our forked copy of mozmill that > just dropping in a newer version seemed non-trivial last time I > looked. It looked like (among other things) there a number of places > where both we and upstream-mozmill made changes to reflect gecko > changes, but in incompatible ways that leak into tests. And I think > there are probably change that we have made to keep up with gecko > since mozmill upstream was deprecated. Also, I haven't looked at the > history of mozilla automation to see whether the logging improvements > and changes happened before or after mozmill was deprecated upstream. > > I do think there is probably a bunch of low-hanging fruit that could > be done to improve the state of our mozmill test-suite. But doing so > is going to diverge us more from upstream, making any future attempts > at upgrading to mozmill 2.0. > Since mozmill is dead I doubt it's worth migrating to 2.0. Probably Andrew's suggestion (the one you linked to) is what makes most sense: https://bugzilla.mozilla.org/show_bug.cgi?id=487380#c5 - TL;DR move to marionette but find a way to expose current mozmill JS logic so current (and future) tests can use it.  -Magnus