Re: [Maildev] Adding dogtail tests?
On 08/03/2019 17:10, Samuel Thibault wrote:
Put another way, overall the Thunderbird testing infrastructure doesn't
seem to allow us to easily write testing scenarii just from our user
testcases. Would it be possible to have the thunderbird testsuite
include a series using dogtail, whose interface is very close to what
the user actually interacts with, and thus allows us to very easily
write usercase tests? The overall testing time would probably be like
Yes, MozMill testing is quite a pain, plus MozMill has been deprecated
by Mozilla for use in Firefox testing; they use Marionette now.
It's always desirable to have simple and complete test coverage, but I'm
not the right guy to discuss this with. We're in the right place for
this discussion on the Maildev mailing list, so I hope others will chime in.
Jörg.
On 3/8/19 10:50 PM, Jörg Knobloch wrote:
On 08/03/2019 17:10, Samuel Thibault wrote:
Put another way, overall the Thunderbird testing infrastructure doesn't
seem to allow us to easily write testing scenarii just from our user
testcases. Would it be possible to have the thunderbird testsuite
include a series using dogtail, whose interface is very close to what
the user actually interacts with, and thus allows us to very easily
write usercase tests? The overall testing time would probably be like
Yes, MozMill testing is quite a pain, plus MozMill has been deprecated
by Mozilla for use in Firefox testing; they use Marionette now.
It's always desirable to have simple and complete test coverage, but
I'm not the right guy to discuss this with. We're in the right place
for this discussion on the Maildev mailing list, so I hope others will
chime in.
My main concern is that dogtail is Linux only (correct me if I am
wrong). If we introduce a new test suite, we'll want one that is cross
platform.
Thunderbird also has mochitests now
(https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest),
and I think the next logical step would be Marionette. I'd be open to
other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
Philipp
Hello,
Philipp Kewisch, le ven. 08 mars 2019 23:14:04 +0100, a ecrit:
On 3/8/19 10:50 PM, Jörg Knobloch wrote:
On 08/03/2019 17:10, Samuel Thibault wrote:
Put another way, overall the Thunderbird testing infrastructure doesn't
seem to allow us to easily write testing scenarii just from our user
testcases. Would it be possible to have the thunderbird testsuite
include a series using dogtail, whose interface is very close to what
the user actually interacts with, and thus allows us to very easily
write usercase tests? The overall testing time would probably be like
Yes, MozMill testing is quite a pain, plus MozMill has been deprecated
by Mozilla for use in Firefox testing; they use Marionette now.
It's always desirable to have simple and complete test coverage, but
I'm not the right guy to discuss this with. We're in the right place
for this discussion on the Maildev mailing list, so I hope others will
chime in.
My main concern is that dogtail is Linux only
Yes, it is. The idea is also to test that the eventual behavior is
alright. I.e. test the glue between thunderbird's accessibility layer
and the actual OS accessibility layer.
(correct me if I am wrong). If we introduce a new test suite, we'll
want one that is cross platform.
The principle of what I propose on
https://git.hypra.fr/hypra/regression-testing/tree/master
is actually quite cross-platform. It currently uses dogtail functions to
actually do tests, but it can be made to use another engine that would
work on windows or MacOS.
Thunderbird also has mochitests now
(https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest),
and I think the next logical step would be Marionette. I'd be open
to other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
But AIUI Marionette's actions are directly given to Thunderbird, without
going through the accessibility layer of Thunderbird and of the OS,
while we do want to test these ones too (notably the glue between the
two).
Samuel
Samuel Thibault, le dim. 10 mars 2019 00:15:46 +0100, a ecrit:
Philipp Kewisch, le ven. 08 mars 2019 23:14:04 +0100, a ecrit:
and I think the next logical step would be Marionette. I'd be open
to other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
But AIUI Marionette's actions are directly given to Thunderbird, without
going through the accessibility layer of Thunderbird and of the OS,
while we do want to test these ones too (notably the glue between the
two).
I could play with Marionette, it indeed sends events directly to
Thunderbird. It however seems to have a simple enough interface that I
can plug my tests both into it and into dogtail, and thus get tests
both running in marionette for Thunderbird regression testing, and
running through dogtail for more general regression testing.
Samuel
Hello,
Samuel Thibault, le mer. 20 mars 2019 18:52:29 +0100, a ecrit:
Samuel Thibault, le dim. 10 mars 2019 00:15:46 +0100, a ecrit:
Philipp Kewisch, le ven. 08 mars 2019 23:14:04 +0100, a ecrit:
and I think the next logical step would be Marionette. I'd be open
to other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
I could play with Marionette, it indeed sends events directly to
Thunderbird. It however seems to have a simple enough interface that I
can plug my tests both into it and into dogtail,
I have experimented with marionette running on thunderbird. I do get
some stuff done (triggering a menu item, that opens a dialog box,
activating a button in it), but there are missing elements in what I am
seeing from marionette. Typically, when opening the "Account Settings"
dialog (Edit->Account Settings), I do get the button "Account Actions",
but I do not get the buttons "Add", "Edit", etc. Typically, running this
code while that dialog is opened:
l = self.marionette.find_elements(By.XPATH, "//*")
for i in l:
print(i.tag_name)
print(i.get_attribute("label"))
brings the list quoted below. We can indeed see the "Account Actions"
button, but not the others. In the "Add Mail account" dialog box, I only
get stringbundle and script elements, while the dialog does contain
labels, entries, and buttons. Could it be that e.g. the script or iframe
parts hide the rest?
Samuel
dialog
stringbundle
stringbundle
script
script
script
script
script
hbox
vbox
tree
treecols
treecol
label
image
treecolpicker
image
menupopup
menuseparator
menuitem
Restore Column Order
treechildren
treeitem
treerow
treecell
Outgoing Server (SMTP)
button
Account Actions
menupopup
menuitem
Add Mail Account…
menuitem
Add Chat Account…
menuitem
Add Feed Account…
menuitem
Add Other Account…
menuseparator
menuitem
Set as Default
menuitem
Remove Account
iframe
Samuel Thibault, le mer. 10 avril 2019 17:00:48 +0200, a ecrit:
but there are missing elements in what I am
seeing from marionette. Typically, when opening the "Account Settings"
dialog (Edit->Account Settings), I do get the button "Account Actions",
but I do not get the buttons "Add", "Edit", etc. Typically, running this
code while that dialog is opened:
l = self.marionette.find_elements(By.XPATH, "//*")
for i in l:
print(i.tag_name)
print(i.get_attribute("label"))
brings the list quoted below. We can indeed see the "Account Actions"
button, but not the others. In the "Add Mail account" dialog box, I only
get stringbundle and script elements, while the dialog does contain
labels, entries, and buttons.
Could it be that e.g. the script or iframe parts hide the rest?
Ok, for the iframe thing it seems it can be looked through with
switch_to_frame and I can access the rest of the dialog.
But for the "Add Mail account" dialog, AIUI stringbundleset and
stringbundle are useless, so I'm left with the script elements only,
which I don't know what to do with.
Samuel
Samuel Thibault, le mer. 10 avril 2019 17:41:46 +0200, a ecrit:
But for the "Add Mail account" dialog, AIUI stringbundleset and
stringbundle are useless, so I'm left with the script elements only,
which I don't know what to do with.
It seems something just prevents most of the content from
showing up: if I add a <p>foo</p> before the <script> tags in
./mail/components/accountcreation/content/emailWizard.xul, I do get it
with find_elements. But if I put it after the
Samuel Thibault, le mer. 10 avril 2019 18:01:58 +0200, a ecrit:
Samuel Thibault, le mer. 10 avril 2019 17:41:46 +0200, a ecrit:
But for the "Add Mail account" dialog, AIUI stringbundleset and
stringbundle are useless, so I'm left with the script elements only,
which I don't know what to do with.
It seems something just prevents most of the content from
showing up: if I add a <p>foo</p> before the <script> tags in
./mail/components/accountcreation/content/emailWizard.xul, I do get it
with find_elements. But if I put it after the
Mmm, waiting a bit with time.sleep() before making the findElements
request apparently allows to get all nodes, so I guess somehow
marionette asks WebDriver too early after the window started to get
created and it's still being constructed. Could there be a way to wait
for the construction to settle down?
Samuel
On 3/9/2019 6:15 PM, Samuel Thibault wrote:
Hello,
Philipp Kewisch, le ven. 08 mars 2019 23:14:04 +0100, a ecrit:
On 3/8/19 10:50 PM, Jörg Knobloch wrote:
On 08/03/2019 17:10, Samuel Thibault wrote:
Put another way, overall the Thunderbird testing infrastructure doesn't
seem to allow us to easily write testing scenarii just from our user
testcases. Would it be possible to have the thunderbird testsuite
include a series using dogtail, whose interface is very close to what
the user actually interacts with, and thus allows us to very easily
write usercase tests? The overall testing time would probably be like
Yes, MozMill testing is quite a pain, plus MozMill has been deprecated
by Mozilla for use in Firefox testing; they use Marionette now.
It's always desirable to have simple and complete test coverage, but
I'm not the right guy to discuss this with. We're in the right place
for this discussion on the Maildev mailing list, so I hope others will
chime in.
My main concern is that dogtail is Linux only
Yes, it is. The idea is also to test that the eventual behavior is
alright. I.e. test the glue between thunderbird's accessibility layer
and the actual OS accessibility layer.
(correct me if I am wrong). If we introduce a new test suite, we'll
want one that is cross platform.
The principle of what I propose on
https://git.hypra.fr/hypra/regression-testing/tree/master
is actually quite cross-platform. It currently uses dogtail functions to
actually do tests, but it can be made to use another engine that would
work on windows or MacOS.
Thunderbird also has mochitests now
(https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest),
and I think the next logical step would be Marionette. I'd be open
to other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
But AIUI Marionette's actions are directly given to Thunderbird, without
going through the accessibility layer of Thunderbird and of the OS,
while we do want to test these ones too (notably the glue between the
two).
Samuel
Samuel, have you been able to make progress in this area?
And what cross platform engine(s) are potential candidate?
Hello,
Wayne Mery, le dim. 13 oct. 2019 10:55:48 -0400, a ecrit:
On 3/9/2019 6:15 PM, Samuel Thibault wrote:
The principle of what I propose on
[1]https://salsa.debian.org/hypra-team/regression-testing.git
is actually quite cross-platform. It currently uses dogtail functions to
actually do tests, but it can be made to use another engine that would
work on windows or MacOS.
Thunderbird also has mochitests now
([2]https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Mochitest),
and I think the next logical step would be Marionette. I'd be open
to other test frameworks, but for Marionette the barrier of entry is
probably small given Firefox already uses it.
But AIUI Marionette's actions are directly given to Thunderbird, without
going through the accessibility layer of Thunderbird and of the OS,
while we do want to test these ones too (notably the glue between the
two).
Samuel, have you been able to make progress in this area?
I have added marionette support to Thunderbird and introduced a couple
of regression tests using it.
And what cross platform engine(s) are potential candidate?
In the hypra repository mentioned above, I have introduced a python
module called "regression" which exposes a very basic API. The "log"
script can emit it from the recording of a user using the application.
The "dogtail2marionette" script can then be used to convert it to a
marionette test script. There are corner cases to fix by hand, but it
works nicely enough to produce tests quite efficiently.
Samuel