This sounds very like, it's simpler for us, is it really important to consider the users,
or the addon developers... Especially after reading Jonathan's comment about how it
doesn't even really work as simply as laid out.... Could this have been depreciated or
something over time... Did it have to be done, bang, all at once?

Blessings
Graeme

I think this can make sense. It seems that Geoff didn't do this lightly. These decisions are based on actual situation in the code, and postponing would mean to stop make significant efforts.

OTOH, we should not treat the time of our add-on developers for free. while the action itselftakes maybe 5 minutes plus 20 minutes testing, the problem needs to be discovered first, there needs to be a release, the dev needs to sit down etc., the overhead of a release is significant.

plus, some devs are simply absent, and users have to deal with a broken add-on, which is a cost of its own.

for all these reasons, i would suggest to make the transition automatic. get all add-ons, reformat the install.rdf into manifest.json, to it up again, put it back on ATN. this is the least effort for everybody.

if we depend on add-on devs for everything, there will soon most add-ons gone. and then users will not update, which is not in the interest of anyone.

so, i think the decision makes sense, but then we should also in exchange help the add-on devs.

Ben

Am 21. Mai 2019 18:19:36 MESZ schrieb Graeme musiquegraeme@gmail.com:

--
Sent from my phone. Please excuse the brevity.

On 5/21/19 6:00 PM, Ben Bucksch wrote:

I would like to speak specifically to this point.

I am currently working as a contractor, which is not my first choice but
it's the work I was able to find when I needed to find work. I'm a
full-time contractor, which means I can consistently bill and get paid
-- rather well, fortunately -- for 40 hours per week if I'm actually
working. But I can't bill my employer for work I do on Thunderbird
add-ons, obviously.

I'm very busy outside of work, which means I really have very little
free time. When there is some warning about changes to add-on
compatibility coming down the pike, and preferably the changes are
introduced gradually in a backward-compatible way, I can usually look at
my schedule and figure out when during my non-work hours I can make the
necessary changes.

When, on the other hand, incompatible changes just appear overnight in
Thunderbird without any prior warning, then my commitment to maintaining
my add-ons makes me feel obligated to address them ASAP. You may think
that this should not be necessary because nightly builds are
bleeding-edge, but in fact every time my add-ons break for nightly
builds I hear about if from multiple users quickly. This is much more
true now that nightly builds update automatically and insistently --
incompatible changes go out within a day to just about everyone who is
running the nightly build.

I also need to address such changes ASAP because I personally run the
nightly builds exactly so that I can find out about incompatibilities in
my add-ons quickly.

Therefore, after these changes were announced yesterday, I had to spend
most of the work day fixing five of my add-ons to be compatible with
them. I literally lost hundreds of dollars in income as a result.

I can't speak for all add-on developers, but I can tell you that for me,
personally, I am trying /really, really hard/ to keep up with all the
changes going into Thunderbird, even at great personal expense. I would
greatly appreciate it if the people maintaining Thunderbird would keep
this in mind. Perhaps if they had, then a way could have found to roll
out this particular change in a gradual, backward-compatible way to give
us more time to deal with it.

I don't think this would work.

I have already pointed out in other emails at least two different
categories of changes that I had to make to my bootstrapped add-ons to
make them work after this change. Neither of these categories of changes
could have been made automatically by ATN. Yes, /some/ bootstrapped
add-ons could have be updated automatically -- several of mine indeed
required only a new manifest.json -- but there would be no way for any
automatic conversion tool to be able to tell the ones that can be
updated automatically from those that can't.

Furthermore, add-on developers have their own source repositories.
Converting install.rdf to manifest.json in the XPI file doesn't get
those changes back into the source repositories, so the next time the
developer updates their add-on it's going to break again.

Leaving aside the fact that it would have been nice if this change had
been rolled out gradually in a backward-compatible way, there are other
ways that the TB devs could have assisted add-on maintainers with
managing this change (and, indeed, many of these still could be done).
These include:

1. Make sure addons.thunderbird.net is prepared to handle the change
   before releasing it. Right now the new version upload tool on ATN
   thinks that bootstrapped add-ons with manifest.json in them are
   WebExtensions, and so do the ATN reviewer tools. This should have
   been anticipated and fixed before this change was released. It
   should be fixed ASAP now.
2. Query the ATN database to identify bootstrapped add-ons marked
   compatible with Thunderbird 68.0a1 and email all of their
   maintainers notifying them about the change.
3. Pick the top n bootstrapped add-ons marked compatible with
   Thunderbird 68.0a1, convert their install.rdf files to manifest.json
   files, and test them to see if they still work. If not, troubleshoot
   any issues that arise and decide whether to address these issues in
   the Thunderbird code or to instead document how to address them.

Honestly, I don't know why I'm bothering to write this message, because
if past experience is any indication no one is going to listen to
anything I've said here, but I guess I'm just stubborn that way.

  jik

On 2019-05-21 5:34 p.m., Jonathan Kamens via Maildev wrote:

I appreciate that you're trying very hard, and I appreciate your
involvement and comments in general. But I honestly don't think it is a
good idea to rush to make add-ons work in Nightly ASAP. I wouldn't even
attempt that myself. I don't know offhand if we've ever stated an
official policy on this, and I don't make policy for Thunderbird
releases, but my instinct is to say Nightly changes every day, add-ons
will break and you should not expect advance warning.

It is absolutely and completely reasonable, on the other hand, to expect
warning and documentation for Beta releases and DOUBLY so for ESR.
I'm happy we made this change immediately. This provides the most time
for people to react for 68 Beta and 68 ESR. If we had said we were going
to make the change, and then left the install.rdf loader in for 68.0b1
or later, it's a much higher risk that people would not have noticed the
change until dangerously close to 68 release. We would be having this
discussion again with 10x as many upset people who have even less time
to fix their add-ons.

The cost of that is that we've caused problems for people using add-ons
on Nightly, but it's hard for me to imagine a world where we have any
kind of add-on support guarantee for that channel. I think that people
who are reliant on add-ons but cannot fix them on their own should not
be using Nightly.

Lets look the reality squarely in the face. There are going to be major,
continuing changes to add-ons with frequent breakage until legacy
add-ons are completely gone. To me, the faster we get there, the better
for everyone. And yes, I am certain it's going to be extremely painful
over the next year for anyone trying to keep up. But I don't see any way
around that, unless someone has a magic time machine we can use to
prevent Mozilla from moving to web extensions in the first place and
constantly deleting code that old systems relied on.

If we lose add-ons or add-on developers because of that transition pain,
that is a cost of doing business that we simply must accept. We can do
our best to minimize the pain or minimize the duration of that pain but
we can't do both, and I'm definitely in the camp of minimizing duration
over intensity.

1. You're going to need to be more specific about this one. They are web
   extensions. If you mean that they're being auto-approved and not going
   in the full review queue, that would be a bug. However, they do need to
   be getting the web extension flag(because the definition of a web
   extension, from the POV of ATN, is 'addon that includes a
   manifest.json') so that their compatibility and attributes are set via
   manifest.json. Please file an issue
   https://github.com/thundernest/addons-server/issues.

In the general case, I can say for certain ATN will not be fixing issues
before they release in Nightly. That's not possible. We don't have a
dedicated developer for ATN, while we have several for Thunderbird
itself. I try to make sure that ATN bugs are fixed for or during betas,
and "ASAP critical fix" to me would be a bug that is breaking add-on
uploads or downloads for a release build.

2. This is a totally reasonable idea, and we ABSOLUTELY need more
   aggressive and easier to use messaging to communicate with add-on
   developers. I will look at getting a better tool and process in place
   for this ASAP. Manually querying the ATN database every time we want to
   send this kind of notification is not going to scale very well, but we
   can definitely do better than that. In a perfect world, we'll have an
   automated tool where we can target add-ons by version and send a
   notification to their maintainers. I think Mozilla/AMO has this
   capability, but that's a piece that we didn't set up because it relies
   on mass messaging services that we can't easily use.

This isn't a frivolous promise, I've filed this
https://github.com/thundernest/addons-server/issues/88 and will try to
work on it within the next month.

3. I am happy to make a system to notify add-on authors that a change
   may break their add-ons but it isn't really practical to do a custom
   testing process every time we make breaking changes, certainly not when
   they hit Nightly.

Hi Andrei,

Thanks for your productive response. Comments below.

On 5/22/19 1:24 AM, Andrei Hajdukewycz wrote:

From my perspective, there is no meaningful distinction between nightly
and beta releases.

A beta release is supposed to mean, "We've tested this to the best of
our ability and we believe it's ready to go. We plan on shipping this,
or something very much like this, after it has had some time to burn in
and serious issues have been addressed. This is stable and functional
and you can use this and port add-ons to it."

Recent Thunderbird beta releases have frequently not meant any of that.
There were no GA releases following the beta cycles for TB52, TB53,
TB54, TB55, TB56, TB57, TB58, TB59, TB63, TB64, TB65, TB66, or TB67.
Many of those beta releases were seriously broken in ways that were
known at the time the beta releases were put out, or not known because
inadequate testing was done of certain functionality (often specifically
related to add-on support). There were significant, incompatible changes
made during several of the beta release periods. For add-on developers
in particular, many of those beta releases were impossible to target for
porting add-ons.

Even now, nightly has just switched from 60.0a1 to 60.0b1 and yet there
are things that are known to be broken on the trunk which the devs
supposedly intend to fix before TB68 is released (e.g., the datepicker
XUL element, which I use in one of my add-ons, is gone, and it's
supposed to be possible to replace it with html:input type=date, but
that doesn't work).

I stopped paying attention to the distinction TB was drawing between
alpha and beta releases when all this became clear. Instead, I would
wait until there was a nightly build that was both stable enough and
complete enough that I could actually target my add-ons to it, switch to
that particular nightly build, and make whatever changes were necessary
to make my add-ons work in it. Lather, rinse, repeat every few months.
But the TB devs have made this /extremely/ difficult to do with the
recent changes to force nightly updates down our throats: the nightly
channel now updates automatically by default, and there is no way to
turn off the updates completely; the best you can do is tell TB to nag
you daily to do the update rather than doing it automatically.

If TB beta releases meant what beta releases are actually supposed to
mean, then I agree with you that tracking beta releases would be a good
strategy for add-on developers. But that's not where we are.

Also, I have a vague recollection that there used to be a real alpha
channel in addition to the nightly and beta channels. As I recall, the
alpha channel was essentially an attempt to do what I started doing
manually as described above ("I would wait until there was a nighly
build..."). As far as I can tell -- please correct me if I'm wrong --
there is no longer an alpha channel. Perhaps bringing it back would help
with the issues we're discussing.

The point is that they're NOT WebExtensions, they're legacy,
bootstrapped add-ons. I.e., I don't believe your definition of
WebExtension above is correct.

I'm talking specifically about three different issues I noticed:

1. When I uploaded the XPI for a bootstrapped add-on with a
   manifest.json in response to the recent TB change, ATN warned me
   that I couldn't go back after switching to WebExtension. This is
   probably harmless, but was incongruous.
2. In a couple of places -- I think on the status page for the add-on
   version in question, and also on the reviewer tools page -- there's
   a message indicating that the add-on wasn't auto-approved (as you
   mention above). But auto-approval shouldn't even be an option
   because it's not really a WebExtension.
3. It's marked as a WebExtension in the reviewer tools, which I believe
   limits which reviewers are allowed to review it. Note that
   non-bootstrapped legacy extensions with a manifest.json are not
   marked this way, so ATN already understands to at least some extent
   the concept that a legacy add-on with a manifest.json is different
   from a WebExtension.

I haven't filed bugs about these because I don't know which of them are
issues that we care about and should be fixed. Perhaps you can comment
on that.

Thank you.

If you're recommending to add-on developers that they target beta
releases for add-on compatibility, but you're not doing any substantive
testing of porting add-ons to a new beta release before you release it,
then either (a) you won't know what problems add-on developers are going
to encounter until it's too late to notify them in enough time for them
to fix their add-ons during the beta period, or (b) you're going to have
to make substantive, breaking changes to TB during beta release periods,
which (as I noted above) is not how beta releases are supposed to work.

This, by the way, is another reason why I'm tracking nightly builds
instead of beta releases: because the TB devs aren't doing enough to
test the impact that changes have on add-ons, and /somebody/ has to do
that to give add-on developers enough warning about those changes.

In other words, "it's not feasible for the TB devs to do real-world
testing of how changes are going to impact add-ons" is in my opinion
incompatible with "add-on maintainers shouldn't be tracking nightly
builds." If you're not testing how changes in nightly builds impact
add-ons, then we need to!

  jik

On 5/22/19 7:01 AM, John Bieling wrote:

I am not sure what this means.

I feel like perhaps the terminology I am using is interfering with the
point I am trying to make. I don't know whether that's because I'm using
the terminology wrong or the terminology is imprecise.

Let me try again without using the terminology...

I do not believe that ATN is treating legacy bootstrapped add-ons with
manifest.json files the way that it should. I explained in my other
emails the ways in which I believe it is treating them incorrectly. If
I'm wrong, then fine, I'm wrong, and no further action is required.

  jik

On 5/22/2019 6:40 AM, Jonathan Kamens via Maildev wrote:

Our betas don't fully match your definition, mostly because our beta
release schedule is not quality driven, is primarily calendar driven**. 
Plus, "WE" don't have a QA team big enough to fully test beta, or even
half of it - we rely on the 70k beta users to do that.  That doesn't
mean beta isn't good when release, nor that we don't delay  releasing at
times because of quality - we do delay, often, and we do closely monitor
quality.

I completely agree with Andre - nightly has always been unstable and not
intended as a target for add-on development, so you will continually be
disappointed in your attempts.  Nightly is not an Alpha version in the
classic sense.  More below

1. Thunderbird doesn't have what real QA people would consider to be an
   Alpha version, and nightly should not be construed as an such.  At one
   time we had Aurora which was notated as N.0a2 (and it is gone because
   Firefox dropped that channel).  But the primary purpose of aurora was
   localization, not IIRC a formal QA step. Aurora did allow time for
   tweaking code, but it never was subjected to a QA process, never had
   release notes, nor were there enough users to rely on it for QA or bug
   reports (about 2k Thunderbird auroa users, which is roughly what nightly
   had). In other words Aurora was just part of the train and was not
   significant in the code development process, and that's a big part of
   why it is gone.  2) We have nightly which is notated as N.0a1, but it's
   main purpose is dropping in new code, making it fairly unstable, and
   suffers all the same ills previously mentioned about Aurora.

2. 80% of the year our betas are not what I would call a true beta (i.e.
   one that has been highly refined and tested), but are a combined
   alpha/beta for three reasons a) we don't have a true alpha version as
   detailed above, b) release of betas is mostly calendar date driven, and
   c) strictly speaking our betas don't need to rise to the level of top
   notch release quality because we don't release until a major ESR every
   10 months - so we're mostly on the continuous beta hamster wheel.  
   (Which is not to say we don't try our best to deliver quality, and we DO
   play loose at times with the schedule so we can fix major issues before
   releasing a beta. And 8 months of beta does ultimately improve the
   quality of our ESR releases.)

Yes, lots of turmoil.  That's the territory we are in, again. And in the
pre-66 era there was a massive gap in manpower vs the amount of work to
be done, and items to document.  Plus the add-ons community was in a big
state of disarray.  The last two items is no longer true. That should
make a big difference relative to current times, and to 68.

Please note, expectations are low exceptions that an add-on will work in
beta. And I don't believe we have EVER dinged addon authors for not
providing updated addons for beta version - expect for the last beta for
a new ESR, for example version 60.    And so I think your expectations
are somewhat high - like of nightly users, most beta users can get on
without add-ons.  And those that cannot always use the release version
where the add-on should work.  It is the last beta before ESR that
counts in this race.  That is where we need add-on authors to step up,
and that is where the Thunderbird development team and the add-on
authors need to document the changed environment and help each other get
ready for the big release.

Presumably you mean 68.0a1 and 68.0b1.  Well, 68.0b1 isn't yet released.

I think you will agree we are living in extraordinary times.  We are
making changes, in a highly time constrained environment, for reasons
that are which are largely forced upon us. It's not pretty. But the show
must go on, and beta releases, ready or not, MUST be shipped largely on
schedule  if 68 isn't going to be a cesspool.  Which would you rather
have, a partly functional beta or a shitty 68?

Version 60 had similar massive changes, had a VERY elongated beta 60
period (such that we didn't ship beta 61 or 62), and had a VERY slow
rollout (such a slow rollout that it took 3-4 months before we fully
unthrottled 60 to all 52 users) - we did that so that the quality could
be boosted AND so that add-on authors who wished to could have
reasonable time to update their addons.

Hopefully we learned a lot from that period, especially in the add-ons
area, that things won't be so bad for 68.  Even so, I predict version 68
will have a similar very long beta period and similarly slow rollout.

I think you are incorrect - aurora was never intended to be a target for
add-on authors.  Regardless, it is not coming back.

Having written what I did above, I will qualify that by agreeing to what
I think you are saying, that it is useful to have add-on authors who
know what they are doing and willing to spend the time, to monitor and
test development changes as soon as possible - with the objective of
having a better quality  beta for all add-on authors.   That seems like
a worthy idea.   Perhaps one that can be explored further in future -
but it's too late do anything about that for version 68.