Choosing a programming language
On 3/19/2019 12:17 PM, Mark Rousell wrote:
I'm not a C++ programmer but isn't one of the key points of modern C++
to use reference semantics and all that cool stuff? :-)
I spent on hour trying to figure out why my program was giving the wrong
result only to realize I was using a value after std::move in
rvalue-references. And the compiler gave me absolutely no warning. I
really have been spoiled by Rust here.
Mozilla generally prohibits the use of the STL
That's rather a shame. I recently made an effort to learn some C++
(coming largely from C#) and it seems to me that STL is where modern
C++ really shines.
The STL maps are bad; one requires a poor-performance hash table
implementation and the other is a red-black tree, which is the wrong
answer 99% of the time. In practice, many projects also choose to extend
the types for vector (with some kind of small vector) and strings.
Mozilla's version of those utilities are 2 decades old at this point,
and date to a time when the STL wasn't really portable. I think there's
been modifications to them to make them use some of the API, and there's
definitely reimplementations of things like <type_traits>,
std::shared_ptr/std::unique_ptr that are nearly drop-in replacements for
the STL. The concepts will generally carry over, but the exact spellings
wrote.
[Rust] As a newer language, we're less likely to see knowledgeable
contributors
This is true but, on the other hand, it seems to me that you are
perhaps more likely to see contributors in general. The reason I say
this is because Rust is the latest cool thing. Enthusiastic coders are
a valuable commodity. It seems to me that Rust is a better language
overall for application development than C++, too.
I considered putting that in, but I didn't want to come across too much
of a Rust fanboy.
Even if we could magically wave that away, the reality of system
integration is that systems require us to have support for native
languages--which include C++, Objective-C, even Java for Android--to
implement necessary features.
This is an interesting statement. Can you suggest some of the things
you're thinking of?
The cross-platform project I mentioned above is being coded in C# on
top of .NET Core and I am not sure that this locks us out of anything.
(I'm not suggesting C#/.NET for Thunderbird, of course, only that
system integration is potentially available to languages other than
native ones). As an aside, if we were to begin the project now then
either Go or Rust might be alternatives to C#.
The major system integration points we need are:
- System mailto: support. This involves registering us via shell
functions, which are all C on the major platforms (although we
already have Mozilla cross-platform glue on this area) - Search indexing support. This is a separate small executable on
Windows; never looked into other platforms. - MAPI support on Windows. You need a separate DLL on Windows for this
to work (which is vaguely COM-based, but it doesn't actually need to
be). - Accessing system addressbook. OS X uses Objective-C for this, GNOME
uses GObject, and Windows uses magic files. Unless you give up and
use extended MAPI, er, now Outlook Office API for Outlook's AB.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
On 21/03/2019 03:37, Joshua Cranmer 🐧 wrote:
I spent on hour trying to figure out why my program was giving the
wrong result only to realize I was using a value after std::move in
rvalue-references. And the compiler gave me absolutely no warning. I
really have been spoiled by Rust here.
The STL maps are bad; one requires a poor-performance hash table
implementation and the other is a red-black tree, which is the wrong
answer 99% of the time. In practice, many projects also choose to
extend the types for vector (with some kind of small vector) and
strings. Mozilla's version of those utilities are 2 decades old at
this point, and date to a time when the STL wasn't really portable. I
think there's been modifications to them to make them use some of the
API, and there's definitely reimplementations of things like
<type_traits>, std::shared_ptr/std::unique_ptr that are nearly drop-in
replacements for the STL. The concepts will generally carry over, but
the exact spellings wrote.
I can definitely see that you prefer Rust (and quite rightly so in my
opinion). :-)
The cross-platform project I mentioned above is being coded in C# on
top of .NET Core and I am not sure that this locks us out of
anything. (I'm not suggesting C#/.NET for Thunderbird, of course,
only that system integration is potentially available to languages
other than native ones). As an aside, if we were to begin the project
now then either Go or Rust might be alternatives to C#.
The major system integration points we need are:
- System mailto: support. This involves registering us via shell
functions, which are all C on the major platforms (although we
already have Mozilla cross-platform glue on this area) - Search indexing support. This is a separate small executable on
Windows; never looked into other platforms. - MAPI support on Windows. You need a separate DLL on Windows for
this to work (which is vaguely COM-based, but it doesn't actually
need to be). - Accessing system addressbook. OS X uses Objective-C for this,
GNOME uses GObject, and Windows uses magic files. Unless you give
up and use extended MAPI, er, now Outlook Office API for Outlook's AB.
Ah, I see what you mean. At least as far as Windows is concerned (where
I am most familiar), none of these are in fact showstoppers in practice
when looking at the project as a whole. Yes, some of the shell
integration components need to be written in a language that will
produce native code (e.g. C/C++/Rust/Pascal) but so be it; this does not
mean that the vast bulk of the project can't be written in the language
of our choice.
In brief, we are not limited to writing the whole project in particular
native languages merely so suit the shell/infrastructure integration
aspects.
--
Mark Rousell
On 3/20/2019 9:24 AM, Wayne Mery wrote:
Perhaps some Thunderbird experiments should be done in Rust, both by
those who have used it and those who have not, and flesh out the
issues and draw some useful conclusions?
What sort of experiments did you have in mind?
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
Build a working prototype that can e.g. check email (in a simple manner). I.e. core logic (accounts, folders, messages), protocol implementation, front-end, and communication between them.
Joshua Cranmer wrote:
What sort of experiments did you have in mind?
--
Sent from my phone. Please excuse the brevity.
I agree with Mark.
Regarding mailto, I have already written a URL scheme handler for Windows and Mac for XUL runner app TomTom HOME 2, and the entire (!) XUL handler implementation was written in JavaScript. There was 0 C code in the URL handler.
The registration of that handler is simply setting an ordinary Windows registration key and can be written a) by running a regedit file, b) in an installer script c) using the existing XPCOM API for the Windows registry, d) using an API of whatever platform we choose (search npm), e) by simply running a shell command, like a powerscript. the is really just a few lines of code. (I'm not hand waving, I've actually written that, in a production application, with 5 million users.)
Mark Rousell wrote:
Joshua Cranmer wrote:
The major system integration points we need are:
- System mailto: support. This involves registering us via shell
functions, which are all C on the major platforms (although we
already have Mozilla cross-platform glue on this area) - Search indexing support. This is a separate small executable on
Windows; never looked into other platforms. - MAPI support on Windows. You need a separate DLL on Windows for
this to work (which is vaguely COM-based, but it doesn't actually
need to be). - Accessing system addressbook. OS X uses Objective-C for this,
GNOME uses GObject, and Windows uses magic files. Unless you give
up and use extended MAPI, er, now Outlook Office API for
Outlook's AB.
Ah, I see what you mean. At least as far as Windows is concerned (where
I am most familiar), none of these are in fact showstoppers in practice
when looking at the project as a whole. Yes, some of the shell
integration components need to be written in a language that will
produce native code (e.g. C/C++/Rust/Pascal) but so be it; this does
not
mean that the vast bulk of the project can't be written in the language
of our choice.
In brief, we are not limited to writing the whole project in particular
native languages merely so suit the shell/infrastructure integration
aspects.
--
Sent from my phone. Please excuse the brevity.
On 3/21/2019 12:43 PM, Mark Rousell wrote:
Ah, I see what you mean. At least as far as Windows is concerned
(where I am most familiar), none of these are in fact showstoppers in
practice when looking at the project as a whole. Yes, some of the
shell integration components need to be written in a language that
will produce native code (e.g. C/C++/Rust/Pascal) but so be it; this
does not mean that the vast bulk of the project can't be written in
the language of our choice.
In brief, we are not limited to writing the whole project in
particular native languages merely so suit the shell/infrastructure
integration aspects.
Where did I ever state that I believed the /entire/ project should be
written in a language like C or C++? My point was that at least some of
our project cannot be in JS, and that we have freedom to choose what the
percentage not in JS from anywhere between <1% and >99%.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
On 3/20/2019 3:12 PM, holger krekel wrote:
On Wed, Mar 20, 2019 at 15:35 +0000, Mark Rousell wrote:
On 20/03/2019 12:37, Paul Morris wrote:
A thought on protocol code: as the Rust community builds out their
ecosystem, I would think they would be interested in having libraries
(crates) for working with common protocols. That might be a good
opportunity for collaboration, and for sharing the load of development
and maintenance.
In the "wouldn't be nice if" department, I would have thought that the
Rust ecosystem could really do with an equivalent of Jeffrey Stedfast's
MailKit and MimeKit.
FWIW we are heading towards porting https://delta.chat (a chat e-mail client)
core dependencies from C to Rust. We are starting by replacing Delta's
current netpgp dependency with https://github.com/dignifiedquire/rpgp
and have also applied for Security code review funds.
Apart from OpenPGP, we are thinking to integrate Rust SMTP and IMAP libraries
and thus substitute libetpan and cyrusasl/openssl with Rust equivalents. To fully
get rid of libetpan we also need to add mime-parsing and generation in Rust.
However, Delta Chat has probably a lot less API requirements than Thunderbird
so even if we succeed, it doesn't mean it's directly integratble with TB.
But it's probably still useful to know that some folks in the e-mail space
are heading for this C->Rust strategy, and there is potential synergy.
MIME is difficult to pull into a generic crate because there is a
spectrum of features and broken-email support that exists, and
Thunderbird in particular sits at one extreme end. You can start with a
separate break-on-newlines, simple folding, and simple detection, but
the real world also tends to start throwing curveballs like 8-bit,
non-UTF-8 headers; incorrect charset information (what, you seriously
think that just because the program said the text was ISO-8859-1 that it
actually meant it?); the inability to actually generically convert any
header to RFC 2047. And then you can reach into the less-wanted features
such as S/MIME support, multiple body part information, asynchronous
decoding for messages, or automatic yEnc support. Parsing emails is hard.
IMAP faces a similar issue in that a simple client is going to have very
different wants from an actual email client. It turns out that IMAP is
actually a database synchronization protocol (that's very bad at its job
of database synchronization), but simple clients don't have a local
database to synchronize, so it collapses into a database view and you
don't need to expose a database even in a high-level API.
Another realistic issue that we face in TB is that many services we do
want to integrate somewhat with alternative system stacks than the
standard library. We want to use NSS for our SSL and crypto, for
example, and we also want to reuse some SASL modules provided by Gecko
(specifically, those that provide Kerberos and NTLM support--Mozilla has
a pretty involved mechanism for figuring out who is actually providing
the SSO credentials in those cases). I'd love to be able to reuse
external libraries, but my experience has been--as I've mentioned
elsewhere--that none really exist that can actually be used.
--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist
First off, thanks for the great summing-up, Joshua!
I've hesitated on this - I know how these kind of discussions can spiral
out if not careful... but here's my two cents as a recent developer,
focusing mainly on the C++ side:
I've come to view Thunderbird as a variant of Firefox, with some extra
modules to handle mail (mostly C++), and a custom UI (mostly JS).
As such, I think we're firmly hitched to the Firefox train and may as
well take advantage of it. So that means essentially going along with
whatever practices FF picks, on the basis that they've already acted as
crash-test dummy and we can benefit from their experience.
My rough impression of the FF direction is (and I could be wrong on this):
- Moving away from exposing XPCOM so much to JS users, in favour of
nice JS-centric APIs - Switching out some C++ modules with Rust, in a conservative way, and
where there is a clear benefit (eg concurrency wins in rendering).
I think point 1 is key for us.
At the moment, C++ and JS are more-or-less considered equal. XPCOM
classes can be implemented in either, and the general level of
abstraction is more or less the same on both sides. It's low level and
very detailed.
I think this really holds us back. It makes the C++ side harder to work
with and debug, and it places an undue burden of detail upon the JS
side. It is responsible for a lot of the fragility and brittleness we
currently see in the codebase, and makes simple things much harder to
fix than they ought to be.
In this FF-like direction, the JS side would be provided with a lovely
set of JS-specific APIs - like a privileged set of APIs to complement
those already emerging for add-ons.
It would free the C++ side from having to maintain the huge surface area
that is currently exposed to JS.
On the JS side, it'd make working in TB much more approachable and
understandable, especially to new developers.
I've got some ideas on how we could head toward this, if anyone thinks
it merits discussing (but not in this thread :- ).
On point 2: I really don't have a good feel for what things might be
good candidates for Rust. My gut feel is that we'd probably be better
sticking with C++ at least until we had point 1 under control. (as much
as I love the idea of jumping to Rust!)
Ben.
--
Ben Campbell.
Thunderbird developer
It looks like WASM may soon be the well-trodden path for Rust and JS
interop:
The Rust and WebAssembly domain working group wants to cultivate a
stable, batteries-available, and production-ready ecosystem for Rust and
Wasm development in 2019 <https://github.com/rustwasm/rfcs/pull/7>.
To further that goal, we are creating Gloo
<https://github.com/rustwasm/gloo>, a modular toolkit for building both:
*
small, targeted Wasm modules that integrate into a larger JavaScript
system, and
*
whole Web applications written in Rust.
Full post here:
https://rustwasm.github.io/2019/03/12/lets-build-gloo-together.html
More Rust + WASM working group posts: https://rustwasm.github.io/
See also: "Standardizing WASI: A system interface to run WebAssembly
outside the web"
https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/
A brief thought on the protocol library question. If there aren't
libraries out there that do all the various things we'd need, cover all
the edge cases, etc., and so we're going to (re-)write code to handle
all that, then maybe we should provide that code as libraries to meet
that need, while we're at it?
But I gladly defer to those more fully steeped in the various concrete
details of the situation.
Cheers,
-Paul
--
Paul Morris
Thunderbird.net
On 04-04-2019 05:52, Paul Morris wrote:
A brief thought on the protocol library question. If there aren't
libraries out there that do all the various things we'd need, cover
all the edge cases, etc., and so we're going to (re-)write code to
handle all that, then maybe we should provide that code as libraries
to meet that need, while we're at it?
That's the idea. When possible we would build on and improve existing
libraries as needed.
The actual protocols is not all though: there's the interaction and
integration to use them that is needed for things like different
authentication schemes. That part is unlikely to be reusable as such, so
there's a weigh-off in how much effort to put into library reusuability.
-Magnus