I find the this idea generally good. IMHO we should try follow the
architecture where you have model interfaces that are only meant to work
with the model objects, repository interfaces where those are supposed
to work with repository (in case there is one! - this is not our case
here) and service interfaces - i.e. only meant for the business logic.
However implementing this will be only be possible after
Mork dies completely and this has not happened so far.  Enrico have you
done some work in that direction already?

Tito

-------- Original Message  --------
Subject: [Maildev] hard dependencies on addrbook component
From: Enrico Weigelt, metux IT consult via Maildev
maildev@lists.thunderbird.net
To: maildev@lists.thunderbird.net
Cc: "Enrico Weigelt, metux IT consult" enrico.weigelt@gr13.net
Date: 12.08.2017 04:14

On 12.08.2017 12:17, Tito wrote:

No, it doesn't need to. It can remain as a backend. We'll need
it for migration anyways (unless someone wants to write an
extra converter).

So far, I've moved out the generic idl's (w/o mdb, ldap, ...)
to it's own component addrbook-api. Pretty easy. Next step is
sorting out which classes are generic enough to go there too.

Then I'd like to move the mdb stuff to its own component
Maybe it will need some kind of 'driver model' where individual
backends can register themselves into.

Collecting my hacks here:

https://github.com/mozilla-cleanup/comm-esr52/tree/wip/mtx

--mtx

On 8/11/2017 9:14 PM, Enrico Weigelt, metux IT consult via Maildev wrote:

Not really. As far as components go, the address book is one of the most
isolated pieces of Thunderbird. Only import really needs access to its
guts, and that's because it's import. The biggest problem is that the
current interface assumes access is fully synchronous, which is problematic.

As far as designing a replacement interface goes, Mike Conley (CC'd)
already made some design for replacing the current APIs for Ensemble.
See https://bugzilla.mozilla.org/show_bug.cgi?id=841603 for more details.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

On 12.08.2017 21:28, Joshua Cranmer 🐧 via Maildev wrote:

IMHO we should split it better. For example we've got lots of places
that directly call MDB-services (even though they just need generic
interfaces).

in nsAddrDatabase::CreateCard():
...
personCard = do_CreateInstance(NS_ABMDBCARD_CONTRACTID, &rv);
...

(anyways, nsAddrDatabase is heavily tied to mdb)

That's orthogonal to my current topic.

Anyways, I really wouldn't like to have the AB storage itself async,
that just complicates things. The async part is just interesting for
javascript side (which doesn't know about threads). So I'd suggest
putting that in front of abmanager+friends (as Mike did), for the
javascript frontend(s).

In the long run, I'd like to move the whole ab storage stuff to an
entirely own library / service - independent from tbird / moz, so
it can be used by other applications, too. Self-contained tbird
install packages (eg. on win32) would just bundle that, while on
packaged distros, we just depend on it and leave the rest to the
distro.

--mtx

On 8/12/2017 2:52 PM, Enrico Weigelt, metux IT consult wrote:

You do realize that nsAddrDatabase is MDB-specific? As in, none of the
other address books construct an nsAddrDatabase (okay, LDAP does it for
offline cache IIRC, but that's an exception that proves the rule)?

No it's not. Either you're proposing revisions to the current API, at
which point all objections to the present API are relevant, or you're
proposing trying to enforce an arbitrary line which doesn't exist, which
isn't really relevant to anybody.

Assume that all file I/O and network I/O can only be asynchronous. That
means the LDAP backends and the local storage backend has to be
asynchronous. If you implement in JS (as is our standing goal), your
choice for APIs are SQLite (asynchronous), IndexedDB (asynchronous), or
run on another thread (where all the cross-thread messaging is
asynchronous). Permitting AB backends to be asynchronous is non-negotiable.

The truth is, no one wants that kind of library. The higher-level a
library is, the more opinionated it has to be about what kind of API it
exposes (e.g., synchronous versus async APIs). Opinionation makes it
harder for consumers to choose to use a library, since they are really
screwed if the library and application have different opinions. This is
all the more apparent in situations like C/C++ and JavaScript, where
basic questions like "how do I access the network?" turn out to be
strongly opinionated due to the lack of these features in the standard
library API.

Thus, the only likely user of such a library is Thunderbird. We've also
generally adhered to a policy in the past that we assume that we're the
only ones who muck about our profile directory, so exposing a public
project to let other applications muck about the profile directory while
Thunderbird is running is likely to cause problems we don't want to fix.
The other unfortunate problem is that there's no real way to extricate
our core Mozilla dependency into a separate package for distros (note
how XULRunner fell through).

Trying to maintain independent fine-grained packages can actually make
application development more, not less, complex. It means more public
APIs that need to be maintained, and it means you either have to shell
out more fine-grained libraries to serve as dependencies for some of the
packages or you have to endlessly repeat code because it's too minor to
be a separate package (after all, who ever coded a binary search
incorrectly? [1]). The trade-off is such that for pretty much anything
higher-level than the very lowest-level protocol/format implementations,
it is not advisable to make it a generic library. And in the context of
the address book, we're largely the consumer of such libraries, not the
originator (the exceptions being CardDAV and our own database
implementation).

[1] This is meant to be a sarcastic question. Binary searches are
probably the single most widely incorrectly-implemented algorithm.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

On 13.08.2017 02:17, Joshua Cranmer 🐧 via Maildev wrote:

Ah, so it is the mdb-based engine ?
Who's the upper layer, that's called by the clients (if there is one) ?

Maybe I wasn't clear enough: I'd like to have an storage-independent
layer, where various backends (whichever they might be) can be plugged
into. Its clients shouldn't have any idea, where the data really comes
from (or goes to). And ideally, this shouldn't be moz specific
(so it can be used in many other applications, too)

Why should I ? I'm not aware of TB targetng any strictly async OS
(which are hard to find anyways). Async IO can give better for
some workloads (usually very high load of pretty simple requests).

But async programming (which is the natural consequence of async IO)
can easily become quite complicated to manage, if you need contiguous
workflows. (eg. dealing w/ synchronous protocols). Then you'll need
to synchronize again (especially when you need transactional behaviour)

Note that we're talking on boringly low load here.

There needs to be an async interface for javascript side. But that
doesn't necessarily mean that everything (down to the low level IO)
needs to be async. Threads are pretty easy and cheap (we're not far
far away from high load scenarios).

You want to implement everything, including ldap and lots of other
backends entirely in javascript ?

My preferred choice is none of them. I need the abook(1) format
(which looks quite like an .ini file). Or mutt alias directives.

I want it. And I know a lot of other people (many of them operators)
want that, too. Or and small and simple (and easy to replace) external
service (eg. something 9P based).

It's a pretty trivial case:

• directory structure w/ contact elements.
• each element can have a limited set of entries
• primary operations are walk, create, read, write, delete

Looks pretty much like a filesystem.

Additional operation: regex-match by pkey

Okay, make it a synthetic filesystem.

Pretty trivial to do in plain boring old C. (even not seriously
difficult in assembler). Ergo: no big deal at all.

Async layer ontop of that also pretty trivial.

This question isn't relevant here - a generic address book API doesn't
ever need to care whether there is anything like a network. It wouldn't
even matter when we'd still using punchcards.

The problem scope so small and simple, so there isn't much room for
serious debates. Simple enough to be easily done on an 8bit machine.

Or mutt, sylpheed, evolution, pidgin, jitsi, ...

I didn't say that this data should necessarily have to be stored in the
profile directory. For my own usecases it should explicitly not there.

Because it's so extremly complicated. My goal is to make things simpler.

In that case it makes it much simpler. The requirements are quite
trivial, so the API can be quite trivial, too. Simple enough for an
CS school homework.

And far less individual code to maintain. Once the API is settled, I
don't see any need for further changes. (except perhaps some tiny
additions).

Aha, so everybody should write his own widget toolkit ?
Maybe also databases (like, eg. mork) ?
Or graphics APIs ?

At the moment, tbird is only the consumer of the ldap library. Actually,
forked it's own private branch of it.

Mozilla also forked it's own private branch of some old cairo.

--mtx

On 8/13/2017 2:18 AM, Enrico Weigelt, metux IT consult via Maildev wrote:

Let me describe a little the address book design, which is represented
by the XPCOM interfaces that describe it (see mailnews/addrbook/public)

The main interface is nsIAbDirectory, which is a collection of address
cards (nsIAbCard).

nsIAbCard is independent of database type. All of the current
nsIAbDirectory implementations are mork-specific, but at the C++ level
there is a mork-independent precursor nsAbDirProperty.cpp that partially
implements nsIAbDirectory that is mork independent. Also at the C++
level, nsAbCardProperty.cpp is a complete implementation of nsIAbCard
that is mork-independent.

So it is possible to implement a mork-independent nsIAbDirectory that
works fine with the existing program, and there is no need to
reimplement nsIAbCard since the existing XPCOM object is fine. The
JsAccount library implements a js-accessible version of
nsAbDirProperty.cpp that can be extended to be a complete nsIAbDirectory
implementation which can be used for an alternate address book directory
implementation, either to replace Mork or to implement a new protocol.
That is how ExQuilla implements an EWS address book. (Note that CardBook
takes a completely different approach, not using the existing address
book interfaces but hooking into Thunderbird at the UI level. Also it is
not really necessary to use the C++ precursor to nsIAbDirectory using
JsAccount, so others simply implement a full nsIAbDirectory in JS).

But there are gotchas. All of nsIAbDirectory appears to be sync. But in
fact, at the UI level, there are listeners that can be called async
after updates and is used by LDAP, but this functionality is very hidden
and only discoverable by pouring over the code. So the address book
itself can be implemented by an async process successfully. But you have
to pretend to be an LDAP address book to do this.

But the real sync/async gotcha is how the rest of mailnews interfaces
with the address book. When data from the address book is needed, it is
accessed through sync methods. What that means is that any
nsIAbDirectory implementation has to read in an entire address book, and
keep that data available for rapid sync access. Practically that means
an in-memory copy of the address book, or at least of any cards in
active use.

What that means, looking at the ExQuilla code, is that the interfaces
that really have to be implemented to have a viable address book that
interfaces with the rest of mailnews are nsIAbDirectory,
nsIAbDirFactory, and nsIAuthCompleteResult. Of course a lot of that code
will end up mapping the new implementation to properties on the existing
nsIAbCard implementation. (This also assumes that you interface
following the existing design, which as I said earlier is not the path
that CardBook chose).

We keep talking about the need for a Carddav implementation. This would
not be a big project. I would approach it by taking the Carddav
implementation from CardBook (which in turn uses Fallen's ical.js for
parsing), and then adapting the nsIAbDirectory implementation in
ExQuilla. (Carddav is now MPL, ExQuilla is not but the code is all mine
and I am willing to relicense the nsIAbDirectory parts as MPL). I would
guess this is a 2-4 week full-time project for myself to do. I wish I
could find that time.

:rkent

R Kent James via Maildev wrote on 14.08.2017 20:07:

That all sounds fine and reasonable to me.

Other than the XPCOM C++ dependency, of course, which makes it hard to
subclass.

A re-work of the AB should probably throw all that away, and just
implement everything in JS, and re-write the UI as well (not much to
savor there, either).

This is an interesting point. For simplicity, I would probably still go
for an in-memory copy. If you have a global corporate AB with 40000
entries, that's still not too bad to keep in RAM. (Assuming you can
delay-load the images :-) .)

I could imagine that server-side search would need to be async. Assuming
we need that at all.

Ben

On 8/14/2017 3:02 PM, Ben Bucksch via Maildev wrote:

If you assume a single entry takes up ~1KB (not too far off the mark, I
think, if you're talking corporate LDAP that's filling in most of the
fields), that means you need 40MB of RAM to load an addressbook. To do
something like autocomplete someone's name. Or reading mail, come to
think of it--we do search for users in the address book then. Were it a
question only of using that much memory for loading the address book
pane, I'd have less issue with that kind of memory usage, but it seems
pretty clear to me that it's unacceptable for autocomplete or looking up
an entry by email address.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist