maildev@lists.thunderbird.net

Thunderbird email developers

View all threads

Google stopping IMAP and SMTP access an moving to proprietary protocols?

JK
Jörg Knobloch
Mon, Aug 26, 2019 8:04 PM

Hi all,

the subject says it all: Google stopping IMAP and SMTP access an moving
to proprietary protocols?

That's what I heard on the grapevine, can anyone confirm or deny?

Jörg.

Hi all, the subject says it all: Google stopping IMAP and SMTP access an moving to proprietary protocols? That's what I heard on the grapevine, can anyone confirm or deny? Jörg.
BB
Ben Bucksch
Wed, Aug 28, 2019 10:38 PM

Where did you hear this? Who are the sources, and what exactly did they say?

From where I stand, Google's push to change IMAP to OAuth2 is a move to
a proprietary protocol. You cannot use IMAP without a web browser (!)
anymore. :-(  Did some people maybe use "proprietary protocol" in this
sense?

Ben

Jörg Knobloch wrote on 26.08.19 22:04:

Hi all,

the subject says it all: Google stopping IMAP and SMTP access an
moving to proprietary protocols?

That's what I heard on the grapevine, can anyone confirm or deny?

Jörg.


Maildev mailing list
Maildev@lists.thunderbird.net
http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net

Where did you hear this? Who are the sources, and what exactly did they say? From where I stand, Google's push to change IMAP to OAuth2 is a move to a proprietary protocol. You cannot use IMAP without a web browser (!) anymore. :-(  Did some people maybe use "proprietary protocol" in this sense? Ben Jörg Knobloch wrote on 26.08.19 22:04: > Hi all, > > the subject says it all: Google stopping IMAP and SMTP access an > moving to proprietary protocols? > > That's what I heard on the grapevine, can anyone confirm or deny? > > Jörg. > > > _______________________________________________ > Maildev mailing list > Maildev@lists.thunderbird.net > http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net > >
JK
Jörg Knobloch
Thu, Aug 29, 2019 7:01 AM

On 29 Aug 2019 00:38, Ben Bucksch wrote:

Where did you hear this? Who are the sources, and what exactly did
they say?

I've heard it from someone ;-) - Those were the words:

... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP
nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle.

Jörg.

On 29 Aug 2019 00:38, Ben Bucksch wrote: > Where did you hear this? Who are the sources, and what exactly did > they say? I've heard it from someone ;-) - Those were the words: ... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle. Jörg.
T
Tanstaafl
Thu, Aug 29, 2019 12:47 PM

On Wed Aug 28 2019 18:38:54 GMT-0400 (Eastern Standard Time), Ben
Bucksch ben.bucksch@beonex.com wrote:

You cannot use IMAP without a web browser (!) anymore.

? I do it all the time.

On Wed Aug 28 2019 18:38:54 GMT-0400 (Eastern Standard Time), Ben Bucksch <ben.bucksch@beonex.com> wrote: > You cannot use IMAP without a web browser (!) anymore. ? I do it all the time.
BB
Ben Bucksch
Thu, Aug 29, 2019 2:28 PM

Was that a user? That's quite unspecific. I wouldn't give much about that.

I think he means that Google generally shows not much respect for
standards, by practically forcing OAUTH2 on IMAP etc.. I say
"practically", because they make using an IMAP password so difficult
that most of our Thunderbird users won't be able to do it, which
practically forces us to use OAUTH2, but technically, the possibility is
still there. It sounds like the user referred to that.

Ben

Jörg Knobloch wrote on 29.08.19 09:01:

On 29 Aug 2019 00:38, Ben Bucksch wrote:

Where did you hear this? Who are the sources, and what exactly did
they say?

I've heard it from someone ;-) - Those were the words:

... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP
nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle.

Jörg.


Maildev mailing list
Maildev@lists.thunderbird.net
http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net

Was that a user? That's quite unspecific. I wouldn't give much about that. I think he means that Google generally shows not much respect for standards, by practically forcing OAUTH2 on IMAP etc.. I say "practically", because they make using an IMAP password so difficult that most of our Thunderbird users won't be able to do it, which practically forces us to use OAUTH2, but technically, the possibility is still there. It sounds like the user referred to that. Ben Jörg Knobloch wrote on 29.08.19 09:01: > On 29 Aug 2019 00:38, Ben Bucksch wrote: >> Where did you hear this? Who are the sources, and what exactly did >> they say? > > I've heard it from someone ;-) - Those were the words: > > ... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP > nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle. > > Jörg. > > > _______________________________________________ > Maildev mailing list > Maildev@lists.thunderbird.net > http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net >
CL
Claudio Luck
Thu, Aug 29, 2019 4:02 PM

On 29.08.19 16:28, Ben Bucksch wrote:

Was that a user? That's quite unspecific. I wouldn't give much about that.

I understood it like Google is apparently calling software companies to
convince them to use their proprietary API [1] instead of IMAP and SMTP
to access Gmail. Given that they also happen to decide which app goes
into their Play Store, their convincing would be ... well ... very
convincing.

[1] https://developers.google.com/gmail/api/guides/

I think he means that Google generally shows not much respect for
standards, by practically forcing OAUTH2 on IMAP etc.. I say
"practically", because they make using an IMAP password so difficult
that most of our Thunderbird users won't be able to do it, which
practically forces us to use OAUTH2, but technically, the possibility is
still there. It sounds like the user referred to that.

That's probably another thing. The OAuth2 authentication "callout" from
SMTP and IMAP to a browser session is standardized with the IETF in the
SASL Framework with RFC 6750. So, it's as proprietary as any other IETF
Proposed Standards. Similar things are done in SSL VPNs and WLAN Logins btw.

An overview of the registered SASL Mechanisms is here:
[2] https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml

RFC 7628 «A Set of Simple Authentication and Security Layer (SASL)
Mechanisms for OAuth»
Search for OAUTHBEARER therein.
[3] https://tools.ietf.org/html/rfc7628

OAuth 2.0 is itself RFC 6750 «The OAuth 2.0 Authorization Framework:
Bearer Token Usage»
[4] https://tools.ietf.org/html/rfc6750

You can implement your own set of services for your own IMAP/SMTP using
OAUTHBEARER, i.e. with Dovecot, it does not need to be talking to Google
or Facebook services at all.
[5] https://wiki.dovecot.org/PasswordDatabase/oauth2

There are open source frameworks like "Apereo CAS", or SimpleSAMLphp
(sigh!) or perhaps OpenStack Keystone which can be used to build your
own IdP. They can work with your existing user+password database.
[6] https://www.apereo.org/projects/cas
[7] https://simplesamlphp.org/

Claudio

Jörg Knobloch wrote on 29.08.19 09:01:

On 29 Aug 2019 00:38, Ben Bucksch wrote:

Where did you hear this? Who are the sources, and what exactly did
they say?

I've heard it from someone ;-) - Those were the words:

... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP
nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle.

Jörg.


Maildev mailing list
Maildev@lists.thunderbird.net
http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net

On 29.08.19 16:28, Ben Bucksch wrote: > > Was that a user? That's quite unspecific. I wouldn't give much about that. I understood it like Google is apparently calling software companies to convince them to use their proprietary API [1] instead of IMAP and SMTP to access Gmail. Given that they also happen to decide which app goes into their Play Store, their convincing would be ... well ... very convincing. [1] https://developers.google.com/gmail/api/guides/ > > I think he means that Google generally shows not much respect for > standards, by practically forcing OAUTH2 on IMAP etc.. I say > "practically", because they make using an IMAP password so difficult > that most of our Thunderbird users won't be able to do it, which > practically forces us to use OAUTH2, but technically, the possibility is > still there. It sounds like the user referred to that. That's probably another thing. The OAuth2 authentication "callout" from SMTP and IMAP to a browser session is standardized with the IETF in the SASL Framework with RFC 6750. So, it's as proprietary as any other IETF Proposed Standards. Similar things are done in SSL VPNs and WLAN Logins btw. An overview of the registered SASL Mechanisms is here: [2] https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml RFC 7628 «A Set of Simple Authentication and Security Layer (SASL) Mechanisms for OAuth» Search for OAUTHBEARER therein. [3] https://tools.ietf.org/html/rfc7628 OAuth 2.0 is itself RFC 6750 «The OAuth 2.0 Authorization Framework: Bearer Token Usage» [4] https://tools.ietf.org/html/rfc6750 You can implement your own set of services for your own IMAP/SMTP using OAUTHBEARER, i.e. with Dovecot, it does not need to be talking to Google or Facebook services at all. [5] https://wiki.dovecot.org/PasswordDatabase/oauth2 There are open source frameworks like "Apereo CAS", or SimpleSAMLphp (sigh!) or perhaps OpenStack Keystone which can be used to build your own IdP. They can work with your existing user+password database. [6] https://www.apereo.org/projects/cas [7] https://simplesamlphp.org/ Claudio > Jörg Knobloch wrote on 29.08.19 09:01: >> On 29 Aug 2019 00:38, Ben Bucksch wrote: >>> Where did you hear this? Who are the sources, and what exactly did >>> they say? >> >> I've heard it from someone ;-) - Those were the words: >> >> ... Probleme mit GMail. Google will nicht mehr, dass man IMAP/SMTP >> nutzt. Sie verlangen den Umstieg auf ihre proprietären Protokolle. >> >> Jörg. >> >> >> _______________________________________________ >> Maildev mailing list >> Maildev@lists.thunderbird.net >> http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net >> > > _______________________________________________ > Maildev mailing list > Maildev@lists.thunderbird.net > http://lists.thunderbird.net/mailman/listinfo/maildev_lists.thunderbird.net
BB
Ben Bucksch
Thu, Aug 29, 2019 7:50 PM

Claudio Luck wrote on 29.08.19 18:02:

I understood it like Google is apparently calling software companies to ```

<pre class="moz-quote-pre" wrap="">convince them to use their proprietary API [1] instead of IMAP and SMTP to access Gmail. Given that they also happen to decide which app goes into their Play Store, their convincing would be ... well ... very convincing. [1] <a class="moz-txt-link-freetext" href="https://developers.google.com/gmail/api/guides/">https://developers.google.com/gmail/api/guides/</a> ```

Thanks for this information.

I hope Google respects Thunderbird to not propose this to us, and respects standards enough to keep IMAP enabled.

I'd be in favor for a replacement of IMAP, but any new standard should first be widely accepted by everybody before the old one is disabled.

<pre class="moz-quote-pre" wrap="">
<pre class="moz-quote-pre" wrap="">
I think he means that Google generally shows not much respect for
standards, by practically forcing OAUTH2 on IMAP etc.. I say
"practically", because they make using an IMAP password so difficult
that most of our Thunderbird users won't be able to do it, which
practically forces us to use OAUTH2, but technically, the possibility is
still there. It sounds like the user referred to that.
<pre class="moz-quote-pre" wrap="">

The OAuth2 authentication "callout" from
SMTP and IMAP to a browser session is standardized with the IETF in the
SASL Framework with RFC 6750. So, it's as proprietary as any other IETF
Proposed Standards.

An overview of the registered SASL Mechanisms is here:
[2] <a class="moz-txt-link-freetext" href="https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml">https://www.iana.org/assignments/sasl-mechanisms/sasl-mechanisms.xhtml</a>

RFC 7628 «A Set of Simple Authentication and Security Layer (SASL)
Mechanisms for OAuth»
Search for OAUTHBEARER therein.
[3] <a class="moz-txt-link-freetext" href="https://tools.ietf.org/html/rfc7628">https://tools.ietf.org/html/rfc7628</a>

OAuth 2.0 is itself RFC 6750 «The OAuth 2.0 Authorization Framework:
Bearer Token Usage»
[4] <a class="moz-txt-link-freetext" href="https://tools.ietf.org/html/rfc6750">https://tools.ietf.org/html/rfc6750</a>

Thanks for these references. I cross-read RFC 6750, but it does not seem to standardize how to integrate OAUTH2 with IMAP.

If you've tried to implement OAUTH2 correctly in a mail application (not a website), then you'll quickly notice serious problems. OAUTH2 leaves tons of things implementation-specific, but these details matter a lot in the implementation. See what the author had to say about it: https://hueniverse.com/oauth-2-0-and-the-road-to-hell-8eec45921529 If you try to implement it as a client, you notice that very quickly.

A few quotes from RFC 6570 that should give immediate laughs to everybody who ever implemented a standard:

  • No response defined
    "4. Example Access Token Response. Typically, a bearer token is returned to the client as part of an OAuth 2.0 [RFC6749] access token response. An example of such a response is:". Now, I love examples in specs. The problem here, there is no actual specification. There is only this example. "Typically" is not good enough, if you implement an app for millions of users.
  • Scope is not defined.

My minimum bar for a "standard" is that I can implement it in a client, and then it will work with every server that implements the same standard. Given that we talk with thousands of ISPs, I cannot "know" or hardcode stuff about the ISP. This is most definitely not the case for OAUTH2, not even for the core spec.

Concrete things that I'm missing in our case:

  • Definition of the specific SASL scheme, so that I can connect IMAP with OAUTH2. (As said, this must work 100% identical for all ISPs.)

  • Knowing how I get the JSON result, given that I'm in an interactive browser session.

  • Google says I should use the system web browser for their OAUTH2 login. I can launch a URL there, but wouldn't know how to follow which URL we're at or get the JSON result from that, given that there is no defined API that works cross-browser. Obviously, we'd need an API for all of Windows, Mac, Linux, Android and iOS.

  • The "scope" is currently completely up to the ISP to define, yet I need to pass the right scope in. I'm dead already with this little detail. This would need to be defined in a standard.

  • Google, Microsoft and others want me to register my application and authenticate the calls from my app. That poses 3 concrete problems:

  • Given that I'm Open-Source, that's theoretically impossible. Thunderbird simply publishes its key and begs people not to use it. The silliness of that should be obvious.

  • I have to enter into a legal contract with Google and Microsoft. That means they can stipulate legal terms on me. That is highly problematic and goes against any idea of an open standard.

  • It's practically impossible to get the secrets from all ISPs in the world, because there are thousands. Note that all app authors would need to get the secrets for all ISPs. Imagine I just want to make an open-source "mail check" app. That's practically impossible. (This "imagine" is not just theoretical. I've created several mail check apps in my life, one of them got 3 million users.)

  • Definition of how expiry works, exactly. It makes a difference whether a session lasts 6 minutes or 6 hours or 6 months or forever. The UI would be different, too. Take, for example, a non-interactive mail check session. And your token expires. This is a serious problem in some scenarios. It has to be clearly defined when the various tokens expires. Note that there are 3 different expiry times involved (server, auth token, refresh token, unless I missed some more).

Before all of these problems are resolved in a standard, OAUTH2 isn't a standard for me. It's a framework, not an actual protocol.

Ben

JC
Joshua Cranmer 🐧
Thu, Aug 29, 2019 11:26 PM

On 8/29/2019 3:50 PM, Ben Bucksch wrote:

Concrete things that I'm missing in our case:

  • Definition of the specific SASL scheme, so that I can connect IMAP
    with OAUTH2. (As said, this must work 100% identical for all ISPs.)

... Every protocol that implements SASL is required to list the
supported mechanisms, and OAUTHBEARER is the correct SASL scheme.

  • Knowing how I get the JSON result, given that I'm in an
    interactive browser session.
  • Google says I should use the system web browser for their OAUTH2
    login. I can launch a URL there, but wouldn't know how to follow
    which URL we're at or get the JSON result from that, given that
    there is no defined API that works cross-browser. Obviously, we'd
    need an API for all of Windows, Mac, Linux, Android and iOS.
  • The "scope" is currently completely up to the ISP to define, yet I
    need to pass the right scope in. I'm dead already with this little
    detail. This would need to be defined in a standard.
  • Google, Microsoft and others want me to register my application
    and authenticate the calls from my app. That poses 3 concrete
    problems:
    o Given that I'm Open-Source, that's theoretically impossible.
    Thunderbird simply publishes its key and begs people not to
    use it. The silliness of that should be obvious.
    o I have to enter into a legal contract with Google and
    Microsoft. That means they can stipulate legal terms on me.
    That is highly problematic and goes against any idea of an
    open standard.
    o It's practically impossible to get the secrets from all ISPs
    in the world, because there are thousands. Note that all app
    authors would need to get the secrets for all ISPs. Imagine I
    just want to make an open-source "mail check" app. That's
    practically impossible. (This "imagine" is not just
    theoretical. I've created several mail check apps in my life,
    one of them got 3 million users.)

There are mechanisms to register the client ID dynamically, and
mechanisms for the authentication process to tell you where to find the
OAuth parameters to do the request (including endpoint locations).
However, these mechanisms are optional. We should have made
implementation in Thunderbird contingent on providers actually
supporting these flows as an incentive to get them to do the work.

  o
  • Definition of how expiry works, exactly. It makes a difference
    whether a session lasts 6 minutes or 6 hours or 6 months or
    forever. The UI would be different, too. Take, for example, a
    non-interactive mail check session. And your token expires. This
    is a serious problem in some scenarios. It has to be clearly
    defined when the various tokens expires. Note that there are 3
    different expiry times involved (server, auth token, refresh
    token, unless I missed some more).

It's no different then Kerberos expiration.

--
Joshua Cranmer
Thunderbird and DXR developer
Source code archæologist

On 8/29/2019 3:50 PM, Ben Bucksch wrote: > Concrete things that I'm missing in our case: > > * Definition of the specific SASL scheme, so that I can connect IMAP > with OAUTH2. (As said, this must work 100% identical for all ISPs.) > ... Every protocol that implements SASL is required to list the supported mechanisms, and OAUTHBEARER is the correct SASL scheme. > > * Knowing how I get the JSON result, given that I'm in an > interactive browser session. > * Google says I should use the system web browser for their OAUTH2 > login. I can launch a URL there, but wouldn't know how to follow > which URL we're at or get the JSON result from that, given that > there is no defined API that works cross-browser. Obviously, we'd > need an API for all of Windows, Mac, Linux, Android and iOS. > * The "scope" is currently completely up to the ISP to define, yet I > need to pass the right scope in. I'm dead already with this little > detail. This would need to be defined in a standard. > * Google, Microsoft and others want me to register my application > and authenticate the calls from my app. That poses 3 concrete > problems: > o Given that I'm Open-Source, that's theoretically impossible. > Thunderbird simply publishes its key and begs people not to > use it. The silliness of that should be obvious. > o I have to enter into a legal contract with Google and > Microsoft. That means they can stipulate legal terms on me. > That is highly problematic and goes against any idea of an > open standard. > o It's practically impossible to get the secrets from all ISPs > in the world, because there are thousands. Note that all app > authors would need to get the secrets for all ISPs. Imagine I > just want to make an open-source "mail check" app. That's > practically impossible. (This "imagine" is not just > theoretical. I've created several mail check apps in my life, > one of them got 3 million users.) > There are mechanisms to register the client ID dynamically, and mechanisms for the authentication process to tell you where to find the OAuth parameters to do the request (including endpoint locations). However, these mechanisms are optional. We should have made implementation in Thunderbird contingent on providers actually supporting these flows as an incentive to get them to do the work. > > o > > > * Definition of how expiry works, exactly. It makes a difference > whether a session lasts 6 minutes or 6 hours or 6 months or > forever. The UI would be different, too. Take, for example, a > non-interactive mail check session. And your token expires. This > is a serious problem in some scenarios. It has to be clearly > defined when the various tokens expires. Note that there are 3 > different expiry times involved (server, auth token, refresh > token, unless I missed some more). > It's no different then Kerberos expiration. -- Joshua Cranmer Thunderbird and DXR developer Source code archæologist
MH
Matt Harris
Fri, Aug 30, 2019 12:31 AM

On 29-Aug-19 10:17 PM, Tanstaafl wrote:

On Wed Aug 28 2019 18:38:54 GMT-0400 (Eastern Standard Time), Ben
Bucksch ben.bucksch@beonex.com wrote:

You cannot use IMAP without a web browser (!) anymore.

? I do it all the time.


/Oh,  Thunderbird acts as a web browser when doing oAuth
authentication,  and your phone apps are all basically browser widgets
of one kind and another.

If as a user you have the temerity to enable less secure apps, Google
will offer you security checkup after security checkup, and helpfully
default to disable less secure apps on the page they load.  All you have
to do is click ok.  So while technically you can enable less secure apps
and just continue to use a password, it is beyond the capacity of many
people and certainly beyond the interests of many more.  So it is for
all practical purposed a dead duck.

How much longer Google will support "less secure apps"  is truly going
to be the question.  I have heard nothing,  but they have yahoo bumbling
along with oAuth,  so a year to two will see it gone is my guess. 
Outlook support for 0Auth in my opinion was the only reason there was a
less secure apps option in the first place. (gsuite integration with MS
Office) Expect it to be deprecated shortly after Outlook offers a
release supporting oAuth.

Matt
/

On 29-Aug-19 10:17 PM, Tanstaafl wrote: > On Wed Aug 28 2019 18:38:54 GMT-0400 (Eastern Standard Time), Ben > Bucksch <ben.bucksch@beonex.com> wrote: >> You cannot use IMAP without a web browser (!) anymore. > ? I do it all the time. > > _______________________________________________ /Oh,  Thunderbird acts as a web browser when doing oAuth authentication,  and your phone apps are all basically browser widgets of one kind and another. If as a user you have the temerity to enable less secure apps, Google will offer you security checkup after security checkup, and helpfully default to disable less secure apps on the page they load.  All you have to do is click ok.  So while technically you can enable less secure apps and just continue to use a password, it is beyond the capacity of many people and certainly beyond the interests of many more.  So it is for all practical purposed a dead duck. How much longer Google will support "less secure apps"  is truly going to be the question.  I have heard nothing,  but they have yahoo bumbling along with oAuth,  so a year to two will see it gone is my guess.  Outlook support for 0Auth in my opinion was the only reason there was a less secure apps option in the first place. (gsuite integration with MS Office) Expect it to be deprecated shortly after Outlook offers a release supporting oAuth. Matt /
T
Tanstaafl
Fri, Aug 30, 2019 2:04 PM

On Thu Aug 29 2019 15:50:36 GMT-0400 (Eastern Standard Time), Ben
Bucksch ben.bucksch@beonex.com wrote:

I hope Google respects Thunderbird to not propose this to us, and
respects standards enough to keep IMAP enabled.

They aren't talking about disabling IMAP, this has to do with OAUTH2,
which is the authentication framework they are trying to force people to
use in order top access their servers over IMAP.

On Thu Aug 29 2019 15:50:36 GMT-0400 (Eastern Standard Time), Ben Bucksch <ben.bucksch@beonex.com> wrote: > I hope Google respects Thunderbird to not propose this to us, and > respects standards enough to keep IMAP enabled. They aren't talking about disabling IMAP, this has to do with OAUTH2, which is the authentication framework they are trying to force people to use in order top access their servers over IMAP.