Hi, I have a script which runs hourly and compares the versions of my add-on displayed on addons.mozilla.org with the versions listed in my user guide, so that if I forget to add the release notes for a new version to the user guide, the script reminds me and I can fix it. This script worked fine with addons.mozilla.org. Now that we've moved over to addons.thunderbird.net, however, it's failing. When I look at the content of the response it's getting back, I see this: <title>Access denied | addons.thunderbird.net used Cloudflare to restrict access</title> Is there some specific reason why it was necessary to introduce this increased level of CloudFlare filtering of addons.thunderbird.net that wasn't being done for addons.mozilla.org? Is there any way I can go back to being able to scrape my add-on's versions so I can continue to check automatically that everything is copacetic on my end?   jik

On 2018-07-17 2:29 p.m., Jonathan Kamens wrote: > > Is there some specific reason why it was necessary to introduce this > increased level of CloudFlare filtering of addons.thunderbird.net that > wasn't being done for addons.mozilla.org? > > Is there any way I can go back to being able to scrape my add-on's > versions so I can continue to check automatically that everything is > copacetic on my end? > addons.mozilla.org doesn't use Cloudflare - in fact they don't use a third party reverse proxy at all. This is because they have essentially unlimited resources, at least compared to us :) Our setup is very different from theirs for cost reasons, so service differences and bugs are quite likely to pop up. Cloudflare has a lot of security and blocking settings, and some that we don't have any control over. I'm not exactly sure what's blocking your requests, but it might be the lack of a reasonable user agent if you're using your own script. I've disabled the "Browser Integrity Check", so that might help. Also, if you're scraping the website, you might be better off using the API https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#versions-list or the versioncheck service.

Thanks, I'll use the API! On 7/17/18 6:26 PM, Andrei Hajdukewycz wrote: > On 2018-07-17 2:29 p.m., Jonathan Kamens wrote: >> >> Is there some specific reason why it was necessary to introduce this >> increased level of CloudFlare filtering of addons.thunderbird.net >> that wasn't being done for addons.mozilla.org? >> >> Is there any way I can go back to being able to scrape my add-on's >> versions so I can continue to check automatically that everything is >> copacetic on my end? >> > addons.mozilla.org doesn't use Cloudflare - in fact they don't use a > third party reverse proxy at all. This is because they have > essentially unlimited resources, at least compared to us :) Our setup > is very different from theirs for cost reasons, so service differences > and bugs are quite likely to pop up. > > Cloudflare has a lot of security and blocking settings, and some that > we don't have any control over. I'm not exactly sure what's blocking > your requests, but it might be the lack of a reasonable user agent if > you're using your own script. I've disabled the "Browser Integrity > Check", so that might help. > > Also, if you're scraping the website, you might be better off using > the API > https://addons-server.readthedocs.io/en/latest/topics/api/addons.html#versions-list > or the versioncheck service. >