Move hosting of beta versions of Adblock Plus for Firefox back to a self-hosted solution

[tlucas]

Background

Mozilla removes the feature to host beta versions of extensions @ AMO, the currently aimed deployment of this change happens on Feb 22nd '18.
We currently use this feature and have to adapt to continue providing beta versions for our users. Extensions STILL need to be signed by AMO, in order to be installable in Firefox.

What to change

• Change createNightlies.py in abpssembly to:
  • upload nightly builds to the AMO (this step is already implemented)
  • download the auto-signed extension
  • let these auto-signed packages be available on our own downloads-page (similar to ​https://downloads.adblockplus.org/devbuilds/adblockpluschrome/ ), i.e. ​https://downloads.adblockplus.org/devbuilds/adblockplusfirefox/

Resources for the aforementioned processes:

​https://developer.mozilla.org/en-US/Add-ons/Distribution
​http://addons-server.readthedocs.io/en/latest/topics/api/signing.html

• change createNightlies.py in abpssembly to create and update an update-manifest https://downloads.adblockplus.org/devbuilds/adblockplusfirefox/updates.json for Adblock Plus for Firefox, always pointing to the latest version in ​https://downloads.adblockplus.org/devbuilds/adblockplusfirefox/, adhering to the following format:

  {
    "addons": {
      "devbuild@adblockplus.org": {
        "updates": [
          {"version": "<LATEST_VERSION>",
           "update_link": "<DOWNLOAD_LINK>" }
        ]
      }
    }
  }
  

Update the buildtools dependency to:

mercurial	git
c830dfa08e2f	3a925dd

Included changes in buildtools

The list of changes imported by this is:

Ticket	Summary	Component
#6383	Overwrite ID and updateURL for Adblock Plus for Firefox when creating a non-release build	Automation
#6037	Allow popup icons without popup	Automation
#6021	Refactor build.py to use modern argparse	Automation
#6020	Unnecessary code being included in webpack bundles	Automation
#5997	Unit test files are included in the release build	Automation
#5996	Let release automation create Firefox builds	Automation
#5994	Release automation produces wrong commit Message	Automation
#5939	Crowdin downloads suffixed translations to the wrong folder	Automation
#5916	Locale strings are broken in adblockpluschrome when building from Windows	Automation
#5915	adblockpluschrome won't build under Windows	Automation
#5913	ensure_dependencies.py doesn't install Node.js dependencies on Windows	Automation
#5877	First run page is missing some text	Automation
#5844	[flake8-eyeo] Detect (more) redundant parentheses	Automation

​8ed981af2b75	Noissue - Drop support for Firefox 50	Manish Jethani
​79688f4a4aff	Noissue - Make comparable_xml safe against ambiguous tags	Tristan Lucas

• Relevant changes for this issue are #6021 and #6383
• Relevant integration notes are found in #6021

Hints for testers

createNightlies should - besides the build-target gecko behave as before.

Note

This issue circular dependent on #6453

[sebastian]

You cannot have unlisted extensions on AMO. I think the way to go is self-distributing the development builds (after having them signed by AMO). This is also what Mozilla seems to recommend.

[fhd]

Note that that's exactly how we currently distribute the Flattr extension, here's the relevant code: ​https://github.com/flattr/flattr-extension/blob/master/build/amo-upload.js

[sergz]

I saw in IRC that it could be eyeofiles for which a couple of projects are awaiting the configuration (basically proper securing of the server) for CI uploading, so I have added myself to CC to be aware if there is some progress in that direction.

[tlucas]

For reference:

​https://discourse.mozilla.org/t/extension-validated-and-signed-via-the-signing-api-fails-to-download/25671

[abpbot]

A commit referencing this issue has landed:
​Issue 6371 - Update buildtools dep. to c830dfa08e2f, use AMO-signing API