Make it possible to release Adblock Plus for Chrome without a build signing key

[sebastian]

Background

Currently, the release automation (i.e. the build.py release command), creates signed and unsigned builds of the Chrome extension, and then adds the signed build to the downloads repository. This means in order to perform a release, you not only need access to the Chrome Web Store, but also need to have the private key used to sign the builds. Since this key cannot be changed, we have been careful with sharing it. As a consequence only Felix (and Wladimir, who is on sabbatical) can perform the final steps of a release.

For reference, for the other target platforms (i.e. Firefox and Microsoft Edge) we only generate unsigned builds, and add them to the downloads repository. For Chrome, however, we have to archive signed builds, mainly so that we can test the update scenario from past versions. The key we use to sign these builds locally during release, is the same key used by the Chrome Web Store, in order to sign versions published there.

What to change

Remove logic that generates signed builds and adds them to the downloads repository from the build.py release command. Add a script to abpsembly that will run as a cronjob, downloading any new release builds from the Chrome Web Store, adding them to the downloads repository.

[fhd]

I fear we might have a chicken egg problem here: We currently use those signed builds to test the update _before_ we publish to the Chrome Web Store. I'm not sure if that's really necessary, but it does seem like a good idea.

Maybe we can find another way to test the update scenario? Perhaps using a key only used for test builds, so we can worry less about giving it to people we want to be able to perform a release?

I guess while at it, we could question whether we really need to keep officially signed Chrome releases in the downloads repo, or if specific test builds signed with a non-critical key will suffice. But then again, with the approach you suggest, it doesn't really hurt either.

[tlucas]

Imo the suggested approach covers all concerns - However, the idea of adding another command to build.py, which would be directly dependent on build.py release being executed beforehand, seems odd to me.
I guess i'd prefer the build.py release performing the download itself.

[sebastian]

Replying to fhd:

I fear we might have a chicken egg problem here: We currently use those signed builds to test the update _before_ we publish to the Chrome Web Store. I'm not sure if that's really necessary, but it does seem like a good idea.

Right, if we don't sign the build before the release, we cannot test the update scenario with the release build, anymore, which isn't great. :/

Maybe we can find another way to test the update scenario? Perhaps using a key only used for test builds, so we can worry less about giving it to people we want to be able to perform a release?

I guess while at it, we could question whether we really need to keep officially signed Chrome releases in the downloads repo, or if specific test builds signed with a non-critical key will suffice. But then again, with the approach you suggest, it doesn't really hurt either.

Using a different key to sign the archived builds, however, isn't great either. First of all, I don't even know if Chrome allows sideloading extensions signed with an unknown key. But even if that would work, we still cannot test update scenarios across key changes. Also even if we have to be less concerned about the key's secrecy, it's still an additional prerequisite to perform a release, which would be nice to avoid.

Replying to tlucas:

Imo the suggested approach covers all concerns

It actually does not, see above. But I don't have a better idea, so far.

However, the idea of adding another command to build.py, which would be directly dependent on build.py release being executed beforehand, seems odd to me.
I guess i'd prefer the build.py release performing the download itself.

Thing is you have to wait for the release to get published (which usually takes 1h, but in rare cases can take even longer) before you can download it. Therefore it needs to be a separate command. Alternatively, we could add a script to abpsembly that will run periodically (as cronjob) and adds any new release build from the Chrome Web Store to the downloads repository.

[tlucas]

Replying to sebastian:

Replying to fhd:

I fear we might have a chicken egg problem here: We currently use those signed builds to test the update _before_ we publish to the Chrome Web Store. I'm not sure if that's really necessary, but it does seem like a good idea.

Right, if we don't sign the build before the release, we cannot test the update scenario with the release build, anymore, which isn't great. :/

Maybe we can find another way to test the update scenario? Perhaps using a key only used for test builds, so we can worry less about giving it to people we want to be able to perform a release?

I guess while at it, we could question whether we really need to keep officially signed Chrome releases in the downloads repo, or if specific test builds signed with a non-critical key will suffice. But then again, with the approach you suggest, it doesn't really hurt either.

Using a different key to sign the archived builds, however, isn't great either. First of all, I don't even know if Chrome allows sideloading extensions signed with an unknown key. But even if that would work, we still cannot test update scenarios across key changes. Also even if we have to be less concerned about the key's secrecy, it's still an additional prerequisite to perform a release, which would be nice to avoid.

Replying to tlucas:

Imo the suggested approach covers all concerns

It actually does not, see above. But I don't have a better idea, so far.

Yes, i had the wrong order in my mind.
One solution might be the following:

The CWS offers a workflow, in which an extension is published privately to e.g. all employees of eyeo.com or trusted testers only. We could add a third channel for Adblock Plus on chrome (i.e. the official releases, the dev-versions and the private versions), from which we only sign the private releases. That way a compromised private key would be useless to everyone outside of eyeo.com, we could still test the update-scenario internally and everyone we trust could officially release the extension, without having to have access to the private key.

However, the idea of adding another command to build.py, which would be directly dependent on build.py release being executed beforehand, seems odd to me.
I guess i'd prefer the build.py release performing the download itself.

Thing is you have to wait for the release to get published (which usually takes 1h, but in rare cases can take even longer) before you can download it. Therefore it needs to be a separate command. Alternatively, we could add a script to abpsembly that will run periodically (as cronjob) and adds any new release build from the Chrome Web Store to the downloads repository.

This would definitely be a solution, but if we could rely on some sort of notification about a publishing, i'd rather do this event-driven (however, a quick research unfortunately yielded nothing).

[sebastian]

Replying to tlucas:

The CWS offers a workflow, in which an extension is published privately to e.g. all employees of eyeo.com or trusted testers only. We could add a third channel for Adblock Plus on chrome (i.e. the official releases, the dev-versions and the private versions), from which we only sign the private releases. That way a compromised private key would be useless to everyone outside of eyeo.com, we could still test the update-scenario internally and everyone we trust could officially release the extension, without having to have access to the private key.

Then we could as well just test the update scenario, during the release, with our devbuilds, which would be much simpler, and has essentially the same effect. Still, the point was to test it, under all circumstances, with the very same build that is effectively released, so that if anything goes wrong (e.g. we end up releasing the wrong revision, which happened once in the past), we would at least have performed this check with whatever is going to be released. Introducing yet a third channel doesn't warrant this. But we might have to live with that.

Thing is you have to wait for the release to get published (which usually takes 1h, but in rare cases can take even longer) before you can download it. Therefore it needs to be a separate command. Alternatively, we could add a script to abpsembly that will run periodically (as cronjob) and adds any new release build from the Chrome Web Store to the downloads repository.

This would definitely be a solution, but if we could rely on some sort of notification about a publishing, i'd rather do this event-driven (however, a quick research unfortunately yielded nothing).

Yeah, that is not possible. But FWIW, a cronjob might be simpler (because we already have some of them), and more reliable (because if it fails it just runs again automatically), anyway.

[sebastian]

I just talked to Felix, and he agrees with our conclusion, removing the part from the build.py release command that generates the signed build and adds it to the downloads repository, and adding a cronjob that downloads any new signed release build from the Chrome Web Store and adds it to the downloads repository. I updated the issue description accordingly. For reference, instead of testing the update scenario with the release builds, we will just test it with the devbuilds, before every release.

[tlucas]

The other day i argued that i'd be going to include this into the release-automation project i'm currently working on - but i figured it might make more sense to get this done beforehand.

[tlucas]

For reference, i created a follow up ticket to handle the abpssembly-parts: #6369

[sebastian]

This is no longer relevant, as we will download and archive the builds from the Chrome Web Store instead (see ​webext#45), so that there is no need to deal locally with any keys anymore.