Introduce CI for Adblock Plus (devbuilds)

[tlucas]

Background

Maintaining our own solution for continuously integrating changes into development builds has grown increasingly complex, while popular tools have grown easy to use and to configure.

What to do

Tasks in our infrastructure:

• ​http://hub.eyeo.com/issues/11192 -> Discussion about storing sensitive data on gitlab.com (outcome: don't)
• Create a skeleton for a VM, which can be registered as a runner for a project on gitlab.com (case study: ​https://gitlab.com/triluc/vagrant-puppet-gitlab-setup )
• (Related to the above: ​http://hub.eyeo.com/issues/11275 -> requesting custom runners for gitlab.com)

After implementing the tickets listed below, add a .gitlab-ci.yml which should invoke a gitlab-pipeline to build and upload a devbuild for adblockpluschrome for Firefox and Chrome any time a new commit is pushed to master, or for Edge, any time a new commit is pushed to edge (as soon as we get rid of the edge-branch, we can also trigger the job for Edge from the master branch as well.). Let these builds be listed in a development ​environment on gitlab.com.

A working case study can be found ​here. (it assumes ./build.py upload ... to be available)

Ticket	Summary	Component
#7456	ABP stops working on Firefox	Platform
#7422	abort-current-inline-script snippet does not work with asynchronous code	Core
#7410	strip-fetch-query-parameter snippet does not work in Firefox Mobile	Core
#7390	Filter content based on HTTP headers	Core
#7337	Create :-abp-contains-visible() advanced selector	Core
#7314	Secure our own snippets code execution	Core
#7269	Do not rewrite preflight OPTIONS requests	Platform
#7142	Add examples to snippet JSDoc	Core
#7055	[Gitlab CI] [Firefox] run "publish -- gecko" though CI	Automation
#7053	[Gitlab CI] [Firefox] Create meta files for Adblock Plus for Firefox development builds	Automation
#7051	Snippet to disable InstartLogic	Core
#7020	[Gitlab CI] Sign Adblock Plus for Firefox development builds via a Node.js script	Automation
#7006	Implement snippet for targeting Base64-decoded versions of image data URIs	Core
#6983	Implement prevent-inline-scripts snippet	Core
#6863	Add support for blocking inline scripts using the StreamFilter API on Firefox and a Content-Security-Policy directive	Platform
#6842	Apply content filtering exceptions to snippet filters	Core
#6826	Use contentScripts API for snippets on Firefox 59+	Platform
#6819	Sandbox snippets	Core
#6813	Reduce default MIN_INVOCATION_INTERVAL for element hiding emulation	Core
#6812	Implement named arguments for snippets	Core
#6762	Implement support for cookie-related filters	Core
#6682	[Gitlab CI] Automate (unsigned) development build creation of Adblock Plus	Automation
#6681	[Gitlab CI] Add linting for a .gitlab-ci.yml	Automation
#6667	Reuse code from ElemHide module for faster lookup of Snippets scripts	Core
#6539	Implement support for snippets	Platform
#6538	Implement support for snippet filters	Core
#6459	[meta] Avoid all types of DOM modifications whenever possible	Platform
#6100	Procedural hiding filter triggers unresponsive page	Platform
#6051	[webextension] Blocked iframes are not collapsed (hidden)	Platform
#5988	[webextension] Devtools resource type dropdown doesn't work	Platform
#5417	Blocking circumvention on major RU domains	Core
#5382	Uponit circumvention - WebRTC blocking	Platform
#5324	PageFair Recovery circumvention ads	Platform
#5308	Tisoomi circumvention on news.de and aachener-zeitung.de with inline script	Platform
#4891	blob connections not blocked in Chrome	Platform
#4781	Chameleonx circumvention	Platform
#4750	Instart Logic advanced circumvention	Platform
#4691	carbuyer.co.uk using interesting detection technique	Platform
#4405	Hiding filters doesn't work on fs.to	Platform
#2095	Prevent background redirects	Core

Note

This ticket is meant for discussion and / or as a parent ticket for more detailed children (on both trac and the hub), once we decided on a tool.

Related tickets

Ticket	Summary	Component
#6220	add CI for adblockpluscore	Core
#5133	Add CI for libadblockplus	Libadblockplus

[sebastian]

How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?

Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?

[tlucas]

Replying to sebastian:

How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?

The stores would be handled by the last point (integrating the upload machinery into buildtools), uploading (or downloading -> AMO) the generated files to our own servers would still be done like it's done now (that is, if we choose to let abp-builds-1 or any other server in our infrastructure be a worker) -> we copy the generated builds to an arbitrary folder on the worker (or even the currently configured), which is then rsynced to the downloads-server OR we scp them (we would have to configure the tools with appropriate private keys and restrictions then)

Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?

It can be generated (e.g. with jinja2), but you would have to commit the changes. (which is naturally also true for static files). The idea was to have a generic layer (metadata.*?) above the CI configuration files. While not necessary, this would be nice to have i guess (so that not everyone who tries to switch the branch an extension is build from has to dig through the CI's documentation).

However, we should still make sure the configuration files are valid (e.g. by integrating linting tools into tox or npm)

[sebastian]

I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?

As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.

[tlucas]

Replying to sebastian:

I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?

If we stick to the scenario with gitlab, were we register our very own machine as a worker for CI, then that machine can be provisioned with the appropriate data (same as with e.g. sitescripts.ini, so the sensitive data would never get touched or passed by anyone. We would of course still need to make sure that no sensitive data is printed in the log files)

As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.

Yes, fair enough - however, i would still add a tool for linting the file(s) (e.g. GitLab CI provides an api over https, also there is ​this, Travis CI and CircleCI provide their own cli).

[sebastian]

Sounds good.

[tlucas]

FWIW, all 3 tools also provide the possibility to pass secret values and / or run secret jobs. That said, we wouldn't depend on our own machines being provisioned with the secret values.

[tlucas]

The blockers were targeting release builds, while this solely is meant for devbuilds (currently) - i removed them.

[tlucas]

Also adding Sergei, as he might be able to provide / gain something to / from this endeavor.

[sergz]

Thanks a lot for adding me, please keep me in the loop.

I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.

[tlucas]

Replying to sergz:

Thanks a lot for adding me, please keep me in the loop.

I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.

I definitely like your thought - however, finding a mixture of gitlab-CI and on premises-CD is most likely even more work than simply implementing everything in one go (FWIW the logic for deploying stuff is already there and i recently shared that case study for provisioning a runner with Matze, Paco and Lior - plan here is to let that be done by Paco / Lior / myself in a cooperation) - i'll also add you to the respective Hub-Ticket (for a more generic gitlab-runner than you requested) once it's created, so that you'll be up to date.

[abpbot]

A commit referencing this issue has landed:
​Issue 6651 - Pt1: Qunit tests through gitlab CI

[abpbot]

A commit referencing this issue has landed:
​Issue 6651,6220 - add gitlab-ci config

[sebastian]

Sorry, but we switched to GitLab. If this issue is still relevant, please file it again in the ​new issue tracker.