Opened 12 months ago

Last modified 3 months ago

#6651 new change

Introduce CI for Adblock Plus (devbuilds)

Reported by: tlucas Assignee:
Priority: Unknown Milestone:
Module: Automation Keywords:
Cc: sebastian, kvas, ferris, kzar, fhd, matze, sergz, erikvold Blocked By:
Blocking: Platform: Unknown / Cross platform
Ready: no Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29862580/

Description (last modified by sergz)

Background

Maintaining our own solution for continuously integrating changes into development builds has grown increasingly complex, while popular tools have grown easy to use and to configure.

What to do

Tasks in our infrastructure:

After implementing the tickets listed below, add a .gitlab-ci.yml which should invoke a gitlab-pipeline to build and upload a devbuild for adblockpluschrome for Firefox and Chrome any time a new commit is pushed to master, or for Edge, any time a new commit is pushed to edge (as soon as we get rid of the edge-branch, we can also trigger the job for Edge from the master branch as well.). Let these builds be listed in a development environment on gitlab.com.

A working case study can be found here. (it assumes ./build.py upload ... to be available)


Note

This ticket is meant for discussion and / or as a parent ticket for more detailed children (on both trac and the hub), once we decided on a tool.

Related tickets


Change History (20)

comment:1 in reply to: ↑ description ; follow-up: Changed 12 months ago by sebastian

How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?

Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?

comment:2 in reply to: ↑ 1 Changed 12 months ago by tlucas

Replying to sebastian:

How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?

The stores would be handled by the last point (integrating the upload machinery into buildtools), uploading (or downloading -> AMO) the generated files to our own servers would still be done like it's done now (that is, if we choose to let abp-builds-1 or any other server in our infrastructure be a worker) -> we copy the generated builds to an arbitrary folder on the worker (or even the currently configured), which is then rsynced to the downloads-server OR we scp them (we would have to configure the tools with appropriate private keys and restrictions then)

Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?

It can be generated (e.g. with jinja2), but you would have to commit the changes. (which is naturally also true for static files). The idea was to have a generic layer (metadata.*?) above the CI configuration files. While not necessary, this would be nice to have i guess (so that not everyone who tries to switch the branch an extension is build from has to dig through the CI's documentation).

However, we should still make sure the configuration files are valid (e.g. by integrating linting tools into tox or npm)

comment:3 follow-up: Changed 12 months ago by sebastian

I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?

As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.

comment:4 in reply to: ↑ 3 Changed 12 months ago by tlucas

Replying to sebastian:

I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?

If we stick to the scenario with gitlab, were we register our very own machine as a worker for CI, then that machine can be provisioned with the appropriate data (same as with e.g. sitescripts.ini, so the sensitive data would never get touched or passed by anyone. We would of course still need to make sure that no sensitive data is printed in the log files)

As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.

Yes, fair enough - however, i would still add a tool for linting the file(s) (e.g. GitLab CI provides an api over https, also there is this, Travis CI and CircleCI provide their own cli).

comment:5 Changed 12 months ago by sebastian

Sounds good.

comment:6 Changed 12 months ago by tlucas

FWIW, all 3 tools also provide the possibility to pass secret values and / or run secret jobs. That said, we wouldn't depend on our own machines being provisioned with the secret values.

comment:7 Changed 12 months ago by tlucas

  • Blocked By 6339, 6369 added

comment:8 Changed 12 months ago by tlucas

  • Description modified (diff)

comment:9 Changed 11 months ago by tlucas

  • Blocked By 6339, 6369 removed

The blockers were targeting release builds, while this solely is meant for devbuilds (currently) - i removed them.

comment:10 Changed 11 months ago by tlucas

  • Cc sergz added
  • Description modified (diff)

Also adding Sergei, as he might be able to provide / gain something to / from this endeavor.

comment:11 Changed 11 months ago by tlucas

  • Description modified (diff)

comment:12 Changed 11 months ago by sergz

  • Description modified (diff)

comment:13 follow-up: Changed 11 months ago by sergz

Thanks a lot for adding me, please keep me in the loop.

I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.

comment:14 in reply to: ↑ 13 Changed 11 months ago by tlucas

Replying to sergz:

Thanks a lot for adding me, please keep me in the loop.

I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.

I definitely like your thought - however, finding a mixture of gitlab-CI and on premises-CD is most likely even more work than simply implementing everything in one go (FWIW the logic for deploying stuff is already there and i recently shared that case study for provisioning a runner with Matze, Paco and Lior - plan here is to let that be done by Paco / Lior / myself in a cooperation) - i'll also add you to the respective Hub-Ticket (for a more generic gitlab-runner than you requested) once it's created, so that you'll be up to date.

comment:15 Changed 8 months ago by tlucas

  • Review URL(s) modified (diff)

comment:16 Changed 8 months ago by tlucas

  • Blocking 6890 added

comment:17 Changed 8 months ago by abpbot

A commit referencing this issue has landed:
Issue 6651 - Pt1: Qunit tests through gitlab CI

comment:18 Changed 8 months ago by tlucas

  • Blocking 6890 removed

comment:19 Changed 8 months ago by abpbot

A commit referencing this issue has landed:
Issue 6651,6220 - add gitlab-ci config

comment:20 Changed 3 months ago by erikvold

  • Cc erikvold added
Note: See TracTickets for help on using tickets.