Opened on 05/07/2018 at 09:41:58 AM
Closed on 08/30/2019 at 05:06:28 PM
#6651 closed change (rejected)
Introduce CI for Adblock Plus (devbuilds)
Reported by: | tlucas | Assignee: | |
---|---|---|---|
Priority: | Unknown | Milestone: | |
Module: | Automation | Keywords: | closed-in-favor-of-gitlab |
Cc: | sebastian, kvas, ferris, kzar, fhd, matze, sergz, erikvold | Blocked By: | |
Blocking: | Platform: | Unknown / Cross platform | |
Ready: | no | Confidential: | no |
Tester: | Unknown | Verified working: | no |
Review URL(s): |
Description (last modified by sergz)
Background
Maintaining our own solution for continuously integrating changes into development builds has grown increasingly complex, while popular tools have grown easy to use and to configure.
What to do
Tasks in our infrastructure:
- http://hub.eyeo.com/issues/11192 -> Discussion about storing sensitive data on gitlab.com (outcome: don't)
- Create a skeleton for a VM, which can be registered as a runner for a project on gitlab.com (case study: https://gitlab.com/triluc/vagrant-puppet-gitlab-setup )
- (Related to the above: http://hub.eyeo.com/issues/11275 -> requesting custom runners for gitlab.com)
After implementing the tickets listed below, add a .gitlab-ci.yml which should invoke a gitlab-pipeline to build and upload a devbuild for adblockpluschrome for Firefox and Chrome any time a new commit is pushed to master, or for Edge, any time a new commit is pushed to edge (as soon as we get rid of the edge-branch, we can also trigger the job for Edge from the master branch as well.). Let these builds be listed in a development environment on gitlab.com.
A working case study can be found here. (it assumes ./build.py upload ... to be available)
Note
This ticket is meant for discussion and / or as a parent ticket for more detailed children (on both trac and the hub), once we decided on a tool.
Related tickets
Ticket | Summary | Component |
---|---|---|
#6220 | add CI for adblockpluscore | Core |
#5133 | Add CI for libadblockplus | Libadblockplus |
Attachments (0)
Change History (21)
comment:1 in reply to: ↑ description ; follow-up: ↓ 2 Changed on 05/07/2018 at 10:45:35 AM by sebastian
comment:2 in reply to: ↑ 1 Changed on 05/07/2018 at 11:01:37 AM by tlucas
Replying to sebastian:
How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?
The stores would be handled by the last point (integrating the upload machinery into buildtools), uploading (or downloading -> AMO) the generated files to our own servers would still be done like it's done now (that is, if we choose to let abp-builds-1 or any other server in our infrastructure be a worker) -> we copy the generated builds to an arbitrary folder on the worker (or even the currently configured), which is then rsynced to the downloads-server OR we scp them (we would have to configure the tools with appropriate private keys and restrictions then)
Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?
It can be generated (e.g. with jinja2), but you would have to commit the changes. (which is naturally also true for static files). The idea was to have a generic layer (metadata.*?) above the CI configuration files. While not necessary, this would be nice to have i guess (so that not everyone who tries to switch the branch an extension is build from has to dig through the CI's documentation).
However, we should still make sure the configuration files are valid (e.g. by integrating linting tools into tox or npm)
comment:3 follow-up: ↓ 4 Changed on 05/07/2018 at 11:43:07 AM by sebastian
I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?
As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.
comment:4 in reply to: ↑ 3 Changed on 05/07/2018 at 12:40:51 PM by tlucas
Replying to sebastian:
I'm wondering how the CI will authenticate at our servers and the stores in order to push new builds (without leaking keys/certificates/passwords)?
If we stick to the scenario with gitlab, were we register our very own machine as a worker for CI, then that machine can be provisioned with the appropriate data (same as with e.g. sitescripts.ini, so the sensitive data would never get touched or passed by anyone. We would of course still need to make sure that no sensitive data is printed in the log files)
As for generating the configuration with a script, I still fail to see any benefits. Perhaps if we are going to support multiple CI systems, there would be a case, but this doesn't seem to be the plan. Otherwise, it just adds an additional/redundant layer of abstraction, and comes with the downside that whenever you changed something, you have to run the script and commit the output.
Yes, fair enough - however, i would still add a tool for linting the file(s) (e.g. GitLab CI provides an api over https, also there is this, Travis CI and CircleCI provide their own cli).
comment:5 Changed on 05/07/2018 at 12:42:15 PM by sebastian
Sounds good.
comment:6 Changed on 05/08/2018 at 08:50:30 AM by tlucas
FWIW, all 3 tools also provide the possibility to pass secret values and / or run secret jobs. That said, we wouldn't depend on our own machines being provisioned with the secret values.
comment:7 Changed on 05/09/2018 at 01:35:49 PM by tlucas
- Blocked By 6339, 6369 added
comment:9 Changed on 05/18/2018 at 06:55:31 AM by tlucas
- Blocked By 6339, 6369 removed
The blockers were targeting release builds, while this solely is meant for devbuilds (currently) - i removed them.
comment:10 Changed on 05/18/2018 at 07:57:08 AM by tlucas
- Cc sergz added
- Description modified (diff)
Also adding Sergei, as he might be able to provide / gain something to / from this endeavor.
comment:11 Changed on 05/18/2018 at 08:00:43 AM by tlucas
- Description modified (diff)
comment:12 Changed on 05/18/2018 at 11:08:56 AM by sergz
- Description modified (diff)
comment:13 follow-up: ↓ 14 Changed on 05/18/2018 at 11:19:12 AM by sergz
Thanks a lot for adding me, please keep me in the loop.
I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.
comment:14 in reply to: ↑ 13 Changed on 05/18/2018 at 12:10:35 PM by tlucas
Replying to sergz:
Thanks a lot for adding me, please keep me in the loop.
I have a lot to say and need to learn a lot, so keeping it short, IMO the first action should be to establish a CI, possibly with minimal continues deployment. Securing of the latter of course should be taken into account but it should not be a blocker so far.
I definitely like your thought - however, finding a mixture of gitlab-CI and on premises-CD is most likely even more work than simply implementing everything in one go (FWIW the logic for deploying stuff is already there and i recently shared that case study for provisioning a runner with Matze, Paco and Lior - plan here is to let that be done by Paco / Lior / myself in a cooperation) - i'll also add you to the respective Hub-Ticket (for a more generic gitlab-runner than you requested) once it's created, so that you'll be up to date.
comment:15 Changed on 08/24/2018 at 12:49:37 PM by tlucas
- Review URL(s) modified (diff)
comment:16 Changed on 08/28/2018 at 08:20:42 AM by tlucas
- Blocking 6890 added
comment:17 Changed on 09/06/2018 at 05:00:56 PM by abpbot
A commit referencing this issue has landed:
Issue 6651 - Pt1: Qunit tests through gitlab CI
comment:18 Changed on 09/06/2018 at 05:31:14 PM by tlucas
- Blocking 6890 removed
comment:19 Changed on 09/07/2018 at 09:12:28 AM by abpbot
A commit referencing this issue has landed:
Issue 6651,6220 - add gitlab-ci config
comment:20 Changed on 02/05/2019 at 03:43:33 AM by erikvold
- Cc erikvold added
comment:21 Changed on 08/30/2019 at 05:06:28 PM by sebastian
- Keywords closed-in-favor-of-gitlab added
- Resolution set to rejected
- Status changed from new to closed
Sorry, but we switched to GitLab. If this issue is still relevant, please file it again in the new issue tracker.
How would the devbuilds generated by the CI get uploaded to our servers (Firefox and Safari) or to the stores (Chrome and Microsoft Edge)?
Any reason the configuration file must be generated, rather than just adding it statically to the repository? Can it even be generated (without requiring the resulting configuration file being updated manually)?