Adding dependencies and build steps to websites

[juliandoucette]

Background

• We would like to share SCSS across websites
• We are currently building SCSS into CSS locally and pushing it to websites
  • Including SCSS from website-defaults, which we are installing locally via npm config

We would like to move the building step onto a staging and/or production server. The easiest (especially in terms of deployment) way to allow this is to make CMS able to perform pre-build and post-build steps. Then website authors could indicate additional actions (i.e. shell commands) to perform before and after generation and generate_static_pages.py would perform those actions during the generation process.

What to change

Introduce a configuration section into settings.ini of the website called [build-steps]. The section would have two configuration variables, each of which is interpreted as a list of strings (separated by newlines):

• before-build -- commands to be run before the generation takes place (in the directory where the sources of the website are located).
• after-build -- commands to be run after the generation takes place (in the directory where the output resides).

Interaction with preview

• Preview should run the before-build steps before launching the web server.

Documentation

New section and options should be added to documentation. It should be emphasized that it's the responsibility of the website developers to ensure that the commands listed in the build steps are present on target production server and that they can be run by the CMS site generation server.

Example

[build-steps]
before-build=
    python ensure_dependencies.py
    npm install
    gulp scss
after-build= html-prettify -r *

[kvas]

@juliandoucette, @saroyanm, @ire: what do you think about the implementation plan for this ticket? In particular, are you comfortable with (1) the responsibility for ensuring that the build steps are available in production (I would imagine a request to Ops should be enough for that) and (2) with the fact that anyone who would want to generate or preview a website that contains build steps would need to make sure the build steps can be run on their computer?

[juliandoucette]

@kvas I think this is backwards. I'd rather rely on infrastructure to run npm install and gulp and call generate_static_pages or runserver (among other things) using gulp.

[kvas]

That would be cool with me (since the CMS developers wouldn't need to do anything ;) but I though it might be useful, as Matze suggested, to support this in CMS, since then these steps would be controlled by website developers rather than infrastructure.

But I think what you're proposing also has advantages: it's more robust, when it's in the infrastructure, and it interacts with preview in a more manual way, which might in some cases also be an advantage.

Anyway, if you would prefer to do this in the infrastructure, we should change this ticket back to "Infrastructure" (or maybe convert to a Hub ticket) and then you should give them exact steps for the websites that need them.

[juliandoucette]

Replying to kvas:

That would be cool with me (since the CMS developers wouldn't need to do anything ;) but I though it might be useful, as Matze suggested, to support this in CMS, since then these steps would be controlled by website developers rather than infrastructure.

I think that just running gulp gives me the exact same control. Am I missing something?

But I think what you're proposing also has advantages: it's more robust, when it's in the infrastructure, and it interacts with preview in a more manual way, which might in some cases also be an advantage.

I don't know what you mean.

Anyway, if you would prefer to do this in the infrastructure, we should change this ticket back to "Infrastructure" (or maybe convert to a Hub ticket) and then you should give them exact steps for the websites that need them.

I agree, and prefer, if I'm not missing anything.

[kvas]

Replying to juliandoucette:

Replying to kvas:

But I think what you're proposing also has advantages: it's more robust, when it's in the infrastructure, and it interacts with preview in a more manual way, which might in some cases also be an advantage.

I don't know what you mean.

For the first part I mean that if you ask the Ops to run gulp for website X, they will probably make sure gulp is available there. If the CMS runs gulp, based on settings.ini, that might not be the case.

And for preview, imagine there's some long build process that takes a while, but the user knows that they don't need to run it now, because what they changed doesn't affect the build artifacts. If CMS would be taking care of build steps, they would run anyway before the preview (unless we'd be very smart about it) but if the build steps are run manually, the user will just do it when they need to.

Anyway, would you prefer to write this ticket as an infrastructure ticket yourself, or would you need some assistance from the CMS team (i.e. from me ;)?

[matze]

For the first part I mean that if you ask the Ops to run gulp for website X, they will probably make sure gulp is available there. If the CMS runs gulp, based on settings.ini, that might not be the case.

Actually the one who operates the CMS in the current environment needs to make sure the dependencies being available anyway. Irregardless of whether those are designated in code, in the documentation, configuration, any combination of the aforementioned or even no proper information.

And for preview, imagine there's some long build process that takes a while, but the user knows that they don't need to run it now, because what they changed doesn't affect the build artifacts. If CMS would be taking care of build steps, they would run anyway before the preview (unless we'd be very smart about it) but if the build steps are run manually, the user will just do it when they need to.

We spoke about that scenario in our call and agreed that a simple flag to activate or suppress configured hooks would render this concern obsolete.

---

However, I strongly recommend to boil the production version down to a single command (be it gulp or something built into the CMS or whatever).

It should not be a problem if that one is for triggering a "clean" or "complete" build, even if that is not what's usually done in development: Just having the option to perform the exact same operation fulfills the associated requirements for being able to reproduce and test.

[juliandoucette]

And for preview, imagine there's some long build process that takes a while, but the user knows that they don't need to run it now, because what they changed doesn't affect the build artifacts. If CMS would be taking care of build steps, they would run anyway before the preview (unless we'd be very smart about it) but if the build steps are run manually, the user will just do it when they need to.

Exactly. Using gulp, we can break down the build steps into pieces and run them individually and in different combinations (the default being a complete build).

Anyway, would you prefer to write this ticket as an infrastructure ticket yourself, or would you need some assistance from the CMS team (i.e. from me ;)?

I don't think I need your help.

[juliandoucette]

This is irrelevant now that we are addressing this and related issues separately in eyeo hub.