Generate docs outside the devbuild build process

[fhd]

Background

We are currently generating the docs for adblockplus and adblockpluschrome in createNightlies.py, i.e. as part of the devbuild build process. That's not the best place for it, because we also need to generate docs for libadblockplus and adblockpluscore (see #3618).

What to change

Trigger doc generation through a separate cron job or post-receive hook, not through createNightlies.py. This should generate docs for adblockplus, adblockpluschrome, adblockpluscore, and libadblockplus (​https://adblockplus.org/docs/libadblockplus/).

Things to consider:

1. We should use hg archive to clone the repositories, it's faster.
2. We shouldn't remove the clones after the script, but keep it around and update it instead.
3. We should keep the clone on the same volume as the target dir (i.e. /srv) to ensure the mv is fast enough.
4. We should move the target docs dir and replace it before removing it - will reduce docs downtime.

[fhd]

Reopening, the approach in the review was too hacky. We should do this in sitescripts.

Notes from the review:

1. We should use hg archive for the clone, it's faster.
2. We shouldn't remove the clone after the script, but keep it around and update it instead.
3. We should keep the clone on the same volume (i.e. /srv) to ensure the mv is fast enough.
4. We should move the target docs dir and replace it before removing it - will reduce docs downtime.

[fhd]

This is getting important now, because we also need it for the adblockpluscore documentation.

[fhd]

Sebastian: I don't think #3635 should block this one. Even if we change ensure_dependencies.py, we cannot update it in ABP for Firefox since that one is still in code freeze until #3621 lands, which is blocked by this issue here. I think we should land this here first, then fix #3635, then remove the hack introduced here.

BUT: I just realised we have to run this on server16 if we stick to the hg archive (see #3637), which is bad. So I think it would be best to just use hg clone and hg update in this script, that would make #3635 irrelevant here. What do you think?

[sebastian]

Replying to fhd:

Sebastian: I don't think #3635 should block this one. Even if we change ensure_dependencies.py, we cannot update it in ABP for Firefox since that one is still in code freeze until #3621 lands, which is blocked by this issue here. I think we should land this here first, then fix #3635, then remove the hack introduced here.

I agree. When I first triaged #3635 I assumed that we are still going to use hg archive but rather want to update the repositories than going with the workaround. I updated the issue.

[fhd]

Landed: ​https://hg.adblockplus.org/sitescripts/rev/5c5437a35082

[fhd]

Had to land a small fix: ​https://hg.adblockplus.org/sitescripts/rev/b1507b694d0f