Fix automation to update subscriptions.xml

[arthur]

How to reproduce

1. Open ​https://hg.adblockplus.org/adblockplusui/file/tip/subscriptions.xml

Observed behaviour

No subscription entry for EasyList Spanish is found.

Expected behaviour

There should be a subscription entry for EasyList Spanish since it's a recommended filter list since ​19 January.

[trev]

@greiner: Why did you change the component? We have automation updating subscriptions.xml in adblockplusui, this automation clearly isn't working right now. How is this not an Infrastructure bug?

[greiner]

@trev I didn't change anything so better ask matze.

[trev]

Sorry, I misread the change. @matze: Question in comment 2 is for you.

[matze]

I didn't know about that stuff being auto-generated and thought this being an Infrastructure ticket is a mistake. Even now I do not recall to ever have stumbled upon this, and I also haven't found any reference in code yet. So I guess it's happening on server16 or something?

[trev]

The script in question is sitescripts/extensions/bin/updateRecommendations.py, and I assume that it is indeed still running on server16. Given that the file path is incorrect there, it's not really surprising that things aren't working. In fact, it seems that this was never updated after subscriptions.xml moved to the adblockplusui repository.

[trev]

Added review for the sitescripts part. Note that this script is reverting some changes that appear to have been added in the adblockplusui repository manually.

[trev]

Never mind, the copy in adblockplusui is bogus. The real one is in adblockpluscore.

[abpbot]

A commit referencing this issue has landed:
​Issue 5171 - Fix automation to update subscriptions.xml

[trev]

@matze: I updated the script but it doesn't seem to be running anywhere. It was definitely running on hg1 but it doesn't any more, I don't know whether it was lost in the server migration or removed intentionally. The following command should be run as a cron job, maybe once a day:

python -m sitescripts.extensions.bin.updateRecommendations

If you add it to hg1 again, then it is best run as user hg. The following entries need to be added to sitescripts.ini:

subscriptions_repository=/home/hg/web/adblockpluscore/
subscriptions_path=chrome/content/ui/subscriptions.xml

It will work with a remote repository as well however, from any server. You should have the ssh:// URL in subscriptions_repository then and the cron user should have SSH access to adblockpluscore repository (name in commits will be "hgbot").

[arthur]

I think this is an urgent issue since there are already two new recommeded filter lists (Spanish and Vietnamese) that new users won't get by default due to this issue. This should improve their user experience for them and the longer it takes this to be fixed, the more new users only get EasyList as the default list which isn't designed for Spanish or Vietnamese sites.

[matze]

Now the core version (​https://hg.adblockplus.org/adblockpluscore/rev/db2002ef7cf5) seems fine after manual update (inclusion with cron or as hook will follow later). But I wonder why there is a "core" and an "UI" version (with the latter being the one this ticket is actually about)?

---

The updateRecommendations.py script requires the following items to be present as well:

[subscriptions]
repository=/home/hg/web/subscriptionlist
recommendationsTemplate=subscriptions/template/recommendations.xml

The aforementioned subscriptions_* directives are associated with the [extensions] group.

[greiner]

Replying to matze:

But I wonder why there is a "core" and an "UI" version (with the latter being the one this ticket is actually about)?

The file in adblockplusui is merely a mock that's used for the testing environment.

[matze]

Replying to greiner:

The file in adblockplusui is merely a mock that's used for the testing environment.

Thank you, I guess we're done here then. Setting up the hook will be done in a follow-up ticket (hub).