Opened 2 years ago

Closed 2 years ago

#5171 closed defect (fixed)

Fix automation to update subscriptions.xml

Reported by: arthur Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: trev, greiner, matze, ferris Blocked By:
Blocking: Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29420575/
https://hg.adblockplus.org/adblockpluscore/rev/db2002ef7cf5

Description

How to reproduce

  1. Open https://hg.adblockplus.org/adblockplusui/file/tip/subscriptions.xml

Observed behaviour

No subscription entry for EasyList Spanish is found.

Expected behaviour

There should be a subscription entry for EasyList Spanish since it's a recommended filter list since 19 January.

Change History (16)

comment:1 Changed 2 years ago by matze

  • Cc greiner added; matze removed
  • Component changed from Infrastructure to User-Interface

comment:2 Changed 2 years ago by trev

@greiner: Why did you change the component? We have automation updating subscriptions.xml in adblockplusui, this automation clearly isn't working right now. How is this not an Infrastructure bug?

comment:3 Changed 2 years ago by greiner

@trev I didn't change anything so better ask matze.

comment:4 Changed 2 years ago by trev

  • Cc matze added

Sorry, I misread the change. @matze: Question in comment 2 is for you.

comment:5 Changed 2 years ago by matze

  • Cc ferris added
  • Component changed from User-Interface to Infrastructure

I didn't know about that stuff being auto-generated and thought this being an Infrastructure ticket is a mistake. Even now I do not recall to ever have stumbled upon this, and I also haven't found any reference in code yet. So I guess it's happening on server16 or something?

comment:6 Changed 2 years ago by trev

The script in question is sitescripts/extensions/bin/updateRecommendations.py, and I assume that it is indeed still running on server16. Given that the file path is incorrect there, it's not really surprising that things aren't working. In fact, it seems that this was never updated after subscriptions.xml moved to the adblockplusui repository.

Last edited 2 years ago by trev (previous) (diff)

comment:7 Changed 2 years ago by matze

  • Priority changed from Unknown to P2
  • Ready set

comment:8 Changed 2 years ago by trev

  • Review URL(s) modified (diff)

Added review for the sitescripts part. Note that this script is reverting some changes that appear to have been added in the adblockplusui repository manually.

comment:9 Changed 2 years ago by trev

Never mind, the copy in adblockplusui is bogus. The real one is in adblockpluscore.

comment:10 Changed 2 years ago by abpbot

A commit referencing this issue has landed:
Issue 5171 - Fix automation to update subscriptions.xml

comment:11 Changed 2 years ago by trev

@matze: I updated the script but it doesn't seem to be running anywhere. It was definitely running on hg1 but it doesn't any more, I don't know whether it was lost in the server migration or removed intentionally. The following command should be run as a cron job, maybe once a day:

python -m sitescripts.extensions.bin.updateRecommendations

If you add it to hg1 again, then it is best run as user hg. The following entries need to be added to sitescripts.ini:

subscriptions_repository=/home/hg/web/adblockpluscore/
subscriptions_path=chrome/content/ui/subscriptions.xml

It will work with a remote repository as well however, from any server. You should have the ssh:// URL in subscriptions_repository then and the cron user should have SSH access to adblockpluscore repository (name in commits will be "hgbot").

comment:12 Changed 2 years ago by arthur

I think this is an urgent issue since there are already two new recommeded filter lists (Spanish and Vietnamese) that new users won't get by default due to this issue. This should improve their user experience for them and the longer it takes this to be fixed, the more new users only get EasyList as the default list which isn't designed for Spanish or Vietnamese sites.

comment:13 Changed 2 years ago by matze

  • Owner set to matze

comment:14 follow-up: Changed 2 years ago by matze

Now the core version (https://hg.adblockplus.org/adblockpluscore/rev/db2002ef7cf5) seems fine after manual update (inclusion with cron or as hook will follow later). But I wonder why there is a "core" and an "UI" version (with the latter being the one this ticket is actually about)?


The updateRecommendations.py script requires the following items to be present as well:

[subscriptions]
repository=/home/hg/web/subscriptionlist
recommendationsTemplate=subscriptions/template/recommendations.xml

The aforementioned subscriptions_* directives are associated with the [extensions] group.

comment:15 in reply to: ↑ 14 ; follow-up: Changed 2 years ago by greiner

Replying to matze:

But I wonder why there is a "core" and an "UI" version (with the latter being the one this ticket is actually about)?

The file in adblockplusui is merely a mock that's used for the testing environment.

comment:16 in reply to: ↑ 15 Changed 2 years ago by matze

  • Resolution set to fixed
  • Review URL(s) modified (diff)
  • Status changed from new to closed

Replying to greiner:

The file in adblockplusui is merely a mock that's used for the testing environment.

Thank you, I guess we're done here then. Setting up the hook will be done in a follow-up ticket (hub).

Note: See TracTickets for help on using tickets.