Opened 4 years ago

Closed 4 years ago

#3168 closed change (fixed)

Add a script for generating new content blocker lists

Reported by: fhd Assignee: fhd
Priority: P3 Milestone:
Module: Sitescripts Keywords:
Cc: sebastian, kzar Blocked By:
Blocking: #3176 Platform: Unknown / Cross platform
Ready: no Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29328894/
https://codereview.adblockplus.org/29329537/

Description (last modified by fhd)

Background

We currently have to create the content blocker lists stored in contentblockerlists by manually downloading/combining the lists, and then running abp2blocklist. We should automate this as much as possible, since it needs to be done frequently, and should eventually happen automatically.

What to change

Add a new script sitescripts.content_blocker_lists.bin.generate_lists, that will:

  1. Download easylist_noadult.txt and exceptionrules.txt.
  2. Fetch abp2blocklist and invoke it to generate easylist_content_blocker.json from easylist_noadult.txt and easylist+exceptionrules_content_blocker.json from the combination of easylist_noadult.txt and exceptionrules.txt.

Change History (12)

comment:1 Changed 4 years ago by fhd

  • Summary changed from [contentblockerlists] Add a script for generating new content blocking lists to [contentblockerlists] Add a script for generating new content blocker lists

comment:2 Changed 4 years ago by fhd

  • Owner set to fhd

comment:3 Changed 4 years ago by fhd

  • Priority changed from Unknown to P3
  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:4 Changed 4 years ago by fhd

  • Cc sebastian added
  • Review URL(s) modified (diff)

Added another patch that avoids using ensure_dependencies.py as discussed with Sebastian.

No strong opinion on what's better, mildly leaning towards the new patch now.

comment:5 Changed 4 years ago by fhd

  • Review URL(s) modified (diff)

Removed the patch that used ensure_dependencies.py.

comment:6 Changed 4 years ago by fhd

  • Cc dave added

comment:7 Changed 4 years ago by fhd

  • Blocking 3176 added

comment:8 Changed 4 years ago by fhd

  • Description modified (diff)
  • Ready unset
  • Summary changed from [contentblockerlists] Add a script for generating new content blocker lists to Add a script for generating new content blocker lists

Updated the description based on our discussion, what do you think? Suggestions on where in sitescripts the script should be and how it should be called welcome. I was wondering if it should go into the subscriptions module, not sure.

comment:9 Changed 4 years ago by kzar

  • Cc kzar added; dave removed

comment:10 Changed 4 years ago by fhd

  • Description modified (diff)

comment:11 Changed 4 years ago by fhd

  • Review URL(s) modified (diff)

comment:12 Changed 4 years ago by fhd

  • Resolution set to fixed
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.