Python filter list parsing API

[kvas]

Background

In the context of separating common functionality from sitescripts into a library, the first step is parsing of the filter lists. While we aim to possibly support many different use cases in the future, our current focus is providing support for rendering of filter lists (see #3849).

What to change

Create a module in python-abp library for parsing of filter lists.

The parser should convert a filter list (coming from a file or an HTTP response) to an iterator of python objects that represent different types of lines found in a filter list. Specifically we need to distinguish the following types of lines (their properties are given in square brackets):

• empty line (line that only has whitespace or nothing in it),
• header [program version],
• regular comments [comment text],
• metadata comments [key, value],
• instructions (e.g. %include ...%) [name of the instruction, arguments].
• filters (not parsed further at the moment, but it should be easy to add this functionality in the future if it's needed) [filter expression].

See ​this spec for a more specific definition of different line types in ​Augmented BNF notation (ABNF). The spec covers filter lists as understood by Adblock Plus plugin, so instructions that are intended for preprocessing are not covered by it. The ABNF for these preprocessing instructions would look like this:

instruction = include
include = "%include" *1WSP include-path "%"
include-path = url / local-path
local-path = [ source-name ":" ] path-in-source

where url is defined in ​rfc3986, source-name is an alphanumeric string and path-in-source is a filesystem path separated by forward slashes.

Each line object produced by the parser should contain all information necessary to produce the original string version of the line (variations in non-significant whitespace are ok). The string version should be produced by line_object.to_string().

Take care to write unit tests.

[kzar]

When creating issues please CC in ​the module owner and don't mark the issue as ready. It's the module owner's decision if the issue is ready and to assign a priority.

In this case I don't think the issue is ready, the description is too vague. It doesn't give me enough background to understand what the script is supposed to do, or for what purpose, or for whom.

• Add a background section, explain the rough idea of this script and what its purpose is. If it's replacing an existing script give some background about that, who uses the script and to what end. Why are we even changing things in the first place?
• Improve the "What to change" section, "The streaming API should convert each line of a filter list to a parsed representation distinguishing different types of filters and supporting basic features of each different type." doesn't mean much to me. (Specifically what is the "parsed representation" and how does it differ from what we're starting with?) Enumerate how the script should transform a filter list and give concrete examples.

Without these kind of details it's going to be really hard or even impossible for me to review your code. I'm not familiar with the combineSubscriptions script at all. It's also important so that other people can understand what's going on, for example the testers.

[kzar]

Their BFN would be: ...

What's BFN? (I also have no idea what the code block below is showing.)

[kvas]

Replying to kzar:

Their BFN would be: ...

What's BFN? (I also have no idea what the code block below is showing.)

Sorry, that was a typo. Should have been BNF. And the actual notation used here is ABNF, which I linked from the text now.

[kzar]

As this issue is blocking #3849 which in turn blocks #3751 we don't need to mark this as blocking #3751.

(See the "Depgraph" button at the top right of the page. If you click on that for #3751 you can see the whole graph of dependencies.)

[kzar]

I added a quick note to write unit tests and tweaked the title, but otherwise this looks OK now. Marking as ready, thanks.

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/python-abp/rev/5bc6a95c0207