[abp2blocklist] Prevent $subdocument filters from blocking top-level documents

[kzar]

Background

Content Blocking rules cannot distinguish between top-level documents and sub-documents in the way that Adblock Plus filters can. This means that filters designed only to block sub-documents are converted into Content Blocking rules that end up blocking both sub-documents and top-level documents. These rules can cause false positives and break whitelisted acceptable adverts. (For example see #4280.)

What to change

For any blocking rule without a domain part remove the "document" resource-type. If the rule only has the "document" resource-type then it should simply be removed.

Where the URL filter does contain the domain part, add an exception for top-level URLs using Safari 11's unless-top-url property, but only if the rule has no if-domain or unless-domain property.

Notes

This will unfortunately reduce the effectiveness of the Content Blocking rules generated, but until Apple provides us the power to distinguish between top-level documents and sub-documents we are left with no choice. The new unless-top-url property, which only works in Safari 11 and onwards, is close even though not exactly what is needed. Once Safari 11 becomes the minimum supported version, unless-top-url should be used in all such cases to prevent the blocking of top-level URLs.

[kzar]

Thinking about it I don't know how good an idea this is, since specific blocking rules have the same problem. What's to stop a blocking rule that matches a domain wrongly matching a request for a top-level document?

[kzar]

In case we do want to go ahead with this I've got the change ready.

I'll also send Mario an example easylist+exceptionrules_content_blocker.json file so he can test if these changes would help.

[kzar]

I still fear even allowing blocking rules with a hostname part to block document requests might be too much. For example take this filter that's in EasyList:

||com/banners/$image,object,subdocument

That will block document requests for any subdomain of .com, even though technically a hostname is included "com":

[
	{
		"trigger": {
			"url-filter": "^https?://([^/]+\\.)?com/banners/",
			"resource-type": [
				"image",
				"media",
				"document"
			]
		},
		"action": {
			"type": "block"
		}
	}
]

Anyway I'll let you guys do some testing, attaching two files:

1. easylist+exceptionrules_content_blocker_issue_4327_specific_only.json.zip
2. easylist+exceptionrules_content_blocker_issue_4327_none_at_all.json.zip

The first has been generated as this issue describes, rules generated without a hostname part will not block "document" requests.

The second allows no rules at all to block "document" requests.

It would be really useful to see how much they help to reduce Acceptable Ads false positives. Then we'll have enough information to decide how to proceed with this change.

[mario]

@kzar, it seems like you attached the file easylist+exceptionrules_content_blocker_issue_4327_specific_only.json.zip twice. Would you mind adding easylist+exceptionrules_content_blocker_issue_4327_none_at_all.json.zip as well? Thanks!

[kzar]

Whoops so I did, there you go.

[kzar]

Unassigning myself from this and adding Manish to Cc since he's been doing a lot of good work with abp2blocklist recently.

For reference Manish we weren't sure if this change was a good idea, we just hoped it would help reduce false positives. The aim was to be able to trust the generated rules enough that we could refresh them automatically.

[mjethani]

I think this is a good idea, for what it's worth.

[kzar]

Sweet it sounds like Apple are finally ​adding what we need to Safari 11

Added if-top-url and unless-top-url, new triggers that execute when a regular expression is matched against the entirety of the main document URL.

I guess we should use those instead of removing the document type from generic request blocking rules?

[mjethani]

Yes, we could simply repeat the URL pattern in the unless-top-url list to restrict the blocking to subdocuments only:

{
  "trigger": {
    "url-filter": "^https?://.*/foo",
    "resource-type": ["document"],
    "unless-top-url": ["^https?://.*/foo"]
  },
  "action": {
    "type": "block"
  }
}

This will not work unfortunately if there's also a domain specified, since if-domain and unless-top-url are mutually exclusive, but in that case we can simply remove the document resource type from the list.

[sebastian]

Let's say the top-level document has the URL https://example.com/foo and there is a sub-frame with the URL https://example.com/bar. Now the filter ||example.com$subdocument should block only the frame. Currently, the top-level document would be blocked as well. But if we duplicate the pattern in url-filter and unless-top-url nothing would be blocked there. So either way the behavior is wrong, not to mention the issue mentioned above when there is a $domain filter option. On the other hand, the false positives we had here were quite bad, so if we can get rid of them I guess we have to accept some false negatives.

[abpbot]

A commit referencing this issue has landed:
​Issue 4327 - Do not apply $subdocument filters to top-level documents

[mjethani]

But if we duplicate the pattern in url-filter and unless-top-url nothing would be blocked there.

I think it's OK that subdocuments don't get blocked because of this, but this will also unblock other resource types. If we want to be proper, we should generate two rules here, a separate one for the $subdocument, and apply this only to that rule.

[abpbot]

A commit referencing this issue has landed:
​Issue 4327 - Prevent filters with no hostname from blocking documents