$document doesn't whitelist everything inside iframes

[arthur]

Environment

• Windows 10 Pro
• Chrome 66.0.3359.181 (Official Build) (64-bit)
• ABP 3.1.0.2053
• EasyList Germany+EasyList
• Acceptable Ads

How to reproduce

1. Enable ABP to run in incognito mode (chrome://extensions/?id=ldcecbkkoecffmfljeihcmifjjdoepkn)
2. Open an incognito window
3. Go to ​https://www.teufel.de/ or ​https://madeleine.de
4. Open like products below "LAUTSPRECHER UND MEHR IM ANGEBOT" and wait until finished loading (this is necessary to target ads for you)
5. Go to ​https://www.autozeitung.de/
6. Open the dev tools panel, click on the "Adblock Plus" tab and refresh the page
7. Choose "whitelisted" in the top left drop-down item so you would only see the whitelisted resources
8. Note the whitelisted iframes https://cdn.stroeerdigitalgroup.de/metatag/libraries/s3p_ab.html are whitelisted by @@||cdn.stroeerdigitalgroup.de/metatag/libraries/s3p_ab.html?$subdocument,document in Acceptable Ads
9. Switch from "whitelisted" to "blocked" in the top left drop-down
10. Note https://js.adscale.de/map.js or more resources (like from criteo.com) are blocked coming from the document domain cdn.stroeerdigitalgroup.de

Observed behaviour

Requests within the iframe are being blocked, no ads appearing above the fold on the right side (next and under the newsletter box).

Expected behaviour

Every request within the iframe should load since it is whitelisted by $subdocument,document. Ads should appear above the fold on the right side (next and under the newsletter box).

Notes

• Keep in mind the ads might stop appearing even when everything works. In that case close the incognito window and start targeting again.
• This is a regression caused by ​this commit 1cf57a70f44b.

Hints for testers

• Verify you can no longer reproduce the symptoms described above.
• Verify that you can still no reproduce the problem described in #6595.
• On both Firefox and Chrome do a bunch of testing, checking things like the block counter, blocking, whitelisting and $sitekey whitelisting still work.

[sebastian]

Reproduced. Since this appears to be a new regression, we should look into it before the next release.

[kzar]

I could reproduce this for a while (the blocked requests, but I never saw the advert) and was attempting to bisect. Unfortunately I can no longer see those requests no matter how many times I try to reproduce the setup again, I guess they logged my IP or something.

I'm pretty sure I saw the request get blocked for 3.1 too, so I'm not convinced this is a regression since the last release. I'll attempt again to reproduce tomorrow, hopefully they'll have forgotten my IP by then.

[arthur]

Instead of teufel.de you might want to try madeleine.de.

[kzar]

Thanks, I could reproduce it more reliably this time with madeleine.de. I've managed to bisect and find ​the problematic commit.

Unfortunately the commit in question was attempting to work around limitations with the Chrome extension APIs we use to keep track of pages and frames, changes to this code tends to fix 2 things and break something else! Such problems are often non-trivial and involve race conditions which will only show up on certain websites or at certain times.

[kzar]

One option might be to restore the old onBeforeNavigate listener whilst keeping the new onCommitted listener. That way we'd update the page frame structure earlier in this case, whilst still catching the Service Worker case (#6595) which the previous change fixed. Hardly ideal to duplicate this work, but then none of this is.

[sebastian]

Did you confirm that this workaround would work (which would mean that the issue here is that the frame structure is updated for the about:blank frames not before they sent any requests otherwise)?

[kzar]

Yes, seems to work OK. I tested that this issue no longer happens and that #6595 still doesn't. But I'm open to suggestions if you can think of a better approach, I'm hardly happy to do it this way.

[sebastian]

I don't have any beter idea either, besides a minor simplification, I ​suggested on the code review.

[abpbot]

A commit referencing this issue has landed:
​Issue 6692 - Update frame structure for about:blank frames earlier

[Ross]

In Chrome: This seems to be fixed in that adverts now show on the right as described. However some requests (such as the https://js.adscale.de/map.js are still listed as blocked in the Adblock Plus devtools panel). Should that still be blocked?

In Firefox: The adverts do not load at all. The frame and the Anzeige title are present, but it looks like requests from the frame are blocked.

ABP 3.1.0.2065
Firefox 60 / Windows 8
Chrome 66 / Windows 8

[arthur]

Replying to Ross:

However some requests (such as the https://js.adscale.de/map.js are still listed as blocked in the Adblock Plus devtools panel). Should that still be blocked?

I think that is correct since it is being loaded from the publisher's site.

This looks fixed to me. It is also working in Firefox for me (60.0.2, Win 10).

[Ross]

Thank you for checking! Since it works for you in Firefox and I can see it's fixed in Chrome I'll mark this verified.

ABP 3.1.0.2065
Firefox 60 / Windows 10
Chrome 66 / 49 / Windows 8
Opera 36 / 52 / Windows 8