Sitekey whitelisting not working reliably after redirect

[esanchez]

Environment

Chrome Version 53.0.2785.116 (64-bit)
Adblock Plus 1.12.2.1661 and 1.12.2

How to reproduce

1. Go to ​http://prepcableinternetservice.com/
2. That site redirects to ​http://ads.prepcableinternetservice.com/ (with a very long query string).
3. Exception list is not respected, specifically @@||google.com/adsense/domains/caf.js$sitekey=
4. Refresh page with current URL.
5. Refresh multiple times with current URL.

Observed behaviour

Ads are not visible most of the times when doing step 1.
Ads are visible most of the times when on Step 4.
Ads are not visible again on few occasions when on Step 5.

Expected behaviour

Whitelisted Accepteble Ads should be visible every time.

[kzar]

Thanks for filing an issue, it mostly looks good. Some feedback:

• The title says "on Chrome and Safari" yet the description mentions that you haven't tested with Safari. Which is it? (The platform should be set to "Unknown / Cross platform" in either case.)
• You don't need to email me to let me know you created an issue if you've also CC'd me in the issue.
• You don't need to CC yourself when creating an issue, the issue creator gets copied in on changes too.

[sebastian]

I cannot see any difference, when using Chrome with the latest Adblock Plus development build (1.12.2.1661) and its default filter lists, and when I use Chrome without Adblock Plus.

• Can you attach a screenshot of what you see?
• Which filter lists are you using?
• Do you have any other extensions installed?

[esanchez]

It is getting harder to reproduce but it is still reproducible.
What I do is open a new tab paste the URL and hit enter, about 1 in 5 times you will see the issue.

The reason why there are no ads is because /caf.js is getting blocked.

• I am attaching some screen shots and a HAR file showing this.
• only easy list and Acceptable ads.
• Testing in Incognito with no extensions enabled but ABP, but I do have lots of extensions installed enabled on not-incognito, let me know if that matters.

[esanchez]

HAR showing caf.js getting blocked

[esanchez]

No ads on chrome

[esanchez]

JS console view showing google is not defined because caf.js did not load

[esanchez]

From OpenMail people

[esanchez]

From OpenMail people

[esanchez]

From OpenMail people

[esanchez]

Hi Guys
OpenMail is really interested on how is this going, do you need more information from us?

[sebastian]

Well, I can reproduce the issue with the current release version Adblock Plus 1.12.2 for Chrome. There are no ads until I reload the page. However, I still cannot reproduce it with the development build. On Safari, I can neither reproduce it with the release version nor with the development build.

kzar, do you have more luck reproducing this issue?

[kzar]

I have not had time to look into this yet.

[esanchez]

I just tested on Chrome's ABP 1.12.2.1662 again, I did see the issue a couple of times.

QQ: Do we have a next-release date? Need to give something to OpenMail.
Thanks

[sebastian]

No, we don't have a release date yet. Perhaps as a (temporary) workaround we could whitelist the effected website(s) by domain, instead of relying on key-based whitelisting which might be the problem here?

[kzar]

Removing myself from CC for now, I don't want to keep getting these nagging emails. I find it quite stressful. I'm still planning to look into this, but can't tell you when that will be. I have a lot to do at the moment and limited time.

[esanchez]

News:
OpenMail tested with dev build, see the issue about 35%, only when copying pasting the url prepcableinternetservice.com

more news:
Tried disabling Acceptable ads and adding domain restrictions and that does not fix the issue (prod build)

New behavior:
Disabling Acceptable ads and Using those domains restrictions works if I open the 'Adblock plus' tab within Chrome's dev tools before loading the page.

Filters used:

@@||google.com/adsense/domains/caf.js$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@*$elemhide,domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
/px.js?ch=2$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@/px.js?ch=1$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@||survey.g.doubleclick.net^$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@||g.doubleclick.net/apps/domainpark/$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ,document,subdocument

[esanchez]

Video Showing how hard it is to repro, but still and issue.

[esanchez]

Original sample changed, no longer a valid use case. this is the new sample: asnagdentalimplants.com

[rraceanu]

Issue still occurs
Chrome 53 (ABP 1.12.2)
Easylist Germany+ Easylist filter

1. Open Chrome with ABP 1.12.2 installed
2. Access asnagdentalimplants.com
3. If you are able to see ads (links in the middle of the page) then refresh the page.

Observed behaviour

The links in the frame are missing (the ads)

Expected behaviour

Ads are always present.

Notes

Could not reproduce on Firefox or Safari.

Please see attached screenshot (AA1) where ads are missing

Errors encountered

Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
​http://ads.asnagdentalimplants.com/favicon.ico Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
​http://bam.nr-data.net/1/af82c74cfe?a=13749613&v=974.7d740e1&to=ZlFSNxNTC0J…l%22:1071,%22le%22:1071%7D,%22navigation%22:%7B%7D%7D&jsonp=NREUM.setToken Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
Navigated to ​http://ads.asnagdentalimplants.com/?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69k…yRkZHdBVkGBxHnmaQ-x6-pvNDfMnDUWwjGkA45a37gKrdKFV6-IIWjpZ0WmX7qRdhu_M8k8XeE
/favicon.ico:1 GET ​http://ads.asnagdentalimplants.com/favicon.ico net::ERR_BLOCKED_BY_CLIENT
Navigated to ​http://ads.asnagdentalimplants.com/?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69k…yRkZHdBVkGBxHnmaQ-x6-pvNDfMnDUWwjGkA45a37gKrdKFV6-IIWjpZ0WmX7qRdhu_M8k8XeE
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:27 GET ​http://ads.asnagdentalimplants.com/px.js?ch=2 net::ERR_BLOCKED_BY_CLIENT
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:35 GET ​http://www.google.com/adsense/domains/caf.js net::ERR_BLOCKED_BY_CLIENT
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:93 Uncaught ReferenceError: google is not defined(anonymous function) @ ?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:93
/favicon.ico:1 GET ​http://ads.asnagdentalimplants.com/favicon.ico net::ERR_BLOCKED_BY_CLIENT

[rraceanu]

Ads should be present inside the frame

[sebastian]

The behavior reproduced on the current release version is most likely a duplicate of #4386. This issue, however, is about the case where those ads are still blocked. Generally, please always try to reproduce issues with the development build first, if they cannot be reproduced there, then they are already fixed.

[rraceanu]

I could also reproduce it on dev build 1.12.2.1664, issue is still valid.

[kzar]

Sorry for the delay with this (and also getting stressed before!), as mentioned in IRC the new test URL ​http://asnagdentalimplants.com is no longer working. Do you have a working one I can use to investigate further?

[esanchez]

Replying to kzar:

Sorry for the delay with this (and also getting stressed before!), as mentioned in IRC the new test URL ​http://asnagdentalimplants.com is no longer working. Do you have a working one I can use to investigate further?

​http://asnagdentalimplants.com does have the issue for me, maybe a regional thing?
I am asking OpenMail for more domains we can use, will post when I have them.

[kzar]

​http://asnagdentalimplants.com is working for me again, I guess they have some kind of rate limiting to avoid abuse. Anyway more URLs would be helpful none the less, in case this one stops working again.

[kzar]

Well I spoke too soon...

It was working earlier, then it started returning "Thank You.", then it had me fill out a captcha, then it started working again, then it started returning a gateway error, now the page seems to just not redirect anywhere.

[esanchez]

understood, will get you more.

[esanchez]

Got them:
asuperbdentist.com
ateachingdegree.com
atravelcreditcardnet.com
coolautoinsurance.com
disabilityattorneyaid.com
gainbackpainrelief.com
greatoilchanges.com
mattressdealsuse.com
onteachingdegree.com
teethwhiteningpage.com

[kzar]

Thanks for those.

This appears to be a timing issue, which also explains why it's so devilish to reproduce or pin down. Good luck bisecting this one...

The onBeforeRequest event for ​http://www.google.com/adsense/domains/caf.js is occasionally firing just before the onComitted event for the redirected URL and also just before the key is recorded after being sent from the content script. (The key is reliably recorded a while earlier by the onHeadersReceived handler, so it seems very possible to fix this.)

Continuing to investigate...

[sebastian]

For reference, we only detect keys in the content script for compatibility with Safari. However, so far it was simpler to do it regardless of the browser, but if it will help with this issue, feel free to only perform that code on Safari.

[kzar]

Sorry for the delay with this, it's a tricky problem to solve correctly. I have a solution now but I'm not too happy with it. Will keep you posted.

[abpbot]

A commit referencing this issue has landed:
​Issue 4483 - Update page structure when sitekey is received

[kzar]

We've decided to go with this less than perfect workaround for now. It's possible that we'll be able to improve things once Safari compatibility is no longer an issue.

Quoting ​Sebastian:

I have a suspicion that this issue might be caused by the fact that we don't
keep track of the whitelisted frame's IDs but of their URL at the time the
sitekey was seen. This was necessary for compatibility with Safari, but once we
stop supporting Safari with the upstream code, we should cleanup this code and
use frame IDs again.

[rraceanu]

Issue no longer occurs on 1.12.2.1670, ads no longer disappear. Verified on Chrome 45-50-53.