Opened 9 months ago

Closed 8 months ago

Last modified 8 months ago

#4483 closed defect (fixed)

Sitekey whitelisting not working reliably after redirect

Reported by: esanchez Assignee: kzar
Priority: P3 Milestone: Adblock-Plus-1.12.4-for-Chrome-Opera-Safari
Module: Platform Keywords:
Cc: sebastian, arthur, esanchez, Caro Blocked By:
Blocking: Platform: Chrome
Ready: yes Confidential: no
Tester: Unknown Verified working: yes
Review URL(s):

https://codereview.adblockplus.org/29357576/

Description (last modified by kzar)

Environment

Chrome Version 53.0.2785.116 (64-bit)
Adblock Plus 1.12.2.1661 and 1.12.2

How to reproduce

  1. Go to http://prepcableinternetservice.com/
  2. That site redirects to http://ads.prepcableinternetservice.com/ (with a very long query string).
  3. Exception list is not respected, specifically @@||google.com/adsense/domains/caf.js$sitekey=
  4. Refresh page with current URL.
  5. Refresh multiple times with current URL.

Observed behaviour

Ads are not visible most of the times when doing step 1.
Ads are visible most of the times when on Step 4.
Ads are not visible again on few occasions when on Step 5.

Expected behaviour

Whitelisted Accepteble Ads should be visible every time.

Attachments (8)

ads.prepcableinternetservice.com.har (232.1 KB) - added by esanchez 9 months ago.
HAR showing caf.js getting blocked
noAds.png (348.5 KB) - added by esanchez 9 months ago.
No ads on chrome
noAds_JS_Console.png (261.9 KB) - added by esanchez 9 months ago.
JS console view showing google is not defined because caf.js did not load
sitekey_header_redirect.png (533.3 KB) - added by esanchez 9 months ago.
From OpenMail people
ff_redirect.png (833.0 KB) - added by esanchez 9 months ago.
From OpenMail people
lander.png (567.3 KB) - added by esanchez 9 months ago.
From OpenMail people
OpenMail.ogv (5.7 MB) - added by esanchez 9 months ago.
Video Showing how hard it is to repro, but still and issue.
AA1.jpg (256.9 KB) - added by rraceanu 8 months ago.
Ads should be present inside the frame

Download all attachments as: .zip

Change History (37)

comment:1 Changed 9 months ago by kzar

  • Description modified (diff)
  • Platform changed from Chrome to Unknown / Cross platform

Thanks for filing an issue, it mostly looks good. Some feedback:

  • The title says "on Chrome and Safari" yet the description mentions that you haven't tested with Safari. Which is it? (The platform should be set to "Unknown / Cross platform" in either case.)
  • You don't need to email me to let me know you created an issue if you've also CC'd me in the issue.
  • You don't need to CC yourself when creating an issue, the issue creator gets copied in on changes too.

comment:2 Changed 9 months ago by sebastian

  • Description modified (diff)

I cannot see any difference, when using Chrome with the latest Adblock Plus development build (1.12.2.1661) and its default filter lists, and when I use Chrome without Adblock Plus.

  • Can you attach a screenshot of what you see?
  • Which filter lists are you using?
  • Do you have any other extensions installed?

comment:3 Changed 9 months ago by esanchez

It is getting harder to reproduce but it is still reproducible.
What I do is open a new tab paste the URL and hit enter, about 1 in 5 times you will see the issue.

The reason why there are no ads is because /caf.js is getting blocked.

  • I am attaching some screen shots and a HAR file showing this.
  • only easy list and Acceptable ads.
  • Testing in Incognito with no extensions enabled but ABP, but I do have lots of extensions installed enabled on not-incognito, let me know if that matters.
Last edited 9 months ago by esanchez (previous) (diff)

Changed 9 months ago by esanchez

HAR showing caf.js getting blocked

Changed 9 months ago by esanchez

No ads on chrome

Changed 9 months ago by esanchez

JS console view showing google is not defined because caf.js did not load

Changed 9 months ago by esanchez

From OpenMail people

Changed 9 months ago by esanchez

From OpenMail people

Changed 9 months ago by esanchez

From OpenMail people

comment:4 Changed 9 months ago by esanchez

Hi Guys
OpenMail is really interested on how is this going, do you need more information from us?

comment:5 Changed 9 months ago by sebastian

Well, I can reproduce the issue with the current release version Adblock Plus 1.12.2 for Chrome. There are no ads until I reload the page. However, I still cannot reproduce it with the development build. On Safari, I can neither reproduce it with the release version nor with the development build.

kzar, do you have more luck reproducing this issue?

comment:6 Changed 9 months ago by kzar

I have not had time to look into this yet.

comment:7 Changed 9 months ago by esanchez

I just tested on Chrome's ABP 1.12.2.1662 again, I did see the issue a couple of times.

QQ: Do we have a next-release date? Need to give something to OpenMail.
Thanks

comment:8 Changed 9 months ago by sebastian

No, we don't have a release date yet. Perhaps as a (temporary) workaround we could whitelist the effected website(s) by domain, instead of relying on key-based whitelisting which might be the problem here?

comment:9 Changed 9 months ago by kzar

  • Cc kzar removed

Removing myself from CC for now, I don't want to keep getting these nagging emails. I find it quite stressful. I'm still planning to look into this, but can't tell you when that will be. I have a lot to do at the moment and limited time.

comment:10 Changed 9 months ago by esanchez

News:
OpenMail tested with dev build, see the issue about 35%, only when copying pasting the url prepcableinternetservice.com

more news:
Tried disabling Acceptable ads and adding domain restrictions and that does not fix the issue (prod build)

New behavior:
Disabling Acceptable ads and Using those domains restrictions works if I open the 'Adblock plus' tab within Chrome's dev tools before loading the page.

Filters used:

@@||google.com/adsense/domains/caf.js$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@*$elemhide,domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
/px.js?ch=2$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@/px.js?ch=1$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@||survey.g.doubleclick.net^$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ
@@||g.doubleclick.net/apps/domainpark/$domain=prepcableinternetservice.com|ads.prepcableinternetservice.com,sitekey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALo4A9ch0h+1WaF7eiREQsF8ZSdjSPKx9KkKjCqabhCJSzV17noE3IU0F05CJ672CxyFRxdONAgr69GDBpn7MRECAwEAAQ,document,subdocument
Last edited 9 months ago by esanchez (previous) (diff)

Changed 9 months ago by esanchez

Video Showing how hard it is to repro, but still and issue.

comment:11 Changed 8 months ago by esanchez

Original sample changed, no longer a valid use case. this is the new sample: asnagdentalimplants.com

comment:12 Changed 8 months ago by rraceanu

Issue still occurs
Chrome 53 (ABP 1.12.2)
Easylist Germany+ Easylist filter

  1. Open Chrome with ABP 1.12.2 installed
  2. Access asnagdentalimplants.com
  3. If you are able to see ads (links in the middle of the page) then refresh the page.

Observed behaviour

The links in the frame are missing (the ads)

Expected behaviour

Ads are always present.

Notes

Could not reproduce on Firefox or Safari.

Please see attached screenshot (AA1) where ads are missing

Errors encountered

Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
http://ads.asnagdentalimplants.com/favicon.ico Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
http://bam.nr-data.net/1/af82c74cfe?a=13749613&v=974.7d740e1&to=ZlFSNxNTC0J…l%22:1071,%22le%22:1071%7D,%22navigation%22:%7B%7D%7D&jsonp=NREUM.setToken Failed to load resource: net::ERR_BLOCKED_BY_CLIENT
Navigated to http://ads.asnagdentalimplants.com/?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69k…yRkZHdBVkGBxHnmaQ-x6-pvNDfMnDUWwjGkA45a37gKrdKFV6-IIWjpZ0WmX7qRdhu_M8k8XeE
/favicon.ico:1 GET http://ads.asnagdentalimplants.com/favicon.ico net::ERR_BLOCKED_BY_CLIENT
Navigated to http://ads.asnagdentalimplants.com/?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69k…yRkZHdBVkGBxHnmaQ-x6-pvNDfMnDUWwjGkA45a37gKrdKFV6-IIWjpZ0WmX7qRdhu_M8k8XeE
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:27 GET http://ads.asnagdentalimplants.com/px.js?ch=2 net::ERR_BLOCKED_BY_CLIENT
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:35 GET http://www.google.com/adsense/domains/caf.js net::ERR_BLOCKED_BY_CLIENT
?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:93 Uncaught ReferenceError: google is not defined(anonymous function) @ ?cc=1fPbI91PgC9oj6TV_G_NxEYKFx9NPDq-A69kEaeMZ_ABIrjQQqwNe0C4QP7eZjy27l-YLaZKQ1T0H2UHCjEIaYP2GpNAIiU…:93
/favicon.ico:1 GET http://ads.asnagdentalimplants.com/favicon.ico net::ERR_BLOCKED_BY_CLIENT

Changed 8 months ago by rraceanu

Ads should be present inside the frame

comment:13 Changed 8 months ago by sebastian

The behavior reproduced on the current release version is most likely a duplicate of #4386. This issue, however, is about the case where those ads are still blocked. Generally, please always try to reproduce issues with the development build first, if they cannot be reproduced there, then they are already fixed.

comment:14 Changed 8 months ago by rraceanu

I could also reproduce it on dev build 1.12.2.1664, issue is still valid.

comment:15 Changed 8 months ago by kzar

  • Description modified (diff)
  • Owner set to kzar
  • Platform changed from Unknown / Cross platform to Chrome
  • Summary changed from On Redirect, Whitelisted by sitekey not working every time on Chrome and Safari to Sitekey whitelisting not working reliably after redirect

comment:16 Changed 8 months ago by esanchez

  • Cc Caro added

comment:17 follow-up: Changed 8 months ago by kzar

Sorry for the delay with this (and also getting stressed before!), as mentioned in IRC the new test URL http://asnagdentalimplants.com is no longer working. Do you have a working one I can use to investigate further?

comment:18 in reply to: ↑ 17 Changed 8 months ago by esanchez

Replying to kzar:

Sorry for the delay with this (and also getting stressed before!), as mentioned in IRC the new test URL http://asnagdentalimplants.com is no longer working. Do you have a working one I can use to investigate further?

http://asnagdentalimplants.com does have the issue for me, maybe a regional thing?
I am asking OpenMail for more domains we can use, will post when I have them.

comment:19 Changed 8 months ago by kzar

http://asnagdentalimplants.com is working for me again, I guess they have some kind of rate limiting to avoid abuse. Anyway more URLs would be helpful none the less, in case this one stops working again.

comment:20 Changed 8 months ago by kzar

Well I spoke too soon...

It was working earlier, then it started returning "Thank You.", then it had me fill out a captcha, then it started working again, then it started returning a gateway error, now the page seems to just not redirect anywhere.

comment:21 Changed 8 months ago by esanchez

understood, will get you more.

comment:22 Changed 8 months ago by esanchez

Got them:
asuperbdentist.com
ateachingdegree.com
atravelcreditcardnet.com
coolautoinsurance.com
disabilityattorneyaid.com
gainbackpainrelief.com
greatoilchanges.com
mattressdealsuse.com
onteachingdegree.com
teethwhiteningpage.com

comment:23 Changed 8 months ago by kzar

Thanks for those.

This appears to be a timing issue, which also explains why it's so devilish to reproduce or pin down. Good luck bisecting this one...

The onBeforeRequest event for http://www.google.com/adsense/domains/caf.js is occasionally firing just before the onComitted event for the redirected URL and also just before the key is recorded after being sent from the content script. (The key is reliably recorded a while earlier by the onHeadersReceived handler, so it seems very possible to fix this.)

Continuing to investigate...

comment:24 Changed 8 months ago by sebastian

For reference, we only detect keys in the content script for compatibility with Safari. However, so far it was simpler to do it regardless of the browser, but if it will help with this issue, feel free to only perform that code on Safari.

comment:25 Changed 8 months ago by kzar

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

Sorry for the delay with this, it's a tricky problem to solve correctly. I have a solution now but I'm not too happy with it. Will keep you posted.

comment:26 Changed 8 months ago by abpbot

A commit referencing this issue has landed:
Issue 4483 - Update page structure when sitekey is received

comment:27 Changed 8 months ago by kzar

  • Milestone set to Adblock-Plus-for-Chrome-Opera-Safari-next
  • Priority changed from Unknown to P3
  • Ready set
  • Resolution set to fixed
  • Status changed from reviewing to closed

We've decided to go with this less than perfect workaround for now. It's possible that we'll be able to improve things once Safari compatibility is no longer an issue.

Quoting Sebastian:

I have a suspicion that this issue might be caused by the fact that we don't
keep track of the whitelisted frame's IDs but of their URL at the time the
sitekey was seen. This was necessary for compatibility with Safari, but once we
stop supporting Safari with the upstream code, we should cleanup this code and
use frame IDs again.

comment:28 Changed 8 months ago by rraceanu

Issue no longer occurs on 1.12.2.1670, ads no longer disappear. Verified on Chrome 45-50-53.

comment:29 Changed 8 months ago by rraceanu

  • Verified working set
Note: See TracTickets for help on using tickets.