Collect filter hit statistics

[sebastian]

Background

See #495

What to change

Add a setting to Adblock Plus for Firefox, in order to collect filter hit statistics and pushing them to our server. The setting must be disabled by default.

The setting should be located in Filter preferences dialog.
Setting label should be Help us improve Adblock Plus by sending anonymous usage data and it should contain link to our privacy policy page.

Regardless of that setting, no data must be collected while in private browsing mode. And collected data must be discarded when the history is cleared.

Right now, a total hit count and date of last hit are already being stored for each filter. In addition to that, we want to store a per-domain hit count and date of last hit - only if the filter hit statistics setting is enabled.

Once a week (we might tweak that later) the client should send this data to our server using ​https://telemetry.adblockplus.org/submit/adblockplus service URL, code for successful response should be 204 No content.

Data:

{
  "version": 1,               // For the server to recognize outdated clients
  "timeSincePush": 12345,     // UTC Time interval (milliseconds) since previous push
  "addonName": "adblockplus", // see require("info")
  "addonVersion": "2.3.4",    // see require("info")
  "application": "firefox",   // see require("info")
  "applicationVersion": "31", // see require("info")
  "platform": "gecko",        // see require("info")
  "platformVersion": "31",    // see require("info")
  "filters": {
    "||example.com^": {
      "firstParty": {
        "example.com": {
          "hits": 12,         // Number of hits
          "latest": 123456789 // UTC Time interval (milliseconds) of last hit
        },
        "example.org": {"hits": 4, "latest": 987654321}
      },
      "thirdParty": {
        "example.com": {"hits": 5, "latest": 123455489}
      },
      "subscriptions": ["https://easylist-downloads.adblockplus.org/easylist.txt", "https:// ..."]  // Subscription source of filter
    },
    "example.com##foo > bar": {
      ...
    }
  }
}

Important: the data should include only filters from filter subscriptions, not the user's own filters. After successfully sending that data to the server the client should clear all the data it saved and start collecting statistics from scratch.

Also we should send "timeSincePush" in seconds instead of milliseconds. Round "timeSincePush" to an hourly interval (i.e. round 8640 seconds (=2.4 hours) to 7200 seconds (=2 hours)). Round "latest" (child of "filters") to an hourly interval (i.e. round 8640 seconds (=2.4 hours) to 7200 seconds (=2 hours)).

[trev]

I've changed the description to specify the exact format for the data to be sent. Some notes:

• It seems that we don't need the overall count, per-domain data should be sufficient.
• I added a timeSincePush field so that the server knows which time interval the data refers to.
• I also added some fields indicating add-on and platform versions - we will need this metadata to investigate issues.

[Kirill]

I think it would be better change it from:

"filters": {
    "||example.com^": {
      "example.com": {"hits": 12, "latest": 123456789},
      "example.org": {"hits": 4, "latest": 987654321}
    },
    "example.com##foo > bar": {
      ...
    }
  }

to

"filters": [
    {
       "domain": "example.com",
       "hits": 12,
       "latest": 124456573
    },
    {
       "domain": "foo.com",
       "hits": 125,
       "latest": 654872312
    },
    ...
  ]    

[saroyanm]

Replying to Kirill:

I think it would be better change it from:

"filters": {
    "||example.com^": {
      "example.com": {"hits": 12, "latest": 123456789},
      "example.org": {"hits": 4, "latest": 987654321}
    },
    "example.com##foo > bar": {
      ...
    }
  }

to

"filters": [
    {
       "domain": "example.com",
       "hits": 12,
       "latest": 124456573
    },
    {
       "domain": "foo.com",
       "hits": 125,
       "latest": 654872312
    },
    ...
  ]    

What about filter itself ? We need to identify the filter.
Ex:

example.com##foo > bar

and also with your example we'll need to iterate through array for appropriate domain, It's not making things simpler I guess.

[trev]

Replying to Kirill:

I think it would be better change it from:

When making suggestions, feel free to explain exactly why you feel it would be better ;)

As discussed before, a data structure that is naturally represented by a map, should actually be represented as a map - it's easy enough to convert it to an array if some tools cannot deal with it.

[Kirill]

For me it is important that keys are not values. So "||example.com^" : {
should be changed to "filter": "||example.com^",... and "example.org": {... should be "domain":"example.org". With this format it is convenient to run aggregations by filter or by domain etc.

The best implemetation for this should be arrays I think.

Here is an example:

> db.data.find().pretty()
{
  "_id" : ObjectId("54353ce3c04dfa1c747c159a"),
  "version" : 1,
  "timeSincePush" : 12345,
  "addonName" : "adblockplus",
  "addonVersion" : "2.3.4",
  "application" : "firefox",
  "applicationVersion" : "31",
  "platform" : "gecko",
  "platformVersion" : "31",
  "filters" : [
    {
      "filter" : "##.ad-text-links",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 17,
          "latest" : 123456789
        },
        {
          "domain" : "tumblr.com",
          "hits" : 17,
          "latest" : 345678912
        }
      ]
    },
    {
      "filter" : "mobifrance.com#@#.advert-horizontal",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 12,
          "latest" : 123456789
        },
        {
          "domain" : "wired.com",
          "hits" : 5,
          "latest" : 234567891
        }
      ]
    },
    {
      "filter" : "||buy.com^*/affiliate/",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 12,
          "latest" : 123456789
        },
        {
          "domain" : "wired.com",
          "hits" : 5,
          "latest" : 234567891
        },
        {
          "domain" : "facebook.com",
          "hits" : 17,
          "latest" : 345678912
        }
      ]
    }
  ]
}
{
  "_id" : ObjectId("54353d2bc04dfa1c747c159b"),
  "version" : 1,
  "timeSincePush" : 12345,
  "addonName" : "adblockplus",
  "addonVersion" : "2.3.4",
  "application" : "firefox",
  "applicationVersion" : "31",
  "platform" : "gecko",
  "platformVersion" : "31",
  "filters" : [
    {
      "filter" : "##.ad-text-links",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 1,
          "latest" : 123456789
        },
        {
          "domain" : "tumblr.com",
          "hits" : 31,
          "latest" : 345678912
        }
      ]
    },
    {
      "filter" : "mobifrance.com#@#.advert-horizontal",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 17,
          "latest" : 123456789
        },
        {
          "domain" : "wired.com",
          "hits" : 5,
          "latest" : 234567891
        }
      ]
    },
    {
      "filter" : "||adn.ebay.com^",
      "domains" : [
        {
          "domain" : "techcrunch.com",
          "hits" : 12,
          "latest" : 123456789
        },
        {
          "domain" : "wired.com",
          "hits" : 51,
          "latest" : 234567891
        },
        {
          "domain" : "facebook.com",
          "hits" : 3,
          "latest" : 345678912
        },
        {
          "domain" : "twitter.com",
          "hits" : 32,
          "latest" : 345678912
        }
      ]
    }
  ]
}

This is an output from MongoDB.

Now it is very easy to aggregate.
To aggregate for filters:

db.data.aggregate([
  {$unwind :"$filters"}, 
  {$unwind:"$filters.domains"}, 
  {$group:{"_id":{filter:"$filters.filter"}, 
           hits: {"$sum":"$filters.domains.hits"}}}
])

result:

{ "_id" : { "filter" : "||buy.com^*/affiliate/" }, "hits" : 34 }
{ "_id" : { "filter" : "||adn.ebay.com^" }, "hits" : 98 }
{ "_id" : { "filter" : "mobifrance.com#@#.advert-horizontal" }, "hits" : 39 }
{ "_id" : { "filter" : "##.ad-text-links" }, "hits" : 66 }

Aggregate by visited websites:

db.data.aggregate([
  {$unwind :"$filters"}, 
  {$unwind:"$filters.domains"}, 
  {$group:{"_id":{filter:"$filters.domains.domain"}, 
           hits: {"$sum":"$filters.domains.hits"}}}
])

Result

{ "_id" : { "filter" : "twitter.com" }, "hits" : 32 }
{ "_id" : { "filter" : "facebook.com" }, "hits" : 20 }
{ "_id" : { "filter" : "wired.com" }, "hits" : 66 }
{ "_id" : { "filter" : "tumblr.com" }, "hits" : 48 }
{ "_id" : { "filter" : "techcrunch.com" }, "hits" : 71 }

And even aggregating by filter and website#

db.data.aggregate([
  {$unwind :"$filters"}, 
  {$unwind:"$filters.domains"}, 
  {$group:{"_id":{domain:"$filters.domains.domain",filter:"$filters.filter"}, 
           hits: {"$sum":"$filters.domains.hits"}}}
])

Result:

{ "_id" : { "domain" : "techcrunch.com", "filter" : "||buy.com^*/affiliate/" }, "hits" : 12 }
{ "_id" : { "domain" : "wired.com", "filter" : "||adn.ebay.com^" }, "hits" : 51 }
{ "_id" : { "domain" : "facebook.com", "filter" : "||adn.ebay.com^" }, "hits" : 3 }
{ "_id" : { "domain" : "facebook.com", "filter" : "||buy.com^*/affiliate/" }, "hits" : 17 }
{ "_id" : { "domain" : "techcrunch.com", "filter" : "||adn.ebay.com^" }, "hits" : 12 }
{ "_id" : { "domain" : "tumblr.com", "filter" : "##.ad-text-links" }, "hits" : 48 }
{ "_id" : { "domain" : "wired.com", "filter" : "||buy.com^*/affiliate/" }, "hits" : 5 }
{ "_id" : { "domain" : "wired.com", "filter" : "mobifrance.com#@#.advert-horizontal" }, "hits" : 10 }
{ "_id" : { "domain" : "twitter.com", "filter" : "||adn.ebay.com^" }, "hits" : 32 }
{ "_id" : { "domain" : "techcrunch.com", "filter" : "mobifrance.com#@#.advert-horizontal" }, "hits" : 29 }
{ "_id" : { "domain" : "techcrunch.com", "filter" : "##.ad-text-links" }, "hits" : 18 }

This computations are easy, fast and you can even do some map reduce.

What do you think?

[kzar]

@kirill this is the raw format of the data as it's transmitted by the client, we could process in the back end to allow you to use it more easily.

@kirill and @trev can I suggest we go ahead with the map format specified, I'll log the raw data as received in plain text log files but additionally have MongoDB store the data in a way that's easy for @kirill to use and store the aggregated form in MySQL as described in the other ticket.

That way we still have all the raw data and we can try out both ways of viewing the data to see what works best in practice. The only question I have is if the data should be put into MongoDB + MySQL in real time as it is received or later in some kind of a batch process.

[Kirill]

That sounds nice, actually I wouldn't insist on using MongoDB, I'm open to other solutions (Cassandra, Postgresql), i just want to be able to query and to aggregate the raw data somehow fast and efficiently. I'm open for a discussion on which is the best solution there. For me it's just important that the data and data structure suits my needs.

[trev]

@kzar: Please don't mangle the data on the backend. As long as we use a proper data structure, it is always possible to convert it to whatever a particular tool needs - we should not store it in such a way that only one tool can work well with it however.

@Kirill: As long as the data is a blob, it doesn't matter where you store it - accessing it won't be any faster with MongoDB or PostgreSQL or whatever than with regular files. The main effort will always be parsing the data and deciding whether it is something you are interested in.

[Kirill]

Replying to trev:

As long as we use a proper data structure, it is always possible to convert it to whatever a particular tool needs

That's what I'm interested in, helping you to make a proper data structure which can be easily and fast be processed by the calculation. And frankly, I don't understand, why saving the values as keys in the json is proper...

As long as the data is a blob, it doesn't matter where you store it - accessing it won't be any faster with MongoDB or PostgreSQL or whatever than with regular files.

The pros for DB are: Indexing for fast queries and centrality, so I dont need to work with many servers and the data is logically stored centrally, so I can directly start querying and analysing it. I don't see, why we can't try out different alternatives in the beginning of the project. I am also willing to work with blobs and if it can be done efficiently I am willing to spare the database solution.

The main effort will always be parsing the data and deciding whether it is something you are interested in.

Yes, you're right and I'm really excited about working with the data and full of ideas! ;)

[saroyanm]

Replying to trev:

I've changed the description to specify the exact format for the data to be sent.

@Wladimir we also had a request about collecting information per filter whether it's "first" or "third party" filter and collect Subscription source of the filter so according to the request the updated format will look like this:

{
  "version": 1,               // For the server to recognize outdated clients
  "timeSincePush": 12345,     // Time interval (milliseconds) since previous push
  "addonName": "adblockplus", // see require("info")
  "addonVersion": "2.3.4",    // see require("info")
  "application": "firefox",   // see require("info")
  "applicationVersion": "31", // see require("info")
  "platform": "gecko",        // see require("info")
  "platformVersion": "31",    // see require("info")
  "filters": {
    "||example.com^": {
      firstParty: {
        "example.com": {"hits": 12, "latest": 123456789},
        "example.org": {"hits": 4, "latest": 987654321}
      },
      thirdParty: {
        "example.com": {"hits": 5, "latest": 123455489}
      },
      subscriptions: ["EasyList", "EasyList Germany+EasyList"]
    },
    "example.com##foo > bar": {
      ...
    }
  }
}

[Kirill]

It's important for me that the domains and filters are stored as values and not as keys!
cheers from The Hague

[saroyanm]

@Kirill as already was mentioned the data can be converted to fit the tool needs in case it's really mandatory to have other structure.

[Kirill]

@saroyanm Can we somehow collect how often a website is visited? So I can see the difference between filter hits and website visits...

[saroyanm]

@Kirill I'm not sure why we need that ? Do you mean track users on each website they visit ?
I don't think that this is something we need to implement, anyway not sure how that will help us to improve filter lists.
We shouldn't collect data more than we need to improve filter lists.

[Lain_13]

@saroyanm Well, it could help to figure out how many hits on average each rule have on pages of specific site. We do collect this data with filter hits anyway since majority of hits happens only once per page. This just will allow to calculate ratios more precisely and especially for pages where only one filter triggers and triggers too much - usually such situations indicates false positives since normally rules doesn't trigger 50 times per page except we have a site with 50 banners on the same page blocked by generic filter (and that is rather rare).

So, collecting site hits will help us detect unusual spikes in activity on generic filters and will have almost none impact on security (in comparison with data which we already decided to collect). We just will know precise number instead of guessing it from available hit counters.

I think there could be more ways to utilize such data to improve our filter lists beside FP detection. For example very low hit ration on generic filters could mean presence of anti-adblock script which changes list of detection rules (I've seen scripts which have different set of hiding traps on each refresh of the page). Also it may highlight banner rotation system and it could be a good idea to block it instead of banners especially when only part of them ends up blocked.

[saroyanm]

Replying to Lain_13:

@saroyanm Well, it could help to figure out how many hits on average each rule have on pages of specific site.

Also it can help us to find which filters are not being used so far.

So, collecting site hits will help us detect unusual spikes in activity on generic filters and will have almost none impact on security (in comparison with data which we already decided to collect). We just will know precise number instead of guessing it from available hit counters.

Sure.

For example very low hit ration on generic filters could mean presence of anti-adblock script which changes list of detection rules (I've seen scripts which have different set of hiding traps on each refresh of the page).

Good point, make sense to keep in mind.

[saroyanm]

I've updated the description with the latest version of the data format including subscription source, FirstParty and ThirdParty filters. As I've mentioned in current comment. This was initially requested so far as I can see from the @Sebastian handover email.

Also Added Addition Notes which was discussed during implementation with @palant, but missing in description Like what database we can use to store the data on client side.

[greiner]

Why do we use the title of the subscription when we usually identify subscriptions by their URL? Using the URL would not only provide consistency and a more unique ID but also the ability to locate subscriptions that we weren't aware of before.

[saroyanm]

Replying to greiner:

Why do we use the title of the subscription when we usually identify subscriptions by their URL? Using the URL would not only provide consistency and a more unique ID but also the ability to locate subscriptions that we weren't aware of before.

Agree on that, I've modified the description will also update the patch accordingly.

[saroyanm]

As ​mentioned in the review. The label of the Filter Hit statistics option item is not descriptive enough and too long, so I suggest to change it to Send anonymous filter hit statistics.
@Dave what you think ?

[kzar]

@saroyanm Honestly I'm not sure, it's shorter at least and is clearer about what the statistics are but it comes at the expense of being less clear about the motive "Help us to improve ABP by...".

Thing is to fit a message that's perfectly clear in such a short string is not easy, but getting it right is pretty important from a UX perspective. Maybe someone else has a better idea?

[saroyanm]

Replying to saroyanm:

I suggest to change it to Send anonymous filter hit statistics.

getting it right is pretty important from a UX perspective. Maybe someone else has a better idea?

@Wladimir, @Sven ?
Guys any suggestion on this ?

[sven]

Thanks to putting me in this discussion. The wording is not optimal like saroyanm mentioned and nearly no one would understand what "filter hit statistics" is. An alternative would be something like: "Help to improve ABP by sending anonymous usage data". The length of the sentence shouldn't be a problem, since we can use more than one row to show the string.

[saroyanm]

Replying to sven:

An alternative would be something like: "Help to improve ABP by sending anonymous usage data". The length of the sentence shouldn't be a problem, since we can use more than one row to show the string.

As mentioned by @Thomas ​here we are not mentioning ABP anywhere in the extension, so I think we should have wording like : Help to improve Adblock Plus by sending anonymous usage data

[kzar]

@sven well unfortunately the text does not wrap onto multiple lines there, hence my concern in the code review. Also, if this is the string we are going to use could we at least replace the word "sending" with "sharing"?

Anyway, I'll leave you to it... don't want to be accused of bike shedding :p

[sven]

1. I'm fine with writing Adblock Plus instead of ABP
2. A lot of users associate with "sharing", sharing it with other people or in social media. Therefore i wouldn't use the word "sharing". IMO it would only confuse the people.

[saroyanm]

The description updated after IRC discussion.

[saroyanm]

@Sven @Palant: What you think where the option should be located in Adblock Plus filter preferences dialog ?
I think it would looks nice under the Allow Acceptable Ads checkbox, but not sure whether it's good place to have it there.

@Palant also what you think does it make sense to have that option in Mobile ? Should we have additional option in Fennec Settings page ?

[kzar]

@saroyanm a quick note before I forget, the response to successful POST requests to the submission API will be "204 No content" instead of "200 OK" now. If you have any code that checks the status code is 200 (even implicitly) before considering the response a success you will need to adjust accordingly.

Edit: (Don't hard code the expectation of 204 obviously, just accept any successful 2xx status codes.)

[sebastian]

If you are just interested in whether the request has been processed successfully you should check for 200 <= status < 300 instead for one particular 2xx status code anyway.

[saroyanm]

Replying to kzar:

@saroyanm a quick note before I forget, the response to successful POST requests to the submission API will be "204 No content" instead of "200 OK" now. If you have any code that checks the status code is 200 (even implicitly) before considering the response a success you will need to adjust accordingly.

Edit: (Don't hard code the expectation of 204 obviously, just accept any successful 2xx status codes.)

Description updated, thanks for commenting that out, will update the review patch accordingly.

[kzar]

Another thing @saroyanm, if you want to test the collection works with the server in practice I've got a testing VM setup that you can use. If you grab an up to date copy of sitescripts, apply the ​changes from this commit and then type vagrant up you'll have a test server at http://localhost:5000 to play with. You can then submit data to http://localhost:5000/submit and query it back out with the interface at http://localhost:5000/static/query.html.

I've already done this to test your work as of patch set 8, I left you a few comments on the review but it basically worked OK. You might want to re-test now that the 204 status code is returned though, or if you make further changes.

[saroyanm]

Replying to kzar:

I've already done this to test your work as of patch set 8, I left you a few comments on the review but it basically worked OK. You might want to re-test now that the 204 status code is returned though, or if you make further changes.

Tested, thanks :)

[saroyanm]

Once a week (we might tweak that later) the client should send this data to our server using https://hitstats.adblockplus.org/submit service URL, code for successful response should be 204 No content.

I assume this link has been changed do we have new URL where the Filter Hit statistics needs to be submitted ?

[mario]

Replying to saroyanm:

Once a week (we might tweak that later) the client should send this data to our server using https://hitstats.adblockplus.org/submit service URL, code for successful response should be 204 No content.

I assume this link has been changed do we have new URL where the Filter Hit statistics needs to be submitted ?

Please have a look at #3261 and its review discussion.
The URL will be: ​https://telemetry.adblockplus.org/submit/adblockplus

[saroyanm]

Replying to mario:

Please have a look at #3261 and its review discussion.
The URL will be: ​https://telemetry.adblockplus.org/submit/adblockplus

Thanks for reference, update the description.

[trev]

Mass-closing all bugs in Adblock Plus for Firefox module, the codebase of Adblock Plus 3.0 belongs into Platform and User-Interface modules. Old bugs are unlikely to still apply.