Opened on 05/15/2014 at 01:27:56 PM

Closed on 12/22/2017 at 09:56:02 AM

#495 closed change (rejected)

[meta] Filter Hit Statistics Tool

Reported by: trev Assignee:
Priority: P2 Milestone:
Module: Unknown Keywords: 2014q4, meta
Cc: famlam, mapx, fiaefuzz, fhd, sebastian, kzar Blocked By: #394, #395, #396, #2220
Blocking: Platform: Firefox
Ready: no Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by philll)


We currently have to ship a growing file with updated filter data. It contains a lot of unused or outdated filters, either because ad implementation changed or websites has been shut off. We are carrying a lot of "waste" in this file which slowly decreases performance.

What to change

We need a tool which extracts relevant data from a large enough sample of ABP users and then analyses filter hit statistics over the whole sample. In the end, the tool should show us which filters can be erased (because of near zero usage).

Users should be able to opt-in into this and send sufficiently anonymized hit statistics to us. We also need to decide whether/how we will ask users to opt in.

Things to consider:

  • We are currently only saving filter hit statistics in Firefox so implementing this in Firefox first is the most logical choice.
  • Private browsing mode: we aren't saving any hit statistics there, meaning that we probably don't want to show the opt-in option for users using private browsing mode permanently. Same goes for users who disabled hit statistics altogether via "Count filter hits" option in Firefox.
  • Clearing browsing history also clears hit statistics, meaning that we probably don't want to show the opt-in option for users clearing history on shutdown either.
  • Moving filter hit statistics out of patterns.ini and into a separate file might be a good idea, the hit count is already responsible for much of the file size there.
  • Sebastian suggested using Nginx Upload Module to receive data. IMHO that's premature optimization, we will unlikely get so many submissions that a regular FCGI script cannot handle them.

Attachments (0)

Change History (17)

comment:1 Changed on 05/15/2014 at 01:35:21 PM by trev

  • Description modified (diff)

comment:2 Changed on 05/21/2014 at 07:19:46 PM by trev

  • Keywords meta added

comment:3 Changed on 05/22/2014 at 01:23:31 PM by philll

  • Description modified (diff)
  • Keywords 2014q3 added

comment:4 Changed on 07/03/2014 at 01:12:45 PM by arthur

  • Cc famlam added
  • Platform set to Unknown

comment:5 Changed on 07/09/2014 at 12:38:11 PM by philll

  • Platform changed from Unknown to Firefox

comment:6 Changed on 10/06/2014 at 08:20:19 AM by fhd

  • Keywords 2014q4 added; 2014q3 removed

comment:7 Changed on 10/13/2014 at 12:27:46 PM by mapx

  • Cc mapx added

comment:8 Changed on 11/21/2014 at 01:05:21 PM by sven

  • Keywords changed from meta,2014q4 to 2014q4, meta

comment:9 Changed on 04/07/2015 at 03:34:18 PM by saroyanm

  • Blocked By 2220 added

comment:10 Changed on 08/12/2015 at 11:38:55 PM by fiaefuzz

How do I add myself to Cc? Hoping commenting does the trick....

comment:11 Changed on 08/13/2015 at 06:50:38 AM by mapx

  • Cc fiaefuzz added
  • Tester set to Unknown

comment:12 Changed on 09/21/2015 at 01:33:07 PM by matze

  • Blocked By 3105 added

comment:13 Changed on 11/03/2015 at 04:27:29 AM by matze

  • Blocked By 3261 added

comment:14 Changed on 11/03/2015 at 04:50:25 AM by matze

  • Blocked By 3105, 3261 removed

comment:15 Changed on 09/23/2017 at 03:21:34 AM by sebastian

  • Cc fhd sebastian added
  • Ready unset

The backend implemented by #395 has been dropped, in favor of a more naive server-side solution, storing the raw data which are then analyzed by other means, IIRC.

And the client-side implementation (#394), which is under review for over 3 years, became obsolete since the legacy Gecko extension it targets is deprecated now.

It seems likely that we will revisit this feature at some point, starting over at zero, but probably not too soon. However, this issue as it stands is outdated, and therefore must be rewritten or rejected.

comment:16 Changed on 12/22/2017 at 09:55:23 AM by kzar

  • Cc kzar added
  • Component changed from Unknown to Platform

Like Sebastian mentioned this issue should be rewritten or rejected, since it's been another 3 months with no rewrite I'll reject for now.

comment:17 Changed on 12/22/2017 at 09:56:02 AM by kzar

  • Component changed from Platform to Unknown
  • Resolution set to rejected
  • Status changed from new to closed

Add Comment

Modify Ticket

Change Properties
as closed .
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from (none).
Note: See TracTickets for help on using tickets.