Opened 4 years ago

Last modified 12 months ago

#396 new change

[abp-backend] Add a filter hit statistics backend server

Reported by: sebastian Assignee:
Priority: P2 Milestone:
Module: Infrastructure Keywords: 2015q3
Cc: arthur, trev, kzar, manvel@… Blocked By: #395, #3102, #3105
Blocking: #495 Platform: Unknown
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by kzar)

Background

See #495 and #395.

What to change

Add a new server to our infrastructure (hitstats.adblockplus.org), running the filter hits statistics backend as implemented by #395. It should also allow running queries on the filter hit statistics, that part needs access restriction (basic auth).

The filter hitstats back end provides two routes via the multiplexer:

  • /submit Which accepts JSON data POSTed to it from users who have opted-in to usage collection.
  • /query Which accepts GET parameters to determine a query of the aggregated data and returns JSON back.

Additionally:

  • /static needs to be set up on the web server to point to the static files in sitescripts/filterhits/static

Both /static and /query should be protected by basic authentication so that only authorised people can view the data. /submit should be open to the public though obviously.

Logged data will be saved in individual files with a name based on the current timestamp and a random string. The files are stored within separate directories for each day. For example 2014-12-19/1418994873-g1sroc.log.

@kirill will need access to the logged data (base path specified by the sitescripts.conf filterhitstats.log_dir option.) He will likely need a daily archive of the past day's logs to be sent to him / available for him to download.

There will also be a log file processing-errors.log created inside the log directory should we fail to process any submitted data. This log should be monitored somehow so that we notice if there's a problem.

Resources

Change History (17)

comment:1 Changed 4 years ago by arthur

  • Cc arthur added

comment:2 Changed 4 years ago by trev

  • Blocking 495 added

comment:3 Changed 4 years ago by trev

  • Blocking 155 removed
  • Cc trev added
  • Description modified (diff)
  • Ready set
  • Sensitive unset

comment:4 Changed 3 years ago by fhd

  • Description modified (diff)
  • Platform set to Unknown
  • Priority changed from Unknown to P2
  • Summary changed from Add a filter hits stats server to our infrastructure to Add a filter hit statistics backend server

comment:5 Changed 3 years ago by trev

  • Cc kzar added
  • Description modified (diff)

comment:6 Changed 3 years ago by saroyanm

  • Cc manvel@… added

comment:7 Changed 3 years ago by sven

  • Keywords 2014q4 added

comment:8 Changed 3 years ago by sven

  • Keywords 2014q4 removed

comment:9 Changed 3 years ago by kzar

  • Description modified (diff)

comment:10 Changed 3 years ago by kzar

  • Description modified (diff)

comment:11 Changed 3 years ago by kzar

  • Description modified (diff)

comment:12 Changed 3 years ago by kzar

  • Description modified (diff)

comment:13 Changed 3 years ago by kzar

  • Description modified (diff)

comment:14 Changed 2 years ago by fhd

  • Keywords 2015q3 added
  • Tester set to Unknown

comment:15 Changed 2 years ago by matze

  • Blocked By 3102 added

comment:16 Changed 2 years ago by matze

  • Blocked By 3105 added

comment:17 Changed 12 months ago by ferris

  • Summary changed from Add a filter hit statistics backend server to [abp-backend] Add a filter hit statistics backend server
Note: See TracTickets for help on using tickets.