[abp-backend] Add a filter hit statistics backend server

[sebastian]

Background

See #495 and #395.

What to change

Add a new server to our infrastructure (hitstats.adblockplus.org), running the filter hits statistics backend as implemented by #395. It should also allow running queries on the filter hit statistics, that part needs access restriction (basic auth).

The filter hitstats back end provides two routes via the multiplexer:

• /submit Which accepts JSON data POSTed to it from users who have opted-in to usage collection.
• /query Which accepts GET parameters to determine a query of the aggregated data and returns JSON back.

Additionally:

• /static needs to be set up on the web server to point to the static files in sitescripts/filterhits/static

Both /static and /query should be protected by basic authentication so that only authorised people can view the data. /submit should be open to the public though obviously.

Logged data will be saved in individual files with a name based on the current timestamp and a random string. The files are stored within separate directories for each day. For example 2014-12-19/1418994873-g1sroc.log.

@kirill will need access to the logged data (base path specified by the sitescripts.conf filterhitstats.log_dir option.) He will likely need a daily archive of the past day's logs to be sent to him / available for him to download.

There will also be a log file processing-errors.log created inside the log directory should we fail to process any submitted data. This log should be monitored somehow so that we notice if there's a problem.

Resources

• I've got ​a branch of sitescripts in GitHub that includes the development VM setup I've been using. If you grab that and type vagrant up the server should be accessible at ​http://localhost:5000. Looking at ​how I've set up the VM might help you set up the real server.

[titaniumtv app]

spam

[nps1337]

spam

[impulsepp]

spam

[nps1337]

spam

[run3unblockedaz]

spam