Create uplink for pushing log files

[matze]

In the context of introducing our new log server, we are going to extend the log rotation's postrotate commands to include the upload to the log master, i.e.:

postrotate
    ssh log@logmaster.adblockplus.org nginx_error < /var/log/nginx/error.log.1.gz
endscript

Requirements

We need the master to recognize hosts sending incoming data and dispatch the IO to the respective uplink (from logstash), i.e.:

    # $adblockplus::log::directory/$host/$name
    /var/adblockplus/log/uplink/$host/nginx_error

This should become implemented using a designated user-account and a small Python script on the master, invoked via command= directives from .ssh/authorized_keys which itself is populated via Puppet and identifies the source hosts based on their RSA key.

Notes

In order to avoid accidental or malicious data insertion, the script should ensure that only the logfile's $name part can be passed on by the client. Furthermore, the script itself should be designed to also allow for manual data import by the administrator, after network outages or similar ("single point of failure").

Finally, we should avoid the uplink logic to interprete the incoming data in any fashion. This will allow for greater flexibility, e.g. using compression and additional encryption on individual channels, or an interval other than 24 hours.

[matze]

​https://hg.adblockplus.org/infrastructure/rev/f775f7683ba8