Opened 4 years ago

Closed 4 years ago

#3306 closed change (fixed)

Create uplink for pushing log files

Reported by: matze Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: fred Blocked By: #3471, #3479
Blocking: #3307 Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29333248/

Description (last modified by matze)

In the context of introducing our new log server, we are going to extend the log rotation's postrotate commands to include the upload to the log master, i.e.:

postrotate
    ssh log@logmaster.adblockplus.org nginx_error < /var/log/nginx/error.log.1.gz
endscript

Requirements

We need the master to recognize hosts sending incoming data and dispatch the IO to the respective uplink (from logstash), i.e.:

    # $adblockplus::log::directory/$host/$name
    /var/adblockplus/log/uplink/$host/nginx_error

This should become implemented using a designated user-account and a small Python script on the master, invoked via command= directives from .ssh/authorized_keys which itself is populated via Puppet and identifies the source hosts based on their RSA key.

Notes

In order to avoid accidental or malicious data insertion, the script should ensure that only the logfile's $name part can be passed on by the client. Furthermore, the script itself should be designed to also allow for manual data import by the administrator, after network outages or similar ("single point of failure").

Finally, we should avoid the uplink logic to interprete the incoming data in any fashion. This will allow for greater flexibility, e.g. using compression and additional encryption on individual channels, or an interval other than 24 hours.

Change History (6)

comment:1 Changed 4 years ago by matze

  • Blocked By 3471 added
  • Cc fred added
  • Description modified (diff)
  • Ready set

comment:2 Changed 4 years ago by matze

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:3 Changed 4 years ago by matze

  • Sensitive unset
  • Type changed from defect to change

comment:4 Changed 4 years ago by matze

  • Blocked By 3479 added

comment:5 Changed 4 years ago by matze

  • Blocking 3307 added

comment:6 Changed 4 years ago by matze

  • Resolution set to fixed
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.