Prevent logprocessor.py children from zombiefication

[matze]

Every time the logprocessor.py script is invoked in production (every morning, via cron(8)), it creates around a dozen zombie processes that won't vanish until the parent process is terminated. This triggers multiple Nagios WARNINGs every day and CRITICALs every second one, at least. That's of course not acceptable. And no; raising the thresholds cannot be a persistent solution.

A short investigation points to the use of ssh(1) and gzip(1) via Python's submodule within function open_stats_file(): Here the processes are spawned, but just their .stdout property is returned. Thus the later .close() outside that scope never terminates the child process, but only stops receiving data. The Garbage collector will eventually pick the now-only-cyclic references up, but here that does not happen before the scripts ends.

Please note that the Python 2.* documentation ​recommends against direct use of the stream properties anyway.

[matze]

Capture of htop(1) during the issue

[matze]

Update: The garbage collection actually does clean up the zombies. It just happens that the script's heavy, buffered I/O prevents that to happen in a reasonable time (most are undead for hours). This behavior is consistent with the brief rationale of the aforementioned recommendation against accessing the subprocess.std* properties directly.

[sebastian]

I agree with your analysis. The way open_stats_file is implemented the .wait() is never called for the ssh and gzip subprocesses. Hence those subprocesses aren't terminated until the Popen objects are garbage collected. Moreover, the inner file isn't closed if it's passed to gzip.

[sebastian]

​https://hg.adblockplus.org/sitescripts/rev/7368858ce430

[trev]

Fixed breakage caused by this commit: ​https://hg.adblockplus.org/sitescripts/rev/0a1f002a0320