Stats processing fails to interprete lastVersion parameter

[matze]

Background

Stats processing on stats1.adblockplus.org is currently failing consistently. The python -m sitescript.stats.bin.logprocessor command produces the following exception for several logs:

Traceback (most recent call last):
  File "/opt/sitescripts/sitescripts/stats/bin/logprocessor.py", line 530, in parse_source
    data = parse_fileobj(mirror_name, fileobj, geo, geov6, ignored)
  File "/opt/sitescripts/sitescripts/stats/bin/logprocessor.py", line 472, in parse_fileobj
    info = parse_record(line, ignored, geo, geov6)
  File "/opt/sitescripts/sitescripts/stats/bin/logprocessor.py", line 421, in parse_record
    parse_downloader_query(info)
  File "/opt/sitescripts/sitescripts/stats/bin/logprocessor.py", line 308, in parse_downloader_query
    last_update = parse_lastversion(last_version)
  File "/opt/sitescripts/sitescripts/stats/bin/logprocessor.py", line 119, in wrapped
    results.popitem(last=False)
  File "/usr/lib/pypy/lib-python/2.7/collections.py", line 162, in popitem
    raise KeyError('dictionary is empty')
KeyError: 'dictionary is empty'

Analysis

The problem is that parse_lastversion() fails to parse the new format for lastVersion (123456789-1/0 instead of 123456789) and throws an exception. This is an expected situation and we handle it, but it also messes up cache_lru decorator - it doesn't expect exceptions and removes an element from the cache but doesn't add one instead. As a result, the cache goes empty because it thinks that there are no free spots left.

[trev]

This error makes little sense so far unfortunately, that cache cannot be empty. I did verify that the cache algorithm is correct by code inspection, and I did run tests with random input. I will try running processing with real logs now...

[trev]

Ok, the cache can be empty - if the processing function throws. And that's what it does, because it cannot process lastVersion parameter.

[trev]

Fixed: ​https://hg.adblockplus.org/sitescripts/rev/1e894f5d1b82