Changes between Initial Version and Version 6 of Ticket #3774


Ignore:
Timestamp:
03/11/2016 03:14:34 PM (4 years ago)
Author:
sebastian
Comment:

I've updated the issue description, in order to fallback to another mirror when the request fails. This seems to be a good first bug. @kvas, do you want to have a try?

Legend:

Unmodified
Added
Removed
Modified
  • Ticket #3774

    • Property Cc kvas sebastian palant added
    • Property Component changed from Infrastructure to Sitescripts
    • Property Owner matze deleted
    • Property Keywords goodfirstbug added
    • Property Type changed from defect to change
  • Ticket #3774 – Description

    initial v6  
    1 **From:** Cron Daemon <root@$FQDN> 
    2 **Date:** Thu, Mar 10, 2016 at 7:15 PM 
    3 **Subject:** Cron <rsync@$HOSTNAME> python -m sitescripts.subscriptions.bin.updateMalwareDomainsList 
    4 **To:** root@localhost, .... 
     1=== Background === 
    52 
    6 {{{ 
    7 Traceback (most recent call last): 
    8   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main 
    9     "__main__", fname, loader, pkg_name) 
    10   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code 
    11     exec code in run_globals 
    12   File "/opt/sitescripts/sitescripts/subscriptions/bin/updateMalwareDomainsList.py", line 40, in <module> 
    13     zip = zipfile.ZipFile(StringIO(data), 'r') 
    14   File "/usr/lib/python2.7/zipfile.py", line 714, in __init__ 
    15     self._GetContents() 
    16   File "/usr/lib/python2.7/zipfile.py", line 748, in _GetContents 
    17     self._RealGetContents() 
    18   File "/usr/lib/python2.7/zipfile.py", line 763, in _RealGetContents 
    19     raise BadZipfile, "File is not a zip file" 
    20 zipfile.BadZipfile: File is not a zip file 
    21 }}} 
     3The [https://hg.adblockplus.org/sitescripts/file/tip/sitescripts/subscriptions/bin/updateMalwareDomainsList.py script] that converts the [https://www.malwaredomainlist.com/ Malware Domains List] into an Adblock Plus filter list, currently hard-codes `mirror3.malwaredomains.com`. 
    224 
    23 **Puppet:** `Class['filtermaster']` / `Cron['update_malware']` 
     5As of now, this mirror blocks out requests sent with Python's `urllib` module's default user agent string, while other mirrors don't have that issue. 
     6 
     7Regardless of this particular issue, it would make sense to support a list of [http://www.malwaredomains.com/?page_id=29 mirrors], so that when one mirror fails we automatically fallback to another mirror from that list. 
     8 
     9=== What to change === 
     10When dowloading the Malware Domains List, pick a random mirror. When the request fails, fallback to another mirror in the list.