Malware Domains list contains punycode

[sebastian]

How to reproduce

Check the list at ​https://easylist-downloads.adblockplus.org/malwaredomains_full.txt

(or run python -m sitescripts.subscriptions.bin.updateMalwareDomainsList)

Observed behaviour

The generated filter lists contains quite some domains encoded in punycode. However, these filters never match as domains get converted to unicode before matching the URL.

Expected behaviour

Domains should be decoded and encoded as UTF-8 in the generated Malware Domains filter list.

[trev]

Makes sense, given that the original list is meant to be applied on DNS level - DNS servers aren't aware of IDNs.

[sebastian]

​https://hg.adblockplus.org/sitescripts/rev/c91be96f1051

[sebastian]

I wonder why Adblock Plus expects IDN domains in filters to be in unicode in the first place. It seems that always when we have to interact with the rest of the world, we end up converting between punycode and unicode. Even at runtime we have to convert the URLs we get from extension APIs (at least on other platforms than Gecko) adding some overhead. So any reason why we don't expect punycode instead?

[trev]

Well, it would be possible to get raw Punycode on Gecko as well - but filters are written by humans, not computers. How do you expect filter maintainers to parse punycode in their lists?

[sebastian]

I see. Though this might be rather confusing sometimes, as Chrome for example only shows domains in unicode when you have enabled the respective language in the preferences and if it doesn't use characters that can be encoded in different ways. AFAIK, also Firefox isn't always showing unicode for domain names in the address bar. So it's inconsistent either way.

I am not going to argue about that. I merely want to point out that having our filters encode domains as unicode, while the rest of the world is using punycode, makes things quite complex, and is prone to bugs, as we have seen with this issue, and not too long ago also with Adblock Plus for Chrome.