Implement a better load balancing approach

[trev]

Background

Filter list downloads currently use DNS-based round robin load balancing. This doesn't distribute the load equally however. More importantly, it limits the number of servers that can be used due to limitations in size of DNS responses.

What to change

Move filter list downloads to their own domain and run our own DNS server. We could use Bind with geolocation capabilities for example (​http://code.google.com/p/bind-geoip/ or ​http://backreference.org/2010/02/01/geolocation-aware-dns-with-bind/). Even if all of our servers are currently in the same location, we could assign some servers to particular countries - so DNS servers in these countries wouldn't get the full list of servers in the response. Later we would move these servers to data centers close to the countries in question.

[trev]

I checked out the information on Tinydns (part of ​djbdns tools) and so far it looks more promising than Bind.

• Last release was in 2001. However, Ubuntu package (a fork named dbndns) contains some patches, in particular to ​add IPv6 support.
• In total there were three vulnerabilities discovered in djbdns, one affecting Tinydns but only exploitable by specially crafted subdomain records (all have been patched in Ubuntu). That's a much better security track record than Bind.
• The configuration files used by ​tinydns-data are unusual but still fairly straightforward. More importantly, generating this configuration file with Puppet should be trivial.
• There is support for conditional responses based on client locations, we could generate locations from GeoIP databases automatically. However, unlike Bind it works with IP prefixes rather than network masks. Given that we don't need exact regions that shouldn't be an issue however.
• Only UDP requests are supported by default. Answering to TCP requests requires setting up axfrdns tool (not sure whether we need it if we keep responses below 512 bytes).
• There is no DNSSEC support. It exists as a ​patch but this one isn't included in Ubuntu packages yet.

[trev]

The other option would be ​gdnsd which is a much more complex tool but with built-in advanced load balancing mechanisms. It also happens to be under active development. Downside here: it's not available for Ubuntu 12.04 but only Ubuntu 13.10.

[matze]

Another possible solution would be the migration to a more sophisticated DNS distribution method, e.g. ​Anycast.

Benefits:

• Implemented on the registrar side, thus applicable quite easy and fast
• No need for additional hard- or software
• The consequences of many attack and overload scenarios are limited to a global region
• Can become extended with custom setups (as mentioned in this ticket already) later

Caveats:

• Would probably increase the volume of our payments / our plan with the registrar
• Does not perform well when balancing an unexpected high load from one global region

[trev]

After reading up on Anycast, there seems to be two possibilities:

1. We use Anycast to have a single IP address for all our download servers. This is rather complicated to set up and would be quite an overkill at this point.
2. We use Anycast to distribute the tasks of our DNS server to many servers across the globe. These DNS servers could then serve different responses depending on their location.

I checked out a few providers of Anycast DNS hosting and it appears that they typically serve "regular" DNS responses (not dependent on client location). Only one provider mentioned GeoDNS and they didn't have any price tag on it which is a good indicator that the price will be very significant. In general, there doesn't seem to be a significant intersection between Anycast and GeoDNS, the latter being what we want.

Now we can look at hosted GeoDNS solutions of course, if there are some significant advantages. The clear disadvantage of this solution will be configuration (no longer possible via Puppet) - and the price should also be significant for our volumes.

[matze]

After excessive tests and evaluation in the last months, a pretty unproductive attempt to establish a better solution with our current provider, and some recent increasingly severe issues, we have decided to move our DNS zones to another service, which seems to not only solve current issues but also support most future tasks planned already or just seen as nice-to-haves. More details will follow.

[matze]

After we finally migrated our adblockplus.org zone to our new DNS provider, this ticket is considered obsolete: The more sophisticated, but still primarily DNS-based balancing features fulfill our requirements as intended, and future improvements (incl. Geo-based casting) are supported as well.