Investigate usage of a local VPN

[fhd]

Background

So far, we assumed it wasn't possible to run a local VPN on Android. However, ​AdGuard seems to be doing just that. We should definitely investigate this, since this would allow us to block ads over all connections and to configure it all automatically.

What to change

At first we should investigate what we need to do to support this, and whether there are any downsides compared to the local HTTP proxy.

[fhd]

Removed the instructions from the description: It's not the right place for this, the process should be clearer than that. And we still don't agree on how it should work anyway.

[hexene]

I've been looking into this for some time now, here are my findings:

Android's VpnService exposes a TUN interface, so the packets we intercept will look something like below:
|IP|UDP|DNS|, |IP|TCP|HTTP|, etc.

Ideally, we would be forwarding these packets as is, but this requires raw sockets, which is not accessible without root. (Incidentally, Java doesn't have a raw sockets API)
To forward packets without root access, we need to carry out layer translation (internet to application layer).

For UDP packets, this is as simple as stripping the internet (IP) and transport (UDP) layer headers, and forwarding the application layer payload over a datagram socket. Once a response is received from the remote server, we only have to add the headers back and forward to the VPN interface.

However, to forward TCP packets, we need to manage the TCP lifecycle ourselves (TCP handshake, packet acknowledgement, etc.). This requires a partial userspace TCP/IP stack.

I've built a POC app with a custom TCP/IP stack (supports only IPv4 for now) here: ​https://github.com/hexene/LocalVPN
There are still a few niggles to iron out, but it works for the most part.

To block ads on top of this, we would need to accumulate TCP packets constituting an HTTP request, examine the payload, and process appropriately.

Downsides:
-VpnService supported only ICS onwards.
-Quite expensive on the CPU, given that we need to process each IP packet (incoming & outgoing) individually.
-Not possible to forward ICMP packets (since ICMP operates at the internet layer, raw sockets are requried), so ping will be broken.
-Starting/stopping the service is allowed only through a system-dialog (VpnDialog), so we cannot auto-start.

[larryp]

I have code that does this now on non-rooted devices, and it works on wireless and 3/4G with browser websites and apps. Yes, it does layer translation on UDP and TCP packets, and yes, there is a performance impact because of this. I find it to be more of a memory impact because of what was said above about maintaining the TCP lifecycle and the 6 MB+ clump of drawables that Android force feeds to every application, but I am working on that now.

My app blocks ads in DNS and never makes the TCP call, but also can do RDNS on IP addresses for those sites that request ads with hardwired IP addresses, and it can be supplied ranges of IP addresses to just block without doing RDNS. Some websites have a problem with this because some of them relentlessly ask for DNS on a blocked URL and won't present content until they can get to that blocked URL. I'm thinking about either scrapping blocking by DNS or letting it get to the blocked URL and then just acking the payload from the blocked site and presenting it "nothing" and seeing if that works.

Most apps work pretty well unless they are media heavy, and Android 5.0+, the VPN interface allows apps to bypass the VPN, and I have some media heavy apps wired to do that like Youtube.

In Android 5.0+, what I have should work pretty well with a proxy that blocks ads in websites if the proxy and web browsers are allowed to bypass the VPN, and if it is a service within the proxy, then there only needs to be one Android clump of drawables instead of 2.

[lohiram]

spam