Opened on 02/14/2017 at 01:35:42 PM

Closed on 04/11/2017 at 08:19:10 PM

Last modified on 09/14/2017 at 09:03:01 AM

#4894 closed change (fixed)

Block requests from mysterious adblocker that concentrates traffic on 21:00 UTC

Reported by: ferris Assignee: paco
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: matze, vickyyu, Kirill, paco, trev Blocked By:
Blocking: Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by ferris)

Every day at 21:00 UTC, our servers suffer an onslaught of traffic coming from some half a million clients downloading the chinese easylist concurrently. They provide no request paramaters or user-agent that would allow us to communicate with or advise the developers of this client.

What to do

As these clients are practically abusing the servers by having all clients fetch concurrently, we have no good choice but to block their traffic. The heuristics for this kind of traffic is:

  • The user agent string is empty
  • There are no request parameters
  • The requested file is easylistchina+easylist.txt
  • No referrer page
  • No language preference

After this has been rolled out on the filter-servers, we expect the daily traffic spike to disappear.

Attachments (0)

Change History (8)

comment:1 Changed on 02/15/2017 at 07:25:01 AM by matze

  • Description modified (diff)
  • Priority changed from Unknown to P2
  • Ready set

Applying the resulting patch-set as a hotfix should suffice for now. Note, however, that "abuse" is not necessarily the correct label for these clients behavior - our servers should be (and are) capable of handling those spikes, and we never published conditions in any form. So when the measures described above are applied and if someone or something pops up, at least we should then be able to provide information on what values are required, and why.

comment:2 Changed on 02/15/2017 at 06:25:44 PM by ferris

  • Description modified (diff)
  • Owner set to paco

We've deployed a hotfix that results in 400 for requests without user agent. A finer match is being researched. A proper patch will follow later.

comment:3 Changed on 04/10/2017 at 08:28:57 PM by paco

  • Review URL(s) modified (diff)

comment:4 Changed on 04/11/2017 at 08:18:13 PM by abpbot

A commit referencing this issue has landed:
Issue 4894 - Mitigate traffic spikes with unknown user-agent

comment:5 Changed on 04/11/2017 at 08:19:10 PM by paco

  • Resolution set to fixed
  • Status changed from new to closed

comment:6 Changed on 07/28/2017 at 03:57:06 PM by abpbot

A commit referencing this issue has landed:
Issue 4894 - Extend blocking pattern

comment:7 Changed on 09/13/2017 at 11:17:21 AM by trev

  • Cc trev added

Why was this added to our default Nginx configuration rather than the configuration for filter servers only?

comment:8 Changed on 09/14/2017 at 09:03:01 AM by ferris

Good question. We'll repair this when we do

Add Comment

Modify Ticket

Change Properties
as closed .
The resolution will be deleted. Next status will be 'reopened'.
to The owner will be changed from paco.
Note: See TracTickets for help on using tickets.