Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#4894 closed change (fixed)

Block requests from mysterious adblocker that concentrates traffic on 21:00 UTC

Reported by: ferris Assignee: paco
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: matze, vickyyu, Kirill, paco, trev Blocked By:
Blocking: Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29408777/

Description (last modified by ferris)

Every day at 21:00 UTC, our servers suffer an onslaught of traffic coming from some half a million clients downloading the chinese easylist concurrently. They provide no request paramaters or user-agent that would allow us to communicate with or advise the developers of this client.

What to do

As these clients are practically abusing the servers by having all clients fetch concurrently, we have no good choice but to block their traffic. The heuristics for this kind of traffic is:

  • The user agent string is empty
  • There are no request parameters
  • The requested file is easylistchina+easylist.txt
  • No referrer page
  • No language preference

After this has been rolled out on the filter-servers, we expect the daily traffic spike to disappear.

Change History (8)

comment:1 Changed 3 years ago by matze

  • Description modified (diff)
  • Priority changed from Unknown to P2
  • Ready set

Applying the resulting patch-set as a hotfix should suffice for now. Note, however, that "abuse" is not necessarily the correct label for these clients behavior - our servers should be (and are) capable of handling those spikes, and we never published conditions in any form. So when the measures described above are applied and if someone or something pops up, at least we should then be able to provide information on what values are required, and why.

comment:2 Changed 3 years ago by ferris

  • Description modified (diff)
  • Owner set to paco

We've deployed a hotfix that results in 400 for requests without user agent. A finer match is being researched. A proper patch will follow later.

comment:3 Changed 3 years ago by paco

  • Review URL(s) modified (diff)

comment:4 Changed 3 years ago by abpbot

A commit referencing this issue has landed:
Issue 4894 - Mitigate traffic spikes with unknown user-agent

comment:5 Changed 3 years ago by paco

  • Resolution set to fixed
  • Status changed from new to closed

comment:6 Changed 2 years ago by abpbot

A commit referencing this issue has landed:
Issue 4894 - Extend blocking pattern

comment:7 Changed 2 years ago by trev

  • Cc trev added

Why was this added to our default Nginx configuration rather than the configuration for filter servers only?

comment:8 Changed 2 years ago by ferris

Good question. We'll repair this when we do http://hub.eyeo.com/issues/652

Note: See TracTickets for help on using tickets.