Opened on 02/14/2017 at 01:35:42 PM
Closed on 04/11/2017 at 08:19:10 PM
Last modified on 09/14/2017 at 09:03:01 AM
#4894 closed change (fixed)
Block requests from mysterious adblocker that concentrates traffic on 21:00 UTC
| Reported by: | ferris | Assignee: | paco |
|---|---|---|---|
| Priority: | P2 | Milestone: | |
| Module: | Infrastructure | Keywords: | |
| Cc: | matze, vickyyu, Kirill, paco, trev | Blocked By: | |
| Blocking: | Platform: | Unknown / Cross platform | |
| Ready: | yes | Confidential: | no |
| Tester: | Unknown | Verified working: | no |
| Review URL(s): | |||
Description (last modified by ferris)
Every day at 21:00 UTC, our servers suffer an onslaught of traffic coming from some half a million clients downloading the chinese easylist concurrently. They provide no request paramaters or user-agent that would allow us to communicate with or advise the developers of this client.
What to do
As these clients are practically abusing the servers by having all clients fetch concurrently, we have no good choice but to block their traffic. The heuristics for this kind of traffic is:
- The user agent string is empty
- There are no request parameters
- The requested file is easylistchina+easylist.txt
- No referrer page
- No language preference
After this has been rolled out on the filter-servers, we expect the daily traffic spike to disappear.
Attachments (0)
Change History (8)
comment:1 Changed on 02/15/2017 at 07:25:01 AM by matze
- Description modified (diff)
- Priority changed from Unknown to P2
- Ready set
comment:2 Changed on 02/15/2017 at 06:25:44 PM by ferris
- Description modified (diff)
- Owner set to paco
We've deployed a hotfix that results in 400 for requests without user agent. A finer match is being researched. A proper patch will follow later.
comment:4 Changed on 04/11/2017 at 08:18:13 PM by abpbot
A commit referencing this issue has landed:
Issue 4894 - Mitigate traffic spikes with unknown user-agent
comment:5 Changed on 04/11/2017 at 08:19:10 PM by paco
- Resolution set to fixed
- Status changed from new to closed
comment:6 Changed on 07/28/2017 at 03:57:06 PM by abpbot
A commit referencing this issue has landed:
Issue 4894 - Extend blocking pattern
comment:7 Changed on 09/13/2017 at 11:17:21 AM by trev
- Cc trev added
Why was this added to our default Nginx configuration rather than the configuration for filter servers only?
comment:8 Changed on 09/14/2017 at 09:03:01 AM by ferris
Good question. We'll repair this when we do http://hub.eyeo.com/issues/652
Applying the resulting patch-set as a hotfix should suffice for now. Note, however, that "abuse" is not necessarily the correct label for these clients behavior - our servers should be (and are) capable of handling those spikes, and we never published conditions in any form. So when the measures described above are applied and if someone or something pops up, at least we should then be able to provide information on what values are required, and why.