Opened on 08/10/2015 at 03:15:10 PM
Closed on 09/18/2015 at 07:26:00 AM
#2880 closed change (fixed)
Decomission single-core filter servers
| Reported by: | matze | Assignee: | matze |
|---|---|---|---|
| Priority: | P2 | Milestone: | |
| Module: | Infrastructure | Keywords: | |
| Cc: | fhd, fred | Blocked By: | #2881, #2907 |
| Blocking: | #2277 | Platform: | Unknown / Cross platform |
| Ready: | yes | Confidential: | no |
| Tester: | Unknown | Verified working: | no |
| Review URL(s): | |||
Description (last modified by matze)
Without major enhancements to our balancing mechanism, our old single-core machines cannot qualify as filter-servers any more. The new notification handlers (#2277), for example, drive the CPU load up to values around 9, which is obviously way too much.
Thus the following servers have been declared obsolete and are to be decommissioned:
- filter1.adblockplus.org
- filter2.adblockplus.org
- filter3.adblockplus.org
- filter4.adblockplus.org
- filter5.adblockplus.org
- server5.adblockplus.org
- server6.adblockplus.org
- server7.adblockplus.org
- server12.adblockplus.org
- server15.adblockplus.org
- server19.adblockplus.org
While they can be removed from balancing as soon as their replacements have been setup, @trev and I agreed on keeping them online (and in the hosts.yaml) for a few more days, so any log processing can finish regularly.
Attachments (0)
Change History (9)
comment:1 Changed on 08/10/2015 at 03:16:32 PM by matze
- Owner set to matze
- Type changed from defect to change
comment:2 Changed on 08/10/2015 at 03:19:19 PM by matze
- Blocked By 2881 added
comment:4 Changed on 08/11/2015 at 03:59:01 PM by matze
- Description modified (diff)
- Resolution set to fixed
- Status changed from new to closed
Just learned that server12 is also a single-core filter server. It slipped through earlier because it's reverse-DNS record is not correct (stating it's server10, which would not even be a filter-server). It has been removed from balancing as well.
comment:5 Changed on 08/15/2015 at 01:22:03 PM by matze
- Resolution fixed deleted
- Status changed from closed to reopened
Decommission has been scheduled for August 28th, the earliest date where all of them can be released at once. Wiping the hard-drives has been scheduled for August 27th.
comment:6 Changed on 08/15/2015 at 01:40:38 PM by matze
- Review URL(s) modified (diff)
- Status changed from reopened to reviewing
Removal from hosts configuration has been prepared, awaiting code-review before roll-out.
Also; the stand-by DNS records have been removed (@Kirill).
comment:7 Changed on 08/15/2015 at 02:27:15 PM by matze
- Blocked By 2907 added
comment:8 Changed on 08/17/2015 at 11:06:54 AM by matze
FYI: In order to minimize load on filtermaster1, I've manually disabled the EasyList synchronization on all of the servers that are to be decommissioned.
comment:9 Changed on 09/18/2015 at 07:26:00 AM by matze
- Cc trev Kirill removed
- Resolution set to fixed
- Status changed from reviewing to closed
Decommissioning has finished a few weeks ago. Thus far the remaining cluster seems to be able to bare the load just fine.
The servers have been removed from balancing. They will idle now until the end of the week, when they become decommissioned.