Opened 4 years ago
Closed 4 years ago
#2880 closed change (fixed)
Decomission single-core filter servers
| Reported by: | matze | Assignee: | matze |
|---|---|---|---|
| Priority: | P2 | Milestone: | |
| Module: | Infrastructure | Keywords: | |
| Cc: | fhd, fred | Blocked By: | #2881, #2907 |
| Blocking: | #2277 | Platform: | Unknown / Cross platform |
| Ready: | yes | Confidential: | no |
| Tester: | Unknown | Verified working: | no |
| Review URL(s): | |||
Description (last modified by matze)
Without major enhancements to our balancing mechanism, our old single-core machines cannot qualify as filter-servers any more. The new notification handlers (#2277), for example, drive the CPU load up to values around 9, which is obviously way too much.
Thus the following servers have been declared obsolete and are to be decommissioned:
- filter1.adblockplus.org
- filter2.adblockplus.org
- filter3.adblockplus.org
- filter4.adblockplus.org
- filter5.adblockplus.org
- server5.adblockplus.org
- server6.adblockplus.org
- server7.adblockplus.org
- server12.adblockplus.org
- server15.adblockplus.org
- server19.adblockplus.org
While they can be removed from balancing as soon as their replacements have been setup, @trev and I agreed on keeping them online (and in the hosts.yaml) for a few more days, so any log processing can finish regularly.
Change History (9)
comment:1 Changed 4 years ago by matze
- Owner set to matze
- Type changed from defect to change
comment:2 Changed 4 years ago by matze
- Blocked By 2881 added
comment:3 Changed 4 years ago by matze
- Description modified (diff)
comment:4 Changed 4 years ago by matze
- Description modified (diff)
- Resolution set to fixed
- Status changed from new to closed
Just learned that server12 is also a single-core filter server. It slipped through earlier because it's reverse-DNS record is not correct (stating it's server10, which would not even be a filter-server). It has been removed from balancing as well.
comment:5 Changed 4 years ago by matze
- Resolution fixed deleted
- Status changed from closed to reopened
Decommission has been scheduled for August 28th, the earliest date where all of them can be released at once. Wiping the hard-drives has been scheduled for August 27th.
comment:6 Changed 4 years ago by matze
- Review URL(s) modified (diff)
- Status changed from reopened to reviewing
Removal from hosts configuration has been prepared, awaiting code-review before roll-out.
Also; the stand-by DNS records have been removed (@Kirill).
comment:7 Changed 4 years ago by matze
- Blocked By 2907 added
comment:8 Changed 4 years ago by matze
FYI: In order to minimize load on filtermaster1, I've manually disabled the EasyList synchronization on all of the servers that are to be decommissioned.
comment:9 Changed 4 years ago by matze
- Cc trev Kirill removed
- Resolution set to fixed
- Status changed from reviewing to closed
Decommissioning has finished a few weeks ago. Thus far the remaining cluster seems to be able to bare the load just fine.
The servers have been removed from balancing. They will idle now until the end of the week, when they become decommissioned.