Opened 4 years ago

Closed 4 years ago

#2880 closed change (fixed)

Decomission single-core filter servers

Reported by: matze Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: fhd, fred Blocked By: #2881, #2907
Blocking: #2277 Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29323721/

Description (last modified by matze)

Without major enhancements to our balancing mechanism, our old single-core machines cannot qualify as filter-servers any more. The new notification handlers (#2277), for example, drive the CPU load up to values around 9, which is obviously way too much.

Thus the following servers have been declared obsolete and are to be decommissioned:

  • filter1.adblockplus.org
  • filter2.adblockplus.org
  • filter3.adblockplus.org
  • filter4.adblockplus.org
  • filter5.adblockplus.org
  • server5.adblockplus.org
  • server6.adblockplus.org
  • server7.adblockplus.org
  • server12.adblockplus.org
  • server15.adblockplus.org
  • server19.adblockplus.org

While they can be removed from balancing as soon as their replacements have been setup, @trev and I agreed on keeping them online (and in the hosts.yaml) for a few more days, so any log processing can finish regularly.

Change History (9)

comment:1 Changed 4 years ago by matze

  • Owner set to matze
  • Type changed from defect to change

comment:2 Changed 4 years ago by matze

  • Blocked By 2881 added

comment:3 Changed 4 years ago by matze

  • Description modified (diff)

The servers have been removed from balancing. They will idle now until the end of the week, when they become decommissioned.

comment:4 Changed 4 years ago by matze

  • Description modified (diff)
  • Resolution set to fixed
  • Status changed from new to closed

Just learned that server12 is also a single-core filter server. It slipped through earlier because it's reverse-DNS record is not correct (stating it's server10, which would not even be a filter-server). It has been removed from balancing as well.

comment:5 Changed 4 years ago by matze

  • Resolution fixed deleted
  • Status changed from closed to reopened

Decommission has been scheduled for August 28th, the earliest date where all of them can be released at once. Wiping the hard-drives has been scheduled for August 27th.

comment:6 Changed 4 years ago by matze

  • Review URL(s) modified (diff)
  • Status changed from reopened to reviewing

Removal from hosts configuration has been prepared, awaiting code-review before roll-out.
Also; the stand-by DNS records have been removed (@Kirill).

comment:7 Changed 4 years ago by matze

  • Blocked By 2907 added

comment:8 Changed 4 years ago by matze

FYI: In order to minimize load on filtermaster1, I've manually disabled the EasyList synchronization on all of the servers that are to be decommissioned.

comment:9 Changed 4 years ago by matze

  • Cc trev Kirill removed
  • Resolution set to fixed
  • Status changed from reviewing to closed

Decommissioning has finished a few weeks ago. Thus far the remaining cluster seems to be able to bare the load just fine.

Note: See TracTickets for help on using tickets.