Opened 4 years ago

Closed 4 years ago

#3016 closed defect (fixed)

Change Nginx Apt::Source in Puppet

Reported by: matze Assignee: matze
Priority: P3 Milestone:
Module: Infrastructure Keywords:
Cc: fhd, fred Blocked By: #3019, #3053, #3062, #3083
Blocking: #3011 Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

https://codereview.adblockplus.org/29326134
https://codereview.adblockplus.org/29326229
https://codereview.adblockplus.org/29328168

Description

Because we try to always use an up-to-date Nginx version whilst still running Ubuntu Precise, we have setup a custom Apt::Source['nginx'] which is linked to http://nginx.org/packages/ubuntu.

Unfortunately, that uplink does not provide a setup built --with-geoip_module. The same vendor (the Nginx team), however, operates a package archive (PPA) at Launchpad, which does not only include recent builds with the aforementioned flag, but is also comparable to the current one regarding reliability.

One should also note that the current uplink is tracking the most recent version only. Thus we've had half a dozen issues in the past where our Provisioning was broken due to a version update at provider side. When switching to the PPA, versions remain avaialble. Which means we can then finally decide when to migrate. And do so without bypassing the obligatorily tests due to release pressure.

Change History (12)

comment:1 Changed 4 years ago by matze

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:2 Changed 4 years ago by matze

  • Blocked By 3019 added

comment:3 Changed 4 years ago by matze

After pushing the change-set for issues 3016 and 3019, I just tried to migrate https://testpages.adblockplus.org/ (which does not include any of the new requirements but should remain as-is regarding behavior).

Despite yesterday's excessive testing efforts, the migration did not work in fully automated fashion via Puppet. It seems like there are some more resource dependencies not configured properly and thus fail under virtually random conditions.

Debugging that one is a bit tricky, yet I do not consider it a show stopper. One can easily perform the failing steps by hand upfront or, when they fail, purge Nginx and install the proper version manually before running another provision to fix possible inconsistencies - as I did with the testpages just now.

Thus I will continue to update the web::server nodes (which are trivial enough to ensure that this will not break anything else), in the hope that while doing so I get enough insight to fix this issue permanently in the context of this very ticket here.

comment:4 Changed 4 years ago by matze

  • Review URL(s) modified (diff)

The new patch-set (on top of the one reviewed and pushed already) is the only one I could come up with that seems to always work, without additional, manual intervention. While it's surely not beautiful, it is also not meant to last forever.

The following boxes have been provisioned with manual intervention:

The following provisioned in regular fashion with the patch-set developed in that process:

The remaining web::server boxes are to be provisioned tomorrow. The more complex setups, however, won't become updated until the review has finished and the patch-set has proven to work on the more trivial ones.

Note, however, that the aforementioned new patch-set is currently applied on the Puppet master.

comment:5 Changed 4 years ago by matze

After pushing the new patch-set and cleaning up on the Puppet master, the following web::server boxes have been provisioned without any further issues:

Thus #3011 is not currently blocked by this ticket any more, and the patch-sets there can become included already.

Last edited 4 years ago by matze (previous) (diff)

comment:6 Changed 4 years ago by matze

More boxes have been provisioned in this context:

There've been a few network issues during the roll-out, but those have proven to be unrelated.

comment:7 Changed 4 years ago by matze

  • Blocked By 3053 added

comment:8 Changed 4 years ago by matze

  • Blocked By 3062 added

comment:9 Changed 4 years ago by matze

  • Blocked By 3083 added

comment:10 Changed 4 years ago by matze

  • Review URL(s) modified (diff)

comment:11 Changed 4 years ago by matze

By now almost all boxes have been migrated. Outstanding ones will follow this week.

Version 0, edited 4 years ago by matze (next)

comment:12 Changed 4 years ago by matze

  • Resolution set to fixed
  • Status changed from reviewing to closed
Note: See TracTickets for help on using tickets.