Opened 4 years ago

Last modified 11 months ago

#1200 new change

Create an APT mirror for internal and production use

Reported by: matze Assignee: matze
Priority: P4 Milestone:
Module: Infrastructure Keywords: infrastructure apt
Cc: fhd Blocked By:
Blocking: Platform: Unknown
Ready: yes Confidential: no
Tester: Verified working: no
Review URL(s):

Description (last modified by fhd)

Description

When provisioning either development boxes or the production system, we always run into the risk of external dependencies being currently unavailable or, in a worst-case scenario, being gone entirely. Especially APT, the Debian/Ubuntu package manager, is something that provides a LOT of dependencies for us (and in fact we have had known issues with it just recently) but should be rather easy to mirror.

Tasks

  1. Determine possible APT mirroring software we could use. Since there's plenty, this probably may require a more detailed set of requirements in advance, but getting an overview should be faily easy.
  2. Integrate the mirror setup with our infrastructure (create Puppet manifests for it).
  3. Migrate at least other modude requiring APT packages, for demonstration purpose.
  4. Close this ticket but continue migrating modules (e.g. in additional tickets or while working on them anyway).

Note that 3. might not be applicable; depending on the software we choose in 1. and the setup we create in 2., it may not support partial migration (at least not without explicit apt-preferences on each client), in which case a full-grown, direct migration may become a better option.

Change History (10)

comment:1 Changed 4 years ago by matze

  • Description modified (diff)

comment:2 Changed 4 years ago by trev

  • Priority changed from Unknown to P4
  • Ready set
  • Summary changed from Create an API mirror for internal and production use to Create an APT mirror for internal and production use

comment:3 Changed 4 years ago by trev

  • Description modified (diff)

comment:4 Changed 4 years ago by fhd

  • Description modified (diff)
  • Priority changed from P4 to P3
  • Ready unset
  • Summary changed from Create an APT mirror for internal and production use to Create an API mirror for internal and production use

I think this would be most useful if we actually control which packages/versions get into the mirror. Like this:

  1. We have only one APT source on all nodes - our mirror
  2. We only mirror packages we actually use, only in the versions we actually use
  3. We regularly do an apt upgrade on all nodes - which will be pretty safe then (there's apt-cron for this e.g.)

That'll make e.g. security update deployments easier. Also, it'll allow us to pin versions, something we hardly do at all so far, and which is a bit messy with both APT pinning and Puppet.

What do you guys think of that?

comment:5 Changed 4 years ago by fhd

  • Description modified (diff)
  • Priority changed from P3 to P4

Sorry, undid your changes thanks to Trac...

comment:6 Changed 4 years ago by fhd

  • Ready set
  • Summary changed from Create an API mirror for internal and production use to Create an APT mirror for internal and production use

comment:7 Changed 4 years ago by matze

Agreed, although it might become a bit tricky do determine which packages are actually in use. Sure, APT can tell us that. But I doubt that there's only *provisioned* packages on the production servers, so they probably need a bunch of other packages in addition to those we'll find if we dump that information on provisioned development boxes only..

comment:8 Changed 4 years ago by fhd

True, if it's too much of a hassle it's not a good option. But I have no idea how/if we can do that in our own APT mirror.

Even if it's doable, I'm not fully convinced it'll be better than pinning/upgrading via Puppet. We're hardly doing any pinning so far - I suppose it's a bit of a different topic.

comment:9 Changed 4 years ago by matze

Fair enough, I agree. It can be considered a different topic than mirroring.

Version pinning is something I associate with requirements, specification, code, dependencies and build-processes (in short: software), whilst mirroring is something motivated by reliability issues and thus operational.

Therefore, I consider pinning via Puppet the better option. It's additional documentation in-source (*.pp), where any changes are then tracked like any other part of the software under version control. It's also independent from the mirror itself, where one can then focus on the operational requirements (e.g. availability). And don't forget that the mirror setup itself is also tracked (*.pp again..).

comment:10 Changed 11 months ago by fhd

  • Cc trev removed
Note: See TracTickets for help on using tickets.