Opened on 08/12/2014 at 02:46:40 PM

Last modified on 12/21/2017 at 11:29:03 AM

#1200 new change

Create an APT mirror for internal and production use

Reported by: matze Assignee: matze
Priority: P4 Milestone:
Module: Infrastructure Keywords: infrastructure apt
Cc: fhd Blocked By:
Blocking: Platform: Unknown
Ready: yes Confidential: no
Tester: Verified working: no
Review URL(s):

Description (last modified by fhd)


When provisioning either development boxes or the production system, we always run into the risk of external dependencies being currently unavailable or, in a worst-case scenario, being gone entirely. Especially APT, the Debian/Ubuntu package manager, is something that provides a LOT of dependencies for us (and in fact we have had known issues with it just recently) but should be rather easy to mirror.


  1. Determine possible APT mirroring software we could use. Since there's plenty, this probably may require a more detailed set of requirements in advance, but getting an overview should be faily easy.
  2. Integrate the mirror setup with our infrastructure (create Puppet manifests for it).
  3. Migrate at least other modude requiring APT packages, for demonstration purpose.
  4. Close this ticket but continue migrating modules (e.g. in additional tickets or while working on them anyway).

Note that 3. might not be applicable; depending on the software we choose in 1. and the setup we create in 2., it may not support partial migration (at least not without explicit apt-preferences on each client), in which case a full-grown, direct migration may become a better option.

Attachments (0)

Change History (10)

comment:1 Changed on 08/12/2014 at 02:48:13 PM by matze

  • Description modified (diff)

comment:2 Changed on 08/12/2014 at 02:49:54 PM by trev

  • Priority changed from Unknown to P4
  • Ready set
  • Summary changed from Create an API mirror for internal and production use to Create an APT mirror for internal and production use

comment:3 Changed on 08/12/2014 at 02:51:16 PM by trev

  • Description modified (diff)

comment:4 Changed on 08/12/2014 at 02:53:27 PM by fhd

  • Description modified (diff)
  • Priority changed from P4 to P3
  • Ready unset
  • Summary changed from Create an APT mirror for internal and production use to Create an API mirror for internal and production use

I think this would be most useful if we actually control which packages/versions get into the mirror. Like this:

  1. We have only one APT source on all nodes - our mirror
  2. We only mirror packages we actually use, only in the versions we actually use
  3. We regularly do an apt upgrade on all nodes - which will be pretty safe then (there's apt-cron for this e.g.)

That'll make e.g. security update deployments easier. Also, it'll allow us to pin versions, something we hardly do at all so far, and which is a bit messy with both APT pinning and Puppet.

What do you guys think of that?

comment:5 Changed on 08/12/2014 at 02:54:37 PM by fhd

  • Description modified (diff)
  • Priority changed from P3 to P4

Sorry, undid your changes thanks to Trac...

comment:6 Changed on 08/12/2014 at 02:55:01 PM by fhd

  • Ready set
  • Summary changed from Create an API mirror for internal and production use to Create an APT mirror for internal and production use

comment:7 Changed on 08/12/2014 at 07:12:32 PM by matze

Agreed, although it might become a bit tricky do determine which packages are actually in use. Sure, APT can tell us that. But I doubt that there's only *provisioned* packages on the production servers, so they probably need a bunch of other packages in addition to those we'll find if we dump that information on provisioned development boxes only..

comment:8 Changed on 08/13/2014 at 05:48:51 AM by fhd

True, if it's too much of a hassle it's not a good option. But I have no idea how/if we can do that in our own APT mirror.

Even if it's doable, I'm not fully convinced it'll be better than pinning/upgrading via Puppet. We're hardly doing any pinning so far - I suppose it's a bit of a different topic.

comment:9 Changed on 08/13/2014 at 06:04:24 AM by matze

Fair enough, I agree. It can be considered a different topic than mirroring.

Version pinning is something I associate with requirements, specification, code, dependencies and build-processes (in short: software), whilst mirroring is something motivated by reliability issues and thus operational.

Therefore, I consider pinning via Puppet the better option. It's additional documentation in-source (*.pp), where any changes are then tracked like any other part of the software under version control. It's also independent from the mirror itself, where one can then focus on the operational requirements (e.g. availability). And don't forget that the mirror setup itself is also tracked (*.pp again..).

comment:10 Changed on 12/21/2017 at 11:29:03 AM by fhd

  • Cc trev removed

Add Comment

Modify Ticket

Change Properties
as new .
as The resolution will be set. Next status will be 'closed'.
to The owner will be changed from matze.
Next status will be 'reviewing'.
Note: See TracTickets for help on using tickets.