Opened 5 years ago

Last modified 4 years ago

#3638 closed change

Refactor hiera('servers') into Puppet class adblockplus::host — at Version 2

Reported by: matze Assignee: matze
Priority: P2 Milestone:
Module: Infrastructure Keywords:
Cc: fred, fhd Blocked By:
Blocking: #6, #3574, #3576 Platform: Unknown / Cross platform
Ready: yes Confidential: no
Tester: Unknown Verified working: no
Review URL(s):

Description (last modified by matze)

The current approach to feed Puppet information about our servers has proven to be quite impractical. Introduced together with the migration to our YAML-based, combined Hiera/ENC solution last year, the use of the hiera("servers") hash setup in base.yaml is quite repetitive, see i.e.:

In addition, this approach does not align well with the Puppet language, especially considering the limitations of the 2.7* version family, but that is an issue for another day.


While managing these in a named hash, available as e.g. hiera("servers"), seems to be a simple way to access the desired information, the common approach using create_resources() to process the returned hash (one of the few tools available in Puppet 2.7) shows various disadvantages:

  • Each use requires an individual Puppet type to be created, and passed on to the create_resources() function, along with the $servers hash. The code and documentation overhead is relatively large, though the results are usually quite practical.
  • The Puppet type must recognize every parameter, even those which are irrelevant to its domain, otherwise the resource creation will result in a Puppet error.
  • The hash key or $title of any hash item has a special meaning as a Puppet namevar, but at the same type any "name" key has no effect when create_resources() is invoked. Instead, the default behavior using $title is applied, which leads to inconsistent results with other approaches (see below).
  • The $dns parameter is optional, by default accumulated by $name and some $domain or $authority value. This is never documented, but implemented in various manifests, templates, and scripts (see below).
  • The $domain or $authority value is not always taken from the same source.

There are other code fragments examining the servers hash, i.e.:

  • Puppet ERB templates
  • Possible hacks with inline templates
  • Ruby functions (not used yet)

For now, the most problematic, most often occurring use-case is a setup where host M needs to recognize each host C[x] of a list of clients C[0..n], like in the examples above, or any other master-slave relationship. More advanced examples include setups where any client C must recognize properties of the master, i.e. the TLS certificate of M (like Puppet itself, or #6 Bacula).


The improvement with the most effect for now seems to be the introduction of a conventional (meta-) resource any other code fragment may access. This allows for both keeping the current schema for now, whilst establishing a single official resource, featuring commonly accumulated properties:

  • Introduce a Puppet type adblockplus::host aligned with base::explicit_host_record()
  • Introduce parameter $adblockplus::hosts to setup hosts based on hiera_hash() and create_resources()
  • Migrate either of the above workarounds to access the newly created resources instead
  • Remove obsolete code fragments

Finally, we should refactor our puppet_node_classifier.rb script to not read the server information from hosts.yaml, but instead require each host to be present as an individual hosts/$HOSTNAME.yaml file. In practice most hosts require individual configuration anyway, i.e. for persistent encryption keys and monitoring quota, and maintaining host information would become more intuitive.


This issue is considered rather urgent because a lot other issues would benefit from it, i.e.:

Namely, each of the above would require some of the aforementioned workarounds or other hacks to get implemented. In the same turn, each of these code fragments would require re-implementation after this ticket has been addressed.

Change History (2)

comment:1 Changed 5 years ago by matze

  • Blocking 6, 3574, 3576 added
  • Cc fred added

comment:2 Changed 5 years ago by matze

  • Description modified (diff)
  • Ready set
Note: See TracTickets for help on using tickets.