Refactor hiera('servers') into Puppet class adblockplus::host

[matze]

The current approach to feed Puppet information about our servers has proven to be quite impractical. Introduced together with the migration to our YAML-based, combined Hiera/ENC solution last year, the use of the hiera("servers") hash setup in base.yaml is quite repetitive, see i.e.:

• ​modules/base/manifests/init.pp at 0c1a7f56317d, line 51
• ​modules/nagios/manifests/server.pp at 0c1a7f56317d, line 163
• ​modules/statsmaster/templates/sitescripts.ini.erb at 0c1a7f56317d

In addition, this approach does not align well with the Puppet language, especially considering the limitations of the 2.7* version family, but that is an issue for another day.

Background

While managing these in a named hash, available as e.g. hiera("servers"), seems to be a simple way to access the desired information, the common approach using create_resources() to process the returned hash (one of the few tools available in Puppet 2.7) shows various disadvantages:

• Each use requires an individual Puppet type to be created, and passed on to the create_resources() function, along with the $servers hash. The code and documentation overhead is relatively large, though the results are usually quite practical.
• The Puppet type must recognize every parameter, even those which are irrelevant to its domain, otherwise the resource creation will result in a Puppet error.
• The hash key or $title of any hash item has a special meaning as a ​Puppet namevar, but at the same type any "name" key has no effect when create_resources() is invoked. Instead, the default behavior using $title is applied, which leads to inconsistent results with other approaches (see below).
• The $dns parameter is optional, by default accumulated by $name and some $domain or $authority value. This is never documented, but implemented in various manifests, templates, and scripts (see below).
• The $domain or $authority value is not always taken from the same source.

There are other code fragments examining the servers hash, i.e.:

• Puppet ​ERB templates
• Possible hacks with ​inline templates
• Ruby functions (not used yet)

For now, the most problematic, most often occurring use-case is a setup where host M needs to recognize each host C[x] of a list of clients C[0..n], like in the examples above, or any other master-slave relationship. More advanced examples include setups where any client C must recognize properties of the master, i.e. the ​TLS certificate of M (like ​Puppet itself, or #6 Bacula).

Todo

The improvement with the most effect for now seems to be the introduction of a conventional (meta-) resource any other code fragment may access. This allows for both keeping the current schema for now, whilst establishing a single official resource, featuring commonly accumulated properties:

• Introduce a Puppet type adblockplus::host aligned with base::explicit_host_record()
• Introduce parameter $adblockplus::hosts to setup hosts based on hiera_hash() and create_resources()
• Migrate either of the above workarounds to access the newly created resources instead
• Remove obsolete code fragments

Finally, we should refactor our puppet_node_classifier.rb script to not read the server information from hosts.yaml, but instead require each host to be present as an individual hosts/$HOSTNAME.yaml file. In practice most hosts require individual configuration anyway, i.e. for persistent encryption keys and monitoring quota, and maintaining host information would become more intuitive.

Status

This issue is considered rather urgent because a lot other issues would benefit from it, i.e.:

Ticket	Status	Resolution	Summary	Component	Owner
#6	new		Set up backups	Infrastructure	matze
#1462	closed	duplicate	Create a development certificate authority	Office-IT	matze
#1568	new		Create DNS nagios tests	Infrastructure
#3305	closed	incomplete	Introduce centralized log management	Infrastructure	matze
#3576	new		[meta] Improve Infrastructure Code-Base	Infrastructure
#3770	closed	duplicate	Generate Key-Pairs and Certificates via Puppet	Infrastructure	matze

Namely, each of the above would require some of the aforementioned workarounds or other hacks to get implemented. In the same turn, each of these code fragments would require re-implementation after this ticket has been addressed.

[matze]

The first patch-set includes the introduction of both class adblockplus::host and parameter $adblockplus::hosts.

[matze]

The former patch-set turned out to be impractical during the implementation of follow-up tickets. Based on the new experience I've created a new version, including examples on how to tackle the most re-occurring tasks, each of which has been proven to work well in further development.

Luckily, the first patch-set has never been pushed. The new one is in review now.

[matze]

There is now ​another patch-set for review, based on the ​new adblockplus::host implementation. I believe it to be a very good example on how the new approach tackles some if the issues discussed in the ticket description above.

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/infrastructure/rev/739d1ea52ea4

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/infrastructure/rev/b4550bec8247

[matze]

The last remaining direct invocation of hiera('servers') outside of class base is in ​the sitescripts.ini template for the stats-master. We can easily migrate the template magic using the newly introduced adblockplus::host::$role mechanism and conditional sitescripts::configfragment definitions, but there are a few other prerequisites which we need to address before:

• Migrate the node definition from the statsserver.pp manifest into a Hiera role
• Migrate the node definitions from the downloadserver.pp manifest into a Hiera role
• Migrate the node definition from the updateserver.pp manifest into a Hiera role
• Finally fix the global Host[] and Sshkey[] realization introduced by accident

The first and last points are covered by the recent patch-set. Note that the latter requires the removal of the resulting configuration fragments in production after it has landed, which could become quite some effort if not done in automated fashion (more later).

[matze]

Two more patch-sets, for aforementioned migration of downloadserver.pp and updateserver.pp into Hiera, respectively.

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/infrastructure/rev/e492c647d6ca

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/infrastructure/rev/c7f0974d6bb5

[abpbot]

A commit referencing this issue has landed:
​https://hg.adblockplus.org/infrastructure/rev/011568177058

[matze]

Last reference is in modules/statsmaster/templates/sitescripts.ini.erb. We probably need to translate the iteration through the hiera('...') hash there into a sitescripts::configfragment defined by adblockplus::host::statsmaster.

[abpbot]

A commit referencing this issue has landed:
​Issue 3638 - Generate statsmaster filter mirror configuration with concat