Provide correct key types in /etc/ssh/ssh_known_hosts

[matze]

Although we use ecdsa-sha2-nistp256 keys exclusively (those are the ones we put in hosts.yaml, for example), the /etc/ssh/ssh_known_hosts file generated by Puppet lists all of them as ssh-rsa:

$ grep -v '^\s*#' /etc/ssh/ssh_known_hosts | cut -d' ' -f2 | uniq -c
     46 ssh-rsa

This also explains the output on /dev/stderr generated by both the ssh(1) and scp(1) commands, e.g.:

# ssh -o UserKnownHostsFile=/tmp/myknownhostsfile filter2.adblockplus.org  
key_read: type mismatch: encoding error
key_read: type mismatch: encoding error
key_read: type mismatch: encoding error
The authenticity of host 'filter2.adblockplus.org (213.239.212.163)' can't be established.
ECDSA key fingerprint is 59:ef:2d:3c:95:a0:3a:db:df:ef:ce:09:fc:9c:27:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'filter2.adblockplus.org,213.239.212.163' (ECDSA) to the list of known hosts.
Permission denied (publickey).

The temporary host file afterwards:

# cat /tmp/myknownhostsfile 
filter2.adblockplus.org,213.239.212.163 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMoFuSVZoEyWWK191/lym1a6ofKKNOvLwiUic2ZMtX9cJ6+oJ5SldW+EtX9X9xNkRl447JtnyVslQWt2Am1Dcs8=

The corresponding record from the system's host file:

# grep 213.239.212.163 /etc/ssh/ssh_known_hosts 
filter2.adblockplus.org,213.239.212.163,2a01:4f8:a0:82e3::2 ssh-rsa AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMoFuSVZoEyWWK191/lym1a6ofKKNOvLwiUic2ZMtX9cJ6+oJ5SldW+EtX9X9xNkRl447JtnyVslQWt2Am1Dcs8=

Same name, same IP, same key, but different type. Altogether, this should be ​easy to fix. Although fetching and verifying each key may require some time, even with fancy shell-foo.

One question remains (and must be answered first, in order to avoid any side-effects): How could this have worked before, ever?

[trev]

How could this have worked before, ever?

Most likely because we still have the file in ~stats/.ssh - so the keys for any old servers can still be taken from there.

[matze]

Indeed, just saw it. Thank's anyway ;-)

[matze]

Note that, although I've published a patch-set for review, we cannot simply roll this out: Our servers use Puppet 2.7.11, which supports only ssh-dsa and ssh-rsa for sshkey {...: type => ...}:
​https://docs.puppetlabs.com/references/2.7.11/type.html#sshkey

The first version supporting the desired type is ​2.7.14. The most recent from the this major version is 2.7.24. While that would be just a minor version upgrade we actually can perform, it's still a way wider impact than expected, especially in testing effort and even if we rely on provisioning-only tests.

An alternative is applying another workaround; running s/ssh-rsa\(\s\+AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBB\)/ecdsa-sha2-nistp256\1/ after each Sshkey<| |>.

I prefer the upgrade (although that'll require a few manual steps per server, updating install_precise.py won't be enough) instead of applying a hack similar to the aforementioned one. But before I spend more time looking into that option: Are there any veto's from either @fhd or @trev?

[matze]

By the way; if you're wondering why the heck the command even cares about the option, especially because it...

1. is just supposed to write the value to that file (even the docs say "This attribute represents concrete state on the target system") and
2. hasn't even checked that the key-type and -prefix match up before, so it's obviously not even using that value any further --

It's just because of their own implementation magic, using descriptor functions to auto-generate documentation and, implicit, validation as well. See /usr/lib/ruby/$version/puppet/type/sshkey.rb, just the first one or two dozen lines, any version between 2.7.0 and 3.8.0.

[matze]

Oh, I almost forgot another option: Switching to use the RSA or DSA host keys. I've verified that all hosts have them already, and verifying that they work can easily be done. While that one is as clean as a Puppet upgrade, it's probably way safer and faster. Well, it's still somehow a step back. But in this case it might be the better choice. Any thoughts?

[trev]

Using other key types requires changing SSH configuration.

Either way, there is exactly one server affected by this. I suggest upgrading Puppet there and applying the patch.

[matze]

That does not work because the patch affects base, and thus every server (although the resources are not realized on others).

[matze]

@trev "Using other key types requires changing SSH configuration." -- Where? Server-side everything works without any changes, one can register any of the ECDSA, DSA and RSA key with the own hosts file and the connection is established perfectly well.

[trev]

Then using RSA keys sounds like a better idea. Upgrading Puppet requires getting it from a different source, this is going to be trouble.

[matze]

Very well, I'll update the keys in hosts.yaml then.

Note, however, that our Puppet source (see hiera/install_precise.py) actually provides Puppet 2.7.24, and it's even compatible with the puppet-hiera package. Otherwise I would not have been able to test the (now obsolete) patch-set and suggest the upgrade that quickly. The now chosen solution to update the keys was just not on my radar before..

Furthermore, please note that I'll update the keys in modules/private-stub/hiera/hosts.yaml as well - they actually match the production ECDSA ones, although they should use the one from the Vagrant box.

[matze]

Uploaded 2nd patch set for production environment --

[matze]

​https://hg.adblockplus.org/infrastructure/rev/0e71c3196d05