Opened 5 years ago

Closed 5 years ago

Last modified 4 years ago

#2588 closed defect (fixed)

Provide correct key types in /etc/ssh/ssh_known_hosts

Reported by: matze Assignee: matze
Priority: P3 Milestone:
Module: Infrastructure Keywords:
Cc: fred, fhd, trev Blocked By: #2585
Blocking: #2589, #2591, #3139 Platform: Unknown
Ready: yes Confidential: no
Tester: Verified working: no
Review URL(s):

http://codereview.adblockplus.org/4835082309206016
http://codereview.adblockplus.org/6256043248058368

Description

Although we use ecdsa-sha2-nistp256 keys exclusively (those are the ones we put in hosts.yaml, for example), the /etc/ssh/ssh_known_hosts file generated by Puppet lists all of them as ssh-rsa:

$ grep -v '^\s*#' /etc/ssh/ssh_known_hosts | cut -d' ' -f2 | uniq -c
     46 ssh-rsa

This also explains the output on /dev/stderr generated by both the ssh(1) and scp(1) commands, e.g.:

# ssh -o UserKnownHostsFile=/tmp/myknownhostsfile filter2.adblockplus.org  
key_read: type mismatch: encoding error
key_read: type mismatch: encoding error
key_read: type mismatch: encoding error
The authenticity of host 'filter2.adblockplus.org (213.239.212.163)' can't be established.
ECDSA key fingerprint is 59:ef:2d:3c:95:a0:3a:db:df:ef:ce:09:fc:9c:27:4f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'filter2.adblockplus.org,213.239.212.163' (ECDSA) to the list of known hosts.
Permission denied (publickey).

The temporary host file afterwards:

# cat /tmp/myknownhostsfile 
filter2.adblockplus.org,213.239.212.163 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMoFuSVZoEyWWK191/lym1a6ofKKNOvLwiUic2ZMtX9cJ6+oJ5SldW+EtX9X9xNkRl447JtnyVslQWt2Am1Dcs8=

The corresponding record from the system's host file:

# grep 213.239.212.163 /etc/ssh/ssh_known_hosts 
filter2.adblockplus.org,213.239.212.163,2a01:4f8:a0:82e3::2 ssh-rsa AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMoFuSVZoEyWWK191/lym1a6ofKKNOvLwiUic2ZMtX9cJ6+oJ5SldW+EtX9X9xNkRl447JtnyVslQWt2Am1Dcs8=

Same name, same IP, same key, but different type. Altogether, this should be easy to fix. Although fetching and verifying each key may require some time, even with fancy shell-foo.

One question remains (and must be answered first, in order to avoid any side-effects): How could this have worked before, ever?

Change History (17)

comment:1 Changed 5 years ago by trev

How could this have worked before, ever?

Most likely because we still have the file in ~stats/.ssh - so the keys for any old servers can still be taken from there.

comment:2 Changed 5 years ago by matze

  • Ready set

Indeed, just saw it. Thank's anyway ;-)

comment:3 Changed 5 years ago by matze

  • Review URL(s) modified (diff)
  • Status changed from new to reviewing

comment:4 Changed 5 years ago by matze

Note that, although I've published a patch-set for review, we cannot simply roll this out: Our servers use Puppet 2.7.11, which supports only ssh-dsa and ssh-rsa for sshkey {...: type => ...}:
https://docs.puppetlabs.com/references/2.7.11/type.html#sshkey

The first version supporting the desired type is 2.7.14. The most recent from the this major version is 2.7.24. While that would be just a minor version upgrade we actually can perform, it's still a way wider impact than expected, especially in testing effort and even if we rely on provisioning-only tests.

An alternative is applying another workaround; running s/ssh-rsa\(\s\+AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBB\)/ecdsa-sha2-nistp256\1/ after each Sshkey<| |>.

I prefer the upgrade (although that'll require a few manual steps per server, updating install_precise.py won't be enough) instead of applying a hack similar to the aforementioned one. But before I spend more time looking into that option: Are there any veto's from either @fhd or @trev?

comment:5 Changed 5 years ago by matze

By the way; if you're wondering why the heck the command even cares about the option, especially because it...

  1. is just supposed to write the value to that file (even the docs say "This attribute represents concrete state on the target system") and
  2. hasn't even checked that the key-type and -prefix match up before, so it's obviously not even using that value any further --

It's just because of their own implementation magic, using descriptor functions to auto-generate documentation and, implicit, validation as well. See /usr/lib/ruby/$version/puppet/type/sshkey.rb, just the first one or two dozen lines, any version between 2.7.0 and 3.8.0.

comment:6 Changed 5 years ago by matze

Oh, I almost forgot another option: Switching to use the RSA or DSA host keys. I've verified that all hosts have them already, and verifying that they work can easily be done. While that one is as clean as a Puppet upgrade, it's probably way safer and faster. Well, it's still somehow a step back. But in this case it might be the better choice. Any thoughts?

comment:7 Changed 5 years ago by trev

Using other key types requires changing SSH configuration.

Either way, there is exactly one server affected by this. I suggest upgrading Puppet there and applying the patch.

comment:8 Changed 5 years ago by matze

That does not work because the patch affects base, and thus every server (although the resources are not realized on others).

comment:9 Changed 5 years ago by matze

@trev "Using other key types requires changing SSH configuration." -- Where? Server-side everything works without any changes, one can register any of the ECDSA, DSA and RSA key with the own hosts file and the connection is established perfectly well.

comment:10 Changed 5 years ago by trev

Then using RSA keys sounds like a better idea. Upgrading Puppet requires getting it from a different source, this is going to be trouble.

comment:11 Changed 5 years ago by matze

Very well, I'll update the keys in hosts.yaml then.

Note, however, that our Puppet source (see hiera/install_precise.py) actually provides Puppet 2.7.24, and it's even compatible with the puppet-hiera package. Otherwise I would not have been able to test the (now obsolete) patch-set and suggest the upgrade that quickly. The now chosen solution to update the keys was just not on my radar before..

Furthermore, please note that I'll update the keys in modules/private-stub/hiera/hosts.yaml as well - they actually match the production ECDSA ones, although they should use the one from the Vagrant box.

comment:12 Changed 5 years ago by matze

  • Review URL(s) modified (diff)

comment:13 Changed 5 years ago by matze

  • Review URL(s) modified (diff)

Uploaded 2nd patch set for production environment --

comment:14 Changed 5 years ago by matze

  • Blocking 2589 added

comment:15 Changed 5 years ago by matze

  • Blocking 2591 added

comment:16 Changed 5 years ago by matze

  • Resolution set to fixed
  • Status changed from reviewing to closed

comment:17 Changed 4 years ago by matze

  • Blocking 3139 added
Note: See TracTickets for help on using tickets.