openssh @7.1p1 +ldns SSHFP DNSSEC validation fails

[scott-macports@…]

ssh performs SSHFP fingerprint lookup when VerifyHostKeyDNS is yes or ask.

The +ldns variant support validating the SSHFP with DNSSEC. SSHFP of host keys validated this way are implicitly trusted. If the resolver has DNSSEC validation, and sets the AD bit on the DNS response, ldns will mark the SSHFP valid without further work. If the resolver does not support DNSSEC, or is authoritative for the domain (eg internal DNS), then ldns must perform the validation locally (the AD bit is not set). For this, ldns needs a trust anchor.

I couldn't find anything in the ldns documentation, but the code in ldns/resolver.c looks for the keyword "anchor" in /etc/resolv.conf to locate a file containing a DS or DNSKEY RR, and loads it as a trust anchor. Without a trust anchor, local DNSSEC validation always fails.

Adding the "anchor" field to resolv.conf allows ldns validation to succeed. However, OSX re-creates resolv.conf from scutil's DNS config whenever the network changes... edits to /etc/resolv.conf are lost.

The only solution to get ldns validation to work on OSX is to add a trust anchor from a specific file (at least when it is needed for local validation)

I've created a patch to openssh to do this... I chose a "well-known" file location for the trust anchor, /etc/trusted-key.key (used by dig, freeipa, etc). The other option was drill's default: /etc/unbound/root.key, but that seemed rather "unbound specific."

I wasn't sure if I should look for the file in /etc directly, or ${prefix}/etc, so the attached patch attempts to load from both locations.

I was able to locate a bug filed upstream in 2013: ​https://bugzilla.mindrot.org/show_bug.cgi?id=2119 - but it's been stale for a long time, and could be worked around on platforms other than OSX by adding the "anchor" key to resolv.conf as discussed above (although again, that appears to be undocumented behavior on ldns's part).

I plan to update my patch for upstream with an option to "configure" the trusted-key.key filename, but a patch for that would be overkill for macports -- and as the fix is very OSX specific, I'm not sure if it'll be accepted upstream.

Relevant logs before the patch: (from ssh -vv <host>)

debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-ed25519 SHA256:a13QzFT87p7agd2MuagyOn7QfsegaUrvgB/M+smt1Zx
debug2: ldns: got 4 answers from DNS
debug2: ldns: trying to validate RRset
debug2: ldns: got 1 signature(s) (RRTYPE 46) from DNS
debug2: ldns: RRset validation failed: General LDNS error
debug1: found 4 insecure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug1: Host '[myhost]:8222' is known and matches the ED25519 host key.
debug1: Found key in /Users/test/.ssh/known_hosts:12
debug2: bits set: 2062/4096

After the patch:

debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-ed25519 SHA256:a13QzFT87p7agd2MuagyOn7QfsegaUrvgB/M+smt1Zx
debug2: ldns: got 4 answers from DNS
debug2: ldns: attempt to load trust anchor from file /etc/trusted-key.key
debug2: ldns: new anchor added to trust chain
debug2: ldns: attempt to load trust anchor from file /opt/local/etc/trusted-key.key
debug2: ldns: file not found
debug2: ldns: trying to validate RRset
debug2: ldns: got 1 signature(s) (RRTYPE 46) from DNS
debug2: ldns: RRset is signed with a valid key
debug1: found 4 secure fingerprints in DNS
debug1: matching host key fingerprint found in DNS
debug2: bits set: 2062/4096

note: this port does not have a maintainer

[scott-macports@…]

Updated patch to candidate for upstream

[raimue (Rainer Müller)]

I agree with the ​upstream answer. Was this ever reported to ldns upstream?

[raimue (Rainer Müller)]

Adding maintainer of ldns to Cc.

[Ionic (Mihai Moldovan)]

Yeah, I'll take a look at it eventually.

[scott-macports@…]

Replying to raimue@…:

I agree with the ​upstream answer. Was this ever reported to ldns upstream?

(sorry I didn't answer earlier, been busy with other projects :)

I looked into submitting a patch to ldns upstream, and created a working patch before realizing that I would break existing users of libldns (for example, drill) if it was applied.

Basically, ldns_resolver_new_frm_file() loads policy from a file (as the name suggests), one specifically documented to be /etc/resolv.conf if the filename parameter is NULL (which is how it's currently called from openssh).

drill uses this function if it's called without a server parameter, and the use of /etc/resolv.conf may be overridden with the -c parameter

If drill is run with a server, ldns_resolver_new() is used and no default anchors are loaded -- and then only if in chase/trace mode, and no other keys are yet loaded, it specifically loads /etc/unbound/root.key as it's default.

Basically, drill expects to the ldns interface to allow all loaded keys to be configurable at runtime, with no "magic policy" loading keys from files that it can't override. If I modified ldns_resolver_new_frm_file() to load keys from, say, /etc/trusted-key.key, then users of drill (and possibly other library clients) couldn't control the keys they were using from parameters, breaking use cases where specific keys need to be loaded (eg testing or private roots).

I could, of course, submit a new interface to ldns that explicitly loaded default keys from /etc/trusted-key.key (or some other default), but then openssh couldn't use that interface until the new library version was available on any particular distribution (as a new api, it can't really be backported as fix to distributions using the current or older libraries), preventing a working openssh + ldns on those systems for some time...

That leaves the final option, which is to use libldns the way drill does, and load policy from the files explicitly specified... and we're back to not being able to modify the default (/etc/resolv.conf) because OSX won't let us.

I agree that having a general purpose "just resolve my hostname" interface that loaded keys and policy from reasonable defaults might be a useful interface for ldns to have, but we'd need to get that added, and then we could use it when compiled against a recent enough version -- in the meantime, the only way we can get a working ldns library on OSX is to load the keys with ldns_resolver_push_dnssec_anchor()

I'll submit this analysis to the upstream openssh bug, but again OSX is hit harder than other platforms at the moment as we can't modify the /etc/resolv.conf config file to add a trust anchor, so we need a fix for that.

[danielluke (Daniel J. Luke)]

Replying to scott-macports@…:

I could, of course, submit a new interface to ldns that explicitly loaded default keys from /etc/trusted-key.key (or some other default), but then openssh couldn't use that interface until the new library version was available on any particular distribution (as a new api, it can't really be backported as fix to distributions using the current or older libraries), preventing a working openssh + ldns on those systems for some time...

It makes the most sense (to me) to get a change like this accepted by ldns and openssh upstream, and then backport it to our release (if necessary because we don't want to wait for upstream to release new versions).

[scott-macports@…]

FYI: I created a new function to load root keys from a "canonical" location for the ldns library, and submitted to upstream as an enhancement request: ​https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=727

Hopefully, if it (or something like it) is accepted, we can backport the patch to macports' ldns, and then submit a patch to call it from openssh.

(I've created the openssh patch too, and it works well, but there's no point submitting it until there's a library to test it against :)