openssh @9.8p1 broke some key types

[fhgwright (Fred Wright)]

MacPro:~ fw$ ssh localhost
/Users/fw/.ssh/config line 77: Bad key types '+ssh-rsa,ssh-dss'.
/Users/fw/.ssh/config line 78: Bad key types '+ssh-rsa,ssh-dss'.
/Users/fw/.ssh/config: terminating, 2 bad configuration options

I think these are still supported, but not by default. And of course, any intentional change of this type requires lots of notice, given that it affects anywhere that keys are installed.

If this isn't just a config change, then it might be best in the short term to go back to 9.7p1 while applying the security patch (which AIUI is simple).

[fhgwright (Fred Wright)]

BTW, IMO the OpenSSH folks are partly responsible for this, by not releasing a 9.7p2 with just the security fix and no functional changes.

[danielluke (Daniel J. Luke)]

It’s mentioned in the release notes: ​https://www.openssh.com/releasenotes.html

“ OpenSSH has disabled DSA keys by default since 2015 but has retained run-time optional support for them.”

I don’t think we should re-enable the dsa keys that have been default disabled since 2015 (it would just push this problem to next year when they remove the option to compile with support), but I’ll leave it to the maintainer to decide.

[fhgwright (Fred Wright)]

It certainly wasn't well publicized. I didn't become aware of it until yesterday, and I have keys of those types all over the place.

Maybe the announcement to users was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard".

[danielluke (Daniel J. Luke)]

You had to specifically enable accepting dsa keys in config since OpenSSH 7 (back in 2015)

[fhgwright (Fred Wright)]

Where "you" means developers. There was no communication to end users that their existing and widely distributed keys would stop working whenever developers decided not to enable that option.

[danielluke (Daniel J. Luke)]

No, I means end users. As far as I can tell, macports’ shipped ssh config doesn’t contain the bits to enable dsa keys for sshd (server) or ssh (client). To make it work after openssh 7 a person would need to configure as described in ​http://www.openssh.com/legacy.html

[artkiver (グレェ)]

DSA key support will be removed entirely in 2025, it's been disabled by default since 2015.

Also see:

Future deprecation notice
=========================

OpenSSH plans to remove support for the DSA signature algorithm in
early 2025 and compile-time disable it later this year.

DSA, as specified in the SSHv2 protocol, is inherently weak - being
limited to a 160 bit private key and use of the SHA1 digest. Its
estimated security level is only 80 bits symmetric equivalent.

OpenSSH has disabled DSA keys by default since 2015 but has retained
run-time optional support for them. DSA was the only mandatory-to-
implement algorithm in the SSHv2 RFCs[3], mostly because alternative
algorithms were encumbered by patents when the SSHv2 protocol was
specified.

This has not been the case for decades at this point and better
algorithms are well supported by all actively-maintained SSH
implementations. We do not consider the costs of maintaining DSA in
OpenSSH to be justified and hope that removing it from OpenSSH can
accelerate its wider deprecation in supporting cryptography
libraries.

This release makes DSA support in OpenSSH compile-time optional,
defaulting to on. We intend the next release to change the default
to disable DSA at compile time. The first OpenSSH release of 2025
will remove DSA support entirely.

From ​https://www.openssh.com/releasenotes.html#9.7p1

I'd heartily recommend searching through ​https://www.openssh.com/releasenotes.html with the phrase "deprecation" to see if there might have been anything else announced but missed?

[artkiver (グレェ)]

If you want to create a variant that enables DSA support, you're welcome to submit a PR, but I would be against it.

[fhgwright (Fred Wright)]

Replying to danielluke:

No, I means end users. As far as I can tell, macports’ shipped ssh config doesn’t contain the bits to enable dsa keys for sshd (server) or ssh (client). To make it work after openssh 7 a person would need to configure as described in ​http://www.openssh.com/legacy.html

OK, but that just means that a long-forgotten config-file tweak that was made out of necessity years ago has been rendered inoperative with no notice. Normally, something of that form should emit obvious warnings for at least a year before action involving significant effort becomes mandatory.

Disallowing new keys based on a deprecated algorithm is one thing, but failing to support existing keys is quite heavy-handed. Perhaps if the OpenSSH folks spent more time learning how the real world works instead of reintroducing old security bugs, we wouldn't be having this issue.

BTW, claims about OpenSSH 7 and 2015 are wrong, at least in the MacPorts context. The openssh port didn't start requiring the config tweak until 8.8p1, issued in Oct-2021. And oddly enough, I don't see anything in the port diffs from 8.4p1 to 8.8p1 relating to key types.

Another issue is that the old key types are necessary to interoperate with the Apple sshd in OS versions prior to 10.12. And since the mechanism for automatically reloading a previously loaded port after a deactivate/activate is extremely unreliable, any dependence on a MacPorts sshd on a headless system risks rendering it incommunicado after a port upgrade. So one would need to keep the Apple sshd active on a different port, and also keep around an alternate ssh client that doesn't suffer from deprecation disease.

Replying to artkiver:

If you want to create a variant that enables DSA support, you're welcome to submit a PR, but I would be against it.

I don't understand this comment. Are you saying that one is free to create, test, and submit a PR which is guaranteed to be rejected due to maintainer opposition?

BTW, the openssl3 port added a legacy variant for somewhat similar reasons in Nov-2021, only a couple of weeks after openssh @8.8p1 came out with the config-file tweak requirement.

[danielluke (Daniel J. Luke)]

Replying to fhgwright:

OK, but that just means that a long-forgotten config-file tweak that was made out of necessity years ago has been rendered inoperative with no notice.

In order to make the config tweak, you had to read the notice (from upstream).

BTW, claims about OpenSSH 7 and 2015 are wrong, at least in the MacPorts context. The openssh port didn't start requiring the config tweak until 8.8p1, issued in Oct-2021. And oddly enough, I don't see anything in the port diffs from 8.4p1 to 8.8p1 relating to key types.

7.0 disabled dsa (2015) 8.8 disabled ssh-rsa using sha-1 signatures (2021)

MacPorts didn't do anything to enable these and if you continued using them you had to manually change your configuration.

Another issue is that the old key types are necessary to interoperate with the Apple sshd in OS versions prior to 10.12.

OpenSSH since 7.2 (2016) supports ssh-rsa keys with RFC8332 RSA/SHA-256/512 signatures and upgrades existing rsa keys to use the newer signature algorithm where possible.

Replying to artkiver:

If you want to create a variant that enables DSA support, you're welcome to submit a PR, but I would be against it.

I don't understand this comment. Are you saying that one is free to create, test, and submit a PR which is guaranteed to be rejected due to maintainer opposition?

It sounds like the maintainer doesn't like the idea and won't pursue it, but probably would accept it.

[artkiver (グレェ)]

It sounds like the maintainer doesn't like the idea and won't pursue it, but probably would accept it.

That is a correct interpretation. I certainly won't be submitting such a PR myself, but if someone else wants to go against upstream and maintain a variant, I can fathom why.

Regarding, "has been rendered inoperative with no notice."?

I would again draw your attention to the release notes for 9.7p1 excerpted previously, italics for emphasis to help with reading comprehension:

We intend the next release to change the default to disable DSA at compile time.

Notice was given in advance, this is not blind siding anyone who has been paying attention.

[fhgwright (Fred Wright)]

Replying to artkiver:

That is a correct interpretation. I certainly won't be submitting such a PR myself, but if someone else wants to go against upstream and maintain a variant, I can fathom why.

I'll probably do that, though not immediately since I'm busy with other things ATM.

Regarding, "has been rendered inoperative with no notice."?

I would again draw your attention to the release notes for 9.7p1 excerpted previously, italics for emphasis to help with reading comprehension:

We intend the next release to change the default to disable DSA at compile time.

Notice was given in advance, this is not blind siding anyone who has been paying attention.

Are you expecting everyone doing a port upgrade outdated to look at the list of upgraded ports, visit their upstream websites, and view the upstream release notes, for every upgraded port? Seriously??? That sort of "notice" isn't much better then "Beware of the Leopard" notice. While referring to upstream release notes is OK in the context of "For more information see XXX", primary notification of upcoming breakage needs to be made directly by the port to its users. Either that or just don't break things (a concept which seems to be lost on upstream).

And the port notes mechanism isn't really adequate, since not only are unsolicited port notes only provided at install and upgrade time, but also notes from an upgrade tend to get buried in a forest of nearly pointless notes from every upgraded port that uses port select.

[danielluke (Daniel J. Luke)]

Replying to fhgwright:

Are you expecting everyone doing a port upgrade outdated to look at the list of upgraded ports, visit their upstream websites, and view the upstream release notes, for every upgraded port?

No, but that wasn't necessary in this case for you to be informed.

When the dsa key was deprecated (in 2015), it would have stopped working and you would have had to read the upstream documentation to find the workaround to re-enable it. That upstream documentation indicated support would go away in the future.

In any case, the maintainer has said they'll accept a PR, so if you're really interested in a variant that will let you go one more year before the support is totally removed, you're welcome to create one.

[artkiver (グレェ)]

Are you expecting everyone doing a port upgrade outdated to look at the list of upgraded ports, visit their upstream websites, and view the upstream release notes, for every upgraded port? Seriously??? That sort of "notice" isn't much better then "Beware of the Leopard" notice. While referring to upstream release notes is OK in the context of "For more information see XXX", primary notification of upcoming breakage needs to be made directly by the port to its users. Either that or just don't break things (a concept which seems to be lost on upstream).

And the port notes mechanism isn't really adequate, since not only are unsolicited port notes only provided at install and upgrade time, but also notes from an upgrade tend to get buried in a forest of nearly pointless notes from every upgraded port that uses port select.

I am not sure why you seem to be lashing out on this, let alone making spurious expectations/"putting words into my mouth" and making up things which I have never written nor said on the subject.

I am not an OpenSSH developer. I am a volunteer.

However, I certainly understand that OpenSSH, is not something which is unique to MacPorts.

Albeit, I am biased.

I am also an editor for undeadly.org, the OpenBSD journal and heck, I even published this story on the OpenSSH 9.7/9.7p1 release announcement: ​https://undeadly.org/cgi?action=article;sid=20240312065313. Every manpage and GitHub and BugZilla link, is something I personally wrote an a href tag for in that story because copy & paste is insufficient to capture all the pertinent nuances for the release notes in a format that is consistent with undeadly.org editorial styles. But again, there too, I am a volunteer. In over 20 years, no one has paid me a penny for any of my contributions to that site.

I should note: I am also not an OpenBSD developer, even if most of the MacPorts on which I toil, are OpenBSD related. That's more a matter of personal interest. As an aside, I have given a brief once over creating a MacPort for OpenBGPD, but it didn't seem to build cleanly from source on macOS, not to mention given that macOS requires /usr/bin/caffeinate to be invoked to make it not sleep, it doesn't really seem as if it is an ideal OS target for core routing protocols, but maybe someday I will tickle that itch again.

Regardless, MacPorts does not exist in a vacuum.

I wouldn't expect that anyone using MacPorts' OpenSSH would be oblivious that OpenSSH itself is an upstream project? Do you think that Other Linux distros shipping OpenSSH 9.8p1 are splitting hairs over DSA's removal as a compile time default? I am guessing, from ​https://repology.org/project/openssh/versions that since only 92 even appear to be at 9.8 out of 768, that the vast majority of Linux distros and their packaging systems, have far bigger problems to worry about; or they're like RHEL which do extremely confusing things such as backporting patches to old version numbers, which I think does everyone a disservice.

If you think it is incumbent upon MacPorts and its maintainers to make its users aware of every nuance that an upstream project is making, rather than focusing on keeping -CURRENT with upstream, I concur that I don't think "port notes" or whatever is going to be adequate, because that isn't what MacPorts has ever been about AFAIK. We don't have anything like the MacPorts' Journal, documenting salient improvements, and far be it from any capitalistic advertising laden blogs such as Apple Insider or MacRumors to pay attention to MacPorts either, apparently. Not that I would expect anything less. There are reasons I contribute to MacPorts and not Homebrew after all, not the least of which is Homebrew defaults to spyware, MacPorts requires users opt-in to installing mpstats if they want to report analytics. I think its wise to avoid surveillance capitalism, by default.

But then, I guess since I was blessed to work with jkh (who seems to be conspicuously absent from MacPorts since around 2016? I have my theories as to why but I haven't seen him in person since the FreeBSD 20 year anniversary party to confirm my suspicions) at iXSystems when we were both there circa 2013 and got to know him personally, I never really saw MacPorts/DarwinPorts as anything more than something like a BSD /usr/ports as might be found in OpenBSD or FreeBSD, etc.? Those typically, also do their best to keep -CURRENT with upstream, but don't make a point of subsuming responsibility from the upstream projects as far as their overarching development goals or objectives. I don't think any BSD /usr/ports user would expect their respective port maintainers to keep ports users abreast of everything upstream changes either, so much as salient release tracking? I don't think most MacPorts users would consider that part of maintainers' responsibilities either?

If it makes you feel any better, it is not as if I am resting on my laurels regarding OpenSSH's development with regards to MacPorts.

For example, I filed ​https://bugzilla.mindrot.org/show_bug.cgi?id=3697 which djm@ rectified, shortly afterwards. I was testing that OpenSSH snapshot, knowing that this commit had occurred: ​https://marc.info/?l=openbsd-cvs&m=171590564107097&w=2

Thankfully, that build issue was rectified before the 9.8p1 release, even if the main things focused on by 9.8p1 were security related, there was at least one other bug I helped identify and djm@ remediated which were pertinent to MacPorts/macOS builds and I like to think I helped make a little bit of a difference in avoiding more potential delays that might have been encountered had I not been testing snapshots and such?

If you want more from me, you probably haven't been paying attention to how dire my living circumstances already are, which I have documented in other Trac tickets. I make a best effort but am barely scraping by as it is. (TL;DR: I am thousands in debt, homeless/sleep in my car and even replying to this is thanks to some library WiFi).

At least personally, I don't think that it is particularly wise to keep the DSA cipher, already deprecated, with future, publicly documented upstream plans to have support removed entirely next year, on any form of life support by MacPorts. I would question, without having done the due diligence, whether potentially impacted systems might not be better off, in every appreciable manner, by simply running ssh-keygen again with a supported key? I believe ssh-keygen currently defaults to ed25519?

I do remember (and I even filed a bug with Apple back when it was still Radar not FeedBack, which Apple closed as a duplicate years after it was opened, not that Apple has ever publicly acknowledged me for any bug I have filed with them [and yes, I filed FeedBack for the vulnerability remediated by 9.8p1 just to raise their awareness internally in hopes maybe they'll ship something saner with Sequoia let alone remunerated me through their bug bounty program]) that Apple was not supporting any elliptic curve keys, so maybe there is some weird edge case which would run afoul of whatever backasswards things Apple's asinine attorneys were deluded about even years after Dragos Ruiu had publicly cited how whatever US Navy patents on such things had long since expired, but I don't seem to have access to the old bug reports I filed with Apple rdar to check those notes at the moment, doubtlessly I could probably dig up the citations again on some BugTraq archive or wherever.

Regardless, that ship sailed.

Even Apple's OpenSSH and ssh-keygen do support elliptic curves currently, and MacPorts' users benefit from more current versions of OpenSSH than Apple ships anyway, so I am kind of confused why there would be much of a need for MacPorts' OpenSSH to support DSA moving forward; but as previously mentioned:

While I personally am not going to submit such a PR, if someone really thinks OpenSSH DSA key support is vital and wants to submit a PR with a variant, so long as it is not a default variant, maybe that will help some folks out and more power to them!

[ryandesign (Ryan Carsten Schmidt)]

Replying to fhgwright:

And since the mechanism for automatically reloading a previously loaded port after a deactivate/activate is extremely unreliable,

This is the first I've heard of that. If there is not already a bug report, please file one.

Replying to artkiver:

If you want more from me, you probably haven't been paying attention to how dire my living circumstances already are, which I have documented in other Trac tickets. I make a best effort but am barely scraping by as it is. (TL;DR: I am thousands in debt, homeless/sleep in my car and even replying to this is thanks to some library WiFi).

I am sorry to hear this! I have not seen your other comments.

[drosehn (Garance A Drosehn)]

Replying to ryandesign:

Replying to fhgwright:

And since the mechanism for automatically reloading a previously loaded port after a deactivate/activate is extremely unreliable,

This is the first I've heard of that. If there is not already a bug report, please file one.

I'm afraid I'm another one who was blindsided by the change to openssh . I'm in the middle of a multiple-level disaster of my own which started with the internal SSD on my Mac going haywire. Once I got things back to where they had been on July 1st, I thought I'd update my ports (partially because I didn't trust some aspects of the Time Machine restore). I updated openssh, and then updated some other port which did run into a problem which was due to the Time Machine restore. Ie, it was not caused by anything in MacPorts. Due to that, I cleaned out all deactivated ports, including the deactivated version of openssh which was left over due to the install I had done just a few minutes earlier.

Okay, so after more pain, I get most ports upgraded. I then went to ssh to a machine which happens to need one of the key-types which had been removed. I hit the errors, and Ugh. Okay, I'll just re-install the version of openssh which had been installed before today. As near as I can tell, it is not possible to install the older version. I'm told that it's not available. And I don't even have the option to just activate the previous version, because I had to clean out the non-active versions of other ports due to an unrelated issue. Is there any version I can install which will still support the now-unsupported keys?

Am I just stuck? I understand that everyone involved is trying to "Do the Right Thing", but it leaves me with many servers that I cannot access. I know there are ways I can get around this by bouncing through other servers of mine, but that's going to get pretty painful.

[drosehn (Garance A Drosehn)]

Replying to drosehn:

Replying to ryandesign:

Replying to fhgwright:

And since the mechanism for automatically reloading a previously loaded port after a deactivate/activate is extremely unreliable,

This is the first I've heard of that. If there is not already a bug report, please file one.

I'm afraid I'm another one who was blindsided by the change to openssh . I'm in the middle of a multiple-level disaster of my own which started with the internal SSD on my Mac going haywire.

FWIW, I should probably add that this is on an Intel-based Mac mini. Not Apple Silicon. I'm thinking the only way out of this is to erase the entire disk, reinstall macOS, redo the restore from TimeMachine, and then upgrade ports without updating openssh! Painful, but it's also going to be painful without those ssh key-times.

[drosehn (Garance A Drosehn)]

Replying to drosehn:

... I'm thinking the only way out of this is to erase the entire disk, reinstall macOS, redo the restore from TimeMachine, and then upgrade ports without updating openssh! Painful, but it's also going to be painful without those ssh key-types.

After some futzing around I have a work-around which I think will work OK. So, ATM I think this issue is not critical for me.

[danielluke (Daniel J. Luke)]

Replying to drosehn:

I'm afraid I'm another one who was blindsided by the change to openssh .

"blindsided" by forgetting that you made a config change in 2015 where the instructions warned you that this day was coming but you didn't migrate to some other key type...

As near as I can tell, it is not possible to install the older version.

​https://trac.macports.org/wiki/howto/InstallingOlderPort

Am I just stuck? I understand that everyone involved is trying to "Do the Right Thing", but it leaves me with many servers that I cannot access. I know there are ways I can get around this by bouncing through other servers of mine, but that's going to get pretty painful.

Using jump servers is super-easy to do with ssh, and it's probably what I'd set up anyway if I had old hardware that I couldn't update (especially since that hardware should be as segmented from the 'regular' internet as possible anyway as it's woefully out of date and clearly isn't receiving security updates any longer).

If you /can/ run newer openssh on your old machines, you should do so. If you install Openssh from 2016 (version 7.2) or newer instead of relying on openssh from 2015 or older, you can use RSA keys which are still supported.

[drosehn (Garance A Drosehn)]

Replying to danielluke:

Replying to drosehn:

I'm afraid I'm another one who was blindsided by the change to openssh .

"blindsided" by forgetting that you made a config change in 2015 where the instructions warned you that this day was coming but you didn't migrate to some other key type...

"Blindsided" in that the desktop computer I use in my office died with hardware errors on July 5th. "Blindsided" in the sense that I got a fixed computer after 5pm on Friday July 12th (in a week I was scheduled to be on vacation), and it would be awfully nice if that computer was mostly-working by 10am on Monday morning. "Blindsided" in that after spending over an hour doing a full restore of the dead SSD onto the new SSD, there were a few minor issues which weren't working right. Most things were working fine, but there were a few minor annoyances. "Blindsided" in that while trying to fix those minor annoyances I decided to upgrade macports. This did indeed did fix one or two minor issues but also resulted in what was a much more serious issue for me. "Blindsided" in that by the time I realized the new ssh had broken things for me, it was already past 2am on Saturday. That kind of "Blindsided".

Also, the feature I'm using is the one which was deprecated in 2021 (during Covid) in OpenSSH 8.8/8.8p1 (2021-09-26), not back in 2015. In 2021 and 2022 I had several more important issues to worry about than an option in .ssh/config (including issues that I'm not going to describe here).

And some added irony is that one of the projects I need to work on this week will hopefully let us unplug some of these very same out-of-date servers before Sept 1st (possibly before Aug 1st?). But oddly enough I need to ssh into those servers to complete that project.

_

As near as I can tell, it is not possible to install the older version.

​https://trac.macports.org/wiki/howto/InstallingOlderPort

Thanks. This part is very helpful. It has been many years since I've had any significant issue with Macports, so I was totally unaware of the options which are currently available. I'm pretty sure that the last time I had a significant problem, Macports wasn't even in github! I ended up going with a different solution, but it's nice to realize that a solution such as this is available.

[danielluke (Daniel J. Luke)]

Replying to drosehn:

Also, the feature I'm using is the one which was deprecated in 2021 (during Covid) in OpenSSH 8.8/8.8p1 (2021-09-26), not back in 2015. In 2021 and 2022 I had several more important issues to worry about than an option in .ssh/config (including issues that I'm not going to describe here).

7.0 disabled dsa (2015) 8.8 disabled ssh-rsa using sha-1 signatures (2021), 7.2 (2016) upgrades rsa keys on a host to use newer signatures. The current release disables dsa keys deprecated in 2015.

[drosehn (Garance A Drosehn)]

Replying to danielluke:

Replying to drosehn:

Also, the feature I'm using is the one which was deprecated in 2021 (during Covid) in OpenSSH 8.8/8.8p1 (2021-09-26), not back in 2015. In 2021 and 2022 I had several more important issues to worry about than an option in .ssh/config (including issues that I'm not going to describe here).

7.0 disabled dsa (2015) 8.8 disabled ssh-rsa using sha-1 signatures (2021), 7.2 (2016) upgrades rsa keys on a host to use newer signatures. The current release disables dsa keys deprecated in 2015.

Ah. I see that I have misunderstood what is going on. I have several entries in my config file which match the initial post in this ticket, which is to say that the entry in config includes the line:
HostKeyAlgorithms +ssh-rsa,ssh-dss

Thus I get the same error message that's in the original entry for this ticket:
line 25: Bad key types '+ssh-rsa,ssh-dss'.

Based on the word "types", I assumed that both of those key types were invalid. None of the servers I connect to actually depend on ssh-dss, so people might wonder why I specified both in the config file entries. That's because the error which comes back when I try to connect to those servers is:
Unable to negotiate with <ip_addr>: no matching host key type found. Their offer: ssh-rsa,ssh-dss

So I just copy&pasted that into the entries in config. As long as ssh-rsa is available, I don't actually need ssh-dss for anything. And I have just confirmed that I can connect to all these servers once I remove the request for type ssh-dss. So it looks like the only problem for me is that I was specifying an option I didn't actually need. Maybe there are other people who have followed the same path I did.

[fhgwright (Fred Wright)]

In 11c25bbb3cd4ae7249c4839bf23d37d35a05d585/macports-ports (master):

openssh: Add legacy_dsa variant.

Also adds message regarding test requirements.

Also fixes new "platforms" lint warning.

Closes: #70319

TESTED:
With +legacy_dsa, accepts "+ssh-rsa,ssh-dss" config options.
Built successfully on OSX 10.4-10.5 ppc, 10.4-10.6 i386, 10.5-12.x
x86_64, and 11.x-14.x arm64. Included all variants compatible with
available dependencies on the respective platforms, except that
testing of +universal was limited to avoid the need to install
+universal versions of many dependencies.