Subject: XZ force downgrade led to removal of many packages

From: Mike Lodispoto <[email-masked]>
Date: Sat, 30 Mar 2024 00:16:57 -0500

Hello,

I was attempting to downgrade XZ because of the SSH backdoor in it.

I'm not the most familiar with Alpine, so this might have been user 
inexperience that screwed me, but I did

|apk add xz-libs=5.4.6 --force|

I had tried it without force before, but it had said something along the 
lines of xz-libs is required by world.

This ended up with a bunch of essential packages removed.

|certbot-nginx-pyc certbot-nginx certbot-pyc certbot 
py3-configargparse-pyc py3-configargparse py3-distro-pyc py3-distro 
py3-distutils-extra-pyc py3-distutils-extra py3-parsedatetime-pyc py3
-parsedatetime py3-future-pyc py3-future py3-acme-pyc py3-acme 
py3-josepy-pyc py3-josepy py3-pyrfc3339-pyc py3-pyrfc3339 py3-tz-pyc 
py3-tz py3-openssl-pyc py3-openssl py3-cryptography-pyc p
y3-cryptography py3-cffi-pyc py3-cffi py3-cparser-pyc py3-cparser 
cloud-init-openrc cloud-init-doc cloud-init-pyc cloud-init 
cloud-utils-growpart partx sfdisk py3-configobj-pyc py3-configob
j py3-six-pyc py3-six py3-jinja2-doc py3-jinja2-pyc py3-jinja2 
py3-markupsafe-pyc py3-markupsafe py3-jsonpatch-pyc py3-jsonpatch 
py3-jsonpointer-pyc py3-jsonpointer py3-jsonschema-pyc py3-j
sonschema py3-jsonschema-specifications-pyc 
py3-jsonschema-specifications py3-referencing-pyc py3-referencing 
py3-attrs-pyc py3-attrs py3-rpds-py-pyc py3-rpds-py py3-requests-pyc 
py3-reques
ts py3-certifi-pyc py3-certifi py3-charset-normalizer-pyc 
py3-charset-normalizer py3-idna-pyc py3-idna py3-urllib3-pyc py3-urllib3 
py3-yaml-pyc py3-yaml shadow-doc shadow tzdata-doc tzdata
eudev-doc eudev-openrc eudev udev-init-scripts-openrc udev-init-scripts 
eudev-libs gdbm-doc grub-bios grub-doc grub ifupdown-ng-iproute2 
iproute2-minimal iotop-pyc iotop-doc iotop json-c-do
c kmod-doc libelf libfdisk libsmartcols libxml2-doc libzip-doc 
linux-pam-doc linux-virt mariadb-doc mariadb-openrc mkinitfs-doc 
mkinitfs kmod lddtree mpdecimal-doc mysql mariadb php7-cgi ph
p7-fpm php7-simplexml php7-soap php7-xmlreader php7-dom php7-xmlrpc 
php7-xml php7-zip php81-doc php81 php81-common py3-packaging-pyc 
py3-parsing-pyc py3-pip-pyc py3-pip-doc py3-pip py3-setu
ptools-pyc py3-setuptools py3-packaging py3-parsing python3-doc 
python3-pyc python3-pycache-pyc0 pyc yaml python3 kmod-libs libzip 
libxml2 xz-libs cryptsetup-libs argon2-libs gdbm json-c li
bpanelw linux-pam mpdecimal|

In my mind, force would have had it told me what packages were going to 
be changed before applying them, so I could approve it like normal.

Am I just completely screwed? I can do a clean install if needed, but 
I'd like to salvage my system.

Thanks,

Mike Lodispoto

---

From: [email-masked]
Date: Sat, 30 Mar 2024 09:21:16 +0300

> In my mind, force would have had it told me what packages were going to be
> changed before applying them, so I could approve it like normal.
> 
> Am I just completely screwed? I can do a clean install if needed, but I'd
> like to salvage my system.

Open /etc/apk/world with your editor. Find xz-libs=5.4.6 and delete it's line.
Then you can do

|apk fix|

This should reinstall every removed other stuff.

But I don't know how you can downgrade xz-libs.

---

From: Mike Lodispoto <[email-masked]>
Date: Sat, 30 Mar 2024 16:46:31 -0500

I was on edge, and had xz 5.6.1.

    |apk fix|

    This should reinstall every removed other

Thank you kurth4cker, your solution saved the server!

---

From: Stuart Longland VK4MSL <[email-masked]>
Date: Sat, 30 Mar 2024 20:15:32 +1000

> I was attempting to downgrade XZ because of the SSH backdoor in it.

While I definitely wouldn't recommend installing the later xz libraries
at this time, looks like the "damage" has been done as there are likely
packages that depend on _that_ version of the xz libs.

Gentoo issued this statement today:
> Impact
> 
> Our current understanding of the backdoor is that is does not affect
> Gentoo systems, because 1. the backdoor only appears to be included
> on specific systems and Gentoo does not qualify; 2. the backdoor as
> it is currently understood targets OpenSSH patched to work with
> systemd-notify support. Gentoo does not suppoxzrt or include these
> patches; Analysis is still ongoing, however, and additional vectors
> may still be identified. For this reason we are still issuing this
> advisory as if that will be the case.
-- https://glsa.gentoo.org/glsa/202403-04

Now, AlpineLinux, like Gentoo…

- uses OpenRC (Gentoo *can* use systemd, but the default is OpenRC)
- does not use RPM or DEB packages
- likely will not be patching OpenSSH to support systemd-notify since 
the latter is not present

I think in the coming days, there will be an audit done on the `xz` 
library and out of that, we should see a release that corrects the 
issues identified.

Let's hold tight on this one.

AlpineLinux 3.19 ships with xz 5.4.5.  It's only "edge" that's affected. 
  Not sure if there is an issue raised on this, I tried searching, but 
when I put "xz" in the search box and click search, gitlab just sits 
there and looks at me stupid -- apparently keyword searching on a bug 
title is not in its feature set.
-- 
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
   ...it's backed up on a tape somewhere.

---

From: Ralf Mardorf <[email-masked]>
Date: Sat, 30 Mar 2024 11:39:53 +0100

> the latter is not present

Hi,

security through naivety is new to me.

For example, Arch uses systemd by default, but does not link openssh
directly to liblzma. Arch uses neither RPM nor DEB. From a naive point
of view, Arch is not affected either.

"However, out of an abundance of caution, we advise users to remove the
malicious code from their system by upgrading either way. This is
because other yet-to-be discovered methods to exploit the backdoor could
exist." -
https://lists.archlinux.org/archives/list/[email-masked]/thread/MX363534MGK44R5UIYPK4GABKHF76TYC/

Regards,
Ralf

---

From: Stuart Longland VK4MSL <[email-masked]>
Date: Sun, 31 Mar 2024 07:25:28 +1000

On 30/3/24 20:39, Ralf Mardorf wrote:
> directly to liblzma. Arch uses neither RPM nor DEB. From a naive
> point of view, Arch is not affected either.
> 
> "However, out of an abundance of caution, we advise users to remove
> the malicious code from their system by upgrading either way. This
> is because other yet-to-be discovered methods to exploit the backdoor
> could exist." - 

If possible.  Now, if you're running 3.19, according to
pkgs.alpinelinux.org, the latest release of xz is 5.4.5:

https://pkgs.alpinelinux.org/packages?name=xz*&branch=v3.19&repo=&arch=&maintainer=

Meaning, the vulnerable version *should not be present*.

The only way a vulnerable version could be there would be if someone
installed it from `edge`: the *unstable* distribution.  It comes with 
this warning:

> Warning: edge is under constant development so be careful using it in
> production. It is possible that bugs in edge could cause data loss or
> could break your system.
-- https://wiki.alpinelinux.org/wiki/Repositories#Edge

I'd imagine that those "bugs" can include security vulnerabilities. 
Especially those from upstream releases, which is what we have here. 
This isn't the fault of the AlpineLinux developers: few of us have the 
resources to manually inspect every release of every package for such 
vulnerabilities.  I've been there: I used to do it on the Mozilla and 
MIPS teams for Gentoo.

So it would seem for the OP, the options are:

1. roll back to release 3.19, which does not include the vulnerability
2. hold tight until a fixed release can trickle down from upstream

Regards,

---

From: Jeffrey Walton <[email-masked]>
Date: Sat, 30 Mar 2024 20:48:53 -0400

> https://pkgs.alpinelinux.org/packages?name=xz*&branch=v3.19&repo=&arch=&maintainer=

Jia Tan started contributing to xz circa the development version 5.3.
To get untainted code, you have to go back to version 5.2. But rolling
back to version 5.2 means ABI and symbol breaks. If you don't want to
go back to 5.2, then it means you have to audit over 700 commits in
xz. Also see <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#5>.

Maybe worth mentioning... Jia Tan also compromised libarchive. See
<https://github.com/libarchive/libarchive/pull/1609>.

> 2. hold tight until a fixed release can trickle down from upstream

Jeff

---

From: Stuart Longland VK4MSL <[email-masked]>
Date: Sun, 31 Mar 2024 11:59:28 +1000

On 31/3/24 10:48, Jeffrey Walton wrote:
> xz. Also see<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#5>.

Exactly, which is why those of us who are unable to assist in that 
audit, should wait before rushing off to do things.

To quote that link:
> Note that reverted to such an old version will break packages that use
> new symbols introduced since then. From a quick look, this is at least:
> - dpkg
> - erofs-utils
> - kmod
> 
> Having dpkg in that list means that such downgrade has to be planned
> carefully.

If I check `kmod` on AlpineLinux:
> gapmx:~# ldd /bin/kmod 
>         /lib/ld-musl-x86_64.so.1 (0x7f5dbc5ab000)
>         libzstd.so.1 => /usr/lib/libzstd.so.1 (0x7f5dbc4db000)
>         liblzma.so.5 => /usr/lib/liblzma.so.5 (0x7f5dbc4a4000)
>         libz.so.1 => /lib/libz.so.1 (0x7f5dbc48a000)
>         libcrypto.so.3 => /lib/libcrypto.so.3 (0x7f5dbc000000)
>         libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f5dbc5ab000)

The same "careful planning" applies here too.
-- 
Stuart Longland (aka Redhatter, VK4MSL)

---

From: Steffen Nurpmeso <[email-masked]>
Date: Sun, 31 Mar 2024 04:03:20 +0200

Jeffrey Walton wrote in
 <[email-masked]>:
 |On Sat, Mar 30, 2024 at 5:26 PM Stuart Longland VK4MSL <[email-masked]> \
 |wrote:
 |>
 |> On 30/3/24 20:39, Ralf Mardorf wrote:
 |>> security through naivety is new to me.
 |>>
 |>> For example, Arch uses systemd by default, but does not link openssh
 |>> directly to liblzma. Arch uses neither RPM nor DEB. From a naive
 |>> point of view, Arch is not affected either.
 |>>
 |>> "However, out of an abundance of caution, we advise users to remove
 |>> the malicious code from their system by upgrading either way. This
 |>> is because other yet-to-be discovered methods to exploit the backdoor
 |>> could exist." -
 |>> https://lists.archlinux.org/archives/list/[email-masked].\
 |>> org/thread/MX363534MGK44R5UIYPK4GABKHF76TYC/
 |>
 |> If possible.  Now, if you're running 3.19, according to
 |> pkgs.alpinelinux.org, the latest release of xz is 5.4.5:
 |>
 |> https://pkgs.alpinelinux.org/packages?name=xz*&branch=v3.19&repo=&arch=&\
 |> maintainer=
 |
 |Jia Tan started contributing to xz circa the development version 5.3.
 |To get untainted code, you have to go back to version 5.2. But rolling
 |back to version 5.2 means ABI and symbol breaks. If you don't want to
 |go back to 5.2, then it means you have to audit over 700 commits in
 |xz. Also see <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068024#5>.

I have downgraded to 5.4.0 locally (CRUX; server not, Alpine
surely will do the right thing).

I have seen the <half a dozen or so commits of jiat75 (would you
pronounce that "dschihad", but this is surely no muslim terror,
no?) and jia tan included by then, they look ok.  Some later too,
but that was too crowded and i cannot nor want to look at them
all.  I use zstd for myself.

 |Maybe worth mentioning... Jia Tan also compromised libarchive. See
 |<https://github.com/libarchive/libarchive/pull/1609>.

Well .. the error string thing i would not call compromised, as Ed
Maste i think it was commented when he reverted (two days, and
late).

Nice Easter to the Christians.  And Muslims have soon passed the
Ramadan!

 |> Meaning, the vulnerable version *should not be present*.
 |>
 |> The only way a vulnerable version could be there would be if someone
 |> installed it from `edge`: the *unstable* distribution.  It comes with
 |> this warning:
 |>
 |>> Warning: edge is under constant development so be careful using it in
 |>> production. It is possible that bugs in edge could cause data loss or
 |>> could break your system.
 |> -- https://wiki.alpinelinux.org/wiki/Repositories#Edge
 |>
 |> I'd imagine that those "bugs" can include security vulnerabilities.
 |> Especially those from upstream releases, which is what we have here.
 |> This isn't the fault of the AlpineLinux developers: few of us have the
 |> resources to manually inspect every release of every package for such
 |> vulnerabilities.  I've been there: I used to do it on the Mozilla and
 |> MIPS teams for Gentoo.
 |>
 |> So it would seem for the OP, the options are:
 |>
 |> 1. roll back to release 3.19, which does not include the vulnerability
 |> 2. hold tight until a fixed release can trickle down from upstream
 |
 |Jeff
 --End of <[email-masked]\
 .com>

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

---

From: steffen <[email-masked]>
Date: Fri, 05 Apr 2024 19:48:02 +0200

P.S.: FreeBSD rolled back to 5.4.5 and took an interesting
approach:

  The branch main has been updated by delphij:

  URL: https://cgit.FreeBSD.org/src/commit/?id=5ffb19ac36c4b5c9e72d69048250c8e56acff739

  commit 5ffb19ac36c4b5c9e72d69048250c8e56acff739
  Author:     Xin LI <[email-masked]>
  AuthorDate: 2024-04-05 06:39:33 +0000
  Commit:     Xin LI <[email-masked]>
  CommitDate: 2024-04-05 06:39:33 +0000

      Backport export of lzma_mt_block_size symbol.

      This restores binary compatibility against liblzma 5.6.0 library.

      PR:             278127
      MFC after:      3 days

---

From: "lauren n. liberda" <[email-masked]>
Date: Sat, 30 Mar 2024 13:25:12 +0100

not noted: the backdoor also required glibc, which alpine famously doesn't use

On 30 March 2024 11:15:32 am GMT+01:00, Stuart Longland VK4MSL <[email-masked]> wrote:
>- likely will not be patching OpenSSH to support systemd-notify since the latter is not present
>
>I think in the coming days, there will be an audit done on the `xz` library and out of that, we should see a release that corrects the issues identified.
>
>AlpineLinux 3.19 ships with xz 5.4.5.  It's only "edge" that's affected.  Not sure if there is an issue raised on this, I tried searching, but when I put "xz" in the search box and click search, gitlab just sits there and looks at me stupid -- apparently keyword searching on a bug title is not in its feature set.
>--

---

From: Jeffrey Walton <[email-masked]>
Date: Sun, 31 Mar 2024 20:08:00 -0400

On Sat, Mar 30, 2024 at 1:18 AM Mike Lodispoto <[email-masked]> wrote:
> I'm not the most familiar with Alpine, so this might have been user inexperience that screwed me, but I did
>
>  apk add xz-libs=5.4.6 --force
>
> I had tried it without force before, but it had said something along the lines of xz-libs is required by world.
>
> certbot-nginx-pyc certbot-nginx certbot-pyc certbot py3-configargparse-pyc py3-configargparse py3-distro-pyc py3-distro py3-distutils-extra-pyc py3-distutils-extra py3-parsedatetime-pyc py3
> -parsedatetime py3-future-pyc py3-future py3-acme-pyc py3-acme py3-josepy-pyc py3-josepy py3-pyrfc3339-pyc py3-pyrfc3339 py3-tz-pyc py3-tz py3-openssl-pyc py3-openssl py3-cryptography-pyc p
> y3-cryptography py3-cffi-pyc py3-cffi py3-cparser-pyc py3-cparser cloud-init-openrc cloud-init-doc cloud-init-pyc cloud-init cloud-utils-growpart partx sfdisk py3-configobj-pyc py3-configob
> j py3-six-pyc py3-six py3-jinja2-doc py3-jinja2-pyc py3-jinja2 py3-markupsafe-pyc py3-markupsafe py3-jsonpatch-pyc py3-jsonpatch py3-jsonpointer-pyc py3-jsonpointer py3-jsonschema-pyc py3-j
> sonschema py3-jsonschema-specifications-pyc py3-jsonschema-specifications py3-referencing-pyc py3-referencing py3-attrs-pyc py3-attrs py3-rpds-py-pyc py3-rpds-py py3-requests-pyc py3-reques
> ts py3-certifi-pyc py3-certifi py3-charset-normalizer-pyc py3-charset-normalizer py3-idna-pyc py3-idna py3-urllib3-pyc py3-urllib3 py3-yaml-pyc py3-yaml shadow-doc shadow tzdata-doc tzdata
> eudev-doc eudev-openrc eudev udev-init-scripts-openrc udev-init-scripts eudev-libs gdbm-doc grub-bios grub-doc grub ifupdown-ng-iproute2 iproute2-minimal iotop-pyc iotop-doc iotop json-c-do
> c kmod-doc libelf libfdisk libsmartcols libxml2-doc libzip-doc linux-pam-doc linux-virt mariadb-doc mariadb-openrc mkinitfs-doc mkinitfs kmod lddtree mpdecimal-doc mysql mariadb php7-cgi ph
> p7-fpm php7-simplexml php7-soap php7-xmlreader php7-dom php7-xmlrpc php7-xml php7-zip php81-doc php81 php81-common py3-packaging-pyc py3-parsing-pyc py3-pip-pyc py3-pip-doc py3-pip py3-setu
> ptools-pyc py3-setuptools py3-packaging py3-parsing python3-doc python3-pyc python3-pycache-pyc0 pyc yaml python3 kmod-libs libzip libxml2 xz-libs cryptsetup-libs argon2-libs gdbm json-c li
> bpanelw linux-pam mpdecimal
>
> In my mind, force would have had it told me what packages were going to be changed before applying them, so I could approve it like normal.
>
> Am I just completely screwed? I can do a clean install if needed, but I'd like to salvage my system.

---

From: Jeffrey Walton <[email-masked]>
Date: Sun, 31 Mar 2024 20:08:41 -0400

> > Am I just completely screwed? I can do a clean install if needed, but I'd like to salvage my system.

Whoops, sorry about the empty reply. Here's an important comment from
oss-security mailing list message,
<https://www.openwall.com/lists/oss-security/2024/03/31/4>:

commit f9cf4c05edd14dedfe63833f8ccbe41b55823b00 (HEAD -> master,
origin/master, origin/HEAD)
Author: Lasse Collin <lasse.collin@...aani.org>
Date:   Sat Mar 30 14:36:28 2024 +0200

    CMake: Fix sabotaged Landlock sandbox check.

    It never enabled it.