Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face
i used to statically link ffmpeg because it would randomly demand new versions of poo poo like x264 without actually using any new functionality, but then they started to while also adding new codecs so i stopped. thats my static linking story thanks for reading

Adbot
ADBOT LOVES YOU

mystes
May 31, 2006

For some reason kde on wayland crashed and now crashes whenever I try to run it even though I didn't change anything but it still works on x11, and I don't really have time to try to debug this right now

big black turnout
Jan 13, 2009



Fallen Rib

fresh_cheese posted:

this bullshit is what they claim is “fixed” by containers

shared libraries my rear end. just statically link everything dude. fuckitall.

welcome to the rustpublican party

FlapYoJacks
Feb 12, 2009
XZ has a nasty backdoor:

https://news.ycombinator.com/item?id=39866275

quote:

Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 & 41 because of it's "great new features". We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.
He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.

https://www.redhat.com/en/blog/urgent-security-alert-fedora-41-and-rawhide-users

quote:

Yesterday, Red Hat Information Risk and Security and Red Hat Product Security learned that the latest versions of the “xz” tools and libraries contain malicious code that appears to be intended to allow unauthorized access. Specifically, this code is present in versions 5.6.0 and 5.6.1 of the libraries - at this time, only Fedora 41 and Fedora Rawhide contain these libraries. This vulnerability was assigned CVE-2024-3094.

Although Fedora 40 beta contained the 5.6 version of xz in an update, the build environment prevents the injection from correctly occurring, and has not been shown to be compromised. Fedora 40 has now reverted to the 5.4.x versions of xz.

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA 41 OR FEDORA RAWHIDE INSTANCES for work or personal activity.

What is xz?
xz is a general purpose data compression format present in nearly every Linux distribution, both community projects and commercial product distributions. Essentially, it helps compress (and then decompress) large file formats into smaller, more manageable sizes for sharing via file transfers.

What is the malicious code?
The malicious injection present in the xz versions 5.6.0 and 5.6.1 libraries is obfuscated and only included in full in the download package - the Git distribution lacks the M4 macro that triggers the build of the malicious code. The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present.

The resulting malicious build interferes with authentication in sshd via systemd. SSH is a commonly used protocol for connecting remotely to systems, and sshd is the service that allows access. Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.

What distributions are affected by this malicious code?
Current investigation indicates that the packages are only present in Fedora 41 and Fedora Rawhide within the Red Hat community ecosystem.

No versions of Red Hat Enterprise Linux (RHEL) are affected.

We have reports and evidence of the injections successfully building in xz 5.6.x versions built for Debian unstable (Sid).

What should I do if I am running an affected distribution?
For both personal and business activities, immediately stop using Fedora 41 or Fedora Rawhide. If you are using an affected distribution in a business setting, we encourage you to contact your information security team for next steps.

Additionally, for those running openSUSE distributions, SUSE has published a fix at https://build.opensuse.org/request/show/1163302.

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face
oh, good thing practically nothing uses xz

FlapYoJacks
Feb 12, 2009

Beeftweeter posted:

oh, good thing practically nothing uses xz

ZSTD superiority. :smug:

sb hermit
Dec 13, 2016





good thing I have only been using gz for everything for the last 25 or so years

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face

FlapYoJacks posted:

ZSTD superiority. :smug:

iirc it has slightly worse compression ratios than xz, but it's definitely faster

i play around with compression formats from time to time and there's certainly some interesting ones, but no clear winner imo

spankmeister
Jun 15, 2008






sb hermit posted:

good thing I have only been using gz for everything for the last 25 or so years

doesn't matter, it tries to backdoor sshd

Clark Nova
Jul 18, 2004

FlapYoJacks posted:

ZSTD superiority. :smug:

it's called zstd but xz is the one that has an std???? linux is so confusing

sb hermit
Dec 13, 2016





spankmeister posted:

doesn't matter, it tries to backdoor sshd

yeah I read it

I’m trying to say that maybe systemd should have just stuck with what works

spankmeister
Jun 15, 2008






sb hermit posted:

yeah I read it

I’m trying to say that maybe systemd should have just stuck with what works

dehumanize yourself and face to poettering

eschaton
Mar 7, 2007

Don't you just hate when you wind up in a store with people who are in a socioeconomic class that is pretty obviously about two levels lower than your own?

Soricidus posted:

… the whole point of giving it a different name is that it guarantees that they cannot possibly break any existing sdl2 software even by accident, why are you so angry about the way they are accomplishing this

because that’s the stupidest possible way to accomplish the goal, instead of just evolving a single codebase reasonably

sb hermit
Dec 13, 2016





spankmeister posted:

dehumanize yourself and face to poettering

:tif:

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face
so hang on a sec, if your distro doesn't use systemd then the backdoor doesn't work? or is redhat's description of the sshd backdoor just tailored to fedora + systemd because it's red hat?

Soricidus
Oct 21, 2010
freedom-hating statist shill

Beeftweeter posted:

so hang on a sec, if your distro doesn't use systemd then the backdoor doesn't work? or is redhat's description of the sshd backdoor just tailored to fedora + systemd because it's red hat?

we know how to trigger the backdoor in apt/yum-based linux distros with systemd, so we know those are vulnerable in specific circumstances.

we don’t know yet whether other distros might be vulnerable with different trigger conditions. they may well be safe, but it would be foolish to assume that.

the only sensible move is to patch everything this dude has ever touched.

shackleford
Sep 4, 2006

Beeftweeter posted:

so hang on a sec, if your distro doesn't use systemd then the backdoor doesn't work? or is redhat's description of the sshd backdoor just tailored to fedora + systemd because it's red hat?

there is a fairly common patch against openssh in the linux world that many distros apply to add systemd notification support ("Type=notify" in the [Service] section of your ssh.service unit). because the openssh developers are BSD partisans and refuse to port their code to other platforms, even in the "portable" version of openssh.

e.g. here's a copy of it in SuSE:

https://build.opensuse.org/projects/network/packages/openssh/files/openssh-7.7p1-systemd-notify.patch

that code adds a runtime dependency to the compiled sshd binary, i.e. an ELF DT_NEEDED tag on the SONAME libsystemd.so.0

libsystemd.so.0 itself has a DT_NEEDED tag on liblzma.so.5

i forget what it's called but the specific way these binaries were compiled or linked together caused all of the shared libraries and symbols to be eagerly loaded and resolved at process startup rather than lazily resolved only when a specific function in a shared library is called. (e: i think it's compiling/linking with the -Wl,z,now flag that does this.) then the malicious liblzma.so.5 DSO used the IFUNC mechanism to run malicious code when the sshd binary starts up. the IFUNC mechanism is typically used to select between alternative implementations of a function, i think they were loving around with different CRC implementations or something. CRC being a pretty good example for this kind of stuff since x86 CPUs with SSE 4.2 have a CRC32C instruction that really speeds things up (iirc it's something like 10X faster than a software CRC32C implementation). to be useful you want to do a CPUID check at process startup and pick the optimal implementation for the running CPU and patch the symbol table or whatever, that way you're not trying to do CPU detection every time the function gets called or indirecting through a function pointer or something like that

so in theory if your sshd didn't have that systemd patch the malicious code wouldn't be called, but if you're on a distro that uses systemd your openssh probably did have that patch

and libsystemd the library is different from systemd the PID 1 process so in theory you could have been running sysvinit on a systemd-supporting distro that still supports sysvinit but the malicious code would still be injected into the sshd process

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face

Soricidus posted:

we know how to trigger the backdoor in apt/yum-based linux distros with systemd, so we know those are vulnerable in specific circumstances.

we don’t know yet whether other distros might be vulnerable with different trigger conditions. they may well be safe, but it would be foolish to assume that.

the only sensible move is to patch everything this dude has ever touched.

right, the question is more "is a system that does not use systemd still vulnerable?"

fwiw i cross-posted the question to the security thread and the answer seems unclear right now. of course it would be foolish to assume anything is safe until a thorough audit is complete, but i was curious about e.g. embedded systems that might have busybox compiled with xz support. those generally don't get many updates so a lot of devices would probably be vulnerable, but at the same time almost certainly wouldn't be using systemd (or a standalone sshd even). but since busybox is so modular it'd be highly configuration-dependent anyway, i.e. someone could even use busybox with systemd and a separate sshd (openssh, dropbear, whatever) if they were so inclined

but i'm not sure if it even uses xz-utils. i do know the busybox xz applets aren't full-featured, so it might be a minimal or entirely custom implementation

same applies to toybox on android i guess. if you could get root through exploiting it somehow that would actually help me out a ton since i have an old huawei with a permanently locked bootloader lol

shackleford
Sep 4, 2006

also there is code in the unreleased main branch of systemd that switches to dlopen()'ing the compression libraries only when needed that apparently would have neutralized this particular attack chain (the Type=notify systemd notification code in libsystemd doesn't depend on the compression functionality, i think that's in there due to a journal thing)

https://github.com/systemd/systemd/pull/31550

if the postgresql guy hadn't noticed the anomalous CPU usage on his benchmark box and the xz backdoor wasn't caught, whether your distro actually executed the backdoored code might have depended on whether your distro had upgraded to systemd ≥ 256 or not

shackleford
Sep 4, 2006

Beeftweeter posted:

fwiw i cross-posted the question to the security thread and the answer seems unclear right now. of course it would be foolish to assume anything is safe until a thorough audit is complete, but i was curious about e.g. embedded systems that might have busybox compiled with xz support. those generally don't get many updates so a lot of devices would probably be vulnerable, but at the same time almost certainly wouldn't be using systemd (or a standalone sshd even). but since busybox is so modular it'd be highly configuration-dependent anyway, i.e. someone could even use busybox with systemd and a separate sshd (openssh, dropbear, whatever) if they were so inclined

the currently known information about the backdoored xz releases is that it was specifically targeting amd64, GNU, linux, and when running under a dpkg/rpm package build

https://www.openwall.com/lists/oss-security/2024/03/29/4 posted:

The attached de-obfuscated script is invoked first after configure, where it
decides whether to modify the build process to inject the code.

These conditions include targeting only x86-64 linux:
if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);then

Building with gcc and the gnu linker
if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
exit 0
fi
if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
exit 0
fi
LDv=$LD" -v"
if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
exit 0

Running as part of a debian or RPM package build:
if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face

shackleford posted:

there is a fairly common patch against openssh in the linux world that many distros apply to add systemd notification support ("Type=notify" in the [Service] section of your ssh.service unit). because the openssh developers are BSD partisans and refuse to port their code to other platforms, even in the "portable" version of openssh.

e.g. here's a copy of it in SuSE:

https://build.opensuse.org/projects/network/packages/openssh/files/openssh-7.7p1-systemd-notify.patch

that code adds a runtime dependency to the compiled sshd binary, i.e. an ELF DT_NEEDED tag on the SONAME libsystemd.so.0

libsystemd.so.0 itself has a DT_NEEDED tag on liblzma.so.5

i forget what it's called but the specific way these binaries were compiled or linked together caused all of the shared libraries and symbols to be eagerly loaded and resolved at process startup rather than lazily resolved only when a specific function in a shared library is called. (e: i think it's compiling/linking with the -Wl,z,now flag that does this.) then the malicious liblzma.so.5 DSO used the IFUNC mechanism to run malicious code when the sshd binary starts up. the IFUNC mechanism is typically used to select between alternative implementations of a function, i think they were loving around with different CRC implementations or something. CRC being a pretty good example for this kind of stuff since x86 CPUs with SSE 4.2 have a CRC32C instruction that really speeds things up (iirc it's something like 10X faster than a software CRC32C implementation). to be useful you want to do a CPUID check at process startup and pick the optimal implementation for the running CPU and patch the symbol table or whatever, that way you're not trying to do CPU detection every time the function gets called or indirecting through a function pointer or something like that

so in theory if your sshd didn't have that systemd patch the malicious code wouldn't be called, but if you're on a distro that uses systemd your openssh probably did have that patch

and libsystemd the library is different from systemd the PID 1 process so in theory you could have been running sysvinit on a systemd-supporting distro that still supports sysvinit but the malicious code would still be injected into the sshd process

ah that's a pretty thorough answer lol, thanks

shackleford posted:

the currently known information about the backdoored xz releases is that it was specifically targeting amd64, GNU, linux, and when running under a dpkg/rpm package build

yeah i gathered from your other post that it didn't seem to impact other architectures. which thankfully limits its potential impact quite a bit i would think, there's probably more non-x86 devices running linux now than x86

Cybernetic Vermin
Apr 18, 2005

i would like to use this opportunity to again whine about how low-level and monolithic the design that made that happen is (i.e. wanting to surface a notification from openssh leading to a bunch of poo poo, almost none performance-critical or low-level in nature, getting loaded up together), but i do also realize that at the point where you're packaging up and shipping the attackers malicious code the defense-in-depth does get pretty hard to do.

fresh_cheese
Jul 2, 2014

MY KPI IS HOW MANY VP NUTS I SUCK IN A FISCAL YEAR AND MY LAST THREE OFFICE CHAIRS COMMITTED SUICIDE
are you advocating that sshd would be better as a collection of microservices?

pseudorandom name
May 6, 2007

he's saying that sshd should be a statically linked unikernel

BobHoward
Feb 13, 2012

The only thing white people deserve is a bullet to their empty skull
i think that perhaps if you want a critical security component like sshd to actually be secure, it should never be linked against arbitrary libraries it doesn't actually use? that's not an argument for microservices it just makes sense

FlapYoJacks
Feb 12, 2009
SSHD should be added TO the kernel. :colbert:

shackleford
Sep 4, 2006

it would have prevented this backdoor given that the kernel implementation of xz wasn't compromised

Beeftweeter
Jun 28, 2005

a medium-format picture of beeftweeter staring silently at the camera, a quizzical expression on his face

FlapYoJacks posted:

SSHD should be added TO the kernel. :colbert:

you know, i'm thinking it really should

Progressive JPEG
Feb 19, 2003

bring back khttpd

fresh_cheese
Jul 2, 2014

MY KPI IS HOW MANY VP NUTS I SUCK IN A FISCAL YEAR AND MY LAST THREE OFFICE CHAIRS COMMITTED SUICIDE
ak ok i took that the wrong way then

yes static linking ssh , scp, sshd makes sense to me

but then what about pam? getty? sssd? how deep is the rabbit hole?

Progressive JPEG
Feb 19, 2003

hey buddy here's a quarter buy yourself a terabyte of disk space

fresh_cheese
Jul 2, 2014

MY KPI IS HOW MANY VP NUTS I SUCK IN A FISCAL YEAR AND MY LAST THREE OFFICE CHAIRS COMMITTED SUICIDE
yea disk is cheap and it isnt even disk anymore

Cybernetic Vermin
Apr 18, 2005

mostly i think there is some thinking one can do here about how to structure these pieces, thinking which probably can go beyond whether we link statically or dynamically. there could be sensible ways of further segregating crypto and auth stuff even further from the rest of the system. but no real clear ideas, mostly just a real annoying path this backdoor took.

sb hermit
Dec 13, 2016





why not just stop development on these things unless it’s critically important

like unless you need to forward port yet another piece of code to be compliant with compiler changes or to take advantage of new hardware, just leave it the hell alone

but I’m sure the next evolution of all this poo poo is to dump the work on the GPU which will lead to yet more insane exploits because of the increased attack surface

sb hermit
Dec 13, 2016





to be serious, though, hopefully any test blobs of any appreciable size will come with tools to show how the blobs were created in the first place or maybe even dynamically generated on the fly

spankmeister
Jun 15, 2008






fresh_cheese posted:

yea disk is cheap and it isnt even disk anymore

it's not just disk, but memory too. the kernel can keep a single copy of each library in memory, shared between all processes using it.

Progressive JPEG
Feb 19, 2003

wow that'll save a whole 4 megs

Athas
Aug 6, 2007

fuck that joker

fresh_cheese posted:

ak ok i took that the wrong way then

yes static linking ssh , scp, sshd makes sense to me

but then what about pam? getty? sssd? how deep is the rabbit hole?

None of these should be dynamically linked. They should offer their services through domain sockets or some other IPC mechanism. Dynamic linking is sometimes necessary for performance reasons, but PAM poo poo and DNS lookups is not that critical. Sharing address spaces sucks.

fresh_cheese
Jul 2, 2014

MY KPI IS HOW MANY VP NUTS I SUCK IN A FISCAL YEAR AND MY LAST THREE OFFICE CHAIRS COMMITTED SUICIDE
if we are not going to fix “rando email address takes maintainership of foundation library because they were there on the day it came up”


then we have to figure out how to compartmentalize dependancies so some stupid “left pad” maintainership change does not affect the entire ecosystem.


im starting to think the altruistic open source movement based on assuming the best about all participants was not a great plan. many eyes do not make all bugs shallow, it turns out.

Adbot
ADBOT LOVES YOU

Athas
Aug 6, 2007

fuck that joker
It is cool & wise to think about how to avoid problems like this, but it is equally lame & foolish to ignore that naive trust in altruistic open source developers has historically gone pretty well most of the time. Thisisfine.jpg, but the house isn't on fire, there's just some mold and maybe a candle next to some drapes.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply