Subject: kernel 5.5 busybox rm: can't remove: Directory not empty

From: [email-masked]
Date: Thu, 23 Jan 2020 23:39:17 +0100

While trying the kernel 5.5 release candidate for Fedora 31 I'm running into an issue with busybox rm where it seems unable to recursively remove directories with specific content, leaving one seemingly random file.

It doesn't happen with fedora's busybox, going back to kernel 5.4 or installing coreutils fixes the problem.

Steps to reproduce:

$ docker run -ti --rm alpine:3.11.3
/ # uname -a
Linux 258313de78b9 5.5.0-0.rc6.git0.1.fc32.x86_64 #1 SMP Mon Jan 13 18:52:03 UTC 2020 x86_64 Linux
/ # wget https://github.com/michaelrsweet/htmldoc/releases/download/v1.9.7/htmldoc-1.9.7-source.tar.gz
Connecting to github.com (140.82.118.4:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.22.11:443)
saving to 'htmldoc-1.9.7-source.tar.gz'
htmldoc-1.9.7-source 100% |****************************************************************************************************************| 4360k  0:00:00 ETA
'htmldoc-1.9.7-source.tar.gz' saved
/ # tar xf htmldoc-1.9.7-source.tar.gz
/ # rm -rf htmldoc-1.9.7
rm: can't remove 'htmldoc-1.9.7/fonts': Directory not empty
rm: can't remove 'htmldoc-1.9.7/jpeg': Directory not empty
/ # rm -rf htmldoc-1.9.7
/ #

Another reproducer where it breaks building docker php based images that add the mysqli extension:

$ docker run --rm php:alpine docker-php-ext-install mysqli
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/25) Installing m4 (1.4.18-r1)
(2/25) Installing libbz2 (1.0.8-r1)
(3/25) Installing perl (5.30.1-r0)
...
(23/25) Purging mpc1 (1.1.0-r1)
(24/25) Purging mpfr4 (4.0.2-r1)
(25/25) Purging gmp (6.1.2-r1)
Executing busybox-1.31.1-r9.trigger
OK: 19 MiB in 32 packages
rm: can't remove '/usr/src/php/sapi/phpdbg/tests': Directory not empty
rm: can't remove '/usr/src/php/ext/sockets/tests': Directory not empty
rm: can't remove '/usr/src/php/ext/ffi/tests': Directory not empty
rm: can't remove '/usr/src/php/ext/gd/libgd': Directory not empty
rm: can't remove '/usr/src/php/ext/soap/tests/soap12': Directory not empty
rm: can't remove '/usr/src/php/ext/soap/tests/interop/Round3/GroupD': Directory not empty
rm: can't remove '/usr/src/php/ext/mysqlnd': Directory not empty
rm: can't remove '/usr/src/php/ext/ftp/tests': Directory not empty
rm: can't remove '/usr/src/php/ext/phar/tests/tar/files': Directory not empty
rm: can't remove '/usr/src/php/ext/hash/tests': Directory not empty
rm: can't remove '/usr/src/php/ext/standard/tests/network': Directory not empty
rm: can't remove '/usr/src/php/Zend/tests/return_types': Directory not empty
$

---

From: Natanael Copa <[email-masked]>
Date: Fri, 24 Jan 2020 09:40:03 +0100

On Thu, 23 Jan 2020 23:39:17 +0100
[email-masked] wrote:

> While trying the kernel 5.5 release candidate for Fedora 31 I'm
> running into an issue with busybox rm where it seems unable to
> recursively remove directories with specific content, leaving one
> seemingly random file.
> 

I wonder if this is related the lchown problem. There was a long thread
about that on musl libc mailing list: https://www.openwall.com/lists/musl/2020/01/22/7

What filesystem do you use?

Does it happen on other filesystems as well?

Can you reproduce it without docker? You can get the busybox-static.apk from 

http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/busybox-static-1.31.1-r9.apk

(you can use `tar -zxf busybox-static-*.apk bin/busybox.static` to extract it)

Can you reproduce it when running `strace busybox rm -rf`?

Thanks!

> Linux 258313de78b9 5.5.0-0.rc6.git0.1.fc32.x86_64 #1 SMP Mon Jan 13
> 18:52:03 UTC 2020 x86_64 Linux / # wget
> https://github.com/michaelrsweet/htmldoc/releases/download/v1.9.7/htmldoc-1.9.7-source.tar.gz
> Connecting to github.com (140.82.118.4:443) Connecting to
> github-production-release-asset-2e65be.s3.amazonaws.com
> (52.216.22.11:443) saving to 'htmldoc-1.9.7-source.tar.gz'
> htmldoc-1.9.7-source 100%
> |****************************************************************************************************************|
> 4360k  0:00:00 ETA 'htmldoc-1.9.7-source.tar.gz' saved / # tar xf
> htmldoc-1.9.7-source.tar.gz / # rm -rf htmldoc-1.9.7 rm: can't remove
> 'htmldoc-1.9.7/fonts': Directory not empty rm: can't remove
> 'htmldoc-1.9.7/jpeg': Directory not empty / # rm -rf htmldoc-1.9.7
> / #
> 
> 
> fetch
> http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
> fetch
> http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
> (1/25) Installing m4 (1.4.18-r1) (2/25) Installing libbz2 (1.0.8-r1)
> (3/25) Installing perl (5.30.1-r0) ...
> rm: can't remove '/usr/src/php/ext/soap/tests/soap12': Directory not
> empty rm: can't remove
> '/usr/src/php/ext/soap/tests/interop/Round3/GroupD': Directory not
> empty rm: can't remove '/usr/src/php/ext/mysqlnd': Directory not
> empty rm: can't remove '/usr/src/php/ext/ftp/tests': Directory not
> empty rm: can't remove '/usr/src/php/ext/phar/tests/tar/files':
> Directory not empty rm: can't remove '/usr/src/php/ext/hash/tests':
> Directory not empty rm: can't remove
> '/usr/src/php/ext/standard/tests/network': Directory not empty rm:
> can't remove '/usr/src/php/Zend/tests/return_types': Directory not
> empty $

---

From: [email-masked]
Date: Sat, 25 Jan 2020 11:39:32 +0100

> Can you reproduce it when running `strace busybox rm -rf`?

Docker uses overlay2 on ext4 but I can reproduce it outside docker on ext4 as well as xfs.

$ strace ./busybox.static rm -rf htmldoc-1.9.7/fonts 2> trace.txt
$ ls -al htmldoc-1.9.7/fonts
total 36
drwxr-xr-x.  2 foo foo    30 Jan 25 11:27 .
drwxr-xr-x. 14 foo foo  4096 Oct  8 20:56 ..
-rw-r--r--.  1 foo foo 31888 Oct  8 20:56 Times-Italic.afm
$