Subject: [alpine-devel] xzdiff in xz package is broken on alpine linux

From: Michael Fox <[email-masked]>
Date: Mon, 29 Jun 2015 08:27:56 -0700

I found this by running make check. It's probably due to xzdiff being a
shell script with some hair regexs and musl having some regex
incompatibilities in the name of standards compliance.

Another one that bit me today was in a script with

    grep '^[a-z]\+'

On alpine you need to say

    grep '^[a-z]+'

-- 

-
Michael

---

From: Michael Fox <[email-masked]>
Date: Mon, 29 Jun 2015 10:07:59 -0700

CC xz maintainer whose email I got wrong the first time

-- 

-
Michael

---

From: Natanael Copa <[email-masked]>
Date: Tue, 30 Jun 2015 16:22:41 +0200

On Mon, 29 Jun 2015 10:07:59 -0700

> > Another one that bit me today was in a script with

\+ is a gnu extension and not a posix Basic Regular Expression (BRE).
You cannot expect those work anywhere except with GNU.

> > On alpine you need to say

You need -E for extended regepx (ERE):

  grep -E '^[a-z]+'

That should be portable.

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Lasse Collin <[email-masked]>
Date: Tue, 30 Jun 2015 18:38:40 +0300

On 2015-06-29 Michael Fox wrote:
> > I found this by running make check. It's probably due to xzdiff
> > being a shell script with some hair regexs and musl having some
> > regex incompatibilities in the name of standards compliance.

Thanks. It wouldn't be a surprise if xzdiff (or xzgrep) has a bug. A
little more information would be useful to find out what exactly is
causing a problem. :-) I already checked for \+ or + in regexes and
there are none in xzdiff or xzgrep.

-- 
Lasse Collin  |  IRC: Larhzu @ IRCnet & Freenode

---

---

From: Michael Fox <[email-masked]>
Date: Tue, 30 Jun 2015 09:20:09 -0700

I have to admit the xzdiff script is a little beyond me but the way xzdiff
fails is you compare two different files and it just returns 0 as if they
were the same. I think the easiest way to see it is to download  an alpine
linux live CD:

http://alpinelinux.org/downloads/

To make it run in a VM takes just a few minutes. Here is complete steps to
make it simple as possible:

In virtualbox create a new VM, Type: Linux, Version: Linux 2.6 (64 bit);
Memory 4096; Do not add a virtual hard drive; Start, select the alpine iso;

Login: root
setup-interfaces (enter, enter, enter)
ifup eth0
setup-dns (enter, enter)
setup-apkrepos (enter)
apk add alpine-sdk
wget http://tukaani.org/xz/xz-5.2.1.tar.gz
tar xf xz-5.2.1.tar.gz
cd xz-5.2.1
./configure
make && make check

-- 

-
Michael

---

From: Michael Fox <[email-masked]>
Date: Tue, 30 Jun 2015 10:52:20 -0700

Here is a diff of the command: /bin/sh -x xzdiff good-1-lzma2-1.xz
good-2-lzma2.xz 2>good.out

diff good.out bad.out

where the bad version comes from alpine:

< + xz='xz --format=auto'
---
> + xz=xz --format=auto
3d2
< + case ${0##*/} in
6,7c5,6
< + version='xzdiff (XZ Utils) 5.1.0alpha'
< + usage='Usage: xzdiff [OPTION]... FILE1 [FILE2]
---
> + version=xzdiff (XZ Utils) 5.2.1
> + usage=Usage: xzdiff [OPTION]... FILE1 [FILE2]
12c11,16
< Do comparisons like '\''diff'\'' does.  OPTIONs are the same as for
'\''diff'\''.
---
> Do comparisons like 'diff' does.  OPTIONs are the same as for 'diff'.
>
> Report bugs to <[email-masked]>.
> + escape=
>   s/'/'\\''/g
>   $s/X$/'/
14,18d17
< Report bugs to <[email-masked]>.'
< + escape='
<   s/'\''/'\''\\'\'''\''/g
<   $s/X$/'\''/
< '
20d18
< + case $1 in
22,28c20,26
< + cmp='diff --'
< + for file in '"$@"'
< + test Xgood-1-lzma2-1.xz = X-
< + for file in '"$@"'
< + test Xgood-2-lzma2.xz = X-
< + xz1='xz --format=auto'
< + xz2='xz --format=auto'
---
> + cmp=diff --
> + test Xfiles/good-1-check-crc32.xz = X-
> +
> + test Xfiles/good-2-lzma2.xz = X-
> +
> + xz1=xz --format=auto
> + xz2=xz --format=auto
33,37c31
< + case $1 in
< + case $2 in
< + case $1 in
< + case "$2" in
< + test good-1-lzma2-1.xzgood-2-lzma2.xz = --
---
> + test files/good-1-check-crc32.xzfiles/good-2-lzma2.xz = --
38a33
> + eval diff -- /dev/fd/5 -
40,51c35,44
< + eval 'diff --' /dev/fd/5 -
< ++ exec
< ++ xz --format=auto -cdfq -- good-1-lzma2-1.xz
< ++ xz --format=auto -cdfq -- good-2-lzma2.xz
< ++ eval 'diff --' /dev/fd/5 -
< +++ diff -- /dev/fd/5 -
< ++ echo 0
< ++ echo 0
< + xz_status='0
< 0'
< + cmp_status=1
< + case $xz_status in
---
> + exec
> + xz --format=auto -cdfq -- files/good-1-check-crc32.xz
> + eval diff -- /dev/fd/5 -
> + diff -- /dev/fd/5 -
> + xz --format=auto -cdfq -- files/good-2-lzma2.xz
> + echo 0
> + echo 0
> + xz_status=0
> 0
> + cmp_status=0
53,54c46,47
< + exit 1
< + cmp_status=1
---
> + exit 0
> + cmp_status=0
56c49
< + exit 1
---
> + exit 0

-- 

-
Michael

---

From: Michael Fox <[email-masked]>
Date: Tue, 30 Jun 2015 10:59:23 -0700

Sorry. I got it wrong. Here is a more exact comparison:

diff ubuntu.out ~/alpine.out
1c1
< + xz='xz --format=auto'
---
3d2
< + case ${0##*/} in
6,7c5,6
12c11,16
14,18d17
22,23c20
< + cmp='diff --'
< + for file in '"$@"'
---
> + cmp=diff --
25c22
< + for file in '"$@"'
---
27,28c24,26
> + xz2=xz --format=auto
33,36d30
< + case $1 in
< + case $2 in
< + case $1 in
< + case "$2" in
38a33
40,51c35,44
> + exec
> + xz --format=auto -cdfq -- good-1-lzma2-1.xz
> + xz --format=auto -cdfq -- good-2-lzma2.xz
> + echo 0
> + echo 0
> + xz_status=0
> 0
> + cmp_status=0
53,54c46,47
< + exit 1
< + cmp_status=1
---
56c49
< + exit 1
---

On Tue, Jun 30, 2015 at 10:52 AM, Michael Fox <[email-masked]> wrote:

>> I have to admit the xzdiff script is a little beyond me but the way
>> xzdiff fails is you compare two different files and it just returns 0 as if
>> they were the same. I think the easiest way to see it is to download  an
>> alpine linux live CD:
>>
>> To make it run in a VM takes just a few minutes. Here is complete steps
>> to make it simple as possible:
>>

-- 

-
Michael

---

From: Lasse Collin <[email-masked]>
Date: Fri, 3 Jul 2015 21:31:48 +0300

On 2015-06-30 Michael Fox wrote:
> xzdiff fails is you compare two different files and it just returns 0
> as if they were the same. I think the easiest way to see it is to
> download  an alpine linux live CD:
> 
> To make it run in a VM takes just a few minutes. Here is complete
> steps to make it simple as possible:

Thanks. The diffs you sent didn't help, so I tested it under Alpine
Linux as you suggested. The problem is BusyBox diff which doesn't work
correctly if both arguments are pipes. Installing GNU diffutils makes
the problem go away.

Here is a simple test case using bash and other shells that support
the <(foo) syntax:

    bash$ echo foo | diff <(echo bar) - ; echo $?
    1c1
    < bar
    ---
    > foo
    1

Switching to BusyBox diff:

    bash$ echo foo | busybox diff <(echo bar) - ; echo $?
    0

For me it would be convenient if you could report the bug to the BusyBox
developers, but I can do it too if you prefer so. Thanks!

---

---

From: Michael Fox <[email-masked]>
Date: Fri, 3 Jul 2015 14:41:13 -0700

Do you think it's a bug in busybox? I think they follow this opengroup
standard

http://pubs.opengroup.org/onlinepubs/009695399/utilities/diff.html

It's hard to tell from the description of diff in that document whether and
how it should deal with two stdin pipes.

I'll CC their mailing list and see what they say.

-- 

-
Michael

---

From: Michael Fox <[email-masked]>
Date: Fri, 3 Jul 2015 15:26:31 -0700

I was hoping this might work:

    diff <(echo foo) <(echo bar)

But no. This part of the standard describes a subshell environment but
scrolling up it gives grammar:

io_file          : '<'       filename
                 | LESSAND   filename
                 | '>'       filename
                 | GREATAND  filename
                 | DGREAT    filename
                 | LESSGREAT filename
                 | CLOBBER   filename

The terminal `filename` sort of implies that it doesn't describe subshell
output to an anonymous file the way bash does.

On Fri, Jul 3, 2015 at 2:41 PM, Michael Fox <[email-masked]> wrote:

> It's hard to tell from the description of diff in that document whether
> and how it should deal with two stdin pipes.
>

-- 

-
Michael

---

From: Lasse Collin <[email-masked]>
Date: Sat, 4 Jul 2015 12:21:54 +0300

On 2015-07-03 Michael Fox wrote:
> It's hard to tell from the description of diff in that document
> whether and how it should deal with two stdin pipes.

There is just one stdin which often (but not always) is a pipe.

Both POSIX.1-2001 (your link) and the latest version (POSIX.1-2008)

    http://pubs.opengroup.org/onlinepubs/9699919799/utilities/diff.html

say that diff should work with pipes:

    Many early implementations of diff require seekable files. Since
    the System Interfaces volume of POSIX.1-2008 supports named pipes,
    the standard developers decided that such a restriction was
    unreasonable. Note also that the allowed filename - almost always
    refers to a pipe.

> scrolling up it gives grammar:

Seems that I confused you with the bash-specific syntax. Sorry about
that. The syntax used with the shell has nothing to do with diff. Here
is a test case using POSIX sh:

    sh$ mkfifo fifo
    sh$ echo foo > fifo &
    sh$ echo bar | diff fifo -
    1c1
    < foo
    ---
    > bar
    sh$ echo $?
    1

    sh$ mkfifo fifo
    sh$ echo foo > fifo &
    sh$ echo bar | busybox diff fifo -
    sh$ echo $?
    0

Or:

    sh$ mkfifo fifo fifo2
    sh$ echo foo > fifo &
    sh$ echo bar > fifo2 &
    sh$ busybox diff fifo fifo2
    sh$ echo $?
    0

diff from BusyBox 1.23.2 doesn't work if both arguments are pipes.

---

---

From: Michael Fox <[email-masked]>
Date: Sat, 4 Jul 2015 04:16:45 -0700

It's easy to get confused on a system with multiple shells and utilities
installed so I'm trying it on a fresh minimal alpline livecd. It is exactly
as you say.

I think busybox has a bug. I hope someone from their list can comment.

-- 

-
Michael

---

From: Michael Fox <[email-masked]>
Date: Sat, 4 Jul 2015 07:49:09 -0700

Looks like it's fixed in busybox/master; I confirmed that it is broken as
of 1.23.2 which is the latest release alpine uses.

-- 

-
Michael