The problem is that gzip refuses to run if it detects that a file has
more than 1 link. Our existing solution (removing hardlinks, compressing
the man page and recreating the hardlinks) made certain assumptions
about inode order that are only given on Unix v7 like filesystems
meaning it didn't work properly on 'tree-based' filesystems like BTRFS
or ZFS.
This patch has a different more bulletproof approach: It simply replaces
all hardlinks with symlinks. This is way easier because symlinks (unlike
hardlinks) can point to a file that doesn't exist, therefore we can
update all links before compressing the file in an easy way.
---
abuild.in | 40 ++++++++++++++++++++++------------------
1 file changed, 22 insertions(+), 18 deletions(-)
diff --git a/abuild.in b/abuild.in
index 4c45504..26b7982 100644
--- a/abuild.in+++ b/abuild.in
@@ -1403,29 +1403,33 @@ default_doc() {
done
# compress man pages
- local previnode= prevname= mandir="$subpkgdir"/usr/share/man- [ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \- -type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \- -exec stat -c "%i %n" {} \; | sort -n \- | while read inode name; do-- if [ "$inode" = "$previnode" ]; then- # update hard link- rm "$name"- ln "$prevname".gz "$name".gz- else- gzip -9 "$name"- fi+ local mandir="$subpkgdir"/usr/share/man+ [ -d "$mandir" ] && find "$mandir" -type f \+ -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \+ -exec stat -c "%i %n" \{\} \; | while read inode name; do++ # Skip hardlinks removed in last iteration.+ [ -f "$name" ] || continue++ local islink=0+ find "$mandir" -type f -links +1 \+ -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \+ -exec stat -c "%i %n" \{\} \; | while read linode lname; do+ if [ "$linode" = "$inode" -a "$lname" != "$name" ]; then+ islink=1+ rm -f "$lname"+ ln -s "${name##*/}".gz "$lname".gz+ fi+ done- previnode="$inode"- prevname="$name"+ [ $islink -eq 0 ] && gzip -9 "$name" done
- [ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \- -type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \+ [ -d "$mandir" ] && find "$mandir" -type l \+ -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \ | while read symlink; do
ln -s $(readlink $symlink).gz "$symlink".gz
- rm "$symlink"+ rm -f "$symlink" done
rm -f "$subpkgdir/usr/share/info/dir"
--
2.6.3
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
[alpine-devel] [PATCH 4/5] abuild: update symlinks before updating hardlinks
On Fri, 27 Nov 2015 00:50:43 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:
> Furthermore compare the UID instead of the username.> ---> abuild.in | 2 +-> 1 file changed, 1 insertion(+), 1 deletion(-)> > diff --git a/abuild.in b/abuild.in> index 38d6efd..697a186 100644> --- a/abuild.in> +++ b/abuild.in> @@ -2144,7 +2144,7 @@ done> shift $(( $OPTIND - 1 ))> > # check so we are not root> -if [ "$(whoami)" = "root" ] && [ -z "$FAKEROOTKEY" ]; then> +if [ $(id -u) -eq 0 ] && [ -z "$FAKEROOTKEY" ]; then> [ -z "$forceroot" ] && die "Do not run abuild as root"> FAKEROOT=> fi
Those patches comes in too late. We have already built world now for
v3.3. I don't want modify abuild unless it fixes something that is
clearly broken at this point as it may result in different reulst of
the built package.
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] [PATCH 2/5] abuild: there is no man page section 9
On Fri, 27 Nov 2015 00:50:45 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:
> The problem is that gzip refuses to run if it detects that a file has> more than 1 link. Our existing solution (removing hardlinks, compressing> the man page and recreating the hardlinks) made certain assumptions> about inode order that are only given on Unix v7 like filesystems> meaning it didn't work properly on 'tree-based' filesystems like BTRFS> or ZFS.> > This patch has a different more bulletproof approach: It simply replaces> all hardlinks with symlinks. This is way easier because symlinks (unlike> hardlinks) can point to a file that doesn't exist, therefore we can> update all links before compressing the file in an easy way.
How well it this tested?
The other approach has build entire world at this point.
I am not very happy about rebuilding world again for this change.
-nc
> ---> abuild.in | 40 ++++++++++++++++++++++------------------> 1 file changed, 22 insertions(+), 18 deletions(-)> > diff --git a/abuild.in b/abuild.in> index 4c45504..26b7982 100644> --- a/abuild.in> +++ b/abuild.in> @@ -1403,29 +1403,33 @@ default_doc() {> done> > # compress man pages> - local previnode= prevname= mandir="$subpkgdir"/usr/share/man> - [ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \> - -type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \> - -exec stat -c "%i %n" {} \; | sort -n \> - | while read inode name; do> -> - if [ "$inode" = "$previnode" ]; then> - # update hard link> - rm "$name"> - ln "$prevname".gz "$name".gz> - else> - gzip -9 "$name"> - fi> + local mandir="$subpkgdir"/usr/share/man> + [ -d "$mandir" ] && find "$mandir" -type f \> + -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \> + -exec stat -c "%i %n" \{\} \; | while read inode name; do> +> + # Skip hardlinks removed in last iteration.> + [ -f "$name" ] || continue> +> + local islink=0> + find "$mandir" -type f -links +1 \> + -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \> + -exec stat -c "%i %n" \{\} \; | while read linode lname; do> + if [ "$linode" = "$inode" -a "$lname" != "$name" ]; then> + islink=1> + rm -f "$lname"> + ln -s "${name##*/}".gz "$lname".gz> + fi> + done
How is performance on this? It looks like this can be horribly slow if
$mandir is huge.
Could you try locate the biggest -doc package and run this on armhf and
post a specific number?
> > - previnode="$inode"> - prevname="$name"> + [ $islink -eq 0 ] && gzip -9 "$name"> done> - [ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \> - -type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]*> \) \> + [ -d "$mandir" ] && find "$mandir" -type l \> + -a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \> | while read symlink; do> > ln -s $(readlink $symlink).gz "$symlink".gz> - rm "$symlink"> + rm -f "$symlink"> done> > rm -f "$subpkgdir/usr/share/info/dir"
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] [PATCH 5/5] add abuild-fetch to the .gitignore file
On 30.11.15, Natanael Copa wrote:
> How well it this tested?> > The other approach has build entire world at this point.> > I am not very happy about rebuilding world again for this change.
I built a few packages with it: bind, nvi, ... so it's not as well
tested as the previous approach but I can guarantee that the previous
approach doesn't work on ZFS and BTRFS.
I would suggest that you apply this after 3.3 is released.
Sören.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] [PATCH 2/5] abuild: there is no man page section 9
On 30.11.15, Natanael Copa wrote:
> Those patches comes in too late. We have already built world now for> v3.3. I don't want modify abuild unless it fixes something that is> clearly broken at this point as it may result in different reulst of> the built package.
Totally understand that, I would suggest that you apply them after 3.3
is released or just apply them to the git repository but don't make a
new release of the main/abuild aport.
Sören.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---