~alpine/devel

11 3

[alpine-devel] [PATCH 1/5] abuild: use id instead of whoami

Sören Tempel
Details
Message ID
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net>
Sender timestamp
1448581843
DKIM signature
missing
Download raw message
Patch: +1 -1
Furthermore compare the UID instead of the username.
---
 abuild.in | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/abuild.in b/abuild.in
index 38d6efd..697a186 100644
--- a/abuild.in
+++ b/abuild.in
@@ -2144,7 +2144,7 @@ done
 shift $(( $OPTIND - 1 ))
 
 # check so we are not root
-if [ "$(whoami)" = "root" ] && [ -z "$FAKEROOTKEY" ]; then
+if [ $(id -u) -eq 0 ] && [ -z "$FAKEROOTKEY" ]; then
 	[ -z "$forceroot" ] && die "Do not run abuild as root"
 	FAKEROOT=
 fi
-- 
2.6.3



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] [PATCH 2/5] abuild: there is no man page section 9

Sören Tempel
Details
Message ID
<1448581847-30376-2-git-send-email-soeren+git@soeren-tempel.net>
In-Reply-To
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448581844
DKIM signature
missing
Download raw message
Patch: +3 -3
---
 abuild.in | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/abuild.in b/abuild.in
index 697a186..4c45504 100644
--- a/abuild.in
+++ b/abuild.in
@@ -591,7 +591,7 @@ postcheck() {
 			warning "Found /usr/share/man but package name doesn't end with -doc"
 		fi
 		# check for uncompressed man pages
-		i=$(find "$dir"/usr/share/man -name '*.[0-9]' -type f | sed "s|^$dir|\t|")
+		i=$(find "$dir"/usr/share/man -name '*.[0-8]' -type f | sed "s|^$dir|\t|")
 		if [ -n "$i" ]; then
 			error "Found uncompressed man pages:"
 			echo "$i"
@@ -1405,7 +1405,7 @@ default_doc() {
 	# compress man pages
 	local previnode= prevname=  mandir="$subpkgdir"/usr/share/man
 	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
-		-type f \( -name \*.[0-9n] -o -name \*.[0-9][a-z]* \) \
+		-type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
 		-exec stat -c "%i %n" {} \; | sort -n \
 		| while read inode name; do
 
@@ -1421,7 +1421,7 @@ default_doc() {
 		prevname="$name"
 	done
 	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
-		-type l \( -name \*.[0-9n] -o -name \*.[0-9][a-z]* \) \
+		-type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
 		| while read symlink; do
 
 		ln -s $(readlink $symlink).gz "$symlink".gz
-- 
2.6.3



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] [PATCH 3/5] abuild: rewrite hardlink handling when compressing man pages

Sören Tempel
Details
Message ID
<1448581847-30376-3-git-send-email-soeren+git@soeren-tempel.net>
In-Reply-To
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448581845
DKIM signature
missing
Download raw message
Patch: +22 -18
The problem is that gzip refuses to run if it detects that a file has
more than 1 link. Our existing solution (removing hardlinks, compressing
the man page and recreating the hardlinks) made certain assumptions
about inode order that are only given on Unix v7 like filesystems
meaning it didn't work properly on 'tree-based' filesystems like BTRFS
or ZFS.

This patch has a different more bulletproof approach: It simply replaces
all hardlinks with symlinks. This is way easier because symlinks (unlike
hardlinks) can point to a file that doesn't exist, therefore we can
update all links before compressing the file in an easy way.
---
 abuild.in | 40 ++++++++++++++++++++++------------------
 1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/abuild.in b/abuild.in
index 4c45504..26b7982 100644
--- a/abuild.in
+++ b/abuild.in
@@ -1403,29 +1403,33 @@ default_doc() {
 	done
 
 	# compress man pages
-	local previnode= prevname=  mandir="$subpkgdir"/usr/share/man
-	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
-		-type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
-		-exec stat -c "%i %n" {} \; | sort -n \
-		| while read inode name; do
-
-		if [ "$inode" = "$previnode" ]; then
-			# update hard link
-			rm "$name"
-			ln "$prevname".gz "$name".gz
-		else
-			gzip -9 "$name"
-		fi
+	local mandir="$subpkgdir"/usr/share/man
+	[ -d "$mandir" ] && find "$mandir" -type f \
+		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
+		-exec stat -c "%i %n" \{\} \; | while read inode name; do
+
+		# Skip hardlinks removed in last iteration.
+		[ -f "$name" ] || continue
+
+		local islink=0
+		find "$mandir" -type f -links +1 \
+			-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
+			-exec stat -c "%i %n" \{\} \; | while read linode lname; do
+			if [ "$linode" = "$inode" -a "$lname" != "$name" ]; then
+				islink=1
+				rm -f "$lname"
+				ln -s "${name##*/}".gz "$lname".gz
+			fi
+		done
 
-		previnode="$inode"
-		prevname="$name"
+		[ $islink -eq 0 ] && gzip -9 "$name"
 	done
-	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
-		-type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
+	[ -d "$mandir" ] && find "$mandir" -type l \
+		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
 		| while read symlink; do
 
 		ln -s $(readlink $symlink).gz "$symlink".gz
-		rm "$symlink"
+		rm -f "$symlink"
 	done
 
 	rm -f "$subpkgdir/usr/share/info/dir"
-- 
2.6.3



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] [PATCH 4/5] abuild: update symlinks before updating hardlinks

Sören Tempel
Details
Message ID
<1448581847-30376-4-git-send-email-soeren+git@soeren-tempel.net>
In-Reply-To
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448581846
DKIM signature
missing
Download raw message
Patch: +7 -7
This should be a minimal performance improvement since hardlinks are
replaced with symlinks and should thus already point to the correct
file.
---
 abuild.in | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/abuild.in b/abuild.in
index 26b7982..65b6377 100644
--- a/abuild.in
+++ b/abuild.in
@@ -1404,6 +1404,13 @@ default_doc() {
 
 	# compress man pages
 	local mandir="$subpkgdir"/usr/share/man
+	[ -d "$mandir" ] && find "$mandir" -type l \
+		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
+		| while read symlink; do
+
+		ln -s $(readlink $symlink).gz "$symlink".gz
+		rm -f "$symlink"
+	done
 	[ -d "$mandir" ] && find "$mandir" -type f \
 		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
 		-exec stat -c "%i %n" \{\} \; | while read inode name; do
@@ -1424,13 +1431,6 @@ default_doc() {
 
 		[ $islink -eq 0 ] && gzip -9 "$name"
 	done
-	[ -d "$mandir" ] && find "$mandir" -type l \
-		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
-		| while read symlink; do
-
-		ln -s $(readlink $symlink).gz "$symlink".gz
-		rm -f "$symlink"
-	done
 
 	rm -f "$subpkgdir/usr/share/info/dir"
 
-- 
2.6.3



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] [PATCH 5/5] add abuild-fetch to the .gitignore file

Sören Tempel
Details
Message ID
<1448581847-30376-5-git-send-email-soeren+git@soeren-tempel.net>
In-Reply-To
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448581847
DKIM signature
missing
Download raw message
Patch: +1 -0
---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitignore b/.gitignore
index 9018782..ac669f6 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 *.tar.bz2
 *.o
 abuild
+abuild-fetch
 abuild-keygen
 abuild-sign
 abuild-sudo
-- 
2.6.3



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Natanael Copa
Details
Message ID
<20151130081331.34a219fd@ncopa-desktop.alpinelinux.org>
In-Reply-To
<1448581847-30376-1-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448867611
DKIM signature
missing
Download raw message
On Fri, 27 Nov 2015 00:50:43 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:

> Furthermore compare the UID instead of the username.
> ---
>  abuild.in | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/abuild.in b/abuild.in
> index 38d6efd..697a186 100644
> --- a/abuild.in
> +++ b/abuild.in
> @@ -2144,7 +2144,7 @@ done
>  shift $(( $OPTIND - 1 ))
>  
>  # check so we are not root
> -if [ "$(whoami)" = "root" ] && [ -z "$FAKEROOTKEY" ]; then
> +if [ $(id -u) -eq 0 ] && [ -z "$FAKEROOTKEY" ]; then
>  	[ -z "$forceroot" ] && die "Do not run abuild as root"
>  	FAKEROOT=
>  fi

Those patches comes in too late. We have already built world now for
v3.3. I don't want modify abuild unless it fixes something that is
clearly broken at this point as it may result in different reulst of
the built package.

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Re: [alpine-devel] [PATCH 2/5] abuild: there is no man page section 9

Natanael Copa
Details
Message ID
<20151130081426.1eb3b240@ncopa-desktop.alpinelinux.org>
In-Reply-To
<1448581847-30376-2-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448867666
DKIM signature
missing
Download raw message
On Fri, 27 Nov 2015 00:50:44 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:

> ---
>  abuild.in | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/abuild.in b/abuild.in
> index 697a186..4c45504 100644
> --- a/abuild.in
> +++ b/abuild.in
> @@ -591,7 +591,7 @@ postcheck() {
>  			warning "Found /usr/share/man but package name doesn't end with -doc"
>  		fi
>  		# check for uncompressed man pages
> -		i=$(find "$dir"/usr/share/man -name '*.[0-9]' -type f | sed "s|^$dir|\t|")
> +		i=$(find "$dir"/usr/share/man -name '*.[0-8]' -type f | sed "s|^$dir|\t|")
>  		if [ -n "$i" ]; then
>  			error "Found uncompressed man pages:"
>  			echo "$i"
> @@ -1405,7 +1405,7 @@ default_doc() {
>  	# compress man pages
>  	local previnode= prevname=  mandir="$subpkgdir"/usr/share/man
>  	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
> -		-type f \( -name \*.[0-9n] -o -name \*.[0-9][a-z]* \) \
> +		-type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
>  		-exec stat -c "%i %n" {} \; | sort -n \
>  		| while read inode name; do
>  
> @@ -1421,7 +1421,7 @@ default_doc() {
>  		prevname="$name"
>  	done
>  	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
> -		-type l \( -name \*.[0-9n] -o -name \*.[0-9][a-z]* \) \
> +		-type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
>  		| while read symlink; do
>  
>  		ln -s $(readlink $symlink).gz "$symlink".gz

What problem does this fix? Does it hurt to check for [0-9] instead of
[0-8] in any way?

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Re: [alpine-devel] [PATCH 3/5] abuild: rewrite hardlink handling when compressing man pages

Natanael Copa
Details
Message ID
<20151130082137.00517e0e@ncopa-desktop.alpinelinux.org>
In-Reply-To
<1448581847-30376-3-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448868097
DKIM signature
missing
Download raw message
On Fri, 27 Nov 2015 00:50:45 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:

> The problem is that gzip refuses to run if it detects that a file has
> more than 1 link. Our existing solution (removing hardlinks, compressing
> the man page and recreating the hardlinks) made certain assumptions
> about inode order that are only given on Unix v7 like filesystems
> meaning it didn't work properly on 'tree-based' filesystems like BTRFS
> or ZFS.
> 
> This patch has a different more bulletproof approach: It simply replaces
> all hardlinks with symlinks. This is way easier because symlinks (unlike
> hardlinks) can point to a file that doesn't exist, therefore we can
> update all links before compressing the file in an easy way.

How well it this tested?

The other approach has build entire world at this point.

I am not very happy about rebuilding world again for this change.

-nc

> ---
>  abuild.in | 40 ++++++++++++++++++++++------------------
>  1 file changed, 22 insertions(+), 18 deletions(-)
> 
> diff --git a/abuild.in b/abuild.in
> index 4c45504..26b7982 100644
> --- a/abuild.in
> +++ b/abuild.in
> @@ -1403,29 +1403,33 @@ default_doc() {
>  	done
>  
>  	# compress man pages
> -	local previnode= prevname=  mandir="$subpkgdir"/usr/share/man
> -	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
> -		-type f \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
> -		-exec stat -c "%i %n" {} \; | sort -n \
> -		| while read inode name; do
> -
> -		if [ "$inode" = "$previnode" ]; then
> -			# update hard link
> -			rm "$name"
> -			ln "$prevname".gz "$name".gz
> -		else
> -			gzip -9 "$name"
> -		fi
> +	local mandir="$subpkgdir"/usr/share/man
> +	[ -d "$mandir" ] && find "$mandir" -type f \
> +		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
> +		-exec stat -c "%i %n" \{\} \; | while read inode name; do
> +
> +		# Skip hardlinks removed in last iteration.
> +		[ -f "$name" ] || continue
> +
> +		local islink=0
> +		find "$mandir" -type f -links +1 \
> +			-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
> +			-exec stat -c "%i %n" \{\} \; | while read linode lname; do
> +			if [ "$linode" = "$inode" -a "$lname" != "$name" ]; then
> +				islink=1
> +				rm -f "$lname"
> +				ln -s "${name##*/}".gz "$lname".gz
> +			fi
> +		done

How is performance on this? It looks like this can be horribly slow if
$mandir is huge.

Could you try locate the biggest -doc package and run this on armhf and
post a specific number?


>  
> -		previnode="$inode"
> -		prevname="$name"
> +		[ $islink -eq 0 ] && gzip -9 "$name"
>  	done
> -	[ -d "$mandir" ] && find "$subpkgdir"/usr/share/man \
> -		-type l \( -name \*.[0-8n] -o -name \*.[0-8][a-z]*
> \) \
> +	[ -d "$mandir" ] && find "$mandir" -type l \
> +		-a \( -name \*.[0-8n] -o -name \*.[0-8][a-z]* \) \
>  		| while read symlink; do
>  
>  		ln -s $(readlink $symlink).gz "$symlink".gz
> -		rm "$symlink"
> +		rm -f "$symlink"
>  	done
>  
>  	rm -f "$subpkgdir/usr/share/info/dir"

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Re: [alpine-devel] [PATCH 5/5] add abuild-fetch to the .gitignore file

Natanael Copa
Details
Message ID
<20151130082734.5e367119@ncopa-desktop.alpinelinux.org>
In-Reply-To
<1448581847-30376-5-git-send-email-soeren+git@soeren-tempel.net> (view parent)
Sender timestamp
1448868454
DKIM signature
missing
Download raw message
On Fri, 27 Nov 2015 00:50:47 +0100
Sören Tempel <soeren+git@soeren-tempel.net> wrote:

> ---
>  .gitignore | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/.gitignore b/.gitignore
> index 9018782..ac669f6 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -1,6 +1,7 @@
>  *.tar.bz2
>  *.o
>  abuild
> +abuild-fetch
>  abuild-keygen
>  abuild-sign
>  abuild-sudo

applied. thanks!

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Re: [alpine-devel] [PATCH 3/5] abuild: rewrite hardlink handling when compressing man pages

Sören Tempel
Details
Message ID
<20151130121911.GA32221@francium.fritz.box>
In-Reply-To
<20151130082137.00517e0e@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1448885951
DKIM signature
missing
Download raw message
On 30.11.15, Natanael Copa wrote:
> How well it this tested?
> 
> The other approach has build entire world at this point.
> 
> I am not very happy about rebuilding world again for this change.

I built a few packages with it: bind, nvi, ... so it's not as well
tested as the previous approach but I can guarantee that the previous
approach doesn't work on ZFS and BTRFS.

I would suggest that you apply this after 3.3 is released.

Sören.


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Re: [alpine-devel] [PATCH 2/5] abuild: there is no man page section 9

Sören Tempel
Details
Message ID
<20151130122040.GB32221@francium.fritz.box>
In-Reply-To
<20151130081426.1eb3b240@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1448886040
DKIM signature
missing
Download raw message
On 30.11.15, Natanael Copa wrote:
> What problem does this fix? Does it hurt to check for [0-9] instead of
> [0-8] in any way?

Nope it doesn't hurt, but it shouldn't be necessary.

Sören.


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Sören Tempel
Details
Message ID
<20151130122153.GC32221@francium.fritz.box>
In-Reply-To
<20151130081331.34a219fd@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1448886113
DKIM signature
missing
Download raw message
On 30.11.15, Natanael Copa wrote:
> Those patches comes in too late. We have already built world now for
> v3.3. I don't want modify abuild unless it fixes something that is
> clearly broken at this point as it may result in different reulst of
> the built package.

Totally understand that, I would suggest that you apply them after 3.3
is released or just apply them to the git repository but don't make a
new release of the main/abuild aport.

Sören.


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---