Subject: [alpine-devel] musl and ARM in 2.7 (and an introduction)

From: "Leslie P. Polzer | PORT ZERO" <[email-masked]>
Date: Fri, 18 Oct 2013 10:52:25 +0200

Hi,

I'm new to Alpine. I've been around GNU/Linux for about 15 years
now as user and developer. During this time I have been working
on various open source projects and have been a community contributor
to Linux From Scratch and ArchLinux.

First let me say that you've done an incredible job! I especially
like the package management system, but there's many more excellent
features that I wouldn't really know where to start. :)

I've come to understand that Alpine 2.7 will be the first version
to feature an ARM port, based on musl as libc. Timo Teräs' efforts
in the past weeks (as per Redmine) also show this. Is this official?

In that case I would like to ask why musl is being used. It certainly
is a great alternative to uclibc, but it appears that the ARM port
is linked to it. I'm seeking to understand the reasoning behind this.

Also, is there an established way already of building Alpine based
on musl and/or cross-compiling for ARM? It doesn't seem to be a big
deal, but I'd like to know the efforts that have already been put
into this so I'm not duplicating any work.

Thanks!

  Leslie

-- 
Dipl.Ing.(BA) Leslie P. Polzer | CTO - PORT ZERO
UG (haftungsbeschränkt)
Softwareentwicklung & IT Engineering
Adalbertstraße 7/8
10999 Berlin
Tel.: +49 (0)30 - 69 200 907 - 0
Fax: +49 (0)30 - 69 200 907 - 9
http://www.port-zero.com

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Timo Teras <[email-masked]>
Date: Fri, 18 Oct 2013 13:16:15 +0300

On Fri, 18 Oct 2013 10:52:25 +0200

> to Linux From Scratch and ArchLinux.

Nice. Welcome!

> features that I wouldn't really know where to start. :)

Thanks :)

> in the past weeks (as per Redmine) also show this. Is this official?

Yes, I am currently doing ARM port. It is undecided if we want to do a
release of it, since it's quite experimental at this point.

> is linked to it. I'm seeking to understand the reasoning behind this.

musl is a lot cleaner, smaller faster and more standards compliant in
many cases. It is also easier to bootstrap to new targets compared to
uclibc/glibc.

Though, uclibc is likely more glibc compatible. We have also quite some
uclibc specific bugs open that are not fixable without doing invasive
uclibc patching. E.g. openjdk7 will not run on standard uclibc. You can
also note that we carry about 40 patches to uclibc already because it
is not maintained as well as glibc/musl. We also have had to fix
ourselves numerous bugs in it, and many of them have been upstreamed.

In the long run we are currently planning to drop uclibc, and replace
it with musl in the primary builds for Alpine 3.0. In case we do not
get everything done for next release date, we might still do
Alpine 2.8 based on uclibc.

> into this so I'm not duplicating any work.

We've patched aports so that a minimal bootstrap system can be cross
compiled. All changes are in git already. And some minor notes + links
to the scripts I used to bootstrap musl builders are at my wiki wall:
http://wiki.alpinelinux.org/wiki/User:Fabled

Currently, about 85-90% of edge/main are built against musl for x86 and
armhf (armv6+vfp target in toolchain). x86_64 is not done yet, but I
expect to do it soon. The remaining 10% of main and testing aports
probably need some patching to build.

The repository for these binary packages is at:
http://nl.alpinelinux.org/alpine/edge-musl/main/
http://nl.alpinelinux.org/alpine/edge-musl/testing/

They are updated sporadically (not synchronized to git like the primary
uclibc builds).

x86 musl build does not have any boot images, but you can do chroot apk
install of it.

For armhf we don't really support any boards yet. But for testing, I've
created a Raspberry Pi image. It's available at:
http://dev.alpinelinux.org/~tteras/alpine-edge-131010-armhf.rpi.tar.gz

I also have Wandboard Quad images but those I have not yet published. I
use Wandboard Quad to native compile the arm binaries.

- Timo

---

---

From: "Leslie P. Polzer | PORT ZERO" <[email-masked]>
Date: Mon, 21 Oct 2013 11:04:17 +0200

> Alpine 2.8 based on uclibc.

Ah, so it's not intimately linked with the ARM port but more of
a general improvement. Good to know.

Thanks a lot for all the information on cross compiling Alpine for ARM!
That's very helpful. I will spend the next days getting acquainted
with the toolchain. I'm also interested in building packages with
clang.

  Leslie

---

---

From: Timo Teras <[email-masked]>
Date: Tue, 22 Oct 2013 08:16:26 +0300

On Mon, 21 Oct 2013 11:04:17 +0200
"Leslie P. Polzer | PORT ZERO" <[email-masked]> wrote:

> > In the long run we are currently planning to drop uclibc, and
> > replace it with musl in the primary builds for Alpine 3.0. In case
> > we do not get everything done for next release date, we might still
> > do Alpine 2.8 based on uclibc.
> 

Correct.

> Thanks a lot for all the information on cross compiling Alpine for
> ARM! That's very helpful. I will spend the next days getting
> acquainted with the toolchain. I'm also interested in building
> packages with clang.

clang is there and built for ARM already. It works since mesa needs it
to build.

However, it's not fully independent build - it uses libgcc, etc. We
have not seen yet the need to build "self hosting" clang. However, this
is an interesting option, and we would appreciate if someone does it.

We are planning to take a look at it at some point too. We might be also
considering switching to clang as main toolchain later when it has
matured a little bit more still.

- Timo

---

---

From: "Leslie P. Polzer | PORT ZERO" <[email-masked]>
Date: Fri, 25 Oct 2013 14:30:48 +0200

> http://wiki.alpinelinux.org/wiki/User:Fabled

That helped! I had to clear out some roadblocks, after which I now have
a successful and reproducible bootstrap crossbuild of i486-alpine-linux-musl
(without KERNEL_PKG and DEBUG_PKG so far; I simply haven't tried).

The problems I encountered, working with your scripts on current edge:

In createcross-toolchain.sh:

* I instructed abuild to clean up the musl build before installing
  headers. This is necessary to prevent the patches from being applied
  multiple times by the "prepare" step if you call the script more
  than once.

In abuild-crossbuild-x86.conf:

* CBUILD is no longer being set in /etc/abuild.conf (this was changed
  pretty recently). This prevented gmp from being built correctly
  (I think it just used the host setting for the build).
  More packages might be affected by this.

In crossbuild-alpine-bootstrap.sh:

* Set up the proper pkgconfig environment for the cross root.
  Without this the relevant .pc files cannot be found. The first
  package that broke in the ordered crossbuild because of this
  was apk-tools.

* Added installation of all packages after they have been built.
  This is a necessary step since later packages require files from
  earlier ones. Also attempt to install $PKG-dev. Some packages need
  special casing here since their -dev packages have a different
  name. Maybe this should be fixed in the packages themselves.
  The packages affected are gmp5 (gmp-dev) and mpfr3 (mpfr-dev).

* Non-essential: also added per-package logging to cut down on the output.

I'm attaching a rough diff against your x86 conf files and the two
scripts. Please let me know how you want to proceed with integrating
these changes.

I wonder how we can test packages for their sanity properly after
we have built them. The usual way is running the testsuite (make check)
after compilation. Obviously this won't work when cross-compiling to
a different machine architecture. Any ideas?

  Leslie

-- 
Dipl.Ing.(BA) Leslie P. Polzer | CTO - PORT ZERO
UG (haftungsbeschränkt)
Softwareentwicklung & IT Engineering
Adalbertstraße 7/8
10999 Berlin
Tel.: +49 (0)30 - 69 200 907 - 0
Fax: +49 (0)30 - 69 200 907 - 9
http://www.port-zero.com
diff --git a/abuild-crossbuild-x86.conf b/abuild-crossbuild-x86.conf
index 205f5f3..7637695 100644
--- a/abuild-crossbuild-x86.conf
+++ b/abuild-crossbuild-x86.conf
@@ -2,7 +2,8 @@
 source /etc/abuild.conf
 unset CARCH CLIBC
 
-# cross compile to ARM
+# cross compile to x86 with musl
+CBUILD=${CHOST} # CBUILD no longer gets set in abuild.conf
 CHOST=i486-alpine-linux-musl
 CTARGET=${CHOST}
 
diff --git a/createcross-toolchain.sh b/createcross-toolchain.sh
old mode 100644
new mode 100755
index 07f1a15..aabff54
--- a/createcross-toolchain.sh
+++ b/createcross-toolchain.sh
@@ -1,6 +1,7 @@
 #!/bin/sh
 
-export TARGET_ARCH=x86_64
+#export TARGET_ARCH=armhf
+export TARGET_ARCH=x86
 export ABUILD_CREATECROSS_CONF=$PWD/abuild-createcross-$TARGET_ARCH.conf
 export ABUILD_CROSSBUILD_CONF=$PWD/abuild-crossbuild-$TARGET_ARCH.conf
 export APORTS=$HOME/aports
@@ -35,7 +36,7 @@ if [ $? -ne 0 ]; then
 	# Step 4. C-library headers for target
 	#
 	cd $APORTS/main/musl
-	ABUILD_CONF=$ABUILD_CROSSBUILD_CONF abuild unpack prepare install_sysroot_headers || return 1
+	ABUILD_CONF=$ABUILD_CROSSBUILD_CONF abuild clean unpack prepare install_sysroot_headers || return 1
 
 	# Step 5. Pass2 GCC for C-library (--with-newlib --enable-threads=no --disable-bootstrap)
 	#
diff --git a/crossbuild-alpine-bootstrap.sh b/crossbuild-alpine-bootstrap.sh
old mode 100644
new mode 100755
index 541065e..8bcd21c
--- a/crossbuild-alpine-bootstrap.sh
+++ b/crossbuild-alpine-bootstrap.sh
@@ -1,6 +1,8 @@
 #!/bin/sh
 
-TARGET_ARCH=x86_64
+#set -x
+
+TARGET_ARCH=x86
 
 export ABUILD_CREATECROSS_CONF=$PWD/abuild-createcross-$TARGET_ARCH.conf
 export ABUILD_CONF=$PWD/abuild-crossbuild-$TARGET_ARCH.conf
@@ -12,12 +14,16 @@ CTARGET="$(source $ABUILD_CONF ; echo $CTARGET)"
 REPODEST="$(source $ABUILD_CONF ; echo $REPODEST)"
 REPODEST_HOST="$(source $ABUILD_CREATECROSS_CONF ; echo $REPODEST)"
 PACKAGER_PRIVKEY="$(source $ABUILD_CONF ; echo $PACKAGER_PRIVKEY)"
+LOGDIR="$HOME/crossbuild-logs"
 
 if [ -z "$CBUILDROOT" ]; then
 	echo "CBUILDROOT needs to be set in $ABUILD_CONF."
 	exit 1
 fi
 
+export PKG_CONFIG_PATH="$CBUILDROOT"/usr/lib/pkgconfig
+export PKG_CONFIG_SYSROOT_DIR="$CBUILDROOT"/
+
 # remove possible old pass2 gcc, and install needed toolchain
 ${SUDO_APK} del gcc-pass2-$CTARGET
 ${SUDO_APK} --repository $REPODEST_HOST/main add gcc-$CTARGET g++-$CTARGET gcc-gnat-$CTARGET || return 1
@@ -33,21 +39,43 @@ ${SUDO_APK} --root "$CBUILDROOT" add libgcc libgnat libstdc++
 #KERNEL_PKG="linux-firmware linux-vanilla"
 #DEBUG_PKG="ncurses gdb"
 
+install_crossbuilt_package() {
+	PKG="$1"
+    echo "Installing crossbuilt package $PKG."
+	${SUDO_APK} add --root "$CBUILDROOT" "$PKG"
+}
+
+mkdir -p "$LOGDIR"
+
 # ordered cross-build
 for PKG in linux-headers musl libc-dev \
-	   busybox busybox-initscripts binutils make pkgconf pkgconfig \
-	   zlib gmp5 mpfr3 mpc1 isl cloog gcc \
+	   busybox busybox-initscripts binutils make pkgconf pkgconfig zlib \
+	   gmp5 mpfr3 mpc1 isl cloog gcc \
 	   openssl apk-tools \
 	   openrc alpine-conf alpine-baselayout alpine-keys alpine-base build-base \
 	   pax-utils patch sudo attr libcap fakeroot \
 	   abuild mkinitfs $KERNEL_PKG $DEBUG_PKG ; do
 
+    echo "Crossbuilding package $PKG."
 	cd $APORTS/main/$PKG
-	abuild -r || exit 1
+	abuild -r 2>&1 1> "$LOGDIR/$PKG.log" || exit 1
+	install_crossbuilt_package "$PKG"
+
+	case "$PKG" in
+	  gmp5)
+		install_crossbuilt_package gmp-dev
+		;;
+	  mpfr3)
+		install_crossbuilt_package mpfr-dev
+		;;
+	  *)
+		install_crossbuilt_package "$PKG-dev"
+		;;
+	esac
 
-	if [ "$PKG" = "libc-dev" ]; then
-		# install libc-dev, as implicit but mandatory dependency
-		${SUDO_APK} -u --root "$CBUILDROOT" add libc-dev || exit 1
-	fi
+	#if [ "$PKG" = "libc-dev" ]; then
+	#	# install libc-dev, as implicit but mandatory dependency
+	#	${SUDO_APK} -u --root "$CBUILDROOT" add libc-dev || exit 1
+	#fi
 done

---

From: "Leslie P. Polzer | PORT ZERO" <[email-masked]>
Date: Fri, 25 Oct 2013 15:15:42 +0200

Another question I forgot to ask:

Why do the scripts install gcc-gnat and build gcc with support
for Ada?

  Leslie

---

---

From: Timo Teräs <[email-masked]>
Date: Fri, 25 Oct 2013 16:54:10 +0300

> Why do the scripts install gcc-gnat and build gcc with support

You need gcc-gnat = ada. You need ada to compile ada. So to port full alpine, ada needs to be cross compiled to bootstrap it.

-Timo

---

---

From: Timo Teras <[email-masked]>
Date: Wed, 30 Oct 2013 11:14:16 +0200

On Fri, 25 Oct 2013 14:30:48 +0200
"Leslie P. Polzer | PORT ZERO" <[email-masked]> wrote:

> > compiled. All changes are in git already. And some minor notes +
> > links to the scripts I used to bootstrap musl builders are at my
> > wiki wall: http://wiki.alpinelinux.org/wiki/User:Fabled
> 
> That helped! I had to clear out some roadblocks, after which I now
> have a successful and reproducible bootstrap crossbuild of
> i486-alpine-linux-musl (without KERNEL_PKG and DEBUG_PKG so far; I
> simply haven't tried).

Very good!

>   multiple times by the "prepare" step if you call the script more

Ah, right. I think this can be affected in abuild.conf - but it's good
to have it explicit.

>   More packages might be affected by this.

Yes, I noticed this too.

>   package that broke in the ordered crossbuild because of this

And this too - I added this to x86_64.conf's earlier.

>   The packages affected are gmp5 (gmp-dev) and mpfr3 (mpfr-dev).

Hmm - this worked for me without this. Though, I did have to fix some
makedepends in the aports. This was done not long ago.

> * Non-essential: also added per-package logging to cut down on the

Ok.

> scripts. Please let me know how you want to proceed with integrating

I need think what to do with the scripts. Might be an idea to bundle
them in abuild git tree. Even if they are not shipped in abuild.
Ultimately I would abuild to have "--target" and "--host" flags that
can be used to trigger the cross build so no additional "kludge" config
files would be needed.

> we have built them. The usual way is running the testsuite (make
> check) after compilation. Obviously this won't work when
> cross-compiling to a different machine architecture. Any ideas?

I don't think we plan to support cross-compiling all of aports. So this
would not be a goal for us.

I believe scratchbox etc. use statically built qemu-user, or some
binfmt_misc magic to launch the files on remote target. In either case
both will be "kludgy" and probably not doing 100% testing.

- Timo

---

---

From: "Leslie P. Polzer | PORT ZERO" <[email-masked]>
Date: Wed, 30 Oct 2013 21:22:39 +0100

> makedepends in the aports. This was done not long ago.

I see, abuild will install the necessary deps automatically so
there's no need to manually install them if the makedeps are
correct.

> them in abuild git tree. Even if they are not shipped in abuild.

Good, please let me know when you've decided what to do.

In the meantime I will test some more and maybe try to automate building
a new cross-compiled image.

> can be used to trigger the cross build so no additional "kludge" config

Yes, that'd be very nice indeed.

Thanks!

  Leslie

---