Subject: [alpine-devel] Attempting x86_64 build

From: Mark Constable <[email-masked]>
Date: Sat, 01 Aug 2009 17:10:05 +1000

I've got an x86_64 busybox/uclibc partition set up, courtesy of buildroot,
and I've started building a few packages. nano built okay so I've moved
on to apk-tools so I can obviously start to use Alpine tools to build
and install more packages. It needs pkgconfig, zlib and openssl so I
managed to build pkgconfig and zlib but I'm stuck on openssl because it
needs perl and the buildroot target only has miniperl (or maybe none at
all, not sure) so a coule of questions...

. would it be possible to build openssl without using perl?

. would it be possible to build apk-tools without openssl?

. should I drop back to pre v1.9 and use the scripted apk tools?

I barely know how to use the 1.9 apk toolset let alone the earlier ones.

***

I've also got an Archlinux ARM chroot and managed to also build nano
but nothing else so far as their is some problem with setting time
in the provided glibc. If I manage to get enough x86_64 Alpine packages
to build, to bootstrap a clean system, I'll post a tarball and follow
the same method for ARM.

--markc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Timo Teräs <[email-masked]>
Date: Sat, 01 Aug 2009 10:59:13 +0300

Hi,

> all, not sure) so a coule of questions...

Right. Bootstrapping new buildroot is always tricky. Especially if core
tools have dependencies like this. As we've moved to the new apk-tools
stuff we've always had older "native" toolchains so it has been easy.
But we need to figure out a way to boot strap from scratch.

> . would it be possible to build openssl without using perl?

I'm not sure. I think openssl uses perl to autogenerate all sorts of
stuff; like the OID tables and some assembly code etc. So this is likely
very tricky.

> . would it be possible to build apk-tools without openssl?

Not as-is. apk-tools uses openssl for packages integrity checking:
digest calculation (sha1, md5) and signature verification (rsa).

We could very easily add "LITE" version that lacks package signature
verification and other stuff. In that version we could provide a
static version of sha1 for calculating the digest.

You do not want openssl at all in your final system, or is it
just inconvenience during the initial build?

I will talk with ncopa about this a bit. The other alternative
I see is: we modify abuild to have "install, no package" mode, so
one can first build the core packages on chroot. That way apk-tools
is not needed until later in the build process. abuild really uses
apk-tools only for updating the repository index; apk-tools does
not contain any package generation code.

> . should I drop back to pre v1.9 and use the scripted apk tools?

For experimental purposes, you might try that. But that branch of
Alpine is more or less declared dead. We will do updates on it if
there's really really important security issues. But we've stopped
the regular updates.

It'd be nicer if we got worked out the new stuff so that it is
usable to boot strap the build root.

> to build, to bootstrap a clean system, I'll post a tarball and follow

Cool.

We appreciate your work on this area a lot. Once I get some VIA
Nano boxes here, I can start working on x86_64 too.

- Timo

---

---

From: Natanael Copa <[email-masked]>
Date: Sat, 01 Aug 2009 11:44:08 +0200

On Sat, 2009-08-01 at 17:10 +1000, Mark Constable wrote:
> and I've started building a few packages. 

Cool!

> nano built okay so I've moved

I doubt. And you will need perl for lot other stuff too. Everything that
involves autoconf/automake and things like kernel. Bootstrapping does
require perl at an early stage.

> . would it be possible to build apk-tools without openssl?

I think the way to do bootstrapping is use a static version of
apk-tools. You could probably use a 32 bit static, or if you want, a 64
bit static built on a glibc host. It will be ugly big but it will get
you started.

> . should I drop back to pre v1.9 and use the scripted apk tools?

No. We want fix bootstrapping for apk-tools.

Does buildroot provide gcc, binutils and gnu make?

Otherwise, it might be that gentoo have a stage1 or stage3 with x86_64
uclibc. They have some binary packages:
http://tinderbox.x86.dev.gentoo.org/uclibc/amd64/All/

Not sure but this looks like gcc with uclibc:
http://tinderbox.x86.dev.gentoo.org/cross-amd64/x86_64-gentoo-linux-uclibc/

You could probably tar -jxf make, binutils, gcc, uclibc, linux-headers,
file, openssl, pax-utils, busybox to get started. Might help to have a
static busybox and apk-tools to get started.

> to build, to bootstrap a clean system, I'll post a tarball and follow

I think gentoo had some ARM stages you could start from.

Thanks for working on this!

-nc

---

---

From: Mark Constable <[email-masked]>
Date: Sat, 01 Aug 2009 22:56:46 +1000

> static version of sha1 for calculating the digest.

That may be a good strategy for multiple archs.

> just inconvenience during the initial build?

I will want it on the final system but it's proving to be the
package that is most difficult to build.

> usable to boot strap the build root.

A script based system would be easier for porting to different
archs but I will stick with the latest apk-tools.

Natanael Copa wrote:
> involves autoconf/automake and things like kernel.

I think the latest kernel has just had the last perl script removed.

> Bootstrapping does require perl at an early stage.

I find it's mostly for things like docs and ancillary stuff that
could be avoided with some care.

> bit static built on a glibc host. It will be ugly big but it will get

Could you spare a hint how to produce a static apk-tools using v1.9?

> Does buildroot provide gcc, binutils and gnu make?

Yes it does.

My first attempt was missing things like file, perl and other basics
but I've managed to build them and install them directly to /usr.
Thanks for the links and suggestions but I think I have just enough
up and running to be worth continuing with. ATM I am stuck on openssl
where it's complaining about a missing ftime reference and I see that
a possible solution is included with the Apline uclibc package but
not with the buildroot version... I just now tried to build uclibc
and install it but everything segfaulted, so I copied back the uclibc
libs from the buildroot tarball and things are working again (whew!).

If anyone feels like creating static apk-tools and abuild packages
then I could probably skip over this part and just continue building
key core packages until a fresh install of those packages is possible.

Also, if anyone is interested in this x86_64 buidlroot system then I
could zip it up to about a 150Mb tarball.

--markc

---

---

From: Timo Teräs <[email-masked]>
Date: Sat, 01 Aug 2009 18:21:36 +0300

> could be avoided with some care.

I'm still thinking that bootstrapping should be doable without
apk-tools. It is generally used by abuild for checking build
dependencies and updating the index. Neither should be needed
while bootstrapping the tool chain.

> Could you spare a hint how to produce a static apk-tools using v1.9?

"make static" and you should get apk.static binary in src directory.

> key core packages until a fresh install of those packages is possible.

Abuild is just scripts so that should be ok.

> could zip it up to about a 150Mb tarball.

No test system yet (except an oldish amd64 desktop machine). However,
a script to produce it would be of high interest for me.

- Timo

---

---

From: Mark Constable <[email-masked]>
Date: Sun, 02 Aug 2009 07:23:56 +1000

> while bootstrapping the tool chain.

Yes, checking not yet existing apk dependencies and indexes
just makes it that much harder.

> "make static" and you should get apk.static binary in src directory.

Too easy, thanks, that probably saved me quite a bit of googling.

> Abuild is just scripts so that should be ok.

Except it depends on openssl which I'm having trouble building...

 cc -o abuild-tar -lssl -lcrypto -ldl -lz   abuild-tar.c
 abuild-tar.c:19:25: error: openssl/evp.h: No such file or directory

> a script to produce it would be of high interest for me.

 git clone git://git.buildroot.net/buildroot
 cd buildroot
 make menuconfig
 make

When I get a working .config I'll make an APKBUILD but it may
require a "real" glibc based build chain which is, FWIW, why I
am putting effort into x86_64 first, so I can install a "real"
system if need be and more easily use the likes of buildroot to
bootstrap ARM (then perhaps PPC and MIPs, if I don't go crazy).

--markc

---

---

From: Timo Teräs <[email-masked]>
Date: Sun, 02 Aug 2009 10:49:13 +0300

> abuild-tar.c:19:25: error: openssl/evp.h: No such file or directory

Oh, right. That utility is only needed to generate .apk files though.
It was added when we did the signing stuff.

I think we need a special version of abuild that does not depend
on openssl or apk-tools: it can be just used to build and install
stuff.

> bootstrap ARM (then perhaps PPC and MIPs, if I don't go crazy).

Ok. I've used buildroot once upon time. But I never looked into
the bootstrapping. Maybe I'll take a look at next week.

Thanks,
  Timo

---

---

From: Mark Constable <[email-masked]>
Date: Mon, 03 Aug 2009 19:26:34 +1000

After another 48 hours of hair pulling I managed to compile openssl
then abuild and now apk-tools.

root@br_x86_64 ~/main/apk-tools/apk-tools-2.0_pre17 make
  CC      src/add.o
  CC      src/apk.o
  CC      src/archive.o
src/archive.c:50: error: expected specifier-qualifier-list before 'uint16_t'
src/archive.c: In function 'apk_tar_parse':
src/archive.c:171: error: 'struct apk_tar_digest_info' has no member named 'size'
src/archive.c:172: error: 'struct apk_tar_digest_info' has no member named 'size'
src/archive.c:173: error: 'struct apk_tar_digest_info' has no member named 'digest'
src/archive.c:173: error: 'struct apk_tar_digest_info' has no member named 'size'
make[1]: *** [src/archive.o] Error 1
make: *** [src/] Error 2

Adding stdint.h to src/archive.c seemed to fix the above, also,
this is a from apk-tools too (fwiw)...

gcc -std=gnu99  -march=x86-64 -mtune=generic -O2 -pipe  -Wl,--as-needed -o tar buffer.o checkpoint.o compare.o create.o delete.o extract.o xheader.o incremen.o list.o misc.o names.o sparse.o suffix.o system.o tar.o transform.o update.o utf8.o ../lib/libtar.a /usr/lib/libintl.a /usr/lib/libc.so -Wl,-rpath -Wl,/usr/lib
../lib/libtar.a(utimens.o): In function `gl_futimens':
utimens.c:(.text+0x6c): warning: the use of LEGACY `utimes' is discouraged, use `utime'
../lib/libtar.a(close-stream.o): In function `close_stream':
close-stream.c:(.text+0x17): warning: This version of __fpending returns bytes remaining in buffer for both narrow and wide streams.  glibc's version returns wide chars in buffer for the wide stream case.

***

To get openssl to compile I resorted to adding the missing ftime()
(a googled hack, not the real thing) directly to apps/speed.c...

int ftime(struct timeb *tp)
{
  time_t fix;
  time(&fix);
  tp->time=fix;
  tp->millitm=tp->timezone=tp->dstflag=0;

  return 0;
}

***

Now, I'm still getting some segfaults but it seems I can start to use
the apk/abuild tools to *start* installing some packages. A question
though is what should be the first most important, say, 2 dozen packages
and in what order should I try to build them?

--markc

---

---

From: Mark Constable <[email-masked]>
Date: Mon, 03 Aug 2009 19:29:11 +1000

Doh, this one was from tar, disregard.

> gcc -std=gnu99  -march=x86-64 -mtune=generic -O2 -pipe  -Wl,--as-needed 
> -o tar buffer.o checkpoint.o compare.o create.o delete.o extract.o 
> xheader.o incremen.o list.o misc.o names.o sparse.o suffix.o system.o 
> tar.o transform.o update.o utf8.o ../lib/libtar.a /usr/lib/libintl.a 
> /usr/lib/libc.so -Wl,-rpath -Wl,/usr/lib
> close-stream.c:(.text+0x17): warning: This version of __fpending returns 
> bytes remaining in buffer for both narrow and wide streams.  glibc's 
> version returns wide chars in buffer for the wide stream case.

--markc

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 03 Aug 2009 11:49:39 +0200

> then abuild and now apk-tools.

wow! cool!

> src/archive.c:50: error: expected specifier-qualifier-list before 'uint16_t'
...
> this is a from apk-tools too (fwiw)...

Committed that upstream (the apk-tools git repo)

>   tp->millitm=tp->timezone=tp->dstflag=0;

probably a uclibc config opt.

> and in what order should I try to build them?

the toolchain (binutils, gcc, linux-headers, uclibc)

tools that help you build stuff:
diffutils, make, bison, flex, perl, autoconf, automake, pax-utils (used
by abuild to trace dynamic deps), file, busybox*, apk-tools, alpine-*,
openrc

Then you should have a working vserver (bootable system without kernel)
that can build stuff for you.

thanks!

-nc

---

---

From: Mark Constable <[email-masked]>
Date: Mon, 03 Aug 2009 21:46:27 +1000

This is a doozy, uclibc is a dependency of uclibc?

default@br_x86_64 ~/main/uclibc abuild
>>> uclibc: Checking sanity of ./APKBUILD...
>>> uclibc: Analyzing dependencies...
>>> ERROR: uclibc: Missing dependency build-base. Use -r to autoinstall or -R to build

default@br_x86_64 ~/main/uclibc abuild -R
>>> uclibc: Checking sanity of ./APKBUILD...
>>> uclibc: Analyzing dependencies...
WARNING: Failed to open index for /home/default/.cache/apks
>>> uclibc: Entering /home/default/main/uclibc
>>> uclibc: Checking sanity of ./APKBUILD...
>>> uclibc: Analyzing dependencies...
/usr/bin/abuild: line 706: 30321 Segmentation fault      $SUDO apk add --repo "$apkcache" --virtual .makedepends-$pkgname $deps
>>> ERROR: uclibc: apkcache failed
/usr/bin/abuild: line 78: 30322 Segmentation fault      $SUDO apk del $uninstall_after
>>> ERROR: uclibc: all failed
/usr/bin/abuild: line 78: 30323 Segmentation fault      $SUDO apk del $uninstall_after

--markc

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 03 Aug 2009 14:40:00 +0200

> This is a doozy, uclibc is a dependency of uclibc?

Yeah, in order to build uclibc you need build-base which depends on gcc
which depends on uclibc.

We need some bootstrap option to abuild that ignores the dependency
check.

-nc

---

---

From: Mark Constable <[email-masked]>
Date: Mon, 03 Aug 2009 20:58:35 +1000

> probably a uclibc config opt.

Best I can determine it's these which are missing from my buildroot
busybox and, just now when I check, seem to be also missing from the
latest busybox/busyboxconfig...

UCLIBC_SUSV3_LEGACY=y
UCLIBC_SUSV3_LEGACY_MACROS=y

I have not been able to successfully rebuild buildroot after about
4 more attempts (each taking many hours) so I'm stuck with the lame
version I managed to build on my first or 2nd attempt (beginners luck).

> that can build stuff for you.

Yep, if I can manage to create any packages at all... my lame
buildroot busybox is missing getopts so abuild won't work. I tried
to manually build busybox but got this...

...
  AR      runit/lib.a
  LD      selinux/built-in.o
  AR      selinux/lib.a
  LD      shell/built-in.o
  CC      shell/ash.o
shell/ash.c: In function 'expsort':
shell/ash.c:6876: error: invalid storage class for function 'expandmeta'
shell/ash.c:6875: warning: no previous prototype for 'expandmeta'
... plus another 100 lines of ugly errors.

So I've manually built bash and attempted abuild for the first time
on strace (so I can get an idea of some segfaults) and got this
(after manually building sudo)...

root@br_x86_64 ~/main/strace abuild
>>> strace: Checking sanity of ./APKBUILD...
>>> strace: Analyzing dependencies...
>>> ERROR: strace: Missing dependency build-base. Use -r to autoinstall or -R to build

root@br_x86_64 ~/main/strace abuild -R
>>> strace: Checking sanity of ./APKBUILD...
>>> strace: Analyzing dependencies...
WARNING: Failed to open index for /root/.cache/apks
WARNING: Failed to open index for /root/.cache/apks
>>> strace: Entering /root/main/uclibc
/usr/bin/abuild: line 706: 29964 Segmentation fault      $SUDO apk add --repo "$apkcache" --virtual .makedepends-$pkgname $deps
>>> ERROR: uclibc: apkcache failed
/usr/bin/abuild: line 78: 29965 Segmentation fault      $SUDO apk del $uninstall_after
>>> ERROR: strace: all failed
/usr/bin/abuild: line 78: 29966 Segmentation fault      $SUDO apk del $uninstall_after

So some kind of "get out of jail" option for abuild to let it
build a package without caring if there are any dependencies would
be helpful for bootstrapping.

Something like makepkg -d, --nodeps     Skip all dependency checks

--markc

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 03 Aug 2009 14:37:43 +0200

> version I managed to build on my first or 2nd attempt (beginners luck).

You might want start from or have a look at those:
http://git.alpinelinux.org/cgit/aports/plain/main/uclibc/uclibcconfig

http://git.alpinelinux.org/cgit/aports/plain/main/busybox/busyboxconfig

...

...

> Something like makepkg -d, --nodeps     Skip all dependency checks

ah.. yeah. good point. Could you please file a bug at
bugs.alpinelinux.org for this? Just so I don't forget.

not a difficult feature so i might be able to implement it today.
(unless i get caught up in something else...)

-nc

---