Subject: Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci

From: Michael Wyraz <[email-masked]>
Date: Wed, 19 Sep 2018 06:19:50 +0200

Manjunath,

I saw that error - it should be fixed in later qemu versions but I did 
not had enough time to get it running on travis' old Ubuntu (would need 
to be backported).

Before I spend lot of time in this idea, I'd first like to hear what 
devs say about it.

Regards,

Michael.

> I love to see something like this working in Travis to avoid 
> dependency on the native requirement, but I see following error in arm 
> emulation mode:
> https://travis-ci.org/micw/aports/jobs/430225451#L836
> Generating RSA private key, 2048 bit long modulus
> qemu: Unsupported syscall: 384
> Unsupported ioctl: cmd=0xffffffff80045200
> This basically means that qemu failed to emulate one of that 
> instruction on amd64 architecture, but I'm not sure which command 
> threw this error message and how that script exited without any error.
> Thanks,
> Manjunath.
>
>     ----- Original message -----
>     From: Michael Wyraz <[email-masked]>
>     To: [email-masked]
>     Cc:
>     Subject: [alpine-devel] Proposal: Multi-Arch matrix builds on
>     travis-ci
>     Date: Wed, Sep 19, 2018 2:51 AM
>     Hello Alpine Devs,
>
>     today I had the idea to add multiple arch support to travis-ci builds,
>     so that CI can be done on other platforms than x86_64.
>
>     So I created a PoC that looks promising:
>     https://github.com/micw/aports/pull/2
>
>     See it in action at:
>     https://travis-ci.org/micw/aports/builds/430225449
>
>     It uses qemu-arm-static and binfmt-support to allow execution of arm
>     code on travis.
>
>     Some work is still to be done to use it productive (use newer qemu,
>     modify the build script to only build packages for the current
>     architecture but that should be easy to achieve).
>
>     Please let me know what you think about it and if we should go
>     this way
>     (as intermediate solution unless we have native CI).
>
>     Best regards,
>
>     Michael.
>
>
>
>
>     ---
>     Unsubscribe:  [email-masked]
>     Help:         [email-masked]
>     ---
>
>
> --- Unsubscribe: [email-masked] Help: 
> [email-masked] ---

---

From: Jakub Jirutka <[email-masked]>
Date: Wed, 19 Sep 2018 11:17:21 +0200

Hi,

I'm sorry to say that, but you're kinda reinventing the wheel. This 
exact approach is already used in [alpine-chroot-install], which is used 
e.g. for building static binaries in [apk-tools] repository.

I didn't use it in aports intentionally, because it's *very* slow. Many 
aports would fail to build due to exceeding the time limit (50 minutes) 
and it would significantly increase waiting time for all builds (due to 
exhausted job queue).

This is not the way to go. The better way is to migrate to a self-hosted 
CI with builders on various architectures. For example sr.ht proposed by 
Drew DeVault looks very promising.

 >> Unsupported ioctl: cmd=0xffffffff80045200

See the note in [1].

Jakub

[alpine-chroot-install]: 
https://github.com/alpinelinux/alpine-chroot-install
[apk-tools]: https://github.com/alpinelinux/apk-tools
[1]: https://github.com/alpinelinux/alpine-chroot-install#requirements

---

---

From: Michael Wyraz <[email-masked]>
Date: Wed, 19 Sep 2018 21:01:26 +0200

Hallo Jakub,

thanks for the information. The time was not wasted, I learned a lot 
about aport build environment and binfmt ;-)

Did you do any performance tests to verify how slow it actually is?

IMO we should provide an easy way for devs to test-build for different 
arches to reduce the hurdle to provide patches and pull requests (which 
is IMO _way_ to high because it needs an alpine setup and many manual 
steps, even for native builds). So if travis is not the way to go and 
unless we have something else, maybe we can find another way to provide 
this (maybe using docker?)

Michael.

> exact approach is already used in [alpine-chroot-install], which is 
> used e.g. for building static binaries in [apk-tools] repository.
>
> I didn't use it in aports intentionally, because it's *very* slow. 
> Many aports would fail to build due to exceeding the time limit (50 
> minutes) and it would significantly increase waiting time for all 
> builds (due to exhausted job queue).
>
> This is not the way to go. The better way is to migrate to a 
> self-hosted CI with builders on various architectures. For example 
> sr.ht proposed by Drew DeVault looks very promising.
>
>> I saw that error - it should be fixed in later qemu versions but I 
>> did not had enough time to get it running on travis' old Ubuntu 
>> (would need to be backported).
>>
>>> dependency on the native requirement, but I see following error in 
>>> arm emulation mode:
>>> --- Unsubscribe: [email-masked] 
>>> Help: [email-masked] --- 
>
>
> ---

---

---

From: Oliver Smith <[email-masked]>
Date: Thu, 20 Sep 2018 05:12:00 +0000

> this (maybe using docker?)

FWIW, pmbootstrap can do this (although the main purpose of the tool is
postmarketOS development).

Run this on any Linux distribution with Python and openssl installed,
and you're ready to compile - and cross compile - Alpine packages:

$ git clone https://gitlab.com/postmarketOS/pmbootstrap.git
$ cd pmbootstrap
$ ./pmbootstrap.py init
$ ./pmbootstrap.py build --arch=armhf hello-world

there's also:
$ ./pmbootstrap.py checksum hello-world
$ ./pmbootstrap.py newapkbuild -h

Best regards,
Oliver

---

---

From: Michael Wyraz <[email-masked]>
Date: Wed, 19 Sep 2018 23:06:00 +0200

I did some performance tests with my cups-epson-escpr pull request:

x86_64: 48s

armhf via qemu: 334s

So it looks that the emulation is out of question for automated CI.

Am 19.09.18 um 21:01 schrieb Michael Wyraz:
> arches to reduce the hurdle to provide patches and pull requests 
> (which is IMO _way_ to high because it needs an alpine setup and many 
> manual steps, even for native builds). So if travis is not the way to 
> go and unless we have something else, maybe we can find another way to 
> provide this (maybe using docker?)
>
> Michael.
>
>
>> Hi,
>>

---

---

From: "Jean-Louis Fuchs" <[email-masked]>
Date: Fri, 21 Sep 2018 21:44:20 +0200

Hi

> and it would significantly increase waiting time for all builds (due to 

There was or is bug in qemu-user-static, that made it terribly slow.  This version:

$> ./qemu-arm-static --version
qemu-arm version 2.9.1(qemu-2.9.1-2.fc26)

is definitely ok, I haven't tested any newer ones. In my opinion most builds should not fail due time limit on travis. BUT the tests sometimes do, after all its not the correct kernel. I am moving towards qemu-system using netboot for my development. netboot boots surprisingly consistent across architectures. You can append apkovl= to provision the VM.

Best,
   Jean-Louis

---

---

From: Michael Wyraz <[email-masked]>
Date: Fri, 21 Sep 2018 22:10:21 +0200

I did my tests with QEMU 3.0:

x86_64: 48s
armhf via qemu: 334s

so far away from "ok" ;-/

---

---

From: "Jean-Louis Fuchs" <[email-masked]>
Date: Fri, 21 Sep 2018 22:49:42 +0200

If you have the bug the ratio is far worse. You can use top to detemine
it, you‘d get like 1-10% for a build that uses all cores on x86.

On Fri, 21 Sep 2018, at 22:10, Michael Wyraz wrote:
> >> I didn't use it in aports intentionally, because it's *very*
> >> slow. Many> >> aports would fail to build due to exceeding the time limit (50
> >> minutes)> >> and it would significantly increase waiting time for all builds
> >> (due to> >> exhausted job queue).
> > There was or is bug in qemu-user-static, that made it terribly slow.
> > This version:> >
> > is definitely ok, I haven't tested any newer ones. In my opinion
> > most builds should not fail due time limit on travis. BUT the tests
> > sometimes do, after all its not the correct kernel. I am moving
> > towards qemu-system using netboot for my development. netboot boots
> > surprisingly consistent across architectures. You can append apkovl=
> > to provision the VM.> >
> > Best,
> >     Jean-Louis
> >
> >
> > ---

---

From: Michael Wyraz <[email-masked]>
Date: Sat, 22 Sep 2018 09:09:19 +0200

Hello devs,

to allow easy building of packages for different archs, i have put 
everthing that is required into docker images and wrote a small wrapper 
script. Please have a look into it and let me know what you think about it.

My primary goal was to enable everyone who wants to contribute to alpine 
to checkout the aports, do some changes and test-build it locally with 
minimal effort.

The wrapper script is at 
https://gist.github.com/micw/87756e35d744e6784ce8d8c66e2619cb - it can 
be put into the aports root directory and can simply be called with arch 
and package.

Docker images for the first 2 arches are 
https://hub.docker.com/r/micwy/aport-builder-x86_64/ and 
https://hub.docker.com/r/micwy/aport-builder-armhf/ .

If you find this usefull, I'll document it a bit more, fine-tune the 
wrapper script and create a PR to add it to the aports.

Regards,

Michael.

---

---

From: Tiago Ilieve <[email-masked]>
Date: Sat, 22 Sep 2018 11:09:02 +0200

Hi Michael,

> to checkout the aports, do some changes and test-build it locally with

It's interesting that you mentioned it, because yesterday I worked on
a similar project: https://github.com/myhro/alpine-builder

The only difference is that it doesn't support multiple architectures,
neither it relies on a shared image. But the goal is exactly the same
you described.

I have been doing Alpine packaging for less than a week, but already
seen how much easier the process can be when you use disposable build
environments.

Regards,
Tiago.

-- 
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://de.linkedin.com/in/myhro
Berlin, Germany

---