Manjunath,
I saw that error - it should be fixed in later qemu versions but I did
not had enough time to get it running on travis' old Ubuntu (would need
to be backported).
Before I spend lot of time in this idea, I'd first like to hear what
devs say about it.
Regards,
Michael.
> I love to see something like this working in Travis to avoid > dependency on the native requirement, but I see following error in arm > emulation mode:> https://travis-ci.org/micw/aports/jobs/430225451#L836> Generating RSA private key, 2048 bit long modulus> qemu: Unsupported syscall: 384> Unsupported ioctl: cmd=0xffffffff80045200> This basically means that qemu failed to emulate one of that > instruction on amd64 architecture, but I'm not sure which command > threw this error message and how that script exited without any error.> Thanks,> Manjunath.>> ----- Original message -----> From: Michael Wyraz <michael@wyraz.de>> To: alpine-devel@lists.alpinelinux.org> Cc:> Subject: [alpine-devel] Proposal: Multi-Arch matrix builds on> travis-ci> Date: Wed, Sep 19, 2018 2:51 AM> Hello Alpine Devs,>> today I had the idea to add multiple arch support to travis-ci builds,> so that CI can be done on other platforms than x86_64.>> So I created a PoC that looks promising:> https://github.com/micw/aports/pull/2>> See it in action at:> https://travis-ci.org/micw/aports/builds/430225449>> It uses qemu-arm-static and binfmt-support to allow execution of arm> code on travis.>> Some work is still to be done to use it productive (use newer qemu,> modify the build script to only build packages for the current> architecture but that should be easy to achieve).>> Please let me know what you think about it and if we should go> this way> (as intermediate solution unless we have native CI).>> Best regards,>> Michael.>>>>> ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->>> --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: > alpine-devel+help@lists.alpinelinux.org ---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
Hi,
I'm sorry to say that, but you're kinda reinventing the wheel. This
exact approach is already used in [alpine-chroot-install], which is used
e.g. for building static binaries in [apk-tools] repository.
I didn't use it in aports intentionally, because it's *very* slow. Many
aports would fail to build due to exceeding the time limit (50 minutes)
and it would significantly increase waiting time for all builds (due to
exhausted job queue).
This is not the way to go. The better way is to migrate to a self-hosted
CI with builders on various architectures. For example sr.ht proposed by
Drew DeVault looks very promising.
>> Unsupported ioctl: cmd=0xffffffff80045200
See the note in [1].
Jakub
[alpine-chroot-install]:
https://github.com/alpinelinux/alpine-chroot-install
[apk-tools]: https://github.com/alpinelinux/apk-tools
[1]: https://github.com/alpinelinux/alpine-chroot-install#requirements
On 9/19/18 6:19 AM, Michael Wyraz wrote:
> Manjunath,> > I saw that error - it should be fixed in later qemu versions but I did > not had enough time to get it running on travis' old Ubuntu (would need > to be backported).> > Before I spend lot of time in this idea, I'd first like to hear what > devs say about it.> > Regards,> > Michael.> > >> I love to see something like this working in Travis to avoid >> dependency on the native requirement, but I see following error in arm >> emulation mode:>> https://travis-ci.org/micw/aports/jobs/430225451#L836>> Generating RSA private key, 2048 bit long modulus>> qemu: Unsupported syscall: 384>> Unsupported ioctl: cmd=0xffffffff80045200>> This basically means that qemu failed to emulate one of that >> instruction on amd64 architecture, but I'm not sure which command >> threw this error message and how that script exited without any error.>> Thanks,>> Manjunath.>>>> ----- Original message ----->> From: Michael Wyraz <michael@wyraz.de>>> To: alpine-devel@lists.alpinelinux.org>> Cc:>> Subject: [alpine-devel] Proposal: Multi-Arch matrix builds on>> travis-ci>> Date: Wed, Sep 19, 2018 2:51 AM>> Hello Alpine Devs,>>>> today I had the idea to add multiple arch support to travis-ci builds,>> so that CI can be done on other platforms than x86_64.>>>> So I created a PoC that looks promising:>> https://github.com/micw/aports/pull/2>>>> See it in action at:>> https://travis-ci.org/micw/aports/builds/430225449>>>> It uses qemu-arm-static and binfmt-support to allow execution of arm>> code on travis.>>>> Some work is still to be done to use it productive (use newer qemu,>> modify the build script to only build packages for the current>> architecture but that should be easy to achieve).>>>> Please let me know what you think about it and if we should go>> this way>> (as intermediate solution unless we have native CI).>>>> Best regards,>>>> Michael.>>>>>>>>>> --->> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>> Help: alpine-devel+help@lists.alpinelinux.org>> --->>>>>> --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: >> alpine-devel+help@lists.alpinelinux.org ---
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
Hallo Jakub,
thanks for the information. The time was not wasted, I learned a lot
about aport build environment and binfmt ;-)
Did you do any performance tests to verify how slow it actually is?
IMO we should provide an easy way for devs to test-build for different
arches to reduce the hurdle to provide patches and pull requests (which
is IMO _way_ to high because it needs an alpine setup and many manual
steps, even for native builds). So if travis is not the way to go and
unless we have something else, maybe we can find another way to provide
this (maybe using docker?)
Michael.
> Hi,>> I'm sorry to say that, but you're kinda reinventing the wheel. This > exact approach is already used in [alpine-chroot-install], which is > used e.g. for building static binaries in [apk-tools] repository.>> I didn't use it in aports intentionally, because it's *very* slow. > Many aports would fail to build due to exceeding the time limit (50 > minutes) and it would significantly increase waiting time for all > builds (due to exhausted job queue).>> This is not the way to go. The better way is to migrate to a > self-hosted CI with builders on various architectures. For example > sr.ht proposed by Drew DeVault looks very promising.>> >> Unsupported ioctl: cmd=0xffffffff80045200>> See the note in [1].>> Jakub>> [alpine-chroot-install]: > https://github.com/alpinelinux/alpine-chroot-install> [apk-tools]: https://github.com/alpinelinux/apk-tools> [1]: https://github.com/alpinelinux/alpine-chroot-install#requirements>> On 9/19/18 6:19 AM, Michael Wyraz wrote:>> Manjunath,>>>> I saw that error - it should be fixed in later qemu versions but I >> did not had enough time to get it running on travis' old Ubuntu >> (would need to be backported).>>>> Before I spend lot of time in this idea, I'd first like to hear what >> devs say about it.>>>> Regards,>>>> Michael.>>>>>>> I love to see something like this working in Travis to avoid >>> dependency on the native requirement, but I see following error in >>> arm emulation mode:>>> https://travis-ci.org/micw/aports/jobs/430225451#L836>>> Generating RSA private key, 2048 bit long modulus>>> qemu: Unsupported syscall: 384>>> Unsupported ioctl: cmd=0xffffffff80045200>>> This basically means that qemu failed to emulate one of that >>> instruction on amd64 architecture, but I'm not sure which command >>> threw this error message and how that script exited without any error.>>> Thanks,>>> Manjunath.>>>>>> ----- Original message ----->>> From: Michael Wyraz <michael@wyraz.de>>>> To: alpine-devel@lists.alpinelinux.org>>> Cc:>>> Subject: [alpine-devel] Proposal: Multi-Arch matrix builds on>>> travis-ci>>> Date: Wed, Sep 19, 2018 2:51 AM>>> Hello Alpine Devs,>>>>>> today I had the idea to add multiple arch support to travis-ci >>> builds,>>> so that CI can be done on other platforms than x86_64.>>>>>> So I created a PoC that looks promising:>>> https://github.com/micw/aports/pull/2>>>>>> See it in action at:>>> https://travis-ci.org/micw/aports/builds/430225449>>>>>> It uses qemu-arm-static and binfmt-support to allow execution of >>> arm>>> code on travis.>>>>>> Some work is still to be done to use it productive (use newer qemu,>>> modify the build script to only build packages for the current>>> architecture but that should be easy to achieve).>>>>>> Please let me know what you think about it and if we should go>>> this way>>> (as intermediate solution unless we have native CI).>>>>>> Best regards,>>>>>> Michael.>>>>>>>>>>>>>>> --->>> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>>> Help: alpine-devel+help@lists.alpinelinux.org>>> --->>>>>>>>> --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>>> Help: alpine-devel+help@lists.alpinelinux.org --- >>> ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
I did some performance tests with my cups-epson-escpr pull request:
x86_64: 48s
armhf via qemu: 334s
So it looks that the emulation is out of question for automated CI.
Am 19.09.18 um 21:01 schrieb Michael Wyraz:
> Hallo Jakub,>> thanks for the information. The time was not wasted, I learned a lot > about aport build environment and binfmt ;-)>> Did you do any performance tests to verify how slow it actually is?>> IMO we should provide an easy way for devs to test-build for different > arches to reduce the hurdle to provide patches and pull requests > (which is IMO _way_ to high because it needs an alpine setup and many > manual steps, even for native builds). So if travis is not the way to > go and unless we have something else, maybe we can find another way to > provide this (maybe using docker?)>> Michael.>>>> Hi,>>>> I'm sorry to say that, but you're kinda reinventing the wheel. This >> exact approach is already used in [alpine-chroot-install], which is >> used e.g. for building static binaries in [apk-tools] repository.>>>> I didn't use it in aports intentionally, because it's *very* slow. >> Many aports would fail to build due to exceeding the time limit (50 >> minutes) and it would significantly increase waiting time for all >> builds (due to exhausted job queue).>>>> This is not the way to go. The better way is to migrate to a >> self-hosted CI with builders on various architectures. For example >> sr.ht proposed by Drew DeVault looks very promising.>>>> >> Unsupported ioctl: cmd=0xffffffff80045200>>>> See the note in [1].>>>> Jakub>>>> [alpine-chroot-install]: >> https://github.com/alpinelinux/alpine-chroot-install>> [apk-tools]: https://github.com/alpinelinux/apk-tools>> [1]: https://github.com/alpinelinux/alpine-chroot-install#requirements>>>> On 9/19/18 6:19 AM, Michael Wyraz wrote:>>> Manjunath,>>>>>> I saw that error - it should be fixed in later qemu versions but I >>> did not had enough time to get it running on travis' old Ubuntu >>> (would need to be backported).>>>>>> Before I spend lot of time in this idea, I'd first like to hear what >>> devs say about it.>>>>>> Regards,>>>>>> Michael.>>>>>>>>>> I love to see something like this working in Travis to avoid >>>> dependency on the native requirement, but I see following error in >>>> arm emulation mode:>>>> https://travis-ci.org/micw/aports/jobs/430225451#L836>>>> Generating RSA private key, 2048 bit long modulus>>>> qemu: Unsupported syscall: 384>>>> Unsupported ioctl: cmd=0xffffffff80045200>>>> This basically means that qemu failed to emulate one of that >>>> instruction on amd64 architecture, but I'm not sure which command >>>> threw this error message and how that script exited without any error.>>>> Thanks,>>>> Manjunath.>>>>>>>> ----- Original message ----->>>> From: Michael Wyraz <michael@wyraz.de>>>>> To: alpine-devel@lists.alpinelinux.org>>>> Cc:>>>> Subject: [alpine-devel] Proposal: Multi-Arch matrix builds on>>>> travis-ci>>>> Date: Wed, Sep 19, 2018 2:51 AM>>>> Hello Alpine Devs,>>>>>>>> today I had the idea to add multiple arch support to travis-ci >>>> builds,>>>> so that CI can be done on other platforms than x86_64.>>>>>>>> So I created a PoC that looks promising:>>>> https://github.com/micw/aports/pull/2>>>>>>>> See it in action at:>>>> https://travis-ci.org/micw/aports/builds/430225449>>>>>>>> It uses qemu-arm-static and binfmt-support to allow execution >>>> of arm>>>> code on travis.>>>>>>>> Some work is still to be done to use it productive (use newer >>>> qemu,>>>> modify the build script to only build packages for the current>>>> architecture but that should be easy to achieve).>>>>>>>> Please let me know what you think about it and if we should go>>>> this way>>>> (as intermediate solution unless we have native CI).>>>>>>>> Best regards,>>>>>>>> Michael.>>>>>>>>>>>>>>>>>>>> --->>>> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>>>> Help: alpine-devel+help@lists.alpinelinux.org>>>> --->>>>>>>>>>>> --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>>>> Help: alpine-devel+help@lists.alpinelinux.org --- >>>>>> --->> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>> Help: alpine-devel+help@lists.alpinelinux.org>> --->>>>> ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
Michael Wyraz:
> IMO we should provide an easy way for devs to test-build for different> arches to reduce the hurdle to provide patches and pull requests (which> is IMO _way_ to high because it needs an alpine setup and many manual> steps, even for native builds). So if travis is not the way to go and> unless we have something else, maybe we can find another way to provide> this (maybe using docker?)
FWIW, pmbootstrap can do this (although the main purpose of the tool is
postmarketOS development).
Run this on any Linux distribution with Python and openssl installed,
and you're ready to compile - and cross compile - Alpine packages:
$ git clone https://gitlab.com/postmarketOS/pmbootstrap.git
$ cd pmbootstrap
$ ./pmbootstrap.py init
$ ./pmbootstrap.py build --arch=armhf hello-world
there's also:
$ ./pmbootstrap.py checksum hello-world
$ ./pmbootstrap.py newapkbuild -h
Best regards,
Oliver
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
Hi
On Wed, 19 Sep 2018, at 11:17, Jakub Jirutka wrote:
> I didn't use it in aports intentionally, because it's *very* slow. Many > aports would fail to build due to exceeding the time limit (50 minutes) > and it would significantly increase waiting time for all builds (due to > exhausted job queue).
There was or is bug in qemu-user-static, that made it terribly slow. This version:
$> ./qemu-arm-static --version
qemu-arm version 2.9.1(qemu-2.9.1-2.fc26)
is definitely ok, I haven't tested any newer ones. In my opinion most builds should not fail due time limit on travis. BUT the tests sometimes do, after all its not the correct kernel. I am moving towards qemu-system using netboot for my development. netboot boots surprisingly consistent across architectures. You can append apkovl= to provision the VM.
Best,
Jean-Louis
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
I did my tests with QEMU 3.0:
x86_64: 48s
armhf via qemu: 334s
so far away from "ok" ;-/
Am 21.09.18 um 21:44 schrieb Jean-Louis Fuchs:
> Hi>> On Wed, 19 Sep 2018, at 11:17, Jakub Jirutka wrote:>>> I didn't use it in aports intentionally, because it's *very* slow. Many>> aports would fail to build due to exceeding the time limit (50 minutes)>> and it would significantly increase waiting time for all builds (due to>> exhausted job queue).> There was or is bug in qemu-user-static, that made it terribly slow. This version:>> $> ./qemu-arm-static --version> qemu-arm version 2.9.1(qemu-2.9.1-2.fc26)>> is definitely ok, I haven't tested any newer ones. In my opinion most builds should not fail due time limit on travis. BUT the tests sometimes do, after all its not the correct kernel. I am moving towards qemu-system using netboot for my development. netboot boots surprisingly consistent across architectures. You can append apkovl= to provision the VM.>> Best,> Jean-Louis>>> ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Re: [alpine-devel] Proposal: Multi-Arch matrix builds on travis-ci
If you have the bug the ratio is far worse. You can use top to detemine
it, you‘d get like 1-10% for a build that uses all cores on x86.
On Fri, 21 Sep 2018, at 22:10, Michael Wyraz wrote:
> I did my tests with QEMU 3.0:>> x86_64: 48s> armhf via qemu: 334s>> so far away from "ok" ;-/>> Am 21.09.18 um 21:44 schrieb Jean-Louis Fuchs:> > Hi> >> > On Wed, 19 Sep 2018, at 11:17, Jakub Jirutka wrote:> >> >> I didn't use it in aports intentionally, because it's *very*> >> slow. Many> >> aports would fail to build due to exceeding the time limit (50> >> minutes)> >> and it would significantly increase waiting time for all builds> >> (due to> >> exhausted job queue).> > There was or is bug in qemu-user-static, that made it terribly slow.> > This version:> >> > $> ./qemu-arm-static --version> > qemu-arm version 2.9.1(qemu-2.9.1-2.fc26)> >> > is definitely ok, I haven't tested any newer ones. In my opinion> > most builds should not fail due time limit on travis. BUT the tests> > sometimes do, after all its not the correct kernel. I am moving> > towards qemu-system using netboot for my development. netboot boots> > surprisingly consistent across architectures. You can append apkovl=> > to provision the VM.> >> > Best,> > Jean-Louis> >> >> > ---> > Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> > Help: alpine-devel+help@lists.alpinelinux.org> > ---> >>>> ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
[alpine-devel] Easy build using docker (was: Proposal: Multi-Arch matrix builds on travis-ci)
Hi Michael,
On Sat, 22 Sep 2018 at 09:10, Michael Wyraz <michael@wyraz.de> wrote:
>> Hello devs,>> to allow easy building of packages for different archs, i have put> everthing that is required into docker images and wrote a small wrapper> script. Please have a look into it and let me know what you think about it.>> My primary goal was to enable everyone who wants to contribute to alpine> to checkout the aports, do some changes and test-build it locally with> minimal effort.
It's interesting that you mentioned it, because yesterday I worked on
a similar project: https://github.com/myhro/alpine-builder
The only difference is that it doesn't support multiple architectures,
neither it relies on a shared image. But the goal is exactly the same
you described.
I have been doing Alpine packaging for less than a week, but already
seen how much easier the process can be when you use disposable build
environments.
Regards,
Tiago.
--
Tiago "Myhro" Ilieve
Blog: https://blog.myhro.info/
GitHub: https://github.com/myhro
LinkedIn: https://de.linkedin.com/in/myhro
Berlin, Germany
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---