From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mx1.tetrasec.net (mx1.tetrasec.net [74.117.190.25]) by lists.alpinelinux.org (Postfix) with ESMTP id 9F0665C4E28 for ; Tue, 6 Feb 2018 00:39:54 +0000 (GMT) Received: from mx1.tetrasec.net (mail.local [127.0.0.1]) by mx1.tetrasec.net (Postfix) with ESMTP id 63B809E247D for ; Tue, 6 Feb 2018 00:39:54 +0000 (GMT) Received: from ncopa-macbook.copa.dup.pw (201-9-120-223.user.veloxzone.com.br [201.9.120.223]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: n@tanael.org) by mx1.tetrasec.net (Postfix) with ESMTPSA id E10DE9E23AD for ; Tue, 6 Feb 2018 00:39:53 +0000 (GMT) Date: Tue, 6 Feb 2018 01:39:44 +0100 From: Natanael Copa To: alpine-devel@lists.alpinelinux.org Subject: [alpine-devel] rethinking the building infra Message-ID: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> X-Mailer: Claws Mail 3.15.1-dirty (GTK+ 2.24.31; x86_64-alpine-linux-musl) X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Hi! I think we need to rethink the building infrastructure. The current build scripts were written as a quick and dirt way to get started and have lived longer way longer than originally planned. It simply does not scale and is very fragile. I guess I don't need to go any deeper into why we need replace it... I'd like to discuss what we need from the build infra and why, before we start talk about how to do it and what implementation etc. Here are some things I want a new building infra should be able to do: - automatic build on git push There should not be needed to do anything more than do a git push to get the package built and uploaded. Like we have today, but with better error reporting. - isolated environment for each build each build should set up an isolated enviroment and destroy it when build is done. This could be a container but it would be nice to be able to set up a disposable build env in a vm in case we want hook it into github PRs or similar. It should also kill everything after build and test is done so we dont get any remains of test suites that does not clean up after themselves (like redis and epmd) - support multi architectures need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would be nice it its not too complicated to add new architectures. - support parallel building would be nice to be able to distribute the workload over available build servers. Should be possible and relatively easy to add new hardware to the pool or remove or replace old without taking everything down. - support cross compile would be nice to cross compile packages that (easily) can be cross compiled. For example, we could let a big x86_64 or ppc64le machine build linux kernel for armhf instead doing that on the slow armhf server. Packages that cannot be cross compiled should be built on native hardware. - separate out signing process of packages and index Would be nice if we could give access to build servers to more people without giving those people access to the private signing keys. - build infra should be able to be used as CI We need do automatic compile checks of contributions, for example via github pull requests or something corresponding. - efficient caching Would be nice to not need to git clone the entire repo for every build for every server. Would be nice if we checkout a shared git repo or do something so data does not goes over the wire more than necessary. Same goes for source and apk packages. Anything else we need from the building infra? -nc --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail-ua0-f180.google.com (mail-ua0-f180.google.com [209.85.217.180]) by lists.alpinelinux.org (Postfix) with ESMTP id 499195C4E2C for ; Tue, 6 Feb 2018 00:43:59 +0000 (GMT) Received: by mail-ua0-f180.google.com with SMTP id e25so128393uan.5 for ; Mon, 05 Feb 2018 16:43:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=duniel-no.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=vn2VKdyv7L253jCohvwixt+5Sb26wkeZ2AugU9T3bX8=; b=CLh6Y0LBxKnl/5jSKk3GKf8buLMJYeMliapidz0v7unP3ba8RNM0xtiawk7CExFh5T 3ZV639/n3A01dTWGIRKLMvQaTyxh3yGIvlS69AxnqHV14eZROETWeJTIVRVi3Gpoetvg CpDBoyyd24EV5AzV1dwv6Cn0p89mYtcUFxNu6PzfW2I75C7HbwOIwjIaR4EcrOPM7nVW SPwdnQp4YcVOhs5xICcdB7naBUgHDHQC7bRoWi5bcG14DQC8Ad7C9wGLazYhdqeyTLwo lcvbYELEevU+YwfY0bd/Tb4gQR22bEEdNHhC2CMz2Q4MJqN2LZUrTgCbgJjyx1I4bp1d z13w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=vn2VKdyv7L253jCohvwixt+5Sb26wkeZ2AugU9T3bX8=; b=j4Wn2R0DznB53Mi0bJtf1AeVTkfBi0W6ejodjZ1YtWN3UoYiXpl1yyzE42dDv+4r19 Qk79tXRH7WDmYk79IlWzHWVcW1bHwWkDhbNvuYyoeMnf1V7e7/3Ous4xvn4sVfMF74k7 MccSmEPEAMAZCBhXt4U7OCkY/UH0WeVT8OLu2R1wEKipdojFQe/aC34OroW5aEXCe1ym P8mkmow/qPKuRhnZPI+3kuYEtzRoew+I+/pWToQH7yYtba7sR+S9fxPqFTJZk3waeLAa sJNLXx75tz5GnRRIDJnWpcg1dLk4w8uXThw8EiJ7uBiqd+2ve8Ckac1q4KpKKiBgAL9F ed9g== X-Gm-Message-State: APf1xPBhpLZXf9LqDBIAQlua2P4il8KN2VIr1kiibo5z8v38V6sLR1r2 5/X7gx0pJMYHDUIr4ASfhCPS0sYIXzIiiO94AG74LQ== X-Google-Smtp-Source: AH8x227C+ZcqrZfa/I4B/Evi6Wqxkh2mO1pm8x5mVYRYbxoYsue49lJxGhW7xpeWLtZraeZIS+P/m9CPd6HrFmOt7Js= X-Received: by 10.176.72.1 with SMTP id b1mr716701uad.22.1517877838525; Mon, 05 Feb 2018 16:43:58 -0800 (PST) X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Received: by 10.176.8.8 with HTTP; Mon, 5 Feb 2018 16:43:58 -0800 (PST) In-Reply-To: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> From: Daniel Isaksen Date: Tue, 6 Feb 2018 01:43:58 +0100 Message-ID: Subject: Re: [alpine-devel] rethinking the building infra To: Natanael Copa Cc: Alpine Development Content-Type: multipart/alternative; boundary="001a114526fe60ccc9056480795a" --001a114526fe60ccc9056480795a Content-Type: text/plain; charset="UTF-8" A small addition - this is in development, and some hands on it would be nice: https://github.com/kaniini/abuildd - Daniel On Tue, Feb 6, 2018 at 1:39 AM, Natanael Copa wrote: > Hi! > > I think we need to rethink the building infrastructure. The current > build scripts were written as a quick and dirt way to get started and > have lived longer way longer than originally planned. It simply does > not scale and is very fragile. > > I guess I don't need to go any deeper into why we need replace it... > > I'd like to discuss what we need from the build infra and why, before we > start talk about how to do it and what implementation etc. > > Here are some things I want a new building infra should be able to > do: > > - automatic build on git push > > There should not be needed to do anything more than do a git push to > get the package built and uploaded. Like we have today, but with > better error reporting. > > - isolated environment for each build > > each build should set up an isolated enviroment and destroy it when > build is done. This could be a container but it would be nice to be > able to set up a disposable build env in a vm in case we want hook it > into github PRs or similar. It should also kill everything after > build and test is done so we dont get any remains of test suites that > does not clean up after themselves (like redis and epmd) > > - support multi architectures > > need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would > be nice it its not too complicated to add new architectures. > > - support parallel building > > would be nice to be able to distribute the workload over available > build servers. Should be possible and relatively easy to add new > hardware to the pool or remove or replace old without taking > everything down. > > - support cross compile > > would be nice to cross compile packages that (easily) can be cross > compiled. For example, we could let a big x86_64 or ppc64le machine > build linux kernel for armhf instead doing that on the slow armhf > server. Packages that cannot be cross compiled should be built on > native hardware. > > - separate out signing process of packages and index > > Would be nice if we could give access to build servers to more > people without giving those people access to the private signing keys. > > - build infra should be able to be used as CI > > We need do automatic compile checks of contributions, for example via > github pull requests or something corresponding. > > - efficient caching > > Would be nice to not need to git clone the entire repo for every > build for every server. Would be nice if we checkout a shared git > repo or do something so data does not goes over the wire more than > necessary. Same goes for source and apk packages. > > Anything else we need from the building infra? > > -nc > > > --- > Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org > Help: alpine-devel+help@lists.alpinelinux.org > --- > > --001a114526fe60ccc9056480795a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
A small addition - this is in development, and some hands = on it would be nice:=C2=A0ht= tps://github.com/kaniini/abuildd

- Daniel

On Tue, Feb 6, 2= 018 at 1:39 AM, Natanael Copa <ncopa@alpinelinux.org> wr= ote:
Hi!

I think we need to rethink the building infrastructure. The current
build scripts were written as a quick and dirt way to get started and
have lived longer way longer than originally planned. It simply does
not scale and is very fragile.

I guess I don't need to go any deeper into why we need replace it...
I'd like to discuss what we need from the build infra and why, before w= e
start talk about how to do it and what implementation etc.

Here are some things I want a new building infra should be able to
do:

- automatic build on git push

=C2=A0 There should not be needed to do anything more than do a git push to=
=C2=A0 get the package built and uploaded. Like we have today, but with
=C2=A0 better error reporting.

- isolated environment for each build

=C2=A0 each build should set up an isolated enviroment and destroy it when<= br> =C2=A0 build is done. This could be a container but it would be nice to be<= br> =C2=A0 able to set up a disposable build env in a vm in case we want hook i= t
=C2=A0 into github PRs or similar. It should also kill everything after
=C2=A0 build and test is done so we dont get any remains of test suites tha= t
=C2=A0 does not clean up after themselves (like redis and epmd)

- support multi architectures

=C2=A0 need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Woul= d
=C2=A0 be nice it its not too complicated to add new architectures.

- support parallel building

=C2=A0 would be nice to be able to distribute the workload over available =C2=A0 build servers. Should be possible and relatively easy to add new
=C2=A0 hardware to the pool or remove or replace old without taking
=C2=A0 everything down.

- support cross compile

=C2=A0 would be nice to cross compile packages that (easily) can be cross =C2=A0 compiled. For example, we could let a big x86_64 or ppc64le machine<= br> =C2=A0 build linux kernel for armhf instead doing that on the slow armhf =C2=A0 server. Packages that cannot be cross compiled should be built on =C2=A0 native hardware.

- separate out signing process of packages and index

=C2=A0 Would be nice if we could give access to build servers to more
=C2=A0 people without giving those people access to the private signing key= s.

- build infra should be able to be used as CI

=C2=A0 We need do automatic compile checks of contributions, for example vi= a
=C2=A0 github pull requests or something corresponding.

- efficient caching

=C2=A0 Would be nice to not need to git clone the entire repo for every
=C2=A0 build for every server. Would be nice if we checkout a shared git =C2=A0 repo or do something so data does not goes over the wire more than =C2=A0 necessary. Same goes for source and apk packages.

Anything else we need from the building infra?

-nc


---
Unsubscribe:=C2=A0 alpine-devel+unsubscribe@lists.alpinelinux.org
Help:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0alpine-devel+help@lists.alpinelinux.org ---


--001a114526fe60ccc9056480795a-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mx1.tetrasec.net (mx1.tetrasec.net [74.117.190.25]) by lists.alpinelinux.org (Postfix) with ESMTP id 5A6D95C4E3E for ; Tue, 6 Feb 2018 14:19:30 +0000 (GMT) Received: from mx1.tetrasec.net (mail.local [127.0.0.1]) by mx1.tetrasec.net (Postfix) with ESMTP id DE0CD9E2490; Tue, 6 Feb 2018 14:19:29 +0000 (GMT) Received: from ncopa-macbook.copa.dup.pw (187-40-174-115.user.veloxzone.com.br [187.40.174.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) (Authenticated sender: n@tanael.org) by mx1.tetrasec.net (Postfix) with ESMTPSA id F3DDC9E20BF; Tue, 6 Feb 2018 14:19:28 +0000 (GMT) Date: Tue, 6 Feb 2018 15:19:19 +0100 From: Natanael Copa To: Daniel Isaksen Cc: Alpine Development Subject: Re: [alpine-devel] rethinking the building infra Message-ID: <20180206151919.5e12b8ae@ncopa-macbook.copa.dup.pw> In-Reply-To: References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> X-Mailer: Claws Mail 3.15.1-dirty (GTK+ 2.24.31; x86_64-alpine-linux-musl) X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Tue, 6 Feb 2018 01:43:58 +0100 Daniel Isaksen wrote: > A small addition - this is in development, and some hands on it would be > nice: https://github.com/kaniini/abuildd I know, I just wanted to list the requirements before discussing implementation. -nc --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail.wilcox-tech.com (mail.wilcox-tech.com [45.32.83.9]) by lists.alpinelinux.org (Postfix) with ESMTP id D75015C5C07 for ; Tue, 6 Feb 2018 18:31:15 +0000 (GMT) Received: (qmail 22566 invoked from network); 6 Feb 2018 18:31:12 -0000 Received: from 107-131-85-28.lightspeed.tulsok.sbcglobal.net (HELO ?192.168.1.237?) (awilcox@wilcox-tech.com@107.131.85.28) by mail.wilcox-tech.com with ESMTPA; 6 Feb 2018 18:31:12 -0000 Subject: Re: [alpine-devel] rethinking the building infra To: alpine-devel@lists.alpinelinux.org References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> From: "A. Wilcox" Organization: =?UTF-8?Q?Ad=c3=a9lie_Linux?= Message-ID: <192872b3-068a-0d7a-d618-e00e543fff3d@adelielinux.org> Date: Tue, 6 Feb 2018 12:31:22 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 In-Reply-To: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="5qemSsk7dSb2HoVMRdVVBQmU0T0ptUp6T" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --5qemSsk7dSb2HoVMRdVVBQmU0T0ptUp6T Content-Type: multipart/mixed; boundary="frhveAuVxHpumXXHUngTN5F0RGgi9XWa1"; protected-headers="v1" From: "A. Wilcox" To: alpine-devel@lists.alpinelinux.org Message-ID: <192872b3-068a-0d7a-d618-e00e543fff3d@adelielinux.org> Subject: Re: [alpine-devel] rethinking the building infra References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> In-Reply-To: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> --frhveAuVxHpumXXHUngTN5F0RGgi9XWa1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi there Natanael, I thought I could go through and say what we (Ad=C3=A9lie) would like to = see too. On 02/05/18 18:39, Natanael Copa wrote: > I'd like to discuss what we need from the build infra and why, before w= e > start talk about how to do it and what implementation etc. >=20 > Here are some things I want a new building infra should be able to > do: >=20 > - automatic build on git push >=20 > There should not be needed to do anything more than do a git push to > get the package built and uploaded. Like we have today, but with > better error reporting. That would be good. One important thing is that it may be desireable to have a "[build skip]" or such for commits that deal with non-build things (like git hooks or such) or a series with interdependencies. > - isolated environment for each build >=20 > each build should set up an isolated enviroment and destroy it when > build is done. This could be a container but it would be nice to be > able to set up a disposable build env in a vm in case we want hook it= > into github PRs or similar. It should also kill everything after > build and test is done so we dont get any remains of test suites that= > does not clean up after themselves (like redis and epmd) It should not kill everything in cases of error. I can't tell you how much time abuild has saved me by leaving src/ alone when it fails to compile or test. > - support multi architectures >=20 > need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would= > be nice it its not too complicated to add new architectures. +1 > - support parallel building >=20 > would be nice to be able to distribute the workload over available > build servers. Should be possible and relatively easy to add new > hardware to the pool or remove or replace old without taking > everything down. That would be a very nice thing. It would also be nice for it to track what builders are idle so it can pick one. I am just thinking of a scenario: Two builders: ppc64-1, ppc64-2 Four packages are modified close together: llvm4, which, unzip, acl If it just goes in sequence, 'unzip' will be held up on llvm4, when ppc64-2 could easily build it. Maybe that is too much of a "implementation detail", but I think it is an important requirement. > - support cross compile >=20 > would be nice to cross compile packages that (easily) can be cross > compiled. For example, we could let a big x86_64 or ppc64le machine > build linux kernel for armhf instead doing that on the slow armhf > server. Packages that cannot be cross compiled should be built on > native hardware. The APKBUILD format doesn't do very well with cross. Sure, some packages can cross easily, but many cannot. Please see coreutils mess in Ad=C3=A9lie repo for an example: https://code.foxkit.us/adelie/aports/blob/master/main/coreutils/APKBUILD I don't really recommend cross to 'speed up' builds, especially since it isn't possible to run tests then. Much better to wait for real hardware. Only use it for things in bootstrap.sh. > - separate out signing process of packages and index >=20 > Would be nice if we could give access to build servers to more > people without giving those people access to the private signing keys= =2E Yes, when a builder is done, it should give the built package to an internal, secure system for signing. I have always agreed with this, and the fact that the abuild signing key has to be present and readable by abuild has prevented me from giving people access to our 32-bit x86 builder. > - build infra should be able to be used as CI >=20 > We need do automatic compile checks of contributions, for example via= > github pull requests or something corresponding. I note here that GitLab has CI too, and Jenkins is open source, so Alpine doesn't always need to be tied to GitHub to still have CI :) > - efficient caching >=20 > Would be nice to not need to git clone the entire repo for every > build for every server. Would be nice if we checkout a shared git > repo or do something so data does not goes over the wire more than > necessary. Same goes for source and apk packages. Well, you could use NFS or such if the build servers are on the same local network. That is what we did at Galapagos Linux. But I'm not sure if Alpine's build systems are all located in the same network. > Anything else we need from the building infra? Ad=C3=A9lie would strongly benefit from having build logs kept in a centralised place. Something like logrotate could be used to keep it from overflowing, so log maintenance is not something that needs to be thought about. The ability to have hooks (like git) for "build started", "build success", and "build failure" would be useful. Preferrably it would allow custom ones written in shell or script, not web hoooks. Then we could have IRC poking if builds fail, and something like #alpine-commits for all build statuses. Best to you and yours, --arw >=20 > -nc --=20 A. Wilcox (awilfox) Project Lead, Ad=C3=A9lie Linux http://adelielinux.org --frhveAuVxHpumXXHUngTN5F0RGgi9XWa1-- --5qemSsk7dSb2HoVMRdVVBQmU0T0ptUp6T Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJMBAEBCAA2FiEEjNyWOYPU1SaTSMHHyynLUZIrnRQFAlp59HoYHGF3aWxmb3hA YWRlbGllbGludXgub3JnAAoJEMspy1GSK50UZYcQAIMIate1dDfgGqAgAmXfExuM eJpXrjZXwJiPtzsz9cld6CrJOfEPjPbD3m6emcDtTyg2fhP8l6WUqKRW2Bl9XMZY E17DfQsALXxOVz1W7ESIrPJJs9ErjUzYClUt1h5+fj8WjZGIQ2OFIG72ZQCzfgeQ fpbcNViA0wdGKI6yPLPEOLxNTNC827Qd1YHcCfb4IIH1waXsF4/01JZyMXllTOvQ VXYwF3HDhKEd8RM1UC2oQ+nb1SkHNFDoflgevSPvQMQig9pVX5f0iFCv0nR/XbVw mzg0RyHYxORVyEZZUzw3At1bcgyGzC3vLvWoph1RyqZcxwA1Vd8AyEpNsOj0LqAP MX9XRLdPxxRaihn82kOpoMWF4i9zYtMJ+GsvWOETcoj7jxPRPIfzyI1REeTRnqOV 0NOo6xeBHhw9wtr7nA18G0Hn/IT5wIJl9G1AeX54Zl+sKbJ5yArdxBaunhisqQaU UG2YLKP/7t2PfKfL5f1wbgvv2QTjH78/Q2Ugzmai/E+Z7Ez7OF5rMnfHD6ARCC50 xR5SRevJOIBgljznaU4SzFZcwpDT+dJvLx5fFkz5WAdxnI6+2lAjZm0lw/cu+KPv np7PDp2+RKMxa9BUZbxyJw0juIjMwpKAe4TIPRTYOmqFSPu0qLsS184Szm0ledoA jnCSfnpql9K+eiPsMesI =gyDu -----END PGP SIGNATURE----- --5qemSsk7dSb2HoVMRdVVBQmU0T0ptUp6T-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail.bitmessage.ch (mail.bitmessage.ch [146.228.112.252]) by lists.alpinelinux.org (Postfix) with ESMTP id F3DA75C4E2C for ; Wed, 7 Feb 2018 19:58:55 +0000 (GMT) dkim-signature: v=1; a=rsa-sha256; d=bitmessage.ch; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=PEeKJABe4Iz9JxhGbZWYGPpuzqZcrB3ytClg+dqhLIs=; b=IJCt+PjQg0YQOXd5WPZKiPPxhuU2vPu11tyNKmpai0YKajSCLWfHlzqO/NKN1PuTsr8fjh6Cf+vUGxI1hAc09SYIDJ4QnXX9/F+ZSpEu/1gGLZMbvdv7xDl/cLdfw0EH7INWoHbZFnxl1G14J22dB6zvrfQ7Ocwu/gu3emzislc= Received: from [127.0.0.1] (BITMESSAGE [127.0.0.1]) by mail.bitmessage.ch with ESMTPA ; Wed, 7 Feb 2018 20:58:49 +0100 Subject: Re: [alpine-devel] rethinking the building infra To: alpine-devel@lists.alpinelinux.org, Natanael Copa References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> From: Oliver Smith Message-ID: <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> Date: Wed, 07 Feb 2018 19:58:00 +0000 X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 In-Reply-To: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Hey Natanael, the list sounds good to me, especially the increased security features! While I agree with A. Wilcox that cross-compiling has the disadvantage of not running tests (or not properly if done through QEMU), I think it *does* make sense at least for kernels (there aren't any tests to execute for them). In my opinion, this is rather a detail, and the important thing would be getting everything else implemented first. With that being said: We have cross-compiling binary packages in the postmarketOS repo since we do lots of cross-compiling: gcc-armhf, binutils-armhf, musl-armhf etc. The aports for these are automatically generated from the upstream ones in Alpine (basically hardcoded variables from the bootstrap script on top). Unless the new build system would approach cross-compiling in a completely different way, you would also need such binary packages to do it, so maybe this feature can be upstreamed properly (that is the long-term plan anyway, if you guys want to have this - maybe with subpackages?). Regarding caching: Maybe ccache makes sense, if it was separated for each package, and incoming pull-requests could only read from that cache, not write to it. Finally, I am not sure if the idea is to replace abuild entirely, or just abuild-rootbld. In case abuild should be replaced, I hope to see the following features preserved (since we make use of them in pmbootstrap, which wraps abuild): - (passing through environment variables) - abuild -r (install depends with abuild) - abuild -d (do not install depends with abuild) - abuild -f (force) - abuild undeps - abuild menuconfig - abuild checksum - abuild unpack - abuild prepare Thanks, Oliver Natanael Copa: > Hi! > > I think we need to rethink the building infrastructure. The current > build scripts were written as a quick and dirt way to get started and > have lived longer way longer than originally planned. It simply does > not scale and is very fragile. > > I guess I don't need to go any deeper into why we need replace it... > > I'd like to discuss what we need from the build infra and why, before we > start talk about how to do it and what implementation etc. > > Here are some things I want a new building infra should be able to > do: > > - automatic build on git push > > There should not be needed to do anything more than do a git push to > get the package built and uploaded. Like we have today, but with > better error reporting. > > - isolated environment for each build > > each build should set up an isolated enviroment and destroy it when > build is done. This could be a container but it would be nice to be > able to set up a disposable build env in a vm in case we want hook it > into github PRs or similar. It should also kill everything after > build and test is done so we dont get any remains of test suites that > does not clean up after themselves (like redis and epmd) > > - support multi architectures > > need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would > be nice it its not too complicated to add new architectures. > > - support parallel building > > would be nice to be able to distribute the workload over available > build servers. Should be possible and relatively easy to add new > hardware to the pool or remove or replace old without taking > everything down. > > - support cross compile > > would be nice to cross compile packages that (easily) can be cross > compiled. For example, we could let a big x86_64 or ppc64le machine > build linux kernel for armhf instead doing that on the slow armhf > server. Packages that cannot be cross compiled should be built on > native hardware. > > - separate out signing process of packages and index > > Would be nice if we could give access to build servers to more > people without giving those people access to the private signing keys. > > - build infra should be able to be used as CI > > We need do automatic compile checks of contributions, for example via > github pull requests or something corresponding. > > - efficient caching > > Would be nice to not need to git clone the entire repo for every > build for every server. Would be nice if we checkout a shared git > repo or do something so data does not goes over the wire more than > necessary. Same goes for source and apk packages. > > Anything else we need from the building infra? > > -nc > > > --- > Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org > Help: alpine-devel+help@lists.alpinelinux.org > --- > > --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail.wilcox-tech.com (mail.wilcox-tech.com [45.32.83.9]) by lists.alpinelinux.org (Postfix) with ESMTP id B2B225C4E2F for ; Wed, 7 Feb 2018 20:33:34 +0000 (GMT) Received: (qmail 24466 invoked from network); 7 Feb 2018 20:33:31 -0000 Received: from 107-131-85-28.lightspeed.tulsok.sbcglobal.net (HELO ?192.168.1.237?) (awilcox@wilcox-tech.com@107.131.85.28) by mail.wilcox-tech.com with ESMTPA; 7 Feb 2018 20:33:31 -0000 Subject: Re: [alpine-devel] rethinking the building infra To: alpine-devel@lists.alpinelinux.org References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> From: "A. Wilcox" Organization: =?UTF-8?Q?Ad=c3=a9lie_Linux?= Message-ID: <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> Date: Wed, 7 Feb 2018 14:33:42 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 In-Reply-To: <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="xCkaaxTvuqoaxLJEFE0qfWGQiBSjlmbDv" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --xCkaaxTvuqoaxLJEFE0qfWGQiBSjlmbDv Content-Type: multipart/mixed; boundary="gxfuElKE4hQMhJThHMncRuQ2xAiLUQIV2"; protected-headers="v1" From: "A. Wilcox" To: alpine-devel@lists.alpinelinux.org Message-ID: <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> Subject: Re: [alpine-devel] rethinking the building infra References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> In-Reply-To: <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> --gxfuElKE4hQMhJThHMncRuQ2xAiLUQIV2 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 02/07/18 13:58, Oliver Smith wrote: > Hey Natanael, >=20 > the list sounds good to me, especially the increased security > features! >=20 > While I agree with A. Wilcox that cross-compiling has the > disadvantage of not running tests (or not properly if done through > QEMU), I think it *does* make sense at least for kernels (there > aren't any tests to execute for them). In my opinion, this is rather > a detail, and the important thing would be getting everything else > implemented first. With that being said: We have cross-compiling > binary packages in the postmarketOS repo since we do lots of > cross-compiling: gcc-armhf, binutils-armhf, musl-armhf etc. The > aports for these are automatically generated from the upstream ones > in Alpine (basically hardcoded variables from the bootstrap script on > top). Unless the new build system would approach cross-compiling in a > completely different way, you would also need such binary packages to > do it, so maybe this feature can be upstreamed properly (that is the > long-term plan anyway, if you guys want to have this - maybe with > subpackages?). I'm not sure what feature you are talking about. There is already bootstrap.sh to generate a working cross environment. I suppose you could make a bootstrap-lite.sh or such that only makes the toolchain and none of the other components, which would work for a kernel build. > Regarding caching: Maybe ccache makes sense, if it was separated for > each package, and incoming pull-requests could only read from that > cache, not write to it. I think the cache point is more for the actual git data and distfiles, not the compilation. > Finally, I am not sure if the idea is to replace abuild entirely, or > just abuild-rootbld. Who said abuild was being replaced? This is about `buildrepo` and friends, not abuild. > In case abuild should be replaced, I hope to see > the following features preserved (since we make use of them in > pmbootstrap, which wraps abuild): > > - (passing through environment variables) This isn't a wise idea. It's better to export them in the build() or package() functions. > - abuild -r (install depends with abuild) This is currently how all Ad=C3=A9lie packages are built, but it is a bit= buggy. It is useful for end users making single contributions, but honestly, I think that this new system is going to be much better than `abuild -r`. > - abuild undeps=20 Due to the way abuild works right now, `apk del .makedepends-$pkgname` has the same effect, if you ever need to do that without abuild. That's an implementation detail and is not guaranteed to always work, but it does work right now if you need it. > - abuild menuconfig ??? This isn't even an abuild feature. It (ab)uses the fact that abuild phases are shell functions by calling a menuconfig() function in the kernel APKBUILD. Honestly, I would be much in favour of /not/ abusing that fact, and making abuild more hardened against that, but that is just me. Best, --arw --=20 A. Wilcox (awilfox) Project Lead, Ad=C3=A9lie Linux http://adelielinux.org --gxfuElKE4hQMhJThHMncRuQ2xAiLUQIV2-- --xCkaaxTvuqoaxLJEFE0qfWGQiBSjlmbDv Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJMBAEBCAA2FiEEjNyWOYPU1SaTSMHHyynLUZIrnRQFAlp7YqYYHGF3aWxmb3hA YWRlbGllbGludXgub3JnAAoJEMspy1GSK50UNjkQAIXWr4Ak6wZBLP/Yt/qpKfvu xL64IAah3I4m9LhRAjojb/ZZWaKhtQK/+ZiMuqDXs2COimaLGtcoy82I+mA/eOnC cw3K8j2qpGjdf3//BKgTvDPf+vkjHM1DcAwSth24eArt93x7Mjl6qwbYOPg4MN7f S277cp3fyzTzmZ+JvQEDxvE3YwUfixbjrnroL2SnL7z5KLIVpMKF2EGp0xNawttd b+xFQ2Jmvt/Nm0kpduE+/Bx2VEqaM11667UUSiEIAbEK1IBHq26daNh6zAAzlbBo IcsIZPXcd8XXBOOgYJLSsRhwL4mbRhDXiSJEjZsfjcaMRqxoWzLsQggQAvDwWiZj 4Y/Y0oHcfKxOMzi2V/q0Uj2L5JVHaAGov3aqbEum9BzNLnLGayHXjHCq3V/nY8Fl SmLuhDbuRkGfGPZImT6V4gFhQtbfW4kV9osXJ3VdJjXLK17CYcp3Pqnz9uD/M94s II6tc8YtooMr5p3QRFgyJxYKCnrHx1+85D5P4Wi4OZ/gZV9ExvqdtW7g3++aACI4 ssUoKyQZ723esU88YkDNwxiFn3by6CYXPKGse4OANe1JTZnXGSudTYd/gdhIwrkW jgNDORkeRbx+MSb5GAfzqsS6Gheg9YbOu8sHEzW7IZPoIP9M7zPZHmxSjojQ2lZt XAO4q39WEcmFmbWM2Eqz =WZBM -----END PGP SIGNATURE----- --xCkaaxTvuqoaxLJEFE0qfWGQiBSjlmbDv-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org --- From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail.bitmessage.ch (mail.bitmessage.ch [146.228.112.252]) by lists.alpinelinux.org (Postfix) with ESMTP id 7985D5C4E4B for ; Wed, 7 Feb 2018 22:53:37 +0000 (GMT) dkim-signature: v=1; a=rsa-sha256; d=bitmessage.ch; s=mail; c=relaxed/relaxed; q=dns/txt; h=From:Subject:Date:Message-ID:To:MIME-Version:Content-Type:Content-Transfer-Encoding:In-Reply-To:References; bh=7pD7S4H/P1PX/Mj9sdU1AParVribsPd44b84w32uXLI=; b=nGuert1o9F1Vp0cFJ3ys9F7TgLjNwPQGTNAno8vvEAer9BctfjF2XAkIGkYIMpc8C5i6+5AxvkoaFZjxzxI/m5XMgYE6TiwpDeAVVld9cyvEqC+VB1ni+kceyM/BRdjNlqmmHWph3DkjiJFHcjvZsVgitsfkPWkTRXTjF2ZGthg= Received: from [127.0.0.1] (BITMESSAGE [127.0.0.1]) by mail.bitmessage.ch with ESMTPA ; Wed, 7 Feb 2018 23:52:54 +0100 Subject: Re: [alpine-devel] rethinking the building infra To: "A. Wilcox" , alpine-devel@lists.alpinelinux.org References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> From: Oliver Smith Message-ID: <5e9111f1-2985-8597-b804-ddcb102a24fc@bitmessage.ch> Date: Wed, 07 Feb 2018 22:53:00 +0000 X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 In-Reply-To: <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: base64 QS4gV2lsY294Og0KPiBPbiAwMi8wNy8xOCAxMzo1OCwgT2xpdmVyIFNtaXRoIHdyb3RlOg0KPj4g SGV5IE5hdGFuYWVsLA0KPj4NCj4+IHRoZSBsaXN0IHNvdW5kcyBnb29kIHRvIG1lLCBlc3BlY2lh bGx5IHRoZSBpbmNyZWFzZWQgc2VjdXJpdHkNCj4+IGZlYXR1cmVzIQ0KPj4NCj4+IFdoaWxlIEkg YWdyZWUgd2l0aCBBLiBXaWxjb3ggdGhhdCBjcm9zcy1jb21waWxpbmcgaGFzIHRoZQ0KPj4gZGlz YWR2YW50YWdlIG9mIG5vdCBydW5uaW5nIHRlc3RzIChvciBub3QgcHJvcGVybHkgaWYgZG9uZSB0 aHJvdWdoDQo+PiBRRU1VKSwgSSB0aGluayBpdCAqZG9lcyogbWFrZSBzZW5zZSBhdCBsZWFzdCBm b3Iga2VybmVscyAodGhlcmUNCj4+IGFyZW4ndCBhbnkgdGVzdHMgdG8gZXhlY3V0ZSBmb3IgdGhl bSkuIEluIG15IG9waW5pb24sIHRoaXMgaXMgcmF0aGVyDQo+PiBhIGRldGFpbCwgYW5kIHRoZSBp bXBvcnRhbnQgdGhpbmcgd291bGQgYmUgZ2V0dGluZyBldmVyeXRoaW5nIGVsc2UNCj4+IGltcGxl bWVudGVkIGZpcnN0LiBXaXRoIHRoYXQgYmVpbmcgc2FpZDogV2UgaGF2ZSBjcm9zcy1jb21waWxp bmcNCj4+IGJpbmFyeSBwYWNrYWdlcyBpbiB0aGUgcG9zdG1hcmtldE9TIHJlcG8gc2luY2Ugd2Ug ZG8gbG90cyBvZg0KPj4gY3Jvc3MtY29tcGlsaW5nOiBnY2MtYXJtaGYsIGJpbnV0aWxzLWFybWhm LCBtdXNsLWFybWhmIGV0Yy4gVGhlDQo+PiBhcG9ydHMgZm9yIHRoZXNlIGFyZSBhdXRvbWF0aWNh bGx5IGdlbmVyYXRlZCBmcm9tIHRoZSB1cHN0cmVhbSBvbmVzDQo+PiBpbiBBbHBpbmUgKGJhc2lj YWxseSBoYXJkY29kZWQgdmFyaWFibGVzIGZyb20gdGhlIGJvb3RzdHJhcCBzY3JpcHQgb24NCj4+ IHRvcCkuIFVubGVzcyB0aGUgbmV3IGJ1aWxkIHN5c3RlbSB3b3VsZCBhcHByb2FjaCBjcm9zcy1j b21waWxpbmcgaW4gYQ0KPj4gY29tcGxldGVseSBkaWZmZXJlbnQgd2F5LCB5b3Ugd291bGQgYWxz byBuZWVkIHN1Y2ggYmluYXJ5IHBhY2thZ2VzIHRvDQo+PiBkbyBpdCwgc28gbWF5YmUgdGhpcyBm ZWF0dXJlIGNhbiBiZSB1cHN0cmVhbWVkIHByb3Blcmx5ICh0aGF0IGlzIHRoZQ0KPj4gbG9uZy10 ZXJtIHBsYW4gYW55d2F5LCBpZiB5b3UgZ3V5cyB3YW50IHRvIGhhdmUgdGhpcyAtIG1heWJlIHdp dGgNCj4+IHN1YnBhY2thZ2VzPykuDQo+IA0KPiANCj4gSSdtIG5vdCBzdXJlIHdoYXQgZmVhdHVy ZSB5b3UgYXJlIHRhbGtpbmcgYWJvdXQuICBUaGVyZSBpcyBhbHJlYWR5DQo+IGJvb3RzdHJhcC5z aCB0byBnZW5lcmF0ZSBhIHdvcmtpbmcgY3Jvc3MgZW52aXJvbm1lbnQuICBJIHN1cHBvc2UgeW91 DQo+IGNvdWxkIG1ha2UgYSBib290c3RyYXAtbGl0ZS5zaCBvciBzdWNoIHRoYXQgb25seSBtYWtl cyB0aGUgdG9vbGNoYWluIGFuZA0KPiBub25lIG9mIHRoZSBvdGhlciBjb21wb25lbnRzLCB3aGlj aCB3b3VsZCB3b3JrIGZvciBhIGtlcm5lbCBidWlsZC4NCg0KV2hhdCBJIG1lYW4gaXMgaGF2aW5n IHBhY2thZ2VzIGRpcmVjdGx5IGluIHRoZSBiaW5hcnkgcmVwb3NpdG9yeSwgc28gb25lIGNhbiBk bzoNCiQgYXBrIGFkZCBnY2MtYXJtaGYNCg0KSnVzdCBsaWtlIGl0J3MgcG9zc2libGUgdG8gaW5z dGFsbCBnY2MtYXZyIHJpZ2h0IG5vdy4gQnV0IGdjYy1hdnIgcGFja2FnZSBuZWVkcyB0byBiZSBt YW51YWxseSBzeW5jZWQgd2l0aCB0aGUgZ2NjIGFwb3J0LiBJbnN0ZWFkIG9mIHRoYXQgaXQgd291 bGQgYmUgbmljZSwgaWYgd2UgaGFkIGNyb3NzLWNvbXBpbGVycyBhdXRvbWF0aWNhbGx5IGJ1aWx0 IGZvciBhbGwgYXJjaGl0ZWN0dXJlcywgd2l0aG91dCBydW5uaW5nIGJvb3RzdHJhcC5zaC4gUG9z c2libHkgYXMgc3VicGFja2FnZSBvZiBnY2MgKGJ1dCB0aGF0J3MgcHJvYmFibHkgbm90IHRoZSBk ZXNpcmVkIHNvbHV0aW9uLCBzaW5jZSB0aGF0IHdpbGwgYmxvdyB1cCB0aGUgYnVpbGQgdGltZSBv ZiBHQ0MgZHJhc3RpY2FsbHkpLiBJIHRoaW5rIHRoYXQgdGhpcyBtaWdodCBiZSByZWxldmFudCB0 byBjcm9zcy1jb21waWxpbmcgaW4gdGhlIGJpbmFyeSByZXBvc2l0b3J5LCBiZWNhdXNlIG9uY2Ug c3VjaCBwYWNrYWdlcyBpbiB0aGUgYmluYXJ5IHJlcG8gZXhpc3QsIGl0IHdvdWxkIGJlIGEgY2xl YW4gd2F5IHRvIGluc3RhbGwgdGhlIGNyb3NzLWNvbXBpbGVyIGluIHRoZSBjb21waWxpbmcgVk0v Y29udGFpbmVyIHdpdGggYXBrLg0KDQo+IA0KPiANCj4+IFJlZ2FyZGluZyBjYWNoaW5nOiBNYXli ZSBjY2FjaGUgbWFrZXMgc2Vuc2UsIGlmIGl0IHdhcyBzZXBhcmF0ZWQgZm9yDQo+PiBlYWNoIHBh Y2thZ2UsIGFuZCBpbmNvbWluZyBwdWxsLXJlcXVlc3RzIGNvdWxkIG9ubHkgcmVhZCBmcm9tIHRo YXQNCj4+IGNhY2hlLCBub3Qgd3JpdGUgdG8gaXQuDQo+IA0KPiANCj4gSSB0aGluayB0aGUgY2Fj aGUgcG9pbnQgaXMgbW9yZSBmb3IgdGhlIGFjdHVhbCBnaXQgZGF0YSBhbmQgZGlzdGZpbGVzLA0K PiBub3QgdGhlIGNvbXBpbGF0aW9uLg0KPiANCg0KSSB1bmRlcnN0b29kIHRoYXQsIHRoaXMgd2Fz IGp1c3QgbWUgdGhyb3dpbmcgYSBkaWZmZXJlbnQgaWRlYSBpbiB0byByZWR1Y2UgY29tcGlsYXRp b24gdGltZXMgZm9yIGNvbW1pdHMgd2hpY2ggb25seSBoYXZlIHNtYWxsIGNoYW5nZXMgKGNvbmZp Z3VyZSBmbGFnIGNoYW5nZWQgZXRjLikuDQoNCj4gDQo+PiBGaW5hbGx5LCBJIGFtIG5vdCBzdXJl IGlmIHRoZSBpZGVhIGlzIHRvIHJlcGxhY2UgYWJ1aWxkIGVudGlyZWx5LCBvcg0KPj4ganVzdCBh YnVpbGQtcm9vdGJsZC4NCj4gDQo+IA0KPiBXaG8gc2FpZCBhYnVpbGQgd2FzIGJlaW5nIHJlcGxh Y2VkPyAgVGhpcyBpcyBhYm91dCBgYnVpbGRyZXBvYCBhbmQNCj4gZnJpZW5kcywgbm90IGFidWls ZC4NCg0KVGhhbmtzIGZvciBjbGVhcmluZyB0aGF0IHVwLg0KDQo+IA0KPiANCj4+IEluIGNhc2Ug YWJ1aWxkIHNob3VsZCBiZSByZXBsYWNlZCwgSSBob3BlIHRvIHNlZQ0KPj4gdGhlIGZvbGxvd2lu ZyBmZWF0dXJlcyBwcmVzZXJ2ZWQgKHNpbmNlIHdlIG1ha2UgdXNlIG9mIHRoZW0gaW4NCj4+IHBt Ym9vdHN0cmFwLCB3aGljaCB3cmFwcyBhYnVpbGQpOg0KPj4NCj4+IC0gKHBhc3NpbmcgdGhyb3Vn aCBlbnZpcm9ubWVudCB2YXJpYWJsZXMpDQo+IA0KPiBUaGlzIGlzbid0IGEgd2lzZSBpZGVhLiAg SXQncyBiZXR0ZXIgdG8gZXhwb3J0IHRoZW0gaW4gdGhlIGJ1aWxkKCkgb3INCj4gcGFja2FnZSgp IGZ1bmN0aW9ucy4NCj4gDQoNCkZvciBwYWNrYWdlIHNwZWNpZmljIHZhcmlhYmxlcywgd2UgZXhw b3J0IHRoZW0gaW4gdGhlIEFQS0JVSUxELiBXZSBvbmx5IHBhc3MgdGhlbSB0aHJvdWdoIGFidWls ZCBmb3IgcGFja2FnZSBpbmRlcGVuZGVudCB2YXJpYWJsZXMsIHN1Y2ggYXM6DQpDQVJDSCwgQ1JP U1NfQ09NUElMRSwgQ0MsIENDQUNIRV9QUkVGSVgsIENDQUNIRV9QQVRILCBDQ0FDSEVfQ09NUElM RVJDSEVDSywgRElTVENDX0hPU1RTDQoNCj4+IC0gYWJ1aWxkIC1yIChpbnN0YWxsIGRlcGVuZHMg d2l0aCBhYnVpbGQpDQo+IA0KPiBUaGlzIGlzIGN1cnJlbnRseSBob3cgYWxsIEFkw6lsaWUgcGFj a2FnZXMgYXJlIGJ1aWx0LCBidXQgaXQgaXMgYSBiaXQNCj4gYnVnZ3kuICBJdCBpcyB1c2VmdWwg Zm9yIGVuZCB1c2VycyBtYWtpbmcgc2luZ2xlIGNvbnRyaWJ1dGlvbnMsIGJ1dA0KPiBob25lc3Rs eSwgSSB0aGluayB0aGF0IHRoaXMgbmV3IHN5c3RlbSBpcyBnb2luZyB0byBiZSBtdWNoIGJldHRl ciB0aGFuDQo+IGBhYnVpbGQgLXJgLg0KPiANCj4+IC0gYWJ1aWxkIHVuZGVwcyANCj4gDQo+IER1 ZSB0byB0aGUgd2F5IGFidWlsZCB3b3JrcyByaWdodCBub3csIGBhcGsgZGVsIC5tYWtlZGVwZW5k cy0kcGtnbmFtZWANCj4gaGFzIHRoZSBzYW1lIGVmZmVjdCwgaWYgeW91IGV2ZXIgbmVlZCB0byBk byB0aGF0IHdpdGhvdXQgYWJ1aWxkLiAgVGhhdCdzDQo+IGFuIGltcGxlbWVudGF0aW9uIGRldGFp bCBhbmQgaXMgbm90IGd1YXJhbnRlZWQgdG8gYWx3YXlzIHdvcmssIGJ1dCBpdA0KPiBkb2VzIHdv cmsgcmlnaHQgbm93IGlmIHlvdSBuZWVkIGl0Lg0KDQpSaWdodCwgdGhhbmtzIQ0KDQo+IA0KPj4g LSBhYnVpbGQgbWVudWNvbmZpZw0KPiANCj4gPz8/ICBUaGlzIGlzbid0IGV2ZW4gYW4gYWJ1aWxk IGZlYXR1cmUuICBJdCAoYWIpdXNlcyB0aGUgZmFjdCB0aGF0DQo+IGFidWlsZCBwaGFzZXMgYXJl IHNoZWxsIGZ1bmN0aW9ucyBieSBjYWxsaW5nIGEgbWVudWNvbmZpZygpIGZ1bmN0aW9uIGluDQo+ IHRoZSBrZXJuZWwgQVBLQlVJTEQuICBIb25lc3RseSwgSSB3b3VsZCBiZSBtdWNoIGluIGZhdm91 ciBvZiAvbm90Lw0KPiBhYnVzaW5nIHRoYXQgZmFjdCwgYW5kIG1ha2luZyBhYnVpbGQgbW9yZSBo YXJkZW5lZCBhZ2FpbnN0IHRoYXQsIGJ1dA0KPiB0aGF0IGlzIGp1c3QgbWUuDQo+IA0KDQpBbHBp bmUncyBsaW51eC12YW5pbGxhIEFQS0JVSUxEIHVzZWQgdG8gaGF2ZSBhIG1lbnVjb25maWcoKSBm dW5jdGlvbiB3aXRoIGEgY29tbWVudCBvbiB0b3Agc2F5aW5nIHNvbWV0aGluZyBsaWtlICIjIFRo aXMgaXMgc28gd2UgY2FuIHVzZSAnYWJ1aWxkIG1lbnVjb25maWcnIi4gQnV0IEkganVzdCByZWFs aXplZCB0aGF0IHRoaXMgd2FzIHJlbW92ZWQuIFdlbGwsIHdlIHN0aWxsIHVzZSB0aGF0IGZlYXR1 cmUgZm9yIHRoYXQgcHVycG9zZS4gQnV0IGlmIHRoYXQgZGlkIG5vdCB3b3JrIGFueW1vcmUgaW4g YWJ1aWxkLCB3ZSBjb3VsZCBjYWxsIG1lbnVjb25maWcgZGlyZWN0bHkuDQoNClRoYW5rcywNCk9s aXZlcg0KDQo+IA0KPiBCZXN0LA0KPiAtLWFydw0KPiANCg0KDQoNCi0tLQ0KVW5zdWJzY3JpYmU6 ICBhbHBpbmUtZGV2ZWwrdW5zdWJzY3JpYmVAbGlzdHMuYWxwaW5lbGludXgub3JnDQpIZWxwOiAg ICAgICAgIGFscGluZS1kZXZlbCtoZWxwQGxpc3RzLmFscGluZWxpbnV4Lm9yZw0KLS0tDQo= From nobody Fri Mar 29 10:11:28 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail.wilcox-tech.com (mail.wilcox-tech.com [45.32.83.9]) by lists.alpinelinux.org (Postfix) with ESMTP id E10C45C4E51 for ; Wed, 7 Feb 2018 23:39:06 +0000 (GMT) Received: (qmail 29045 invoked from network); 7 Feb 2018 23:39:03 -0000 Received: from 107-131-85-28.lightspeed.tulsok.sbcglobal.net (HELO ?192.168.1.237?) (awilcox@wilcox-tech.com@107.131.85.28) by mail.wilcox-tech.com with ESMTPA; 7 Feb 2018 23:39:03 -0000 Subject: Re: [alpine-devel] rethinking the building infra To: Oliver Smith , alpine-devel@lists.alpinelinux.org References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> <5e9111f1-2985-8597-b804-ddcb102a24fc@bitmessage.ch> From: "A. Wilcox" Organization: =?UTF-8?Q?Ad=c3=a9lie_Linux?= Message-ID: <45c16fdd-4091-0a50-e1b6-b025f949ae37@adelielinux.org> Date: Wed, 7 Feb 2018 17:39:13 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 In-Reply-To: <5e9111f1-2985-8597-b804-ddcb102a24fc@bitmessage.ch> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="O21xauSAIcQlDJdVUaMhDH4xXTothInbQ" This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --O21xauSAIcQlDJdVUaMhDH4xXTothInbQ Content-Type: multipart/mixed; boundary="C1xEmd5fgTplttFppu5HJORvqsqGDgRCl"; protected-headers="v1" From: "A. Wilcox" To: Oliver Smith , alpine-devel@lists.alpinelinux.org Message-ID: <45c16fdd-4091-0a50-e1b6-b025f949ae37@adelielinux.org> Subject: Re: [alpine-devel] rethinking the building infra References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw> <7676a963-d2b0-bf15-4f51-f5aa0d034e9a@bitmessage.ch> <290b7bc4-203d-dad4-21e5-3892537a2a76@adelielinux.org> <5e9111f1-2985-8597-b804-ddcb102a24fc@bitmessage.ch> In-Reply-To: <5e9111f1-2985-8597-b804-ddcb102a24fc@bitmessage.ch> --C1xEmd5fgTplttFppu5HJORvqsqGDgRCl Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 02/07/18 16:53, Oliver Smith wrote: > What I mean is having packages directly in the binary repository, so > one can do: $ apk add gcc-armhf >=20 > Just like it's possible to install gcc-avr right now. But gcc-avr > package needs to be manually synced with the gcc aport. Instead of > that it would be nice, if we had cross-compilers automatically built > for all architectures, without running bootstrap.sh. Possibly as > subpackage of gcc (but that's probably not the desired solution, > since that will blow up the build time of GCC drastically). I think > that this might be relevant to cross-compiling in the binary > repository, because once such packages in the binary repo exist, it > would be a clean way to install the cross-compiler in the compiling > VM/container with apk. We (Ad=C3=A9lie) do this on x86_64 and ppc64 arches already (both of them= have various gcc-* binutils-* etc). It would be cool if Alpine wanted to do that. And necessary, if kernels are cross-built. > For package specific variables, we export them in the APKBUILD. We > only pass them through abuild for package independent variables, such > as: CARCH, CROSS_COMPILE, CC, CCACHE_PREFIX, CCACHE_PATH, > CCACHE_COMPILERCHECK, DISTCC_HOSTS CARCH and CROSS_COMPILE makes sense, since they are used by abuild. CC, CCACHE_*, and DISTCC_* make more sense in /etc/abuild.conf imo. > Alpine's linux-vanilla APKBUILD used to have a menuconfig() function > with a comment on top saying something like "# This is so we can use > 'abuild menuconfig'". But I just realized that this was removed. > Well, we still use that feature for that purpose. But if that did not > work anymore in abuild, we could call menuconfig directly. Yes, I personally think this is a bad idea and abuses the fact that APKBUILD files are `source`d by abuild. It is better to remove from the APKBUILD. I would even be in favour of having a little shell script in the linux-vanilla package or such that takes CARCH and CROSS_COMPILE like abuild does, and calls menuconfig properly for you. But not directly from the APKBUILD. Best, --arw --=20 A. Wilcox (awilfox) Project Lead, Ad=C3=A9lie Linux http://adelielinux.org --C1xEmd5fgTplttFppu5HJORvqsqGDgRCl-- --O21xauSAIcQlDJdVUaMhDH4xXTothInbQ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJMBAEBCAA2FiEEjNyWOYPU1SaTSMHHyynLUZIrnRQFAlp7jiEYHGF3aWxmb3hA YWRlbGllbGludXgub3JnAAoJEMspy1GSK50Ur2cP/2dfEdNy3kK1xAaNVuBpd4wP 62aMA24mF875xQH5e6l5tAPsNV/5yeVeKlOKJE4AYTLLq/g5oAA9AqGOOjDhNLaA PvJCIY5GUfCgbOPg5PRUdPF3bS2gHasw8v5GlQMQQc9pzfOLUecjnuKmEv0ao9P2 Quq7/D7iFqw94HT0As/Uh/VlTzW+AfUg73+qAzqlsuDeMGFtshFCpCUhad48uYdM Pq/Hge3l2Mf8It4vGOiYqg40Xra4KMCrWLzIuhOYgaLOnpjgd1bB2Q4VMj+I7KoC GrskBKwJjWDlxtjZFUyI5CJP3EbnLrPAaV1MF25nFWupxHZAYhEkP4+ApEvg/mij FO4bqXDX4P8SBzmIUIjXdD6y5ikzq+ayLalJUiAKBqj0HHfl8rRwBbz762U+hZLb 7FPJe6PudyF7woZCU1ZE9r/dp8RJfCjsF/H65pOxedRTZkU/KC4TlSWlvygj0PoP bN0zwNWlxRaWZf/gHRIilZmCc09U70cZ7ahC0ArlaeCg66//jGNbyZiya2aRTRgJ OsKRkM6vOmcfEShCpwhvn16D/ZfZOVJTJs0MPuJVZaTnekLoW/uRdzThnB4evQ7m XecG7hvIJpKK8KFG69oEFGOYCeEvs/jo+jvFP0s1z2tFz0mu0dt3Qu4ha7L+nkKh mLfktEfda5dhR9nefCvy =uAGj -----END PGP SIGNATURE----- --O21xauSAIcQlDJdVUaMhDH4xXTothInbQ-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org ---