Precedence: list
MIME-Version: 1.0
In-Reply-To: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw>
References: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw>
From: Daniel Isaksen <d@duniel.no>
Date: Tue, 6 Feb 2018 01:43:58 +0100
Message-ID: <CAFWK1CDG0Vncg2axfwSUy7jgTszHgw8SFu244q8Y7SzgyzNVrw@mail.gmail.com>
Subject: Re: [alpine-devel] rethinking the building infra
To: Natanael Copa <ncopa@alpinelinux.org>
Cc: Alpine Development <alpine-devel@lists.alpinelinux.org>
Content-Type: multipart/alternative; boundary="001a114526fe60ccc9056480795a"

--001a114526fe60ccc9056480795a
Content-Type: text/plain; charset="UTF-8"

A small addition - this is in development, and some hands on it would be
nice: https://github.com/kaniini/abuildd

- Daniel

On Tue, Feb 6, 2018 at 1:39 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:

> Hi!
>
> I think we need to rethink the building infrastructure. The current
> build scripts were written as a quick and dirt way to get started and
> have lived longer way longer than originally planned. It simply does
> not scale and is very fragile.
>
> I guess I don't need to go any deeper into why we need replace it...
>
> I'd like to discuss what we need from the build infra and why, before we
> start talk about how to do it and what implementation etc.
>
> Here are some things I want a new building infra should be able to
> do:
>
> - automatic build on git push
>
>   There should not be needed to do anything more than do a git push to
>   get the package built and uploaded. Like we have today, but with
>   better error reporting.
>
> - isolated environment for each build
>
>   each build should set up an isolated enviroment and destroy it when
>   build is done. This could be a container but it would be nice to be
>   able to set up a disposable build env in a vm in case we want hook it
>   into github PRs or similar. It should also kill everything after
>   build and test is done so we dont get any remains of test suites that
>   does not clean up after themselves (like redis and epmd)
>
> - support multi architectures
>
>   need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would
>   be nice it its not too complicated to add new architectures.
>
> - support parallel building
>
>   would be nice to be able to distribute the workload over available
>   build servers. Should be possible and relatively easy to add new
>   hardware to the pool or remove or replace old without taking
>   everything down.
>
> - support cross compile
>
>   would be nice to cross compile packages that (easily) can be cross
>   compiled. For example, we could let a big x86_64 or ppc64le machine
>   build linux kernel for armhf instead doing that on the slow armhf
>   server. Packages that cannot be cross compiled should be built on
>   native hardware.
>
> - separate out signing process of packages and index
>
>   Would be nice if we could give access to build servers to more
>   people without giving those people access to the private signing keys.
>
> - build infra should be able to be used as CI
>
>   We need do automatic compile checks of contributions, for example via
>   github pull requests or something corresponding.
>
> - efficient caching
>
>   Would be nice to not need to git clone the entire repo for every
>   build for every server. Would be nice if we checkout a shared git
>   repo or do something so data does not goes over the wire more than
>   necessary. Same goes for source and apk packages.
>
> Anything else we need from the building infra?
>
> -nc
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
>
>

--001a114526fe60ccc9056480795a
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">A small addition - this is in development, and some hands =
on it would be nice:=C2=A0<a href=3D"https://github.com/kaniini/abuildd">ht=
tps://github.com/kaniini/abuildd</a><div><br></div><div>- Daniel</div></div=
><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Tue, Feb 6, 2=
018 at 1:39 AM, Natanael Copa <span dir=3D"ltr">&lt;<a href=3D"mailto:ncopa=
@alpinelinux.org" target=3D"_blank">ncopa@alpinelinux.org</a>&gt;</span> wr=
ote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border=
-left:1px #ccc solid;padding-left:1ex">Hi!<br>
<br>
I think we need to rethink the building infrastructure. The current<br>
build scripts were written as a quick and dirt way to get started and<br>
have lived longer way longer than originally planned. It simply does<br>
not scale and is very fragile.<br>
<br>
I guess I don&#39;t need to go any deeper into why we need replace it...<br=
>
<br>
I&#39;d like to discuss what we need from the build infra and why, before w=
e<br>
start talk about how to do it and what implementation etc.<br>
<br>
Here are some things I want a new building infra should be able to<br>
do:<br>
<br>
- automatic build on git push<br>
<br>
=C2=A0 There should not be needed to do anything more than do a git push to=
<br>
=C2=A0 get the package built and uploaded. Like we have today, but with<br>
=C2=A0 better error reporting.<br>
<br>
- isolated environment for each build<br>
<br>
=C2=A0 each build should set up an isolated enviroment and destroy it when<=
br>
=C2=A0 build is done. This could be a container but it would be nice to be<=
br>
=C2=A0 able to set up a disposable build env in a vm in case we want hook i=
t<br>
=C2=A0 into github PRs or similar. It should also kill everything after<br>
=C2=A0 build and test is done so we dont get any remains of test suites tha=
t<br>
=C2=A0 does not clean up after themselves (like redis and epmd)<br>
<br>
- support multi architectures<br>
<br>
=C2=A0 need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Woul=
d<br>
=C2=A0 be nice it its not too complicated to add new architectures.<br>
<br>
- support parallel building<br>
<br>
=C2=A0 would be nice to be able to distribute the workload over available<b=
r>
=C2=A0 build servers. Should be possible and relatively easy to add new<br>
=C2=A0 hardware to the pool or remove or replace old without taking<br>
=C2=A0 everything down.<br>
<br>
- support cross compile<br>
<br>
=C2=A0 would be nice to cross compile packages that (easily) can be cross<b=
r>
=C2=A0 compiled. For example, we could let a big x86_64 or ppc64le machine<=
br>
=C2=A0 build linux kernel for armhf instead doing that on the slow armhf<br=
>
=C2=A0 server. Packages that cannot be cross compiled should be built on<br=
>
=C2=A0 native hardware.<br>
<br>
- separate out signing process of packages and index<br>
<br>
=C2=A0 Would be nice if we could give access to build servers to more<br>
=C2=A0 people without giving those people access to the private signing key=
s.<br>
<br>
- build infra should be able to be used as CI<br>
<br>
=C2=A0 We need do automatic compile checks of contributions, for example vi=
a<br>
=C2=A0 github pull requests or something corresponding.<br>
<br>
- efficient caching<br>
<br>
=C2=A0 Would be nice to not need to git clone the entire repo for every<br>
=C2=A0 build for every server. Would be nice if we checkout a shared git<br=
>
=C2=A0 repo or do something so data does not goes over the wire more than<b=
r>
=C2=A0 necessary. Same goes for source and apk packages.<br>
<br>
Anything else we need from the building infra?<br>
<br>
-nc<br>
<br>
<br>
---<br>
Unsubscribe:=C2=A0 <a href=3D"mailto:alpine-devel%2Bunsubscribe@lists.alpin=
elinux.org">alpine-devel+unsubscribe@<wbr>lists.alpinelinux.org</a><br>
Help:=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"mailto:alpine-devel%2Bhel=
p@lists.alpinelinux.org">alpine-devel+help@lists.<wbr>alpinelinux.org</a><b=
r>
---<br>
<br>
</blockquote></div><br></div>

--001a114526fe60ccc9056480795a--


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---