Date: Tue, 6 Feb 2018 01:39:44 +0100
From: Natanael Copa <ncopa@alpinelinux.org>
To: alpine-devel@lists.alpinelinux.org
Subject: [alpine-devel] rethinking the building infra
Message-ID: <20180206013944.7fa393b6@ncopa-macbook.copa.dup.pw>
Precedence: list
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hi!

I think we need to rethink the building infrastructure. The current
build scripts were written as a quick and dirt way to get started and
have lived longer way longer than originally planned. It simply does
not scale and is very fragile.

I guess I don't need to go any deeper into why we need replace it...

I'd like to discuss what we need from the build infra and why, before we
start talk about how to do it and what implementation etc.

Here are some things I want a new building infra should be able to
do:

- automatic build on git push

  There should not be needed to do anything more than do a git push to
  get the package built and uploaded. Like we have today, but with
  better error reporting.

- isolated environment for each build

  each build should set up an isolated enviroment and destroy it when
  build is done. This could be a container but it would be nice to be
  able to set up a disposable build env in a vm in case we want hook it
  into github PRs or similar. It should also kill everything after
  build and test is done so we dont get any remains of test suites that
  does not clean up after themselves (like redis and epmd)

- support multi architectures

  need to support x86_64, x86, armhf, aarch64, ppc64le and s390x. Would
  be nice it its not too complicated to add new architectures.

- support parallel building

  would be nice to be able to distribute the workload over available
  build servers. Should be possible and relatively easy to add new
  hardware to the pool or remove or replace old without taking
  everything down.

- support cross compile

  would be nice to cross compile packages that (easily) can be cross
  compiled. For example, we could let a big x86_64 or ppc64le machine
  build linux kernel for armhf instead doing that on the slow armhf
  server. Packages that cannot be cross compiled should be built on
  native hardware.

- separate out signing process of packages and index

  Would be nice if we could give access to build servers to more
  people without giving those people access to the private signing keys.

- build infra should be able to be used as CI

  We need do automatic compile checks of contributions, for example via
  github pull requests or something corresponding.

- efficient caching

  Would be nice to not need to git clone the entire repo for every
  build for every server. Would be nice if we checkout a shared git
  repo or do something so data does not goes over the wire more than
  necessary. Same goes for source and apk packages.

Anything else we need from the building infra?

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---