Date: Sat, 20 Nov 2021 15:59:39 -0600 (CST)
From: Ariadne Conill <ariadne@dereferenced.org>
To: =?ISO-8859-15?Q?S=F6ren_Tempel?= <soeren@soeren-tempel.net>
cc: ~alpine/devel@lists.alpinelinux.org
Subject: Re: Thoughts on self-hosting compilers in Alpine
In-Reply-To: <2DGGIA6U0PY5D.3SO8BVCBQLL47@8pit.net>
Message-ID: <c983f33-3858-37c1-ba1b-8218c23254f0@dereferenced.org>
References: <33KG0XO61I4IL.2Z7RTAZ5J3SY6@8pit.net> <9c3ab565-3489-62e4-c15e-a7af3c2ef569@dereferenced.org> <2DGGIA6U0PY5D.3SO8BVCBQLL47@8pit.net>
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="0-296694885-1637445580=:10302"

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.

--0-296694885-1637445580=:10302
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8BIT

Hi,

On Sat, 20 Nov 2021, Sören Tempel wrote:

> Ariadne Conill <ariadne@dereferenced.org> wrote:
>> Hi,
>
> Hello,
>
> Thanks for your input, comments below.
>
>> Also gcc and clang.
>
> Yes. I would also like to stress that the goal of the proposal (and the
> goal of the Bootstrapple Builds project) is minimizing the amount of
> bootstrap binaries and making use of bootstrap binaries transparent, not
> eliminating bootstrap binaries entirely.
>
>> The solution here is to version key system libraries like libffi3.3 and so
>> on.  This isn't done as much as it needs to be done.  Every time I bring
>> this up, I am told that we don't need to do something we need to be doing.
>> This proposal would be far more expensive (rebootstrapping) than just
>> doing the right thing.
>
> Sure, that works as well for this particular problem but doesn't solve
> the other two problems I outlined in the original post.

Yes, but we need to do it regardless.  I don't think this proposal really 
solves that problem.

>>> 3. Builder setup: When setting up builders for a new Alpine release
>>>   (e.g. as we did a few weeks ago for 3.15), manual intervention is
>>>   necessary to install an initial version for every packaged self-hosting
>>>   compiler from Alpine Edge.
>>
>> An alternative would be to provide a bootstrap repo providing only the
>> bootstrap compilers and their dependencies, and no other packages from
>> edge.
>
> Could you elaborate on that? Because self-hosted compilers we are
> shipping presently have dependencies on other packages in Edge (e.g. GHC
> depending on libffi).

Build a repo that is a subset of the edge repos for bootstrap, e.g. `apk 
fetch rust-bootstrap`, `apk index *.apk`, `abuild-sign APKINDEX.tar.gz` 
...

>
>> What happens for cases where architectures were added *late* into the
>> bootstrap path, such as ppc64le?  A stage0 would not exist for these.
>
> Depends on the self-hosting compiler. Regarding Go, there are two Go
> implementations: The Google implementation written in Go and the GCC
> implementation written in C/C++ (which we package as gcc-go). A stage0
> for ppc64le could use the GCC implementation (instead of using Go 1.4).
>
> Generally speaking, the stage0 should use whatever upstream recommends
> for bootstrapping their compiler on a given architecture. I personally
> wouldn't mind if the stage0 is cross-compiled from Ubuntu using a Docker
> file in an APKBUILD with a snapshot() rule (as we did with
> ghc-bootstrap). The goal of the stage0 is simply making the initial
> bootstrap transparent.

I would suggest that we should use gcc-go to generate the stage0 
regardless.  Keep things simple and portable.

>> As noted above, rebootstrapping the whole chain of compilers would be
>> highly expensive for languages like Rust.  Each step would take hours.
>
> I am not suggesting to rebootstrap the entire chain. Ideally
> ${self_hosting_compiler}-stage0 should always be capable of building
> ${self_hosting_compiler} at any point in time. As such, the chain would
> be:
>
> 	${self_hosting_compiler}-stage0 -> ${self_hosting_compiler}
>
> which shouldn't be expensive. I don't know Rust so I don't know what
> upstream suggests regarding bootstrap, but as an example it would be
> possible for the Rust stage0 to use (static?) binaries provided by Rust
> upstream.

For Rust, this would mean that -stage0 has to be current version minus 
one.  That means to produce -stage0, we would have to:

* Compile rust 1.29 with mrustc.
* Compile rust 1.30 with the rust 1.29, 1.31 with 1.30, so on, up to 1.54.

This would literally take days of CPU time, even on our fastest builders.

>> This proposal is not possible for Rust: mrustc is not capable of serving
>> as stage0 for several architectures, at least riscv64, s390x, as 1.29 did
>> not have musl support for them.
>
> I am not proposing use of mrustc *right now*. This proposal isn't about
> any self-hosting compiler in particular. I am just saying that using
> mrustc may become feasible in the future, thereby eliminating the need
> to trust pre-existing Rust binaries for bootstrapping rustc entirely.
>
> BTW: Just as a case-study I started implemented this proposal for the
> Cyclone C compiler were upstream-provided autogenerated C files are used
> for the -stage0 <https://github.com/justinethier/cyclone-bootstrap>:
>
> 	https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27636

I think Cyclone and Go are good candidates for proof of concept.  I don't 
think it will be possible to do it with Rust until mrustc or gcc-rust is 
more capable.

Ariadne
--0-296694885-1637445580=:10302--