Hello,
The recent issue with building GHC [1] lead to an interesting discussion
regarding the general handling of self-hosting compilers in Alpine on
IRC. I want to further discuss a proposal for improving the current
packaging of self-hosting compilers in Alpine in this ML thread.
## Current Situation
Alpine actually packages a lot of self-hosting compilers (e.g.
community/rust, community/go, community/ghc, testing/cyclone, …). Most
of these depend on a previous version of the compiler (as available in
the Alpine repositories) for building. For example, community/go
provides a virtual package called "go-bootstrap" and depends itself on
this virtual package.
## Problem Statement
This leads to three problems:
1. Trust Chain Transparency: Unless you want to go all the way back to
an initial non-self-hosting version of a self-hosting compiler, building
self-hosting compilers practically requires pre-existing binaries at some
point. For example, our initial version of GHC was cross-compiled from
the binary provided by Ubuntu using Docker [2]. However, with
self-hosting compilers, you need to trust not just the previous version
of the compiler but the entire chain of versions used [3]. With our
present setup, this chain is obscured and (without looking at the Git
history) it is in unclear that GHC (for example) was initially bootstrapped
from the Ubuntu binary.
2. Dynamic Libraries: If the previous version of the compiler is linked
dynamically against libraries packaged by Alpine, the previous
version of the compiler (as required for the build process) will stop
working if these libraries are upgraded to a non-ABI compatible
version. This is the root cause of the GHC problem mentioned above: When
libffi (against which GHC links dynamically) was upgraded from
libffi.so.7 to libffi.so.8 the version of GHC (as available in the
repositories) became defunct and could not be used to rebuild GHC
against libffi.so.8.
3. Builder setup: When setting up builders for a new Alpine release
(e.g. as we did a few weeks ago for 3.15), manual intervention is
necessary to install an initial version for every packaged self-hosting
compiler from Alpine Edge.
## Proposal
I propose improving packaging of self-hosting compilers by providing two
packages for each self-hosting compiler in aports.git:
1. ${repo}/${self_hosting_compiler}-stage0 which builds and packages
$compiler without requiring a previous version of this compiler to be
available in the Alpine repository. This package should basically follow
the bootstrap path recommended by upstream. This bootstrap path depends on the
specific compiler. For GHC, upstream recommends using provided binaries
(the ghc-stage0 package would thus package those). For Go, upstream
recommends bootstrapping from Go 1.4 (which is still written in C) for
most architectures.
2. ${repo}/${self_hosting_compiler} which is initially (i.e. each time a
new builder is set up) compiled using ${self_hosting_compiler}-stage0
but from that point onward build from the previous version available in
the Alpine repositories.
This could be implemented by having both ${self_hosting_compiler} and
${self_hosting_compiler}-stage0 provide a virtual package called
${self_hosting_compiler}-bootstrap. The latter would be assigned the
lower $provider_priority (see APKBUILD(5)) to ensure it is only used if
the former isn't available.
This approach would make the trust chain for self-hosting compilers more
transparent, it would ease setting up new builders, and it should also
allow us to deal with soname-rebuilds for dynamic libraries by
reboostrapping the compiler using -stage0 in these cases.
## Discussion
For many self-hosting compilers, the -stage0 package would likely depend on
upstream binaries. In this regard I want to stress that—even with the
current situation—we already trust pre-existing binaries (e.g. the
initial Ubuntu GHC binary for our GHC package). Furthermore, there is an
existing effort on bootstrappable builds which attempts to minimize the
amount of pre-existing binaries required for bootstrapping self-hosting
compilers [4].
For example, for rust people are working on providing an implementation
of the rust programming language in C++ (mrustc) [5]. Presently, it
does not seem to be possible to build the most recent rust version with
mrustc. However, technically it would be possible to use mrustc for
community/rust-stage0 as soon as mrustc is reliably capable of
initially bootstrapping our packaged version of community/rust.
Thoughts on this proposal?
## Further Reading
* Ken's Turing Award lecture "Reflections on Trusting Trust":
https://doi.org/10.1145/358198.358210
* The bootstrappable builds project https://bootstrappable.org/
[1]: https://lists.alpinelinux.org/~alpine/devel/%3C20211021133615.32f08070%40ncopa-desktop.lan%3E
[2]: https://gitlab.alpinelinux.org/alpine/aports/-/commit/8488e8747aa7cb275882157b8a4a53c274c71927
[3]: https://doi.org/10.1145/358198.358210
[4]: https://bootstrappable.org/
[5]: https://github.com/thepowersgang/mrustc
Hi,
On Sat, 20 Nov 2021, Sören Tempel wrote:
> Hello,>> The recent issue with building GHC [1] lead to an interesting discussion> regarding the general handling of self-hosting compilers in Alpine on> IRC. I want to further discuss a proposal for improving the current> packaging of self-hosting compilers in Alpine in this ML thread.>> ## Current Situation>> Alpine actually packages a lot of self-hosting compilers (e.g.> community/rust, community/go, community/ghc, testing/cyclone, …). Most> of these depend on a previous version of the compiler (as available in> the Alpine repositories) for building. For example, community/go> provides a virtual package called "go-bootstrap" and depends itself on> this virtual package.
Also gcc and clang.
>> ## Problem Statement>> This leads to three problems:>> 1. Trust Chain Transparency: Unless you want to go all the way back to> an initial non-self-hosting version of a self-hosting compiler, building> self-hosting compilers practically requires pre-existing binaries at some> point. For example, our initial version of GHC was cross-compiled from> the binary provided by Ubuntu using Docker [2]. However, with> self-hosting compilers, you need to trust not just the previous version> of the compiler but the entire chain of versions used [3]. With our> present setup, this chain is obscured and (without looking at the Git> history) it is in unclear that GHC (for example) was initially bootstrapped> from the Ubuntu binary.
Right.
> 2. Dynamic Libraries: If the previous version of the compiler is linked> dynamically against libraries packaged by Alpine, the previous> version of the compiler (as required for the build process) will stop> working if these libraries are upgraded to a non-ABI compatible> version. This is the root cause of the GHC problem mentioned above: When> libffi (against which GHC links dynamically) was upgraded from> libffi.so.7 to libffi.so.8 the version of GHC (as available in the> repositories) became defunct and could not be used to rebuild GHC> against libffi.so.8.
The solution here is to version key system libraries like libffi3.3 and so
on. This isn't done as much as it needs to be done. Every time I bring
this up, I am told that we don't need to do something we need to be doing.
This proposal would be far more expensive (rebootstrapping) than just
doing the right thing.
> 3. Builder setup: When setting up builders for a new Alpine release> (e.g. as we did a few weeks ago for 3.15), manual intervention is> necessary to install an initial version for every packaged self-hosting> compiler from Alpine Edge.
An alternative would be to provide a bootstrap repo providing only the
bootstrap compilers and their dependencies, and no other packages from
edge.
>> ## Proposal>> I propose improving packaging of self-hosting compilers by providing two> packages for each self-hosting compiler in aports.git:>> 1. ${repo}/${self_hosting_compiler}-stage0 which builds and packages> $compiler without requiring a previous version of this compiler to be> available in the Alpine repository. This package should basically follow> the bootstrap path recommended by upstream. This bootstrap path depends on the> specific compiler. For GHC, upstream recommends using provided binaries> (the ghc-stage0 package would thus package those). For Go, upstream> recommends bootstrapping from Go 1.4 (which is still written in C) for> most architectures.
What happens for cases where architectures were added *late* into the
bootstrap path, such as ppc64le? A stage0 would not exist for these.
> 2. ${repo}/${self_hosting_compiler} which is initially (i.e. each time a> new builder is set up) compiled using ${self_hosting_compiler}-stage0> but from that point onward build from the previous version available in> the Alpine repositories.>> This could be implemented by having both ${self_hosting_compiler} and> ${self_hosting_compiler}-stage0 provide a virtual package called> ${self_hosting_compiler}-bootstrap. The latter would be assigned the> lower $provider_priority (see APKBUILD(5)) to ensure it is only used if> the former isn't available.>> This approach would make the trust chain for self-hosting compilers more> transparent, it would ease setting up new builders, and it should also> allow us to deal with soname-rebuilds for dynamic libraries by> reboostrapping the compiler using -stage0 in these cases.
As noted above, rebootstrapping the whole chain of compilers would be
highly expensive for languages like Rust. Each step would take hours.
> ## Discussion>> For many self-hosting compilers, the -stage0 package would likely depend on> upstream binaries. In this regard I want to stress that—even with the> current situation—we already trust pre-existing binaries (e.g. the> initial Ubuntu GHC binary for our GHC package). Furthermore, there is an> existing effort on bootstrappable builds which attempts to minimize the> amount of pre-existing binaries required for bootstrapping self-hosting> compilers [4].>> For example, for rust people are working on providing an implementation> of the rust programming language in C++ (mrustc) [5]. Presently, it> does not seem to be possible to build the most recent rust version with> mrustc. However, technically it would be possible to use mrustc for> community/rust-stage0 as soon as mrustc is reliably capable of> initially bootstrapping our packaged version of community/rust.
This proposal is not possible for Rust: mrustc is not capable of serving
as stage0 for several architectures, at least riscv64, s390x, as 1.29 did
not have musl support for them.
Ariadne
Ariadne Conill <ariadne@dereferenced.org> wrote:
> Hi,
Hello,
Thanks for your input, comments below.
> Also gcc and clang.
Yes. I would also like to stress that the goal of the proposal (and the
goal of the Bootstrapple Builds project) is minimizing the amount of
bootstrap binaries and making use of bootstrap binaries transparent, not
eliminating bootstrap binaries entirely.
> The solution here is to version key system libraries like libffi3.3 and so > on. This isn't done as much as it needs to be done. Every time I bring > this up, I am told that we don't need to do something we need to be doing. > This proposal would be far more expensive (rebootstrapping) than just > doing the right thing.
Sure, that works as well for this particular problem but doesn't solve
the other two problems I outlined in the original post.
> > 3. Builder setup: When setting up builders for a new Alpine release> > (e.g. as we did a few weeks ago for 3.15), manual intervention is> > necessary to install an initial version for every packaged self-hosting> > compiler from Alpine Edge.> > An alternative would be to provide a bootstrap repo providing only the > bootstrap compilers and their dependencies, and no other packages from > edge.
Could you elaborate on that? Because self-hosted compilers we are
shipping presently have dependencies on other packages in Edge (e.g. GHC
depending on libffi).
> What happens for cases where architectures were added *late* into the > bootstrap path, such as ppc64le? A stage0 would not exist for these.
Depends on the self-hosting compiler. Regarding Go, there are two Go
implementations: The Google implementation written in Go and the GCC
implementation written in C/C++ (which we package as gcc-go). A stage0
for ppc64le could use the GCC implementation (instead of using Go 1.4).
Generally speaking, the stage0 should use whatever upstream recommends
for bootstrapping their compiler on a given architecture. I personally
wouldn't mind if the stage0 is cross-compiled from Ubuntu using a Docker
file in an APKBUILD with a snapshot() rule (as we did with
ghc-bootstrap). The goal of the stage0 is simply making the initial
bootstrap transparent.
> As noted above, rebootstrapping the whole chain of compilers would be > highly expensive for languages like Rust. Each step would take hours.
I am not suggesting to rebootstrap the entire chain. Ideally
${self_hosting_compiler}-stage0 should always be capable of building
${self_hosting_compiler} at any point in time. As such, the chain would
be:
${self_hosting_compiler}-stage0 -> ${self_hosting_compiler}
which shouldn't be expensive. I don't know Rust so I don't know what
upstream suggests regarding bootstrap, but as an example it would be
possible for the Rust stage0 to use (static?) binaries provided by Rust
upstream.
> This proposal is not possible for Rust: mrustc is not capable of serving > as stage0 for several architectures, at least riscv64, s390x, as 1.29 did > not have musl support for them.
I am not proposing use of mrustc *right now*. This proposal isn't about
any self-hosting compiler in particular. I am just saying that using
mrustc may become feasible in the future, thereby eliminating the need
to trust pre-existing Rust binaries for bootstrapping rustc entirely.
BTW: Just as a case-study I started implemented this proposal for the
Cyclone C compiler were upstream-provided autogenerated C files are used
for the -stage0 <https://github.com/justinethier/cyclone-bootstrap>:https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27636
Greetings,
Sören
Hi,
On Sat, 20 Nov 2021, Sören Tempel wrote:
> Ariadne Conill <ariadne@dereferenced.org> wrote:>> Hi,>> Hello,>> Thanks for your input, comments below.>>> Also gcc and clang.>> Yes. I would also like to stress that the goal of the proposal (and the> goal of the Bootstrapple Builds project) is minimizing the amount of> bootstrap binaries and making use of bootstrap binaries transparent, not> eliminating bootstrap binaries entirely.>>> The solution here is to version key system libraries like libffi3.3 and so>> on. This isn't done as much as it needs to be done. Every time I bring>> this up, I am told that we don't need to do something we need to be doing.>> This proposal would be far more expensive (rebootstrapping) than just>> doing the right thing.>> Sure, that works as well for this particular problem but doesn't solve> the other two problems I outlined in the original post.
Yes, but we need to do it regardless. I don't think this proposal really
solves that problem.
>>> 3. Builder setup: When setting up builders for a new Alpine release>>> (e.g. as we did a few weeks ago for 3.15), manual intervention is>>> necessary to install an initial version for every packaged self-hosting>>> compiler from Alpine Edge.>>>> An alternative would be to provide a bootstrap repo providing only the>> bootstrap compilers and their dependencies, and no other packages from>> edge.>> Could you elaborate on that? Because self-hosted compilers we are> shipping presently have dependencies on other packages in Edge (e.g. GHC> depending on libffi).
Build a repo that is a subset of the edge repos for bootstrap, e.g. `apk
fetch rust-bootstrap`, `apk index *.apk`, `abuild-sign APKINDEX.tar.gz`
...
>>> What happens for cases where architectures were added *late* into the>> bootstrap path, such as ppc64le? A stage0 would not exist for these.>> Depends on the self-hosting compiler. Regarding Go, there are two Go> implementations: The Google implementation written in Go and the GCC> implementation written in C/C++ (which we package as gcc-go). A stage0> for ppc64le could use the GCC implementation (instead of using Go 1.4).>> Generally speaking, the stage0 should use whatever upstream recommends> for bootstrapping their compiler on a given architecture. I personally> wouldn't mind if the stage0 is cross-compiled from Ubuntu using a Docker> file in an APKBUILD with a snapshot() rule (as we did with> ghc-bootstrap). The goal of the stage0 is simply making the initial> bootstrap transparent.
I would suggest that we should use gcc-go to generate the stage0
regardless. Keep things simple and portable.
>> As noted above, rebootstrapping the whole chain of compilers would be>> highly expensive for languages like Rust. Each step would take hours.>> I am not suggesting to rebootstrap the entire chain. Ideally> ${self_hosting_compiler}-stage0 should always be capable of building> ${self_hosting_compiler} at any point in time. As such, the chain would> be:>> ${self_hosting_compiler}-stage0 -> ${self_hosting_compiler}>> which shouldn't be expensive. I don't know Rust so I don't know what> upstream suggests regarding bootstrap, but as an example it would be> possible for the Rust stage0 to use (static?) binaries provided by Rust> upstream.
For Rust, this would mean that -stage0 has to be current version minus
one. That means to produce -stage0, we would have to:
* Compile rust 1.29 with mrustc.
* Compile rust 1.30 with the rust 1.29, 1.31 with 1.30, so on, up to 1.54.
This would literally take days of CPU time, even on our fastest builders.
>> This proposal is not possible for Rust: mrustc is not capable of serving>> as stage0 for several architectures, at least riscv64, s390x, as 1.29 did>> not have musl support for them.>> I am not proposing use of mrustc *right now*. This proposal isn't about> any self-hosting compiler in particular. I am just saying that using> mrustc may become feasible in the future, thereby eliminating the need> to trust pre-existing Rust binaries for bootstrapping rustc entirely.>> BTW: Just as a case-study I started implemented this proposal for the> Cyclone C compiler were upstream-provided autogenerated C files are used> for the -stage0 <https://github.com/justinethier/cyclone-bootstrap>:>> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27636
I think Cyclone and Go are good candidates for proof of concept. I don't
think it will be possible to do it with Rust until mrustc or gcc-rust is
more capable.
Ariadne
Ariadne Conill <ariadne@dereferenced.org> wrote:
> Hi,
Hi,
> On Sat, 20 Nov 2021, Sören Tempel wrote:> Yes, but we need to do it regardless. I don't think this proposal really > solves that problem.
I think this depends on a case-by-case basis on the specifics of the
self-hosted compiler. So, for example, for the Cyclone Scheme compiler,
there is no overhead involved in re-bootstrapping from the stage0. As
such, I believe it to be easier (i.e. less work) to re-bootstrap from
stage0 instead of adding versioned packages for the dynamic libraries on
which it depends. Same thing with Go if we were to be using gcc-go as a
stage0.
> For Rust, this would mean that stage0 has to be current version minus > one. That means to produce stage0, we would have to:> > * Compile rust 1.29 with mrustc.> * Compile rust 1.30 with the rust 1.29, 1.31 with 1.30, so on, up to 1.54.> > This would literally take days of CPU time, even on our fastest builders.
To me, this more of an argument against the current Rust bootstrapping
process rather than argument against my proposal. If Rust always needs
the most recent compiler version to build rustc itself, my proposal may
not be feasible for Rust for the time being. Other self-hosted compiler
mitigate this problem through language standards and by not using recent
language features in the compiler implementation (e.g. the Go compiler
can still be built with Go 1.4).
> I think Cyclone and Go are good candidates for proof of concept. I don't > think it will be possible to do it with Rust until mrustc or gcc-rust is > more capable.
I can only make qualified comments for packages I have worked on in the
past. From the top of my head, I can think of the following additional
packages for which this proposal is also feasible:
* community/ghc where the stage0 should eventually be able to use the
musl bindist provided by upstream or, alternatively, we could revive
the Docker based cross-compilation setup as a stage0 [1].
* testing/idris2 where upstream provides autogenerated Chez Scheme files
for bootstrapping purposes which could be used in the stage0.
* community/cabal for which we already have community/cabal-bootstrap
and could thus easily have them provide a virtual package with
different provider_priorities.
Greetings,
Sören
[1]: https://gitlab.alpinelinux.org/alpine/aports/-/commit/8488e8747aa7cb275882157b8a4a53c274c71927
On Sat, 20 Nov 2021 14:34:56 +0100
Sören Tempel <soeren@soeren-tempel.net> wrote:
> ## Current Situation> > Alpine actually packages a lot of self-hosting compilers (e.g.> community/rust, community/go, community/ghc, testing/cyclone, …). Most> of these depend on a previous version of the compiler (as available in> the Alpine repositories) for building. For example, community/go> provides a virtual package called "go-bootstrap" and depends itself on> this virtual package.
So there used to be separate go-bootstrap package which enabled
bootstrapping with C-compiler, but that seems to have been removed in
commit 895fcea724edcb357cd2b0da3b64d36155b159d4. This was the latest go
that did not require go to build.
I am not sure if it would be possible to bootstrap go with gcc-go? But
there might exist also some other paths.
But as discussed in the thread, it seems each compiler has unique
requirements on how to bootstrap it. Granted, this is getting painful
as the number of self-dependent programming languages increase. And for
some language full bootstrap just is not feasible which makes things
quite annoying.
Timo
Timo Teras <timo.teras@iki.fi> wrote:
> So there used to be separate go-bootstrap package which enabled> bootstrapping with C-compiler, but that seems to have been removed in> commit 895fcea724edcb357cd2b0da3b64d36155b159d4. This was the latest go> that did not require go to build.
Correct, I was the maintainer of this package. As discussed in this ML
thread, bootstrapping via Go 1.4 (as previously packaged by
go-bootstrap) does not work with architectures that are not supported by
the 1.4 release (e.g. riscv64 or ppc64le). For this reason, I added a
package for gcc-go in commit b93b0b134f7d10720a4359579ff4f55575ac90e3.
> I am not sure if it would be possible to bootstrap go with gcc-go? But> there might exist also some other paths.
Yes, that should be possible and that's exactly what I would like to do
for Go. I just haven't gotten around to fixing all the minor issues with
the gcc-go based bootstrap process yet.
> But as discussed in the thread, it seems each compiler has unique> requirements on how to bootstrap it. Granted, this is getting painful> as the number of self-dependent programming languages increase. And for> some language full bootstrap just is not feasible which makes things> quite annoying.
Right, I would just like to make the bootstrap process transparent. So
for example, just by looking at community/go/APKBUILD it is entirely
unclear how Go was bootstrapped initially for the riscv64 or ppc64le
architecture.
Greetings,
Sören
On Wed, 24 Nov 2021 09:21:57 +0100
Sören Tempel <soeren@soeren-tempel.net> wrote:
> > I am not sure if it would be possible to bootstrap go with gcc-go?> > But there might exist also some other paths. > > Yes, that should be possible and that's exactly what I would like to> do for Go. I just haven't gotten around to fixing all the minor> issues with the gcc-go based bootstrap process yet.
Cool!
> > But as discussed in the thread, it seems each compiler has unique> > requirements on how to bootstrap it. Granted, this is getting> > painful as the number of self-dependent programming languages> > increase. And for some language full bootstrap just is not feasible> > which makes things quite annoying. > > Right, I would just like to make the bootstrap process transparent. So> for example, just by looking at community/go/APKBUILD it is entirely> unclear how Go was bootstrapped initially for the riscv64 or ppc64le> architecture.
So community/go supports cross-building for bootstrapping. x86_64
system was used to cross-build go for those target as go-bootstrap.
I am not sure if the cross-built go has functional differences or not.
But during first native build round go would then get rebuilt natively
by the earlier cross-built compiler.
This kind of cross-built bootstrapping is often used for new targets.
And this is how e.g. gcc is bootstrapped for new systems too.
Timo
Hello,
Little status update specifically on Go: I made some improvements to
gccgo in the past few weeks [1, 2, 3, 4, 5, 6]. With these changes
applied, it is now possible to bootstrap Google's Go implementation with
gccgo on all architectures supported by Alpine [7].
I also submitted a few of our libgo/gccgo related patches upstream in
order to improve gccgo's upstream musl support [8, 9, 10]. Ideally, I
would like to see all of our gccgo patches integrated upstream.
The only remaining problem I am presently aware of is that 32-bit
architectures seem to run out-of-memory on the CI when used with
multiple OS threads which execute user-level Go code simultaneously
(hence bootstrapped with GOMAXPROCS=1). Nonetheless, gccgo should now be
a viable option to bootstrap community/go if we want to go down this
path.
Greetings,
Sören
[1]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27772
[2]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27875
[3]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28185
[4]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/28525
[5]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/29185
[6]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/29096
[7]: https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/27944
[8]: https://gcc.gnu.org/pipermail/gcc-patches/2021-December/587138.html
[9]: https://gcc.gnu.org/pipermail/gcc-patches/2021-November/585536.html
[10]: https://gcc.gnu.org/pipermail/gcc-patches/2022-January/587520.html
Timo Teras <timo.teras@iki.fi> wrote:
> On Wed, 24 Nov 2021 09:21:57 +0100> Sören Tempel <soeren@soeren-tempel.net> wrote:> > > > I am not sure if it would be possible to bootstrap go with gcc-go?> > > But there might exist also some other paths. > > > > Yes, that should be possible and that's exactly what I would like to> > do for Go. I just haven't gotten around to fixing all the minor> > issues with the gcc-go based bootstrap process yet.> > Cool!> > > > But as discussed in the thread, it seems each compiler has unique> > > requirements on how to bootstrap it. Granted, this is getting> > > painful as the number of self-dependent programming languages> > > increase. And for some language full bootstrap just is not feasible> > > which makes things quite annoying. > > > > Right, I would just like to make the bootstrap process transparent. So> > for example, just by looking at community/go/APKBUILD it is entirely> > unclear how Go was bootstrapped initially for the riscv64 or ppc64le> > architecture.> > So community/go supports cross-building for bootstrapping. x86_64> system was used to cross-build go for those target as go-bootstrap.> > I am not sure if the cross-built go has functional differences or not.> But during first native build round go would then get rebuilt natively> by the earlier cross-built compiler.> > This kind of cross-built bootstrapping is often used for new targets.> And this is how e.g. gcc is bootstrapped for new systems too.> > Timo
Hi,
On Fri, 7 Jan 2022, Sören Tempel wrote:
> Hello,>> Little status update specifically on Go: I made some improvements to> gccgo in the past few weeks [1, 2, 3, 4, 5, 6]. With these changes> applied, it is now possible to bootstrap Google's Go implementation with> gccgo on all architectures supported by Alpine [7].>> I also submitted a few of our libgo/gccgo related patches upstream in> order to improve gccgo's upstream musl support [8, 9, 10]. Ideally, I> would like to see all of our gccgo patches integrated upstream.
This is really great. I can talk with some upstream GCC folks about
refactoring the rest of the patchset if wanted. I'm trying to split up
some of our patches into logical changesets to faciitate this kind of
upstreaming already.
> The only remaining problem I am presently aware of is that 32-bit> architectures seem to run out-of-memory on the CI when used with> multiple OS threads which execute user-level Go code simultaneously> (hence bootstrapped with GOMAXPROCS=1). Nonetheless, gccgo should now be> a viable option to bootstrap community/go if we want to go down this> path.
This is likely due to running out of address space. I wonder if we could
do GOMAXPROCS="$((JOBS / 2))" and have it be fine. Bootstrapping Go from
gcc-go however is a welcome addition for 3.16!
Ariadne
Ariadne Conill <ariadne@dereferenced.org> wrote:
> I can talk with some upstream GCC folks about refactoring the rest of> the patchset if wanted.
Sure, if you are in touch with any GCC developers that could help us
refactor our patches in order to get them integrated upstream that would
much appreciated.
> I'm trying to split up some of our patches into logical changesets to> faciitate this kind of upstreaming already.
Great! Let me know if I can be of assistance.
> This is likely due to running out of address space. I wonder if we could > do GOMAXPROCS="$((JOBS / 2))" and have it be fine.
Yes, GOMAXPROCS=$((JOBS / 2)) may also work. Nonetheless, this needs
some further investigation since I don't really understand why Go would
run out of address space with certain GOMAXPROCS values on 32-bit
architectures. However, for now just limiting GOMAXPROCS should be fine.
Greetings,
Sören