Hi,

I’ve updated community/cargo package and removed dependency on external prebuilt binary, so it now (make)depends on itself (you need cargo to build cargo… *#^@$~$+). This is not a new situation, we did the same with community/rust some time ago. IIRC it needed some manual intervention on build server, but don’t know any details. Unfortunately I cannot do anything with it, ’cause I don’t have access to the builder.

I changed the abuild to not use external prebuilt binary primarily because the old URI of prebuilt binaries built by upstream’s CI doesn’t work anymore. I didn’t find any other and if I understand their CI configuration, they don’t upload cargo built with musl anywhere (again!). However, we need cargo to build rust and we need rust to build cargo (*put some very angry and vulgar comment here*), so we must take cargo binary from somewhere when bootstrapping from scratch anyway.

So please someone make it work and it be nice if you’d also write what exactly was needed.

Jakub

Hi,

Since nobody fixed this on the builders, I wound up reverting this change.

However, I have more to say.

On Sat, Oct 28, 2017 at 1:44 PM, Jakub Jirutka <jakub@jirutka.cz> wrote:
> Hi,
>
> I’ve updated community/cargo package and removed dependency on external prebuilt binary, so it now (make)depends on itself (you need cargo to build cargo… *#^@$~$+). This is not a new situation, we did the same with community/rust some time ago. IIRC it needed some manual intervention on build server, but don’t know any details. Unfortunately I cannot do anything with it, ’cause I don’t have access to the builder.

A manual intervention on each builder is not an acceptable solution.
Policy requires that builds be reproducible, in other words, that they
can build with `abuild rootbld`.
So this change should not have been pushed to begin with.

> I changed the abuild to not use external prebuilt binary primarily because the old URI of prebuilt binaries built by upstream’s CI doesn’t work anymore. I didn’t find any other and if I understand their CI configuration, they don’t upload cargo built with musl anywhere (again!). However, we need cargo to build rust and we need rust to build cargo (*put some very angry and vulgar comment here*), so we must take cargo binary from somewhere when bootstrapping from scratch anyway.

A solution would be to take a copy of the cargo package from
distfiles.alpinelinux.org (the old version would still exist there),
and make a cargo-bootstrap package.

Look at the haskell + cabal packaging for inspiration, it is a similar
situation, and is policy compliant.

William

Hello,

On Mon, Oct 30, 2017 at 11:39 AM, Jakub Jirutka <jakub@jirutka.cz> wrote:
>> A manual intervention on each builder is not an acceptable solution.
>> Policy requires that builds be reproducible, in other words, that they can build with `abuild rootbld`.
>> So this change should not have been pushed to begin with.
>
>
> No, I thought this too, but it’s not how it really works. See below.
>
>> Look at the haskell + cabal packaging for inspiration, it is a similar situation, and is policy compliant.
>
> Have *you* looked at ghc (haskell) package? I did some time ago (and now), tried hard to figure out how does it build. There is `makedepends_build="$pkgname"`, so it depends on itself. There used to be package ghc-bootstrap, but it was not used anymore (and even that couldn’t work without manual intervention). I asked fabled about it and he explained me that ghc-bootstrap was only temporary for initial bootstrap, now ghc depends on itself and it must be probably manually installed on the builder when bootstrapping.

I did some time ago, when we had ghc-bootstrap package.
If this is the case, then it, too, is no longer policy compliant.

> The same situation is community/rust. I even mentioned this in my email.

The same situation of being policy non-compliant?

> Unfortunately the bootstrapping (and cross-compilation) process is still not documented anywhere, so I can refer only to information that I was told by more competent people like fabled.

Manual intervention on the builders does not seem like a good idea to
me, which is probably why it's not compliant with policy.

I suspect the ghc-bootstrap package was dropped because of the
depsolver bug I solved back in April.
The bootstrap package would have a provides entry for the compiler
with a version of 0.
This would allow the real compiler to always "win" the preference.

>> A solution would be to take a copy of the cargo package from distfiles.alpinelinux.org
>
> This is actually a good idea. I can keep it as it was before, just use our cargo binary instead of now non-existence binary from the upstream.

You could do that, or what I was inferring was that the binary itself
was still available on distfiles.alpinelinux.org.
All the builders copy their sources to that server.

> fabled or ncopa, could you please explain how is ghc and similar packages handled? Ideally directly on wiki, or just via email and I will copy it to wiki.

Instead, lets solve these problems correctly, please.
Indeed, the only thing we have learned today is that Haskell is also broken.

William

Hello,

On Mon, Oct 30, 2017 at 12:07 PM, Jakub Jirutka <jakub@jirutka.cz> wrote:
>> If this is the case, then it, too, is no longer policy compliant.
>
> Said who? These policies are (still) not written anywhere, so I follow what I was told by competent people like ncopa or fabled.

Thank you for your continued implication that I'm not competent.

I am working on fixing the fact that policy is not properly
documented, please be patient.

However, the entire point of adding `abuild rootbld` was to ensure
that builds were reproducible, so clearly reproducibility is a policy
goal.

>> Manual intervention on the builders does not seem like a good idea to me, which is probably why it’s not compliant with policy.
>
> I agree, but I haven’t created this approach.

If you know in your gut, it is a bad approach, then why continue to follow it?

>> I suspect the ghc-bootstrap package was dropped because of the depsolver bug I solved back in April.
>> The bootstrap package would have a provides entry for the compiler with a version of 0.
>> This would allow the real compiler to always „win" the preference.
>
> No, it was dropped because it was partially broken (I reviewed it) and actually not used (per fabled). I remember that quite well, ’cause I wasted amount of time trying to figure out how does it work, to be then told it’s not used at all.

Yes, what I am implying, is that it was never used because it
triggered the same bug.
It was never used because it resulted in erroneous conflicts.

>> Instead, lets solve these problems correctly, please.
>
> The problem is that I don’t know what is the correct solution. That’s why I asked ncopa or fabled.

I explained the correct solution already, but will do so again: you
have to break the dependency cycle, by using a bootstrap package.
There is no other way to do it that would allow the build to be reproducible.
But since I am in your eyes incompetent, I am sure you will continue
to ignore this.

>
>> Indeed, the only thing we have learned today is that Haskell is also broken.
>
> And maybe even gcc, ’cause it also depends on itself… Or you have just wrong information. gcc is also handled somehow specially, if I remember correctly what fabled told me.

It is not.
The GCC contains enough information in the APKBUILD to bootstrap
itself, given an already present C compiler.
In fact, in the process of bootstrapping, *two* GCC packages are created.
See $_bootstrap_configure, $_cross_configure and $_arch_configure for
more details on how these are different.

> So please, don’t interfere into it and wait for reaction from ncopa or fabled.

Please stop implying everyone other than you is incompetent when in
reality you are the one who pushed a broken package that you did not
bother to test under rootbld.
Had you tested under rootbld, you would have known that the builders
would not have accepted the package.
There is no need to wait for bad advice when the correct advice is
already known.

William

> A manual intervention on each builder is not an acceptable solution.
> Policy requires that builds be reproducible, in other words, that they can build with `abuild rootbld`.
> So this change should not have been pushed to begin with.


No, I thought this too, but it’s not how it really works. See below.

> Look at the haskell + cabal packaging for inspiration, it is a similar situation, and is policy compliant.

Have *you* looked at ghc (haskell) package? I did some time ago (and now), tried hard to figure out how does it build. There is `makedepends_build="$pkgname"`, so it depends on itself. There used to be package ghc-bootstrap, but it was not used anymore (and even that couldn’t work without manual intervention). I asked fabled about it and he explained me that ghc-bootstrap was only temporary for initial bootstrap, now ghc depends on itself and it must be probably manually installed on the builder when bootstrapping.

The same situation is community/rust. I even mentioned this in my email.

Unfortunately the bootstrapping (and cross-compilation) process is still not documented anywhere, so I can refer only to information that I was told by more competent people like fabled.

> A solution would be to take a copy of the cargo package from distfiles.alpinelinux.org

This is actually a good idea. I can keep it as it was before, just use our cargo binary instead of now non-existence binary from the upstream.


fabled or ncopa, could you please explain how is ghc and similar packages handled? Ideally directly on wiki, or just via email and I will copy it to wiki.


Jakub

> On 30. Oct 2017, at 4:14, William Pitcock <nenolod@dereferenced.org> wrote:
> 
> Hi,
> 
> Since nobody fixed this on the builders, I wound up reverting this change.
> 
> However, I have more to say.
> 
> On Sat, Oct 28, 2017 at 1:44 PM, Jakub Jirutka <jakub@jirutka.cz> wrote:
>> Hi,
>> 
>> I’ve updated community/cargo package and removed dependency on external prebuilt binary, so it now (make)depends on itself (you need cargo to build cargo… *#^@$~$+). This is not a new situation, we did the same with community/rust some time ago. IIRC it needed some manual intervention on build server, but don’t know any details. Unfortunately I cannot do anything with it, ’cause I don’t have access to the builder.
> 
> A manual intervention on each builder is not an acceptable solution.
> Policy requires that builds be reproducible, in other words, that they
> can build with `abuild rootbld`.
> So this change should not have been pushed to begin with.
> 
>> I changed the abuild to not use external prebuilt binary primarily because the old URI of prebuilt binaries built by upstream’s CI doesn’t work anymore. I didn’t find any other and if I understand their CI configuration, they don’t upload cargo built with musl anywhere (again!). However, we need cargo to build rust and we need rust to build cargo (*put some very angry and vulgar comment here*), so we must take cargo binary from somewhere when bootstrapping from scratch anyway.
> 
> A solution would be to take a copy of the cargo package from
> distfiles.alpinelinux.org (the old version would still exist there),
> and make a cargo-bootstrap package.
> 
> Look at the haskell + cabal packaging for inspiration, it is a similar
> situation, and is policy compliant.
> 
> William

> If this is the case, then it, too, is no longer policy compliant.

Said who? These policies are (still) not written anywhere, so I follow what I was told by competent people like ncopa or fabled.

> Manual intervention on the builders does not seem like a good idea to me, which is probably why it’s not compliant with policy.

I agree, but I haven’t created this approach.

> I suspect the ghc-bootstrap package was dropped because of the depsolver bug I solved back in April.
> The bootstrap package would have a provides entry for the compiler with a version of 0.
> This would allow the real compiler to always „win" the preference.

No, it was dropped because it was partially broken (I reviewed it) and actually not used (per fabled). I remember that quite well, ’cause I wasted amount of time trying to figure out how does it work, to be then told it’s not used at all.

> Instead, lets solve these problems correctly, please.

The problem is that I don’t know what is the correct solution. That’s why I asked ncopa or fabled.

> Indeed, the only thing we have learned today is that Haskell is also broken.

And maybe even gcc, ’cause it also depends on itself… Or you have just wrong information. gcc is also handled somehow specially, if I remember correctly what fabled told me.

So please, don’t interfere into it and wait for reaction from ncopa or fabled.

Jakub

> On 30. Oct 2017, at 17:53, William Pitcock <nenolod@dereferenced.org> wrote:
> 
> Hello,
> 
> On Mon, Oct 30, 2017 at 11:39 AM, Jakub Jirutka <jakub@jirutka.cz> wrote:
>>> A manual intervention on each builder is not an acceptable solution.
>>> Policy requires that builds be reproducible, in other words, that they can build with `abuild rootbld`.
>>> So this change should not have been pushed to begin with.
>> 
>> 
>> No, I thought this too, but it’s not how it really works. See below.
>> 
>>> Look at the haskell + cabal packaging for inspiration, it is a similar situation, and is policy compliant.
>> 
>> Have *you* looked at ghc (haskell) package? I did some time ago (and now), tried hard to figure out how does it build. There is `makedepends_build="$pkgname"`, so it depends on itself. There used to be package ghc-bootstrap, but it was not used anymore (and even that couldn’t work without manual intervention). I asked fabled about it and he explained me that ghc-bootstrap was only temporary for initial bootstrap, now ghc depends on itself and it must be probably manually installed on the builder when bootstrapping.
> 
> I did some time ago, when we had ghc-bootstrap package.
> If this is the case, then it, too, is no longer policy compliant.
> 
>> The same situation is community/rust. I even mentioned this in my email.
> 
> The same situation of being policy non-compliant?
> 
>> Unfortunately the bootstrapping (and cross-compilation) process is still not documented anywhere, so I can refer only to information that I was told by more competent people like fabled.
> 
> Manual intervention on the builders does not seem like a good idea to
> me, which is probably why it's not compliant with policy.
> 
> I suspect the ghc-bootstrap package was dropped because of the
> depsolver bug I solved back in April.
> The bootstrap package would have a provides entry for the compiler
> with a version of 0.
> This would allow the real compiler to always "win" the preference.
> 
>>> A solution would be to take a copy of the cargo package from distfiles.alpinelinux.org
>> 
>> This is actually a good idea. I can keep it as it was before, just use our cargo binary instead of now non-existence binary from the upstream.
> 
> You could do that, or what I was inferring was that the binary itself
> was still available on distfiles.alpinelinux.org.
> All the builders copy their sources to that server.
> 
>> fabled or ncopa, could you please explain how is ghc and similar packages handled? Ideally directly on wiki, or just via email and I will copy it to wiki.
> 
> Instead, lets solve these problems correctly, please.
> Indeed, the only thing we have learned today is that Haskell is also broken.
> 
> William
> 

>> You could do that, or what I was inferring was that the binary itself was still available on distfiles.alpinelinux.org.

There’s only cargo-0.22.0.tar.gz, that’s the source tarball from the upstream, not binary…

Jakub

> On 30. Oct 2017, at 18:07, Jakub Jirutka <jakub@jirutka.cz> wrote:
> 
>> If this is the case, then it, too, is no longer policy compliant.
> 
> Said who? These policies are (still) not written anywhere, so I follow what I was told by competent people like ncopa or fabled.
> 
>> Manual intervention on the builders does not seem like a good idea to me, which is probably why it’s not compliant with policy.
> 
> I agree, but I haven’t created this approach.
> 
>> I suspect the ghc-bootstrap package was dropped because of the depsolver bug I solved back in April.
>> The bootstrap package would have a provides entry for the compiler with a version of 0.
>> This would allow the real compiler to always „win" the preference.
> 
> No, it was dropped because it was partially broken (I reviewed it) and actually not used (per fabled). I remember that quite well, ’cause I wasted amount of time trying to figure out how does it work, to be then told it’s not used at all.
> 
>> Instead, lets solve these problems correctly, please.
> 
> The problem is that I don’t know what is the correct solution. That’s why I asked ncopa or fabled.
> 
>> Indeed, the only thing we have learned today is that Haskell is also broken.
> 
> And maybe even gcc, ’cause it also depends on itself… Or you have just wrong information. gcc is also handled somehow specially, if I remember correctly what fabled told me.
> 
> So please, don’t interfere into it and wait for reaction from ncopa or fabled.
> 
> Jakub
> 
>> On 30. Oct 2017, at 17:53, William Pitcock <nenolod@dereferenced.org> wrote:
>> 
>> Hello,
>> 
>> On Mon, Oct 30, 2017 at 11:39 AM, Jakub Jirutka <jakub@jirutka.cz> wrote:
>>>> A manual intervention on each builder is not an acceptable solution.
>>>> Policy requires that builds be reproducible, in other words, that they can build with `abuild rootbld`.
>>>> So this change should not have been pushed to begin with.
>>> 
>>> 
>>> No, I thought this too, but it’s not how it really works. See below.
>>> 
>>>> Look at the haskell + cabal packaging for inspiration, it is a similar situation, and is policy compliant.
>>> 
>>> Have *you* looked at ghc (haskell) package? I did some time ago (and now), tried hard to figure out how does it build. There is `makedepends_build="$pkgname"`, so it depends on itself. There used to be package ghc-bootstrap, but it was not used anymore (and even that couldn’t work without manual intervention). I asked fabled about it and he explained me that ghc-bootstrap was only temporary for initial bootstrap, now ghc depends on itself and it must be probably manually installed on the builder when bootstrapping.
>> 
>> I did some time ago, when we had ghc-bootstrap package.
>> If this is the case, then it, too, is no longer policy compliant.
>> 
>>> The same situation is community/rust. I even mentioned this in my email.
>> 
>> The same situation of being policy non-compliant?
>> 
>>> Unfortunately the bootstrapping (and cross-compilation) process is still not documented anywhere, so I can refer only to information that I was told by more competent people like fabled.
>> 
>> Manual intervention on the builders does not seem like a good idea to
>> me, which is probably why it's not compliant with policy.
>> 
>> I suspect the ghc-bootstrap package was dropped because of the
>> depsolver bug I solved back in April.
>> The bootstrap package would have a provides entry for the compiler
>> with a version of 0.
>> This would allow the real compiler to always "win" the preference.
>> 
>>>> A solution would be to take a copy of the cargo package from distfiles.alpinelinux.org
>>> 
>>> This is actually a good idea. I can keep it as it was before, just use our cargo binary instead of now non-existence binary from the upstream.
>> 
>> You could do that, or what I was inferring was that the binary itself
>> was still available on distfiles.alpinelinux.org.
>> All the builders copy their sources to that server.
>> 
>>> fabled or ncopa, could you please explain how is ghc and similar packages handled? Ideally directly on wiki, or just via email and I will copy it to wiki.
>> 
>> Instead, lets solve these problems correctly, please.
>> Indeed, the only thing we have learned today is that Haskell is also broken.
>> 
>> William
>> 
> 

On Mon, 30 Oct 2017 17:39:19 +0100
Jakub Jirutka <jakub@jirutka.cz> wrote:

> > Look at the haskell + cabal packaging for inspiration, it is a
> > similar situation, and is policy compliant.  
> 
> Have *you* looked at ghc (haskell) package? I did some time ago (and
> now), tried hard to figure out how does it build. There is
> `makedepends_build="$pkgname"`, so it depends on itself. There used
> to be package ghc-bootstrap, but it was not used anymore (and even
> that couldn’t work without manual intervention). I asked fabled about
> it and he explained me that ghc-bootstrap was only temporary for
> initial bootstrap, now ghc depends on itself and it must be probably
> manually installed on the builder when bootstrapping.
> 
> The same situation is community/rust. I even mentioned this in my
> email.
> 
> Unfortunately the bootstrapping (and cross-compilation) process is
> still not documented anywhere, so I can refer only to information
> that I was told by more competent people like fabled.

There's two separate "bootstrap" problems.

1) bootstrapping new stable branches, or new architectures. These are
usually cross-compiled using some existing Alpine build with the
scripts/bootstrap.sh. Doing this follows relatively working set of
rules.

2) bootstrapping packages that are self-dependent. This is annoying,
and starting to be increasing number with different languages. rust,
ghc etc. just to name a few.

Now go and java have been special, because for them there's been a way
to bootstrap them without binary builds. E.g. early go versions is
still go-bootstrap package for x86_64, which can be used to bootstrap
go. Or java can be done via gcc-java -> openjdk7 -> openjdk8. But
usually they now require themselves to build a version of compiler with
all features. So go wants to build it self with go. Especially when
targeting more "exotic" architectures than x86_64.

Now rust, ghc and some others are nasty. To have been able to produce
the first Alpine package has been impossible without existing ghc
compiler. So for ghc, we added ghc-bootstrap which was basically
Debian/docker produced binary blobs packages. And used to that produce
the first ghc compiler on x86_64. Since we don't want binary blobs like
that in aports, we dropped it right after ghc was built. All other
architectures were cross-compiled using alpine x86_64 ghc to target new
architecture.

> > A solution would be to take a copy of the cargo package from
> > distfiles.alpinelinux.org  
> 
> This is actually a good idea. I can keep it as it was before, just
> use our cargo binary instead of now non-existence binary from the
> upstream.
> 
> fabled or ncopa, could you please explain how is ghc and similar
> packages handled? Ideally directly on wiki, or just via email and I
> will copy it to wiki.

This is complex topic as the answer to previous question implies. And
it needs package specific insights.

But basically, if there's maintainable bootstrap procedure without
self-dependency, we are happy to have the "$pkg-bootstrap" package
doing this. But if the package is hardwired to require itself for
compilation, the only practical option is to do "$pkg-bootstrap" which
is basically binaries produced from foreign system, use that to build
the package first time, and then drop "$pkg-bootstrap" and makedepend
the package to build itself.

Hope this explains my standpoint. And reasoning on the different
approaches between different packages.

Timo