For discussion of Alpine Linux development and developer support

4 3

[alpine-devel] checkdepends and bootstrapping new releases

Natanael Copa
Details
Message ID
<20170429142041.74314721@ncopa-macbook.copa.dup.pw>
Sender timestamp
1493468441
DKIM signature
missing
Download raw message
Hi,

I think we have an issue with checkdepends in APKBUILDs. The problem is
that lua-aports does not pull in the the checkdepends as build time
dependency while calculating the build order.

We have 2 alternatives:

- always pull in the checkdepends.

  This will make sure that the checkdepends are always built before the
  package needing it. But this also means that we need to make sure
  that we don't introduce any circular dependencies via checkdepends.

- bootstrap new releases without running tests. By building the repo
  without any tests on the first run will allow us to have circular
  dependencies, but then we will not run any tests for our stable
  release (only updates will get the testing), or do we want rebuild all
  the package which checkdepends?

Opinions?

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
William Pitcock
Details
Message ID
<CA+T2pCF412gC-bdby+2Yh1GsukwyRYK3jrXjaxedabKfrAW+Eg@mail.gmail.com>
In-Reply-To
<20170429142041.74314721@ncopa-macbook.copa.dup.pw> (view parent)
Sender timestamp
1493502712
DKIM signature
missing
Download raw message
Hi,

On Sat, Apr 29, 2017 at 7:20 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:
> Hi,
>
> I think we have an issue with checkdepends in APKBUILDs. The problem is
> that lua-aports does not pull in the the checkdepends as build time
> dependency while calculating the build order.
>
> We have 2 alternatives:
>
> - always pull in the checkdepends.
>
>   This will make sure that the checkdepends are always built before the
>   package needing it. But this also means that we need to make sure
>   that we don't introduce any circular dependencies via checkdepends.
>
> - bootstrap new releases without running tests. By building the repo
>   without any tests on the first run will allow us to have circular
>   dependencies, but then we will not run any tests for our stable
>   release (only updates will get the testing), or do we want rebuild all
>   the package which checkdepends?
>
> Opinions?

I think we should prefer bootstrapping without tests.
The reason why is that the tests may fail *anyway* under bootstrapping
conditions (due to cross compiling, etc).
So tests should probably just be run when we're not bootstrapping.

William


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Natanael Copa
Details
Message ID
<20170501150510.36d36590@ncopa-macbook.copa.dup.pw>
In-Reply-To
<CA+T2pCF412gC-bdby+2Yh1GsukwyRYK3jrXjaxedabKfrAW+Eg@mail.gmail.com> (view parent)
Sender timestamp
1493643910
DKIM signature
missing
Download raw message
Hi,

On Sat, 29 Apr 2017 16:51:52 -0500
William Pitcock <nenolod@dereferenced.org> wrote:

> Hi,
> 
> On Sat, Apr 29, 2017 at 7:20 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:
> > Hi,
> >
> > I think we have an issue with checkdepends in APKBUILDs. The problem is
> > that lua-aports does not pull in the the checkdepends as build time
> > dependency while calculating the build order.
> >
> > We have 2 alternatives:
> >
> > - always pull in the checkdepends.
> >
> >   This will make sure that the checkdepends are always built before the
> >   package needing it. But this also means that we need to make sure
> >   that we don't introduce any circular dependencies via checkdepends.
> >
> > - bootstrap new releases without running tests. By building the repo
> >   without any tests on the first run will allow us to have circular
> >   dependencies, but then we will not run any tests for our stable
> >   release (only updates will get the testing), or do we want rebuild all
> >   the package which checkdepends?
> >
> > Opinions?  
> 
> I think we should prefer bootstrapping without tests.
> The reason why is that the tests may fail *anyway* under bootstrapping
> conditions (due to cross compiling, etc).
> So tests should probably just be run when we're not bootstrapping.

I'm not talking about the initial boostrap of toolchain and abuild. I
am talking about the next step, when we have a working abuild and
aports-build and will build the rest of the repositories. If we disable
checks, it means that the releases are built without checks.

This is actually not only affect when we build new releases from
scratch, but also affects edge.

What happens if you push 2 commits, one with new aport for a new
language (lets call it newlang). The checkdepends needs a new library
implemented in newlang. The second commit has this library (lets callit
newlib).

---
pkgname=newlang
checkdepends=newlib
---
pkgname=newlib
depends=newlang

Now the question is, how should the build resolver handle this?

The correct thing to do would be to build a temporary newlang with
tests disabled, use this to build newlib and finally re-build newlang
with tests enabled. But this is not implemented on lua-aports and is
sort of complicated since it needs to be auto detected.

In other words, its the classic bootstrap problem. Same as we had when
pkg-config who depended on glib and glib depended on pkg-config.

Also, what if an upgrade needs a newer version of the checkdepends?
then we need to build the checkdepends before the package needing it,
which means that we need to pull in the checkdepends in the dependency
tree to figure out the build order. This means that the build-order
resolver needs to figure out a good way to handle circular dependencies
anyway.

One option is that we could make the 'check' opportunistic, that we
only run check after verified that checkdepends is not circular (for
example when there are no checkdepends at all), but i dont like that
either because then it becomes unpredicatble if checks are run or not.

I think the check is something we really want, Jirutka's detect of the
libressl issue clearly shows that.

But I don't know how. Might be we need disable the checks for 3.6
release and solve those issues properly after the release.

-nc

> William
> 
> 
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
> 



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
William Pitcock
Details
Message ID
<CA+T2pCE35zuGBEYazVY936Sbfdb-t4A3_2ynEtHNrnWreLkAkw@mail.gmail.com>
In-Reply-To
<20170501150510.36d36590@ncopa-macbook.copa.dup.pw> (view parent)
Sender timestamp
1493682512
DKIM signature
missing
Download raw message
Hello,

On Mon, May 1, 2017 at 8:05 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:
> Hi,
>
> On Sat, 29 Apr 2017 16:51:52 -0500
> William Pitcock <nenolod@dereferenced.org> wrote:
>
>> Hi,
>>
>> On Sat, Apr 29, 2017 at 7:20 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:
>> > Hi,
>> >
>> > I think we have an issue with checkdepends in APKBUILDs. The problem is
>> > that lua-aports does not pull in the the checkdepends as build time
>> > dependency while calculating the build order.
>> >
>> > We have 2 alternatives:
>> >
>> > - always pull in the checkdepends.
>> >
>> >   This will make sure that the checkdepends are always built before the
>> >   package needing it. But this also means that we need to make sure
>> >   that we don't introduce any circular dependencies via checkdepends.
>> >
>> > - bootstrap new releases without running tests. By building the repo
>> >   without any tests on the first run will allow us to have circular
>> >   dependencies, but then we will not run any tests for our stable
>> >   release (only updates will get the testing), or do we want rebuild all
>> >   the package which checkdepends?
>> >
>> > Opinions?
>>
>> I think we should prefer bootstrapping without tests.
>> The reason why is that the tests may fail *anyway* under bootstrapping
>> conditions (due to cross compiling, etc).
>> So tests should probably just be run when we're not bootstrapping.
>
> I'm not talking about the initial boostrap of toolchain and abuild. I
> am talking about the next step, when we have a working abuild and
> aports-build and will build the rest of the repositories. If we disable
> checks, it means that the releases are built without checks.
>
> This is actually not only affect when we build new releases from
> scratch, but also affects edge.
>
> What happens if you push 2 commits, one with new aport for a new
> language (lets call it newlang). The checkdepends needs a new library
> implemented in newlang. The second commit has this library (lets callit
> newlib).
>
> ---
> pkgname=newlang
> checkdepends=newlib
> ---
> pkgname=newlib
> depends=newlang
>
> Now the question is, how should the build resolver handle this?
>
> The correct thing to do would be to build a temporary newlang with
> tests disabled, use this to build newlib and finally re-build newlang
> with tests enabled. But this is not implemented on lua-aports and is
> sort of complicated since it needs to be auto detected.
>
> In other words, its the classic bootstrap problem. Same as we had when
> pkg-config who depended on glib and glib depended on pkg-config.
>
> Also, what if an upgrade needs a newer version of the checkdepends?
> then we need to build the checkdepends before the package needing it,
> which means that we need to pull in the checkdepends in the dependency
> tree to figure out the build order. This means that the build-order
> resolver needs to figure out a good way to handle circular dependencies
> anyway.
>
> One option is that we could make the 'check' opportunistic, that we
> only run check after verified that checkdepends is not circular (for
> example when there are no checkdepends at all), but i dont like that
> either because then it becomes unpredicatble if checks are run or not.
>
> I think the check is something we really want, Jirutka's detect of the
> libressl issue clearly shows that.
>
> But I don't know how. Might be we need disable the checks for 3.6
> release and solve those issues properly after the release.

I think we make aports rebuild scripts calculate checkdepends as part
of the dependency, unless it is doing an initial build of the
distribution.
If it is an initial build, then we do not run check like I said before.
What may be interesting is to build the distribution and then use that
distribution to rebuild itself with checks enabled.

In short, it should be a policy option instead of opportunistic.

William


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Jean-Louis Fuchs
Details
Message ID
<20170501175955.GA7124@angua.1042.ch>
In-Reply-To
<20170501150510.36d36590@ncopa-macbook.copa.dup.pw> (view parent)
Sender timestamp
1493661595
DKIM signature
missing
Download raw message
Hi Natanael

On Mon, May 01, 2017 at 03:05:10PM +0200, Natanael Copa wrote:
> The correct thing to do would be to build a temporary newlang with
> tests disabled, use this to build newlib and finally re-build newlang
> with tests enabled. But this is not implemented on lua-aports and is
> sort of complicated since it needs to be auto detected.

If you don't want to implement a complex logic, what about retrying
_once_ if it has checkdepends?

Depend fail:

* Aport a check fails -> schedule retry an hour later
* Aport b build success
* Aport a check success, because b has been built in the meantime

Real fail:

* Aport a check fails -> schedule retry an hour later
* Aport b build success
* Aport a check fails -> because something is broken

Best,
    Jean-Louis