Subject: [alpine-devel] Calling build() check() package() might be a bad order

From: Jean-Louis Fuchs <[email-masked]>
Date: Sat, 4 Mar 2017 16:34:49 +0100

Hi

I wanted to point out that there might be a problem with the order in
which abuild calls build(), check() and package().

My project has to be built different for testing and releasing. Since
abuild will call build() first, then check() and then package(), it is
built the wrong way either in check() or in package(). I am in the
fortunate position that my project can do out-of-source builds, but if
it couldn't I would have to do the following:

build()
    Build for release

check()
    Clean, build for test, test, clean and build for release

package()
    Package

I know that any project that does coverage has to be rebuilt for
testing, so my project shouldn't be the only case. Also tests often
only work if assert is active, which it isn't in release mode.

Here is the out-of-source version of the package of my project:
https://github.com/ganwell/chirp/blob/build_cleanup/dist/alpine/chirp/APKBUILD

If check() would run at the end:

build()
    Build for release

package()
    Package

check()
    Clean, build for test and test

May be there is a much better way that I am not aware of, in that case
we should document this in the wiki.

Best,
    Jean-Louis

---

From: Ivan Tham <[email-masked]>
Date: Sun, 5 Mar 2017 13:08:29 +0800

Yes, I think build() -> package() -> check() would be a nice idea. I
suggest that check() can be optional and not be included as well if we
are building for abuild since there is not much reason to do so.

-- 
Do what you like, like what you do.  -- Pickfire

---

From: William Pitcock <[email-masked]>
Date: Tue, 7 Mar 2017 00:34:27 -0600

Hello,

> are building for abuild since there is not much reason to do so.

The proposal which brought us check made it explicit that packages are
required to run testsuites if possible prior to generating a package.
This was on purpose: we want to ensure that what we are shipping (the
binaries generated by build()) properly passes upstream tests -- this
gives us a significant advantage in catching bugs before they affect
end users.
If a package does not wish to use check(), it can declare that it does
not want to use check() by declaring "!check" in $options.
In the future (2018), it will be considered a hard failure if the
check() function is missing and "!check" is not in $options.

>>    Clean, build for test, test, clean and build for release

This is not what check() is for.  The point of check() is that what
you are putting in the package itself should be tested.
If a release build cannot be tested then you should opt-out of testing
by declaring "!check" in $options.
However, as an aside, I would strongly suggest that you make your
release builds formally testable, because that will allow you to make
sure that what you are actually releasing is in good shape.

>> only work if assert is active, which it isn't in release mode.

Coverage tests and checking for assert() hits is not really what the
scope of check() is.
check() is for running formal testsuites against a "release" build,
using a set of tests and a test runner such as kyua (see main/pkgconf
package for an example of what a kyua-based testsuite looks like), or
the autotools test runner, etc.
If this does not apply to a package, then they should just skip tests
as passing/failing on binaries other than the ones we are going to
ship isn't directly relevant to us.

A way to look at it is that check() is for checking that the binaries
actually going into the package are kosher.

>> we should document this in the wiki.

There is no other way to do it: the point of check() coming after
build() and before package() is so that we can fail the build before
we package it.  This is an intentional design decision.
In your case, I would simply not bother running the testsuite at all,
because you're not testing the actual binaries going into the package,
making the testsuite ineffective (from a distribution point of view).

William

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Jean-Louis Fuchs <[email-masked]>
Date: Wed, 8 Mar 2017 14:19:51 +0100

Hi William

On Tue, Mar 07, 2017 at 12:34:27AM -0600, William Pitcock wrote:

> I would simply not bother running the testsuite at all,

I think I got it, I will change my build system the following way:

make check
    Not instrumented (release mode), goal: checking compatibility 

make test
    Instrumented (dev mode), goal: helping developers to find bugs

Of course most of tests are shared between both test-suites, but some
may not make sense in either test-suite.

Best,
    Jean-Louis

---

From: William Pitcock <[email-masked]>
Date: Thu, 9 Mar 2017 11:28:19 -0600

Hi,

> may not make sense in either test-suite.

Absolutely this is the way to go.  Then run "make check" from the APKBUILD.

William

---