~alpine/devel

11 7

Proposal: replace ninja with samurai

Details
Message ID
<BYZDB5LH4OVY.9FDVF5I3J954@homura>
DKIM signature
missing
Download raw message
Ninja is a build tool which fills the same niche as Make, and samurai is
a third-party implementation on of Ninja.

https://github.com/ninja-build/ninja
https://github.com/michaelforney/samurai

samu is much simpler than ninja, clocking in at around ~3,000 lines of
C, compared to Ninja's ~20,000 lines of C++. samu also supports a
SAMUFLAGS environment variable which we can use to tune the -j flag in
abuild, similar to Make. Samurai is a drop-in replacement, but is not
bug-for-bug compatible with Ninja.

Ninja upstream has been hostile to appeals from distros and users alike
to add a similar environment variable. Their argument seems to boil down
to being proud to only support one environment variable, which can
change the format string their progress meter is printed with. Their
attitude is non-cooperative towards distros, and I would rather use an
implementation which plays more nicely with others.

I've been using samu as my daily driver for a few years now, and I've
yet to meet a project it couldn't compile. Its performance is also
comparable to ninja. A samurai package is available in community on edge
if you would like to try it for yourself.

I propose dropping the ninja package and adding provides=replaces=ninja
to the samurai package, as well as a symlink from /usr/bin/ninja to
/usr/bin/samu.
Details
Message ID
<20191207183917.74342cbb@Impreza>
In-Reply-To
<BYZDB5LH4OVY.9FDVF5I3J954@homura> (view parent)
DKIM signature
missing
Download raw message
+1 from Me, i packaged it for Exherbo and Void Linux and i had plans to
make this same change on Void Linux.
Rasmus Thomsen <oss@cogitri.dev>
Details
Message ID
<fb158baeba5b4454c7e0b01755c42a3e145686b3.camel@cogitri.dev>
In-Reply-To
<BYZDB5LH4OVY.9FDVF5I3J954@homura> (view parent)
DKIM signature
missing
Download raw message
On Sat, 2019-12-07 at 12:37 -0500, Drew DeVault wrote:
> I propose dropping the ninja package and adding
> provides=replaces=ninja
> to the samurai package, as well as a symlink from /usr/bin/ninja to
> /usr/bin/samu.

Although I do like the rest of your proposal (using Samu instead of
Ninja for building packages on Alpine) I oppose dropping the ninja
package. As you've mentioned Samu isn't bug compatible with Ninja, so
users may very well want to keep using ninja (if only to use what a
project's upstream uses for bug reports etc.).

Regards,

Rasmus
Details
Message ID
<BZ10HOAT7Z8A.1DMTQFIGETY9K@homura>
In-Reply-To
<fb158baeba5b4454c7e0b01755c42a3e145686b3.camel@cogitri.dev> (view parent)
DKIM signature
missing
Download raw message
Perhaps it can be moved to unmaintained. The cases where anyone hits a
bug-for-bug compatibility issue should be _exceptionally_ rare, because
no one writes ninja build files by hand - they're always written by
tools, and anything which hits these bugs will be the fault of the
higher-level tool, which will have surely received a bugfix. Multiple
unrelated systems have to fail for someone to hit one of these edge
cases (corner cases?).
Details
Message ID
<20191211094116.4908c616@ncopa-desktop.copa.dup.pw>
In-Reply-To
<BYZDB5LH4OVY.9FDVF5I3J954@homura> (view parent)
DKIM signature
missing
Download raw message
On Sat, 07 Dec 2019 12:37:56 -0500
"Drew DeVault" <sir@cmpwn.com> wrote:

> Ninja is a build tool which fills the same niche as Make, and samurai is
> a third-party implementation on of Ninja.
> 
> https://github.com/ninja-build/ninja
> https://github.com/michaelforney/samurai
> 
> samu is much simpler than ninja, clocking in at around ~3,000 lines of
> C, compared to Ninja's ~20,000 lines of C++. samu also supports a
> SAMUFLAGS environment variable which we can use to tune the -j flag in
> abuild, similar to Make. Samurai is a drop-in replacement, but is not
> bug-for-bug compatible with Ninja.

What I think is nice with ninja compared to make, is that we don't need
set -j to make it use all cores. Will samurai not use all detected
cores without setting -j?

> Ninja upstream has been hostile to appeals from distros and users
> alike to add a similar environment variable. Their argument seems to
> boil down to being proud to only support one environment variable,
> which can change the format string their progress meter is printed
> with. Their attitude is non-cooperative towards distros, and I would
> rather use an implementation which plays more nicely with others.

Do you have any example with hostility or non-cooperative against
Alpine?

I am against this change if it is purely for political reasons.

> I've been using samu as my daily driver for a few years now, and I've
> yet to meet a project it couldn't compile. Its performance is also
> comparable to ninja. A samurai package is available in community on edge
> if you would like to try it for yourself.

Does chromium build with samurai? Does chromium developers support
builds with samurai?

> I propose dropping the ninja package and adding provides=replaces=ninja
> to the samurai package, as well as a symlink from /usr/bin/ninja to
> /usr/bin/samu.

I'd like to have them both available so we can easily pick one
depending on which works or is supported.

I also saw that there was a comparison with pkg-config vs pkgconf on
IRC. The main motivator for pkgconf switch was that pkg-config
introduced a circular dependency (pkg-config -> glib -> pkg-config).
The pkgconf author is also an Alpine developer so any issues we had
with pkgconf were fixed very quick upstream.

Do we have similar problems with ninja?

Can we expect the same level of support from samurai that we have with
pkgconf?

-nc
Details
Message ID
<CAML-Udta6PjdKJEAYvee6rvOxRi-ywYB7Uj5nDkp9K0fM1FZzA@mail.gmail.com>
In-Reply-To
<20191211094116.4908c616@ncopa-desktop.copa.dup.pw> (view parent)
DKIM signature
missing
Download raw message
On Wed, Dec 11, 2019 at 3:41 AM Natanael Copa <ncopa@alpinelinux.org> wrote:
>
> On Sat, 07 Dec 2019 12:37:56 -0500
> "Drew DeVault" <sir@cmpwn.com> wrote:
>
> > Ninja is a build tool which fills the same niche as Make, and samurai is
> > a third-party implementation on of Ninja.
> >
> > https://github.com/ninja-build/ninja
> > https://github.com/michaelforney/samurai
> >
> > samu is much simpler than ninja, clocking in at around ~3,000 lines of
> > C, compared to Ninja's ~20,000 lines of C++. samu also supports a
> > SAMUFLAGS environment variable which we can use to tune the -j flag in
> > abuild, similar to Make. Samurai is a drop-in replacement, but is not
> > bug-for-bug compatible with Ninja.
>
> What I think is nice with ninja compared to make, is that we don't need
> set -j to make it use all cores. Will samurai not use all detected
> cores without setting -j?

The primary reason to include -j$JOBS is not because samurai won't use
all cores by default, but rather because, in some circumstances, using
all cores may not be desirable (e.g. if one lacks the RAM to utilize
all available cores at once on large projects).
Details
Message ID
<BZ2O2OMRVNZG.38Y8EDZCUSIQH@homura>
In-Reply-To
<20191211094116.4908c616@ncopa-desktop.copa.dup.pw> (view parent)
DKIM signature
missing
Download raw message
On Wed Dec 11, 2019 at 9:41 AM, Natanael Copa wrote:
> What I think is nice with ninja compared to make, is that we don't need
> set -j to make it use all cores. Will samurai not use all detected
> cores without setting -j?

Yes, it uses all cores even without -j being set.

> Do you have any example with hostility or non-cooperative against
> Alpine?
>
> I am against this change if it is purely for political reasons.

It's not purely for political reasons, though they are a factor. I
explained that samurai is a much smaller and simpler implementation, an
order of magnitude smaller and written in C rather than C++.

The non-cooperation with requests from distros and users to add a means
of controlling the jobs via the environment is here:

https://github.com/ninja-build/ninja/issues/1482

> Does chromium build with samurai? Does chromium developers support
> builds with samurai?

I've asked around to see if anyone has built Chromium with samurai. I
expect it works fine.

Does Chromium support builds with musl? Busybox?

> I'd like to have them both available so we can easily pick one
> depending on which works or is supported.

As discussed on IRC, the chances of hitting a bug where samurai doesn't
work and ninja does is extremely slim, and has never been seen in
practice.

> I also saw that there was a comparison with pkg-config vs pkgconf on
> IRC. The main motivator for pkgconf switch was that pkg-config
> introduced a circular dependency (pkg-config -> glib -> pkg-config).
> The pkgconf author is also an Alpine developer so any issues we had
> with pkgconf were fixed very quick upstream.
> 
> Do we have similar problems with ninja?
>
> Can we expect the same level of support from samurai that we have with
> pkgconf?

There are no circular dependency issues with ninja that I'm aware of.

We can probably expect a good level of support with samurai, Michael
has always been responsive and is an active Alpine Linux user.
Details
Message ID
<LPZWmJ3431mZH6Y4Ob8JOKuDApbz6dmFBOmCU4omhYUQ2vyu4UZ3dUXW7rrtl2W3rMHWCMNCSPEnUxyh5f_h1nj8VOUYO__GC6hRPulrCAw=@protonmail.com>
In-Reply-To
<20191211094116.4908c616@ncopa-desktop.copa.dup.pw> (view parent)
DKIM signature
missing
Download raw message
> Does chromium developers support builds with samurai?

I think if we start asking that for packages, we will end up just like all other distros and use the most common (but not the best) dependencies.
What I like about Alpine is the use of smaller and simpler solutions.



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, December 11, 2019 5:41 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:

> On Sat, 07 Dec 2019 12:37:56 -0500
> "Drew DeVault" sir@cmpwn.com wrote:
>
> > Ninja is a build tool which fills the same niche as Make, and samurai is
> > a third-party implementation on of Ninja.
> > https://github.com/ninja-build/ninja
> > https://github.com/michaelforney/samurai
> > samu is much simpler than ninja, clocking in at around ~3,000 lines of
> > C, compared to Ninja's ~20,000 lines of C++. samu also supports a
> > SAMUFLAGS environment variable which we can use to tune the -j flag in
> > abuild, similar to Make. Samurai is a drop-in replacement, but is not
> > bug-for-bug compatible with Ninja.
>
> What I think is nice with ninja compared to make, is that we don't need
> set -j to make it use all cores. Will samurai not use all detected
> cores without setting -j?
>
> > Ninja upstream has been hostile to appeals from distros and users
> > alike to add a similar environment variable. Their argument seems to
> > boil down to being proud to only support one environment variable,
> > which can change the format string their progress meter is printed
> > with. Their attitude is non-cooperative towards distros, and I would
> > rather use an implementation which plays more nicely with others.
>
> Do you have any example with hostility or non-cooperative against
> Alpine?
>
> I am against this change if it is purely for political reasons.
>
> > I've been using samu as my daily driver for a few years now, and I've
> > yet to meet a project it couldn't compile. Its performance is also
> > comparable to ninja. A samurai package is available in community on edge
> > if you would like to try it for yourself.
>
> Does chromium build with samurai? Does chromium developers support
> builds with samurai?
>
> > I propose dropping the ninja package and adding provides=replaces=ninja
> > to the samurai package, as well as a symlink from /usr/bin/ninja to
> > /usr/bin/samu.
>
> I'd like to have them both available so we can easily pick one
> depending on which works or is supported.
>
> I also saw that there was a comparison with pkg-config vs pkgconf on
> IRC. The main motivator for pkgconf switch was that pkg-config
> introduced a circular dependency (pkg-config -> glib -> pkg-config).
> The pkgconf author is also an Alpine developer so any issues we had
> with pkgconf were fixed very quick upstream.
>
> Do we have similar problems with ninja?
>
> Can we expect the same level of support from samurai that we have with
> pkgconf?
>
> -nc
Details
Message ID
<BZ64G5WZ6AER.RTBMRG2COZ30@homura>
In-Reply-To
<LPZWmJ3431mZH6Y4Ob8JOKuDApbz6dmFBOmCU4omhYUQ2vyu4UZ3dUXW7rrtl2W3rMHWCMNCSPEnUxyh5f_h1nj8VOUYO__GC6hRPulrCAw=@protonmail.com> (view parent)
DKIM signature
missing
Download raw message
Michael has confirmed that he does build Chromium with samurai before
every release as an acceptance test.
Michael Forney <mforney@mforney.org>
Details
Message ID
<CAGw6cBsAdr=hfyk_vKrUyuBhkmWBf9RB=CT0+T08v76Pif5RAg@mail.gmail.com>
In-Reply-To
<BYZDB5LH4OVY.9FDVF5I3J954@homura> (view parent)
DKIM signature
missing
Download raw message
On 2019-12-07, Drew DeVault <sir@cmpwn.com> wrote:
> I've been using samu as my daily driver for a few years now, and I've
> yet to meet a project it couldn't compile. Its performance is also
> comparable to ninja. A samurai package is available in community on edge
> if you would like to try it for yourself.

In my testing, samurai is slightly faster than ninja and uses less
memory. Of course, this is a negligible part of complete builds, but
it is nice during development for incremental builds.

In preparation for releasing samurai 1.0, I built all Alpine packages
that used ninja. The aports branch I used is at [0]. Here are the
results:

## community/chromium

The tools/gn/bootstrap/bootstrap.py script hardcoded `ninja`, and it
passed options to ninja after the build target. samurai doesn't do
GNU-style argument permutation, so this failed. I submitted a CL to
fix this, which is now submitted at [1].

I wasn't patient enough to let the build finish after fixing this, but
I've built chromium multiple times in the past from the git
repository, so I don't expect any other issues.

## community/fwupd

The coreboot module was missing a dependency on the generated
fu-hash.h. This is now fixed upstream in [2].

## community/grilo

The APKBUILD had $source containing a patch auto-generated by gitlab,
which doesn't appear to be stable, so the checksum failed. This is
unrelated to samurai, but I submitted a patch to fix this at [3]. The
build succeeded after this.

## community/knot-resolver

The APKBUILD invoked ninja with options (`-C build`) following the
build target (`install`). As explained above, samurai doesn't do
argument permutation, so the arguments needed to be switched around.

## community/qt5-qtwebengine

Similar to chromium, this package hard-coded 'ninja', and needed some
changes to allow 'samu' instead. This should not be a problem if you
provide a compatibility symlink /usr/bin/ninja -> samu. I submitted a
change to make the ninja command configurable at [8].

## community/weston

The presentation-shm demo client was missing a dependency on the
generated xdg-shell-client-protocol.h. This is now fixed upstream in
[4].

## testing/wob

This package was missing dependencies on the generated
wlr-layer-shell-unstable-v1.h. I submitted a pull request to fix this
at [5].

## community/gtk4.0

The build failed because the linker used too much memory and was
killed, but the build was progressing smoothly up until that point.

## main/libinput

This package failed its test due to missing /dev/fd, presumably
because I wasn't running udev. I created a symlink /dev/fd ->
../proc/self/fd, and the tests succeeded.

## community/libhandy, testing/libzim

These packages failed their tests (logs at [6][7]). I don't believe
this is related to samurai.

## community/gom

The tests hang for me. I don't believe this is related to samurai.


Note that any instance of missing dependencies can also affect ninja
depending on the maximum number of jobs and how long they take to run.

-Michael

[0] https://github.com/alpinelinux/aports/compare/master...michaelforney:samurai
[1] https://chromium-review.googlesource.com/c/chromium/src/+/1975453
[2] https://github.com/fwupd/fwupd/commit/9bdb96f768f1318ccaac4ecbcac68200a36ee1be
[3] https://lists.alpinelinux.org/~alpine/aports/patches/3179
[4] https://gitlab.freedesktop.org/wayland/weston/commit/ba3b384b930f9ab40f94d125d4d26bc4b636821f
[5] https://github.com/francma/wob/pull/23
[6] http://ix.io/24Qr
[7] http://ix.io/24Qq
[8] https://codereview.qt-project.org/c/qt/qtwebengine/+/285134
Michael Forney <mforney@mforney.org>
Details
Message ID
<CAGw6cBuBUqL3e6Mvrq5tvCsKRanMDyYKEVHTRyHqVh-zj9EUpA@mail.gmail.com>
In-Reply-To
<20191211094116.4908c616@ncopa-desktop.copa.dup.pw> (view parent)
DKIM signature
missing
Download raw message
On 2019-12-11, Natanael Copa <ncopa@alpinelinux.org> wrote:
> Does chromium build with samurai? Does chromium developers support
> builds with samurai?

chromium, llvm, and mesa are my usual sanity check build targets
before I make a release. That covers large projects for most major
build systems using ninja (gn, cmake, and meson).

> I also saw that there was a comparison with pkg-config vs pkgconf on
> IRC. The main motivator for pkgconf switch was that pkg-config
> introduced a circular dependency (pkg-config -> glib -> pkg-config).
> The pkgconf author is also an Alpine developer so any issues we had
> with pkgconf were fixed very quick upstream.
>
> Do we have similar problems with ninja?
>
> Can we expect the same level of support from samurai that we have with
> pkgconf?

ninja requires itself to build, but also comes with a configure.py
script that, when passed --bootstrap, first builds a version of ninja
as it generates .ninja files, then uses that to build itself. So,
there is no issue bootstrapping ninja in practice.

The current patch you have for ninja-1.9.0 was an issue I reported in
[0] over a month before the release was made as I was looking through
the ninja commits to make corresponding behavior changes in samurai.
nykula submitted a patch for this shortly afterward. The patch did not
get applied until 8 months later, and is still not in any release of
ninja.

I've been using musl enthusiastically since 2013, and am a musl
release sponsor, so you can expect samurai's musl support to be better
than ninja's.

I'm not sure what level of support you get with pkgconf, but I have
fixed all reported bugs so far within a couple days (usually on the
same day).

-Michael

[0] https://github.com/ninja-build/ninja/issues/1510
Rasmus Thomsen <oss@cogitri.dev>
Details
Message ID
<7e1f2d6953dee88df53f9c02aa2a27fe4245e72d.camel@cogitri.dev>
In-Reply-To
<CAGw6cBsAdr=hfyk_vKrUyuBhkmWBf9RB=CT0+T08v76Pif5RAg@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
> ## community/libhandy, testing/libzim
> 
> These packages failed their tests (logs at [6][7]). I don't believe
> this is related to samurai.
> 
> [6] http://ix.io/24Qr

At least libhandy is failing due to you not running dbus, so this isn't
on samurai.

From my point of view using Samu should be fine then, thank you for
your work!

Regards, Rasmus
Reply to thread Export thread (mbox)