Subject: Proposal: replace ninja with samurai

From: "Drew DeVault" <[email-masked]>
Date: Sat, 07 Dec 2019 12:37:56 -0500

Ninja is a build tool which fills the same niche as Make, and samurai is
a third-party implementation on of Ninja.

https://github.com/ninja-build/ninja
https://github.com/michaelforney/samurai

samu is much simpler than ninja, clocking in at around ~3,000 lines of
C, compared to Ninja's ~20,000 lines of C++. samu also supports a
SAMUFLAGS environment variable which we can use to tune the -j flag in
abuild, similar to Make. Samurai is a drop-in replacement, but is not
bug-for-bug compatible with Ninja.

Ninja upstream has been hostile to appeals from distros and users alike
to add a similar environment variable. Their argument seems to boil down
to being proud to only support one environment variable, which can
change the format string their progress meter is printed with. Their
attitude is non-cooperative towards distros, and I would rather use an
implementation which plays more nicely with others.

I've been using samu as my daily driver for a few years now, and I've
yet to meet a project it couldn't compile. Its performance is also
comparable to ninja. A samurai package is available in community on edge
if you would like to try it for yourself.

I propose dropping the ninja package and adding provides=replaces=ninja
to the samurai package, as well as a symlink from /usr/bin/ninja to
/usr/bin/samu.

---

From: Leo <[email-masked]>
Date: Sat, 7 Dec 2019 18:39:17 +0100

+1 from Me, i packaged it for Exherbo and Void Linux and i had plans to
make this same change on Void Linux.

---

From: Rasmus Thomsen <[email-masked]>
Date: Sun, 08 Dec 2019 19:55:20 +0000

> to the samurai package, as well as a symlink from /usr/bin/ninja to

Although I do like the rest of your proposal (using Samu instead of
Ninja for building packages on Alpine) I oppose dropping the ninja
package. As you've mentioned Samu isn't bug compatible with Ninja, so
users may very well want to keep using ninja (if only to use what a
project's upstream uses for bug reports etc.).

Regards,

Rasmus

---

From: "Drew DeVault" <[email-masked]>
Date: Mon, 09 Dec 2019 11:00:32 -0500

Perhaps it can be moved to unmaintained. The cases where anyone hits a
bug-for-bug compatibility issue should be _exceptionally_ rare, because
no one writes ninja build files by hand - they're always written by
tools, and anything which hits these bugs will be the fault of the
higher-level tool, which will have surely received a bugfix. Multiple
unrelated systems have to fail for someone to hit one of these edge
cases (corner cases?).

---

From: Natanael Copa <[email-masked]>
Date: Wed, 11 Dec 2019 09:41:16 +0100

On Sat, 07 Dec 2019 12:37:56 -0500

> bug-for-bug compatible with Ninja.

What I think is nice with ninja compared to make, is that we don't need
set -j to make it use all cores. Will samurai not use all detected
cores without setting -j?

> Ninja upstream has been hostile to appeals from distros and users
> alike to add a similar environment variable. Their argument seems to
> boil down to being proud to only support one environment variable,
> which can change the format string their progress meter is printed
> with. Their attitude is non-cooperative towards distros, and I would
> rather use an implementation which plays more nicely with others.

Do you have any example with hostility or non-cooperative against
Alpine?

I am against this change if it is purely for political reasons.

> if you would like to try it for yourself.

Does chromium build with samurai? Does chromium developers support
builds with samurai?

> to the samurai package, as well as a symlink from /usr/bin/ninja to

I'd like to have them both available so we can easily pick one
depending on which works or is supported.

I also saw that there was a comparison with pkg-config vs pkgconf on
IRC. The main motivator for pkgconf switch was that pkg-config
introduced a circular dependency (pkg-config -> glib -> pkg-config).
The pkgconf author is also an Alpine developer so any issues we had
with pkgconf were fixed very quick upstream.

Do we have similar problems with ninja?

Can we expect the same level of support from samurai that we have with
pkgconf?

-nc

---

From: ecsx <[email-masked]>
Date: Sun, 15 Dec 2019 11:13:12 +0000

> Does chromium developers support builds with samurai?

I think if we start asking that for packages, we will end up just like all other distros and use the most common (but not the best) dependencies.
What I like about Alpine is the use of smaller and simpler solutions.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, December 11, 2019 5:41 AM, Natanael Copa <[email-masked]> wrote:

> "Drew DeVault" [email-masked] wrote:
>

---

From: "Drew DeVault" <[email-masked]>
Date: Sun, 15 Dec 2019 11:09:58 -0500

Michael has confirmed that he does build Chromium with samurai before
every release as an acceptance test.

---

From: Kiyoshi Aman <[email-masked]>
Date: Wed, 11 Dec 2019 08:58:44 -0500

> cores without setting -j?

The primary reason to include -j$JOBS is not because samurai won't use
all cores by default, but rather because, in some circumstances, using
all cores may not be desirable (e.g. if one lacks the RAM to utilize
all available cores at once on large projects).

---

From: "Drew DeVault" <[email-masked]>
Date: Wed, 11 Dec 2019 09:42:04 -0500

> cores without setting -j?

Yes, it uses all cores even without -j being set.

> I am against this change if it is purely for political reasons.

It's not purely for political reasons, though they are a factor. I
explained that samurai is a much smaller and simpler implementation, an
order of magnitude smaller and written in C rather than C++.

The non-cooperation with requests from distros and users to add a means
of controlling the jobs via the environment is here:

https://github.com/ninja-build/ninja/issues/1482

> Does chromium build with samurai? Does chromium developers support

I've asked around to see if anyone has built Chromium with samurai. I
expect it works fine.

Does Chromium support builds with musl? Busybox?

> depending on which works or is supported.

As discussed on IRC, the chances of hitting a bug where samurai doesn't
work and ninja does is extremely slim, and has never been seen in
practice.

> Can we expect the same level of support from samurai that we have with

There are no circular dependency issues with ninja that I'm aware of.

We can probably expect a good level of support with samurai, Michael
has always been responsive and is an active Alpine Linux user.

---

From: Michael Forney <[email-masked]>
Date: Thu, 19 Dec 2019 22:37:29 -0800

> Does chromium build with samurai? Does chromium developers support

chromium, llvm, and mesa are my usual sanity check build targets
before I make a release. That covers large projects for most major
build systems using ninja (gn, cmake, and meson).

> Can we expect the same level of support from samurai that we have with

ninja requires itself to build, but also comes with a configure.py
script that, when passed --bootstrap, first builds a version of ninja
as it generates .ninja files, then uses that to build itself. So,
there is no issue bootstrapping ninja in practice.

The current patch you have for ninja-1.9.0 was an issue I reported in
[0] over a month before the release was made as I was looking through
the ninja commits to make corresponding behavior changes in samurai.
nykula submitted a patch for this shortly afterward. The patch did not
get applied until 8 months later, and is still not in any release of
ninja.

I've been using musl enthusiastically since 2013, and am a musl
release sponsor, so you can expect samurai's musl support to be better
than ninja's.

I'm not sure what level of support you get with pkgconf, but I have
fixed all reported bugs so far within a couple days (usually on the
same day).

-Michael

[0] https://github.com/ninja-build/ninja/issues/1510

---

From: Michael Forney <[email-masked]>
Date: Thu, 19 Dec 2019 21:56:18 -0800

> if you would like to try it for yourself.

In my testing, samurai is slightly faster than ninja and uses less
memory. Of course, this is a negligible part of complete builds, but
it is nice during development for incremental builds.

In preparation for releasing samurai 1.0, I built all Alpine packages
that used ninja. The aports branch I used is at [0]. Here are the
results:

## community/chromium

The tools/gn/bootstrap/bootstrap.py script hardcoded `ninja`, and it
passed options to ninja after the build target. samurai doesn't do
GNU-style argument permutation, so this failed. I submitted a CL to
fix this, which is now submitted at [1].

I wasn't patient enough to let the build finish after fixing this, but
I've built chromium multiple times in the past from the git
repository, so I don't expect any other issues.

## community/fwupd

The coreboot module was missing a dependency on the generated
fu-hash.h. This is now fixed upstream in [2].

## community/grilo

The APKBUILD had $source containing a patch auto-generated by gitlab,
which doesn't appear to be stable, so the checksum failed. This is
unrelated to samurai, but I submitted a patch to fix this at [3]. The
build succeeded after this.

## community/knot-resolver

The APKBUILD invoked ninja with options (`-C build`) following the
build target (`install`). As explained above, samurai doesn't do
argument permutation, so the arguments needed to be switched around.

## community/qt5-qtwebengine

Similar to chromium, this package hard-coded 'ninja', and needed some
changes to allow 'samu' instead. This should not be a problem if you
provide a compatibility symlink /usr/bin/ninja -> samu. I submitted a
change to make the ninja command configurable at [8].

## community/weston

The presentation-shm demo client was missing a dependency on the
generated xdg-shell-client-protocol.h. This is now fixed upstream in
[4].

## testing/wob

This package was missing dependencies on the generated
wlr-layer-shell-unstable-v1.h. I submitted a pull request to fix this
at [5].

## community/gtk4.0

The build failed because the linker used too much memory and was
killed, but the build was progressing smoothly up until that point.

## main/libinput

This package failed its test due to missing /dev/fd, presumably
because I wasn't running udev. I created a symlink /dev/fd ->
../proc/self/fd, and the tests succeeded.

## community/libhandy, testing/libzim

These packages failed their tests (logs at [6][7]). I don't believe
this is related to samurai.

## community/gom

The tests hang for me. I don't believe this is related to samurai.

Note that any instance of missing dependencies can also affect ninja
depending on the maximum number of jobs and how long they take to run.

-Michael

[0] https://github.com/alpinelinux/aports/compare/master...michaelforney:samurai
[1] https://chromium-review.googlesource.com/c/chromium/src/+/1975453
[2] https://github.com/fwupd/fwupd/commit/9bdb96f768f1318ccaac4ecbcac68200a36ee1be
[3] https://lists.alpinelinux.org/~alpine/aports/patches/3179
[4] https://gitlab.freedesktop.org/wayland/weston/commit/ba3b384b930f9ab40f94d125d4d26bc4b636821f
[5] https://github.com/francma/wob/pull/23
[6] http://ix.io/24Qr
[7] http://ix.io/24Qq
[8] https://codereview.qt-project.org/c/qt/qtwebengine/+/285134

---

From: Rasmus Thomsen <[email-masked]>
Date: Fri, 20 Dec 2019 07:34:45 +0000

> this is related to samurai.

At least libhandy is failing due to you not running dbus, so this isn't
on samurai.

From my point of view using Samu should be fine then, thank you for
your work!

Regards, Rasmus