Date: Tue, 21 Jun 2022 11:02:41 -0500 (CDT)
From: Ariadne Conill <ariadne@dereferenced.org>
To: Tomas Kolda <koldat@gmail.com>
cc: ~alpine/devel@lists.alpinelinux.org
Subject: Re: Native Alpine GLibc support (NEW)
In-Reply-To: <CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com>
Message-ID: <a3367d7-d3c5-4755-289e-37507033405f@dereferenced.org>
References: <CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; format=flowed; charset=US-ASCII

Hi,

On Tue, 21 Jun 2022, Tomas Kolda wrote:

> Trying to send email again from my personal email as my previous post
> was marked as invalid.
>
> Hi,
>
> I know that this topic is very sensitive in Alpine, but please let me
> describe first what and why we would like to achieve.
>
> 1. We love to use Alpine for our containers. Alpine has very fresh
> versions packages, it is very easy to use, versatile, customizable and
> clean. Basically only superlatives.
> 2. Our applications are mostly Java. But many of our jars are JNIs and
> are compiled against glibc.
> 3. We were using Alpine with Glibc hacks like
> https://github.com/sgerrand/alpine-pkg-glibc It was working nice until
> we ran into a problem where some other library was trying to load
> other native library and it started crashing.

Yes, I have written about this before[0].  Mixing glibc and musl
environments will lead to instability -- this can be overcome with proper 
multi-arch support, but we have not implemented this in Alpine yet (and 
apk-tools would need some minor changes to the solver to become 
multi-arch aware).

> 4. We have switched to native MUSL based everywhere (Java, libs,
> etc.). I was happy that it was possible, but I was surprised that the
> performance of our applications degraded a lot. Our application is
> heavily multithreaded. After long investigation I have found that
> degradation is because of poor implementation of pthread locks in
> muslc.

Locks in musl are done with atomics.  The advantage is probably due to the 
fact that glibc has lock elision, which musl does not yet have. 
Investigating the addition of lock elision to musl is on my list of things 
to do eventually.

Which JVM are you using?  OpenJDK 17 and later have their own internal 
implementation of lock elision for Java/JNI locks, which should improve 
performance.

> Probably the advantage of glibc  is the adaptive locking
> feature and maybe other types of tunings. In our example application
> spends with muslc build 600% CPU  in kernel and 200% in user space,
> but on glibc 100% CPU in kernel and 700% in user space (where 100%
> means 1 CPU core). Yes it is that huge difference.

These statistics do not actually mean anything useful.  The increased 
kernel wait load is simply due to futex contention, which is already known 
due to the lack of lock elision support in musl.

> At this point I had many options of what to do and one I would like to describe.
>
> - I started thinking that I would like to have a variant of Alpine
> with glibc. Everything else is the same and great as usual.
> - After the initial painful bootstrap to get bare minimum to be able
> to compile I got the first 300 packages built (enough for us) with
> only a few tweaks in some packages that are explicitly applying MUSL
> patches. I simply disabled these.
> - Now our application is running fast again with native builds and no tweaks.
> - I know glibc is maybe a little bit, but we simply need performance
> and compatibility.
>
> Example of change in procps. There is a tweak to make it work in MUSL
> so I simply conditionally disable it:
> if [ "$CLIBC" != "glibc" ]; then
> export LIBS="$LIBS -lintl"
> fi
>
> I would love to have the patches part of the main branch so I do not
> need to do merges from time to time. I can only dream that the Alpine
> community will provide the official build as well. I think that more
> people would like that when I look at a number of projects around
> hacking glibc in Alpine.

If you believe Alpine should offer a glibc build, you should propose it as 
a system change proposal to the Alpine TSC.  This is because you will 
either need to:

- change abuild to make variation-specific changes easier to do (e.g. 
restricting musl/glibc-specific patches to their own variants)

- customize hundreds of APKBUILDs which will require the cooperation of 
package maintainers (who may be skeptical)

Speaking in personal capacity, I do not think the Alpine TSC will approve 
this proposal because the project is already overwhelmed with work to do.

Ariadne

[0]: https://ariadne.space/2021/08/26/there-is-no-such-thing-as-a-glibc-based-alpine-image/