Subject: Native Alpine GLibc support (NEW)

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 09:32:44 +0200

Trying to send email again from my personal email as my previous post
was marked as invalid.

Hi,

I know that this topic is very sensitive in Alpine, but please let me
describe first what and why we would like to achieve.

1. We love to use Alpine for our containers. Alpine has very fresh
versions packages, it is very easy to use, versatile, customizable and
clean. Basically only superlatives.
2. Our applications are mostly Java. But many of our jars are JNIs and
are compiled against glibc.
3. We were using Alpine with Glibc hacks like
https://github.com/sgerrand/alpine-pkg-glibc It was working nice until
we ran into a problem where some other library was trying to load
other native library and it started crashing.
4. We have switched to native MUSL based everywhere (Java, libs,
etc.). I was happy that it was possible, but I was surprised that the
performance of our applications degraded a lot. Our application is
heavily multithreaded. After long investigation I have found that
degradation is because of poor implementation of pthread locks in
muslc. Probably the advantage of glibc  is the adaptive locking
feature and maybe other types of tunings. In our example application
spends with muslc build 600% CPU  in kernel and 200% in user space,
but on glibc 100% CPU in kernel and 700% in user space (where 100%
means 1 CPU core). Yes it is that huge difference.

At this point I had many options of what to do and one I would like to describe.

- I started thinking that I would like to have a variant of Alpine
with glibc. Everything else is the same and great as usual.
- After the initial painful bootstrap to get bare minimum to be able
to compile I got the first 300 packages built (enough for us) with
only a few tweaks in some packages that are explicitly applying MUSL
patches. I simply disabled these.
- Now our application is running fast again with native builds and no tweaks.
- I know glibc is maybe a little bit, but we simply need performance
and compatibility.

Example of change in procps. There is a tweak to make it work in MUSL
so I simply conditionally disable it:
if [ "$CLIBC" != "glibc" ]; then
export LIBS="$LIBS -lintl"
fi

I would love to have the patches part of the main branch so I do not
need to do merges from time to time. I can only dream that the Alpine
community will provide the official build as well. I think that more
people would like that when I look at a number of projects around
hacking glibc in Alpine.

Please do not take it as a heretical thought. It came as a need.

What do you think about the project? I can share my changes in some
branch or in github. I can also share initial 3.16 binaries to have an
easier bootstrap. It was really a pain.

Thank you very much for your project and the hard work you put into it.
Tomas

---

From: Natanael Copa <[email-masked]>
Date: Tue, 21 Jun 2022 09:56:53 +0200

On Tue, 21 Jun 2022 09:32:44 +0200

> Trying to send email again from my personal email as my previous post
hi!

...
> means 1 CPU core). Yes it is that huge difference.

Do you have any smallish testcases where we can test this?

> with glibc. Everything else is the same and great as usual.

Would it be an idea to try fix the performance problems in musl libc
instead? It would probably be significantly less maintenance work long
term.

> export LIBS="$LIBS -lintl"

I believe someone did this some years ago, but stopped due to lack of
resources.

And this might also be of interest:
https://twitter.com/ariadneconill/status/1527642520167108609

> easier bootstrap. It was really a pain.

We are already struggling with resources, full disks, many incoming
issues, feature requests, technical debt (lack of tests and/or
documentation) etc. I am personally feeling I'm almost able to keep my
head above the surface.

So if this project can help solve our current issues, and contribute
with more resources for the project as a whole, then sure. But if this
is going to add more work for us long term, or even worse, someone do
the initial work and then dump the maintenance work on our laps after
there are a significant amount of users depending on it, then please no
thanks.

I would rather spend the resources on fixing the issues with musl.

> Thank you very much for your project and the hard work you put into
> it. Tomas

---

From: Dominique Martinet <[email-masked]>
Date: Tue, 21 Jun 2022 17:42:46 +0900

> https://twitter.com/ariadneconill/status/1527642520167108609

I assume this tweet refers to these two, if anyone else is curious and
wants to save 5 minutes searching:

https://github.com/chainguard-dev/melange
and
https://github.com/chainguard-dev/apko

(With that said, I agree improving musl might be better, but.. any
reason to stick to alpine for this particular usecase? I don't know what
you need to rebuild your JNI dependencies but it might be even less work
to slim down another distro's container)

--
Asmadeus | Dominique Martinet

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 10:54:26 +0200

> to slim down another distro's container)
We are using Redhat UBI minimal 8 as well. But there is so many
packages with many vulnerabilities so security scans are showing many
issues. Even if it is sometimes false positives it still does not look
good to customers.

I tried many distros, but Alpine install to different root (-p
/newroot) allows me to have just a tiny bit of packages (only what I
need) with zero issues.

So yes we can do that. I just like the simplicity that is already available.

---

From: Olliver Schinagl <[email-masked]>
Date: Tue, 21 Jun 2022 20:56:44 +0200

Hey Tomas,

> So yes we can do that. I just like the simplicity that is already available.

I have also thought of a glibc-based alpine (like). For most things, 
muslc is great; for some things, glibc is needed/desired. So why stick 
with alpine?

TBH, because alpine is really amazing. I'm migrated from 
gentoo/ubuntu/debian to Arch Linux on my desktop/laptop; and while Arch 
shares quite a few similarities with Alpine, yet, strangely enough 
Alpine is still far superior. How?

Well for one, the team is amazing.

But technically also, security updates are often quickly addressed, 
plenty of package, ease of packaging. There's just so many things Alpine 
does 'better' then all of the distro's it has learned from. So if not 
alpine, what distro then? That's a good question, one I haven't been 
able to answer.

Void seems to kind of offer however what we'd be after, musl OR glibc at 
least. But void is even more understaffed, slower, 'smaller', so 
stepping that direction is also not helpful, and lacks the alpine 
awesomeness :)

Just some thoughts ...

Olliver

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 10:49:07 +0200

> Do you have any smallish testcases where we can test this?
I do not have a small test. Synthetic applications does not give the
best results. I am sure it is the issue as the only change is glibc
vs. musl and kernel profiler point to pthread locking. There is a spin
lock that wastes the CPU time in system space. Glibc have some
heuristics to not get into the kernel that often (adaptive locking)
which as a result brigs a lot of speed. JDK has no other dependency
than C library.

Anyway I will try to prepare some Java app that can show this.
> instead? It would probably be significantly less maintenance work long
Yes this was one of my options. I was looking in glibc pthread
implementation and it looks to me that they save some statistical
features for adaptive locking in the structure. That as a result would
require change of size of mutex object and that would require
recompilation of all binaries. That is I think not acceptable.
Performance tuning these low level functions is more like a magic and
very sensitive (what helps one workload can break other). I am not
sure how long glibc was tuning that so yes it looks like the best
solution, but I am not sure if it is really doable under one year or
so. We can ask MUSL community after I would be able to create a test.
> I believe someone did this some years ago, but stopped due to lack of
I definitely do not want to create new distro. It is really only about
conditionally disabling some line here and there. I think it is like
around 50 lines of changes in 10 packages to be able to compile 300
packages so far. I was actually surprised how easy it is now after I
have compiled build-base, abuild and apk-tools, because now I can
reuse it to build it itself again easily (new versions). I build in
docker.
> I would rather spend the resources on fixing the issues with musl.
I completely agree and understand your view. Unfortunately I am not in
a position to offer or fund the resources. Yes it can bring users and
yes it can generate additional issues.

My intention really was to have a pure Alpine built against glibc. It
is a C library as a MUSL. Everything else is exactly same. I remember
how much we were fighting with all variants of dependencies that are
coming as glibc based, because many users are still on
CentOS/Redhat/Debian/Ubuntu/etc. Glibc is mostly the blocker. That is
also why I was not going to try to improve the MUSL as at the end I
would hit the wall, because of another glibc based component.

Anyway if there is some interest just let me know.

Thanks,
Tomas

---

From: Natanael Copa <[email-masked]>
Date: Tue, 21 Jun 2022 11:45:59 +0200

On Tue, 21 Jun 2022 10:49:07 +0200
Tomas Kolda <[email-masked]> wrote:

> a position to offer or fund the resources. 

This is what I was afraid of.

> Yes it can bring users and yes it can generate additional issues.

So more work, but not not more resources.

> is a C library as a MUSL. Everything else is exactly same.

I actually don't believe this, even if that is how it would start.

Adding glibc support will open new possibilities, like adding support
for systemd (see
https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33329).

I believe this would trigger big discussions and possible conflicts.
After months of conflicts and 3-5 (or more) developers leaving the
project systemd will be merged. It would be kind of stupid to not do
so.

Now we will have to maintain both openrc and systemd services. systemd
will add its own set of new dependencies (together with many awesome
features of course). The initramfs scripts will need work to handle
this, the installer script will need to be adapted to this.

We would now effectively be supporting two different OSes in one
project and at some point we will realize that we simply don't have
resources for supporting both. The most realistic outcome would be that
we'd need to drop openrc/busybox and musl libc support. How many other
distros that have added systemd support has *not* ended up with it as
the only supported init?

At this point there would not really be any big difference from
CentOS/Redhat/Debian/Ubuntu/etc.

So why not just use CentOS/Redhat/Debian/Ubuntu in the first place? I
believe there is where we'd end up anyway eventually.

> I remember how much we were fighting with all variants of
> dependencies that are coming as glibc based, because many users are
> still on CentOS/Redhat/Debian/Ubuntu/etc. Glibc is mostly the
> blocker. That is also why I was not going to try to improve the MUSL
> as at the end I would hit the wall, because of another glibc based
> component.
>

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 12:20:05 +0200

> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33329).
Adding more options what becomes possible does not mean that it needs
to be done. I tried to say that when you look on glibc as another
"libc" then there does not need to be any change. It is just different
build. Every component must work with glibc as it is the very first
one people support. I do not want to argue about where Alpine should
go. It is your decisions and obviously good ones as I like Alpine. If
it helps with systemd, it is good. Does it push anyone for systemd?
No.
> believe there is where we'd end up anyway eventually.
- Alpine serves very fresh packages. Other distros go too slow.
- Alpine has very little vulnerabilities reported compared to other distros.
- Alpine is so easy to customize compared to other distros.
- Alpine is perfect for docker use case (mine)

I know that I defend my use case so I understand it is just one vote.
Just explaining why I do not like to use saint four.

I did not pick Alpine, because of MUSL. I am not sure it brings that
much for non embedded use cases. I have picked Alpine, because of its
packaging and simplicity to customize it. It can serve easily any use
case. It is not important if C library is couple of megabytes smaller.

---

From: Thomas Lamprecht <[email-masked]>
Date: Tue, 21 Jun 2022 14:00:06 +0200

> - Alpine is perfect for docker use case (mine)

Basically all of above is also true for Arch Linux which supports glibc, so
why not use that? I'm using it in a hosted VM for some personal important
services since over six years, without any issue whatsoever and quite good
experience - packaging is also simple and straight forward, so not hard to
re-package one's application for Arch Linux.

FWIW, if your working at a company that would profit from such a thing why
not try to implement glibc integration and build CI yourself, you wouldn't
even need to cover the whole archive available in Alpine Linux, just a java
build and the external library dependencies of your app. Basically doing the
approach of Nico's (a) proposal yourself.

That would serve as good proof of concept and give a much better idea about
practicability and maintenance effort required, it also could proof that
this change alone was really responsible for the performance hit your app
took and you wouldn't need to wait for any such hypothetical change being
released in upstream Alpine Linux.

IMO, and that really is an outside view, as while I'm dabbling with Alpine
Linux and its inner workings on and off since years I'm really not active in
the community in any form, it feels rather misguided to add glibc into
Alpine Linux itself in general as complete musl alternative, just feels like
going against core characteristics and the reasoning for doing such really
doesn't outweigh Natanael's arguments at all.

cheers,
Thomas

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 14:51:01 +0200

> re-package one's application for Arch Linux.
Maybe ... :)
> approach of Nico's (a) proposal yourself.
Yes that is what we have and what we use already. We are building it
based on incoming changes. I am just merging it with my changes. Any
difference to community is not good so I am just trying to have it
official. Maybe more people needs something like this. So it is not
just "idea". It is working project.
> released in upstream Alpine Linux.
- This introduce some changes in some of the ports (mostly only
disable MUSL tweaks) - so far like 5% of packages require change
- This introduce one more build as it behaves like new architecture
- I agree that having build of all is hard to estimate. But after
initial bootstrap (nightmare at least for me) I was able to "fix" that
300 packages in two days.

---

From: Jakub Panek <[email-masked]>
Date: Tue, 21 Jun 2022 13:05:32 +0000

> doesn't outweigh Natanael's arguments at all.

As a person that dabbles with Alpine and is somewhat active in community,
I agree that glibc doesn't feel right with Alpine. It's not a component
that you can simply replace in ecosystem as everything has to be rebuilt
against it. IMO, libc and package manager are base components which make
OS distinct so calling it Alpine with glibc would be a disservice as
Alpine focuses on size, simplicity and security, which for glibc I
would say neither are a focus.

> easier bootstrap. It was really a pain.

I would be happy to contribute towards apk-tools + glibc OS (option b))
and it's ports as much as I can but would really recommend for it
to define itself as separate project from Alpine (e.g. different name).
I could also provide some compute and storage for it.

Regards,
Jakub P.

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 15:39:21 +0200

> would say neither are a focus.
So the Alpine can be first distro that supports two libc libraries! ;-)
> I could also provide some compute and storage for it.
As Natanael said it would increase the resource needs (people) even
more. I think... Changing one library does not need to generate
another distro. Maybe I should share it somehow so you can try it in
chroot or docker to see there is really actually almost no change
which I like most.

Maybe Jakube we can setup a repository of packages somewhere on your
web server? I can copy it there. Then it is easy to start to play with
that as you have build toolchain ready.

---

From: "alice" <[email-masked]>
Date: Tue, 21 Jun 2022 15:42:14 +0200

> So the Alpine can be first distro that supports two libc libraries! ;-)
void linux already does, unless you specifically mean multilib'd at the
same time.

and gentoo, of course :)

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 16:05:01 +0200

> void linux already does, unless you specifically mean multilib'd at the
Nice I did not know that. From what I see there they are also not
combining. Both are having it as separate flavor builds (one or the
other).

---

From: "Laurent Bercot" <[email-masked]>
Date: Tue, 21 Jun 2022 15:50:45 +0000

I find it interesting that when thinking about such things (making
a distro supporting two libcs, or supporting two init systems, or
patching some software to work with the better-known alternative, or...)
the frame of reference is always "adding the support of $biggerthing
to a project that has $smallthing as a characteristic", and never the
opposite.

  It always ends up meaning more burden for the project that typically
has fewer resources, less manpower, and that is only kept afloat by
the superhuman good will and effort of its maintainers.

  Why does nobody ever say "ohey, let's add musl support to Fedora!" ?

  No, it has to be Alpine, whose defining characteristic is to be a
musl-based distribution, which is the main reason why it's small and
efficient and has the success it has, that needs to grow glibc support,
i.e. diluting its meager resources away from its central vision and
into something *every other distro under the sun* already does, which
means decreasing its value in the software ecosystem.

  I see this pattern happening over, and over, and over.
  I don't get it.
  That's a lie. I kinda get it, but I really don't like it.

  I would prefer more energy to be spent in understanding why the
musl pthread thing is so inefficient for your use case, and having a
discussion in musl about how to improve it, and whether the use case
is common enough to be worth trade-offs, if necessary.
  I would prefer technical issues to be solved rather than translated
into organizational burden and vision dilution put onto projects.

--
  Laurent

---

From: Jakub Panek <[email-masked]>
Date: Tue, 21 Jun 2022 15:54:35 +0000

> No, it has to be Alpine

https://github.com/NixOS/nixpkgs/issues/90147 ?

R, Jakub. P

---

From: Ariadne Conill <[email-masked]>
Date: Tue, 21 Jun 2022 11:07:44 -0500 (CDT)

Hi,

> is common enough to be worth trade-offs, if necessary.

+1.  The solution is to add support for lock elision to musl, as glibc 
have done.  Or to use a newer version of the JDK which adds lock elision 
at the language level.

But the problem with lock elision in libc is that it is very difficult to 
prove that an implementation is correct.  When it is implemented at the 
language level, it is easy to prove.  This is why it has not happened in 
musl yet, in my opinion.

Ariadne

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 14:12:26 +0200

> means decreasing its value in the software ecosystem.
You are right. Alpine can definitely pick their way. I do not want to
create yet another distro. I was asking for cooperation that was
outlined by Oliver as well. And maybe after some time Alpine will see
market for this change. There is a good reason main distros are using
glibc. I am trying to be pragmatic and not just say MUSL is great.
There are so many variables in the world. And maybe that is why we see
this type of request time to time.
> into organizational burden and vision dilution put onto projects.
Yes I would like that as well. This would potentially solve this one
particular issue (time frame is questionable). It does not solve the
others.

---

From: "alice" <[email-masked]>
Date: Wed, 22 Jun 2022 18:57:12 +0200

On Wed Jun 22, 2022 at 2:12 PM CEST, Tomas Kolda wrote:
> outlined by Oliver as well.
you don't want to make another distro incorporating $hugesetofchanges,
so you just want others to work on it for free incorporating them
instead?
> And maybe after some time Alpine will see
> market for this change.
last i checked nobody working on alpine was working for 'a market' to
make money out of (with alpine)- personally i do it for free, but maybe
we are all supposed to be getting paid? certainly, if someone was
throwing free money my way maybe this would be an actual interesting
project to work towards.
> There is a good reason main distros are using
then use a glibc distro? i don't get it, again, as Laurent has said,
nothing whatsoever is forcing you to use musl or alpine.

i understand you believe you are not asking for much, just a 'small' set
of compatibility patches to easily allow someone else to 'build alpine
but with glibc'.. but if they are as easy as you have demonstrated and
claim, then.. you're done? you successfully made all the changes, and
can now build your forked+patched aports whenever you wish with glibc,
and can keep it that way. sounds like you already found the solution,
and can keep working on it there, as your own project. sadly though,
this is indeed 'an entire other distro'.. though you are basically
already done making it.

> particular issue (time frame is questionable). It does not solve the
a very easy solution (for all the others): use literally any other
glibc-based distribution.

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 19:06:17 +0200

> so you just want others to work on it for free incorporating them

I wanted to help other people that would like to have glibc in Alpine.
If there is none then it is the answer as well. That is why I ask
here.

> throwing free money my way maybe this would be an actual interesting

I am not sure I understand. I also send patches to open source in my
time. We are doing it for fun. And it is more fun when there are more
users.

> nothing whatsoever is forcing you to use musl or alpine.

I know and I mentioned. Please do not take it offensive. I am just
asking if Alpine users would like it.

> this is indeed 'an entire other distro'.. though you are basically

I am not sure if I was rude. I hope I was not. Yes I can do all of
that. But there is nothing wrong with trying to find better options.
If that options are not viable I understand it.

> glibc-based distribution.

Yes it is definitely one of the options.

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 19:27:05 +0200

Thank you all for your patience with giving me the feedback. Based on
the answers of biggest contributors I do not see this attempt as the
one that will be accepted. I was actually expecting that, but you know
who does not try will not achieve anything :-)

Anyway thank you for this great distro that I like very much even
without glibc. Your build system rocks (apk, abuild....).

I wil keep the github for a while. I also posted binaries to Jakub in
case someone will want to take a look.

Thanks again!
Tomas

---

From: Ariadne Conill <[email-masked]>
Date: Tue, 21 Jun 2022 13:42:50 -0500 (CDT)

Hi,

> combining. Both are having it as separate flavor builds (one or the

Combining two runtime environments without appropriate namespacing (as 
Debian has done for example) results in a lot of stability problems.

In general, I don't think this idea is very good.  I would avoid it, and 
instead use the appropriate runtime environment for the workload. 
Sometimes, that is glibc, other times it is musl.

Ariadne

---

From: Jakub Panek <[email-masked]>
Date: Tue, 21 Jun 2022 15:49:23 +0000

> út 21. 6. 2022 v 15:05 odesílatel Jakub Panek [email-masked] napsal:
>

Void Linux and Gentoo as mentioned by alice have done that before so
not really "first" :)
As for simultaneous multilibc on host it would be a terrible idea
as you have to monitor/maintain both libcs at the same time with (IMO)
very little benefit.

> that as you have build toolchain ready.

Sure, if you can provide a public endpoint (HTTP/SFTP/FTP(S)/RSYNC/other)
I can grab and host right it now. If not possible, it will take me
some time to set up it for multi-user purpose so you can throw it there
yourself :)

Whichever the way, you can send me an email directly (as to not clutter
mailing list even more) but it would be good to see the modified
ports source available publicly so the bootstrap could be verified as
well. I'm fine with GitHub, GitLab or wherever it would be hosted with
preference for something that could be easily integrated for
collaboration and CI/CD.

Regards,
Jakub P.

---

From: Olliver Schinagl <[email-masked]>
Date: Tue, 21 Jun 2022 21:09:04 +0200

> chroot or docker to see there is really actually almost no change

That is often the problem anyway, resources, be it $work or foss. The 
sad thing about open source in general (which I know, is also its 
strength), is the duplication of effort. Oh I don't like how you do XYZ, 
lets fork/re-write. Just look at the number of distro's/applications in 
C or C++ flavor, KDE or Gnome flavor. It kind of is sad that we waste so 
much resources here. Sometimes, a new gem is born of course.

On topic, it would have been nice, to add glibc support ot the APKBUILD 
files, keep alpine an musl project, but share the package base with a 
second 'project'.

The previous effort of glibc alpine failed, because it was a one man 
show, lack of resources. Getting something of the ground would be great; 
but then we'd have something that Void tries to do; but struggling even 
more for resources, though sharing the package base, kind of like we can 
use the same package for both x86 and s390x.

It does make me wonder, what is the 'raison d'etre' of void? Why does 
the world need a 'pretty similar distro' and can't we all work together 
on a common/better distro together :) One can dream ...

Olliver

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 13:49:00 +0200

I completely agree with all you said Oliver. Very well written. It
should be set in stone.
> files, keep alpine an musl project, but share the package base with a
Yes please, this would be great. Maybe after some time people will see
it just works it can be reconsidered to have it as another platform.
This will simplify merges a lot.

Anyway, I uploaded sources to github. Here is the PR of all my changes
so far to be able to build that 300 packages (build stack + good base
we need to build our dockers).
- https://github.com/koldat/alpine/pull/1
- It is based on 3.16 branch
- I will work with Jakub to upload first compiled binaries so when you
are interested you can just "apk add" it to some root.
- When base is installed abuild works as usual for most packages.
- I have docker builder as well so one can easily compile/test musl vs
glibc without any external dependencies (ok only docker is needed).

---

From: Ariadne Conill <[email-masked]>
Date: Tue, 21 Jun 2022 13:41:20 -0500 (CDT)

Hi,

> believe there is where we'd end up anyway eventually.

Most likely he is looking for a "componentized" Linux distribution. 
Alpine, for example, generally avoids pre-declared dependencies in favor 
of automatically discovered ones.  This allows the packages to be treated 
as distinct components, and is one of the key reasons why Alpine 
containers are small.

For cases like these, the most likely alternative at the moment is probably 
Microsoft's CBL-Mariner.  My team at work is building a glibc-based 
companion distribution to Alpine, but it is not in a state where I can 
recommend it yet.

Ariadne

---

From: Nico Schottelius <[email-masked]>
Date: Tue, 21 Jun 2022 13:00:20 +0200

Hey Tomas,

I have to say I find this an interesting idea, because at ungleich we
also sometimes have the need to run glibc based apps, which at the
moment, we simply run inside containers.

I see multiple paths here and before even diving deeper into the topic,
I would like to understand which one you are suggesting:

- a) Adding glibc support as an add-on/additionally to musl
     While some things need to be clarified (which is libc.so...), most
     things could stay the same and there would simply be an additional
     libc installed and available.

     Using this approach there might be *some* packages which are
     available as -glibc variants (like a JVM), but most of Alpine would
     stay as-is.

- b) Replacing musl with glibc
     This would be more the scenario of a "fork" or "glibc version" of
     Alpine. All packages would need to be compiled, similar to adding a
     new architecture, but potentially doubling the builds.

     This might be more coherent and easier to understand and
     potentially less confusing, if you only have software linked
     against *one* libc.

Both variants come with their disadvantages, I personally would have a
small a preference for (a), keeping Alpine simple and lean as is, but
adding glibc support in case it makes sense.

Best regards,

Nico

--
Sustainable and modern Infrastructures by ungleich.ch

---

From: Tomas Kolda <[email-masked]>
Date: Tue, 21 Jun 2022 14:20:32 +0200

>      available as -glibc variants (like a JVM), but most of Alpine would
I am not sure it is possible to have it in a stable way. There are
more projects trying to achieve it. Some bring libraries from other
distros (sgerrand), some does compatibility layer (gcompat). In both
cases it is not 100% same so all applications does not work. First
does not work when you dynamically want to use other library from OS
that is musl. The other does not work when it relies on exact
functionality as it is only proxy. Some functionality also relies on
"headers" as well that affects it during compile time. That is why I
tried (b).
>      potentially less confusing, if you only have software linked
Yes that is what I propose. To have a new "architecture" build that
will simply have everything with glibc enabled. That is what we do. We
have Alpine build with CLIBC=glibc. We have around 300 packages (what
we need) built with a full set of build tools (build-base + abuild +
apk-tools + git).
> adding glibc support in case it makes sense.
I would prefer (a) as well in case it can work for 100% cases. I do
not know how to achieve it. Any try to mix these libraries end up with
a crash.

---

From: Ariadne Conill <[email-masked]>
Date: Tue, 21 Jun 2022 11:02:41 -0500 (CDT)

Hi,

> other native library and it started crashing.

Yes, I have written about this before[0].  Mixing glibc and musl
environments will lead to instability -- this can be overcome with proper 
multi-arch support, but we have not implemented this in Alpine yet (and 
apk-tools would need some minor changes to the solver to become 
multi-arch aware).

> muslc.

Locks in musl are done with atomics.  The advantage is probably due to the 
fact that glibc has lock elision, which musl does not yet have. 
Investigating the addition of lock elision to musl is on my list of things 
to do eventually.

Which JVM are you using?  OpenJDK 17 and later have their own internal 
implementation of lock elision for Java/JNI locks, which should improve 
performance.

> Probably the advantage of glibc  is the adaptive locking

These statistics do not actually mean anything useful.  The increased 
kernel wait load is simply due to futex contention, which is already known 
due to the lack of lock elision support in musl.

> people would like that when I look at a number of projects around

If you believe Alpine should offer a glibc build, you should propose it as 
a system change proposal to the Alpine TSC.  This is because you will 
either need to:

- change abuild to make variation-specific changes easier to do (e.g. 
restricting musl/glibc-specific patches to their own variants)

- customize hundreds of APKBUILDs which will require the cooperation of 
package maintainers (who may be skeptical)

Speaking in personal capacity, I do not think the Alpine TSC will approve 
this proposal because the project is already overwhelmed with work to do.

Ariadne

[0]: https://ariadne.space/2021/08/26/there-is-no-such-thing-as-a-glibc-based-alpine-image/

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 14:04:12 +0200

> apk-tools would need some minor changes to the solver to become
I know. That is why I want to have glibc build. Many JNI dependencies
in Java is glibc based for example. It is really hard to keep up.
> implementation of lock elision for Java/JNI locks, which should improve
I tried JDK 8, 11, 17. Latest versions. No difference. Yes it is in
futex as you said. From user perspective it matters, because it is 5
times slower. As I said I do not believe MUSL can dramatically change
pthread implementation as it is most sensitive part and very hard to
test. It is very long run.

> package maintainers (who may be skeptical)
I do not agree with this. Most changes in that 300 set was in the
basic packages that was requiring MUSL tweaks. There will be less
tweaks after we will go further as more complex applications are using
libraries that are solved already. Anyway even from statistical
perspective it is 17 changes for 300 packages -> 5-6%. I am not sure
how many packages are in Alpine, but we can extrapolate.

I also do not agree that someone would just for one "if" be pushing
change out. There are many patches for all other reasons. I am
positive that people tend to help others. And this change does not
generate any work for them if they will not need to support it. They
will just try to keep it in working way.
> this proposal because the project is already overwhelmed with work to do.
Agree. I think that for start it would be great to at least allow "if"
patches to be able to merge it. It will be noop for Alpine.

See example changes here:
https://github.com/koldat/alpine/pull/1

---

From: "alice" <[email-masked]>
Date: Wed, 22 Jun 2022 18:39:30 +0200

---

From: "alice" <[email-masked]>
Date: Wed, 22 Jun 2022 18:44:45 +0200

> patches to be able to merge it. It will be noop for Alpine.
the example patches you have posted on github are very far from no-ops
at all, they require constant vigilance on how to maintain the
musl-specific patches to be separate and still work. i don't think
anyone wants to add musl specific patches as musl && msg && patch -p1,
and they were all also removed from checksums.

if they were part of aports, then it becomes the responsibility of the
people that work on aports to ensure that everything still works. but
since aports itself does not feed an actual glibc apk system.. that
doesn't really make sense. they only make sense as an entire separate
project where you either keep specific patches to apply to get aports to
build, or as a fork of aports (named something else) where you rebase
your changes on top every once in a while.

is it a lot of work? sure. it's not any more work than making them in
the first place, though, and it gives zero additional work to anyone
that maintains aports that they didn't ask for.

> https://github.com/koldat/alpine/pull/1

apologies- i accidentally sent an empty reply before.

---

From: Tomas Kolda <[email-masked]>
Date: Wed, 22 Jun 2022 18:57:35 +0200

> and they were all also removed from checksums.
Main reason is that abuild does not allow to conditionally exclude
patch as it requires it (build fails as checksum exists for something
not used). So I used this approach (current functionality). Otherwise
a build script would need a tweak like:

sources_exclude=.....
or even without condition
sources_exclude_glibc=.....

Ofcourse there are many ways of having it nice and current POC is not
final. I first tested if it is possible (it is). Then I tried to ask
if community even would want to do something like that (in
anyacceptable  way).