~alpine/devel

32 11

Native Alpine GLibc support (NEW)

Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com>
DKIM signature
missing
Download raw message
Trying to send email again from my personal email as my previous post
was marked as invalid.

Hi,

I know that this topic is very sensitive in Alpine, but please let me
describe first what and why we would like to achieve.

1. We love to use Alpine for our containers. Alpine has very fresh
versions packages, it is very easy to use, versatile, customizable and
clean. Basically only superlatives.
2. Our applications are mostly Java. But many of our jars are JNIs and
are compiled against glibc.
3. We were using Alpine with Glibc hacks like
https://github.com/sgerrand/alpine-pkg-glibc It was working nice until
we ran into a problem where some other library was trying to load
other native library and it started crashing.
4. We have switched to native MUSL based everywhere (Java, libs,
etc.). I was happy that it was possible, but I was surprised that the
performance of our applications degraded a lot. Our application is
heavily multithreaded. After long investigation I have found that
degradation is because of poor implementation of pthread locks in
muslc. Probably the advantage of glibc  is the adaptive locking
feature and maybe other types of tunings. In our example application
spends with muslc build 600% CPU  in kernel and 200% in user space,
but on glibc 100% CPU in kernel and 700% in user space (where 100%
means 1 CPU core). Yes it is that huge difference.

At this point I had many options of what to do and one I would like to describe.

- I started thinking that I would like to have a variant of Alpine
with glibc. Everything else is the same and great as usual.
- After the initial painful bootstrap to get bare minimum to be able
to compile I got the first 300 packages built (enough for us) with
only a few tweaks in some packages that are explicitly applying MUSL
patches. I simply disabled these.
- Now our application is running fast again with native builds and no tweaks.
- I know glibc is maybe a little bit, but we simply need performance
and compatibility.

Example of change in procps. There is a tweak to make it work in MUSL
so I simply conditionally disable it:
if [ "$CLIBC" != "glibc" ]; then
export LIBS="$LIBS -lintl"
fi

I would love to have the patches part of the main branch so I do not
need to do merges from time to time. I can only dream that the Alpine
community will provide the official build as well. I think that more
people would like that when I look at a number of projects around
hacking glibc in Alpine.

Please do not take it as a heretical thought. It came as a need.

What do you think about the project? I can share my changes in some
branch or in github. I can also share initial 3.16 binaries to have an
easier bootstrap. It was really a pain.

Thank you very much for your project and the hard work you put into it.
Tomas
Details
Message ID
<20220621095653.71773d59@ncopa-desktop.lan>
In-Reply-To
<CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Tue, 21 Jun 2022 09:32:44 +0200
Tomas Kolda <koldat@gmail.com> wrote:

> Trying to send email again from my personal email as my previous post
> was marked as invalid.
> 
> Hi,
> 
hi!

...
> 4. We have switched to native MUSL based everywhere (Java, libs,
> etc.). I was happy that it was possible, but I was surprised that the
> performance of our applications degraded a lot. Our application is
> heavily multithreaded. After long investigation I have found that
> degradation is because of poor implementation of pthread locks in
> muslc. Probably the advantage of glibc  is the adaptive locking
> feature and maybe other types of tunings. In our example application
> spends with muslc build 600% CPU  in kernel and 200% in user space,
> but on glibc 100% CPU in kernel and 700% in user space (where 100%
> means 1 CPU core). Yes it is that huge difference.

Do you have any smallish testcases where we can test this?

> At this point I had many options of what to do and one I would like to describe.
> 
> - I started thinking that I would like to have a variant of Alpine
> with glibc. Everything else is the same and great as usual.

Would it be an idea to try fix the performance problems in musl libc
instead? It would probably be significantly less maintenance work long
term.

> - After the initial painful bootstrap to get bare minimum to be able
> to compile I got the first 300 packages built (enough for us) with
> only a few tweaks in some packages that are explicitly applying MUSL
> patches. I simply disabled these.
> - Now our application is running fast again with native builds and no tweaks.
> - I know glibc is maybe a little bit, but we simply need performance
> and compatibility.
> 
> Example of change in procps. There is a tweak to make it work in MUSL
> so I simply conditionally disable it:
> if [ "$CLIBC" != "glibc" ]; then
> export LIBS="$LIBS -lintl"
> fi

I believe someone did this some years ago, but stopped due to lack of
resources.

And this might also be of interest:
https://twitter.com/ariadneconill/status/1527642520167108609

> I would love to have the patches part of the main branch so I do not
> need to do merges from time to time. I can only dream that the Alpine
> community will provide the official build as well. I think that more
> people would like that when I look at a number of projects around
> hacking glibc in Alpine.
> 
> Please do not take it as a heretical thought. It came as a need.
> 
> What do you think about the project? I can share my changes in some
> branch or in github. I can also share initial 3.16 binaries to have an
> easier bootstrap. It was really a pain.

We are already struggling with resources, full disks, many incoming
issues, feature requests, technical debt (lack of tests and/or
documentation) etc. I am personally feeling I'm almost able to keep my
head above the surface.

So if this project can help solve our current issues, and contribute
with more resources for the project as a whole, then sure. But if this
is going to add more work for us long term, or even worse, someone do
the initial work and then dump the maintenance work on our laps after
there are a significant amount of users depending on it, then please no
thanks.

I would rather spend the resources on fixing the issues with musl.

> Thank you very much for your project and the hard work you put into
> it. Tomas
Dominique Martinet <asmadeus@codewreck.org>
Details
Message ID
<YrGEhsJi+AJ+eUVx@codewreck.org>
In-Reply-To
<20220621095653.71773d59@ncopa-desktop.lan> (view parent)
DKIM signature
missing
Download raw message
Natanael Copa wrote on Tue, Jun 21, 2022 at 09:56:53AM +0200:
> And this might also be of interest:
> https://twitter.com/ariadneconill/status/1527642520167108609

I assume this tweet refers to these two, if anyone else is curious and
wants to save 5 minutes searching:

https://github.com/chainguard-dev/melange
and
https://github.com/chainguard-dev/apko



(With that said, I agree improving musl might be better, but.. any
reason to stick to alpine for this particular usecase? I don't know what
you need to rebuild your JNI dependencies but it might be even less work
to slim down another distro's container)

--
Asmadeus | Dominique Martinet
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmniJdjO54SHR+Js=5pyuB70A_M0Ge92=pFpxjbBHYC9w@mail.gmail.com>
In-Reply-To
<20220621095653.71773d59@ncopa-desktop.lan> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 9:57 odesílatel Natanael Copa <ncopa@alpinelinux.org> napsal:
>
> Do you have any smallish testcases where we can test this?
>
I do not have a small test. Synthetic applications does not give the
best results. I am sure it is the issue as the only change is glibc
vs. musl and kernel profiler point to pthread locking. There is a spin
lock that wastes the CPU time in system space. Glibc have some
heuristics to not get into the kernel that often (adaptive locking)
which as a result brigs a lot of speed. JDK has no other dependency
than C library.

Anyway I will try to prepare some Java app that can show this.
>
> Would it be an idea to try fix the performance problems in musl libc
> instead? It would probably be significantly less maintenance work long
> term.
>
Yes this was one of my options. I was looking in glibc pthread
implementation and it looks to me that they save some statistical
features for adaptive locking in the structure. That as a result would
require change of size of mutex object and that would require
recompilation of all binaries. That is I think not acceptable.
Performance tuning these low level functions is more like a magic and
very sensitive (what helps one workload can break other). I am not
sure how long glibc was tuning that so yes it looks like the best
solution, but I am not sure if it is really doable under one year or
so. We can ask MUSL community after I would be able to create a test.
>
> I believe someone did this some years ago, but stopped due to lack of
> resources.
>
I definitely do not want to create new distro. It is really only about
conditionally disabling some line here and there. I think it is like
around 50 lines of changes in 10 packages to be able to compile 300
packages so far. I was actually surprised how easy it is now after I
have compiled build-base, abuild and apk-tools, because now I can
reuse it to build it itself again easily (new versions). I build in
docker.
>
> We are already struggling with resources, full disks, many incoming
> issues, feature requests, technical debt (lack of tests and/or
> documentation) etc. I am personally feeling I'm almost able to keep my
> head above the surface.
>
> So if this project can help solve our current issues, and contribute
> with more resources for the project as a whole, then sure. But if this
> is going to add more work for us long term, or even worse, someone do
> the initial work and then dump the maintenance work on our laps after
> there are a significant amount of users depending on it, then please no
> thanks.
>
> I would rather spend the resources on fixing the issues with musl.
>
I completely agree and understand your view. Unfortunately I am not in
a position to offer or fund the resources. Yes it can bring users and
yes it can generate additional issues.

My intention really was to have a pure Alpine built against glibc. It
is a C library as a MUSL. Everything else is exactly same. I remember
how much we were fighting with all variants of dependencies that are
coming as glibc based, because many users are still on
CentOS/Redhat/Debian/Ubuntu/etc. Glibc is mostly the blocker. That is
also why I was not going to try to improve the MUSL as at the end I
would hit the wall, because of another glibc based component.

Anyway if there is some interest just let me know.

Thanks,
Tomas
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrM=RUOpE+NfBS0k-My-Sk81okHJHBF0cnO4Z9_VLzuyYeQ@mail.gmail.com>
In-Reply-To
<YrGEhsJi+AJ+eUVx@codewreck.org> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 10:43 odesílatel Dominique Martinet
<asmadeus@codewreck.org> napsal:
>
> (With that said, I agree improving musl might be better, but.. any
> reason to stick to alpine for this particular usecase? I don't know what
> you need to rebuild your JNI dependencies but it might be even less work
> to slim down another distro's container)
>
We are using Redhat UBI minimal 8 as well. But there is so many
packages with many vulnerabilities so security scans are showing many
issues. Even if it is sometimes false positives it still does not look
good to customers.

I tried many distros, but Alpine install to different root (-p
/newroot) allows me to have just a tiny bit of packages (only what I
need) with zero issues.

So yes we can do that. I just like the simplicity that is already available.
Details
Message ID
<20220621114559.4d38d98e@ncopa-desktop.lan>
In-Reply-To
<CAGFcrMmniJdjO54SHR+Js=5pyuB70A_M0Ge92=pFpxjbBHYC9w@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Tue, 21 Jun 2022 10:49:07 +0200
Tomas Kolda <koldat@gmail.com> wrote:


> > I would rather spend the resources on fixing the issues with musl.
> >  
> I completely agree and understand your view. Unfortunately I am not in
> a position to offer or fund the resources. 

This is what I was afraid of.

> Yes it can bring users and yes it can generate additional issues.

So more work, but not not more resources.

> My intention really was to have a pure Alpine built against glibc. It
> is a C library as a MUSL. Everything else is exactly same.

I actually don't believe this, even if that is how it would start.

Adding glibc support will open new possibilities, like adding support
for systemd (see
https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33329).

I believe this would trigger big discussions and possible conflicts.
After months of conflicts and 3-5 (or more) developers leaving the
project systemd will be merged. It would be kind of stupid to not do
so.

Now we will have to maintain both openrc and systemd services. systemd
will add its own set of new dependencies (together with many awesome
features of course). The initramfs scripts will need work to handle
this, the installer script will need to be adapted to this.

We would now effectively be supporting two different OSes in one
project and at some point we will realize that we simply don't have
resources for supporting both. The most realistic outcome would be that
we'd need to drop openrc/busybox and musl libc support. How many other
distros that have added systemd support has *not* ended up with it as
the only supported init?

At this point there would not really be any big difference from
CentOS/Redhat/Debian/Ubuntu/etc.

So why not just use CentOS/Redhat/Debian/Ubuntu in the first place? I
believe there is where we'd end up anyway eventually.

> I remember how much we were fighting with all variants of
> dependencies that are coming as glibc based, because many users are
> still on CentOS/Redhat/Debian/Ubuntu/etc. Glibc is mostly the
> blocker. That is also why I was not going to try to improve the MUSL
> as at the end I would hit the wall, because of another glibc based
> component.
> 
> Anyway if there is some interest just let me know.
>
> Thanks,
> Tomas
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmddidhEknrSUtydvocghN2pm30pUQ8eGsuqEd9eNpLSg@mail.gmail.com>
In-Reply-To
<20220621114559.4d38d98e@ncopa-desktop.lan> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 11:46 odesílatel Natanael Copa <ncopa@alpinelinux.org> napsal:
>
> Adding glibc support will open new possibilities, like adding support
> for systemd (see
> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33329).
>
Adding more options what becomes possible does not mean that it needs
to be done. I tried to say that when you look on glibc as another
"libc" then there does not need to be any change. It is just different
build. Every component must work with glibc as it is the very first
one people support. I do not want to argue about where Alpine should
go. It is your decisions and obviously good ones as I like Alpine. If
it helps with systemd, it is good. Does it push anyone for systemd?
No.
>
> At this point there would not really be any big difference from
> CentOS/Redhat/Debian/Ubuntu/etc.
>
> So why not just use CentOS/Redhat/Debian/Ubuntu in the first place? I
> believe there is where we'd end up anyway eventually.
>
- Alpine serves very fresh packages. Other distros go too slow.
- Alpine has very little vulnerabilities reported compared to other distros.
- Alpine is so easy to customize compared to other distros.
- Alpine is perfect for docker use case (mine)

I know that I defend my use case so I understand it is just one vote.
Just explaining why I do not like to use saint four.

I did not pick Alpine, because of MUSL. I am not sure it brings that
much for non embedded use cases. I have picked Alpine, because of its
packaging and simplicity to customize it. It can serve easily any use
case. It is not important if C library is couple of megabytes smaller.
Details
Message ID
<87bkum1div.fsf@ungleich.ch>
In-Reply-To
<CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Hey Tomas,

I have to say I find this an interesting idea, because at ungleich we
also sometimes have the need to run glibc based apps, which at the
moment, we simply run inside containers.

I see multiple paths here and before even diving deeper into the topic,
I would like to understand which one you are suggesting:

- a) Adding glibc support as an add-on/additionally to musl
     While some things need to be clarified (which is libc.so...), most
     things could stay the same and there would simply be an additional
     libc installed and available.

     Using this approach there might be *some* packages which are
     available as -glibc variants (like a JVM), but most of Alpine would
     stay as-is.

- b) Replacing musl with glibc
     This would be more the scenario of a "fork" or "glibc version" of
     Alpine. All packages would need to be compiled, similar to adding a
     new architecture, but potentially doubling the builds.

     This might be more coherent and easier to understand and
     potentially less confusing, if you only have software linked
     against *one* libc.

Both variants come with their disadvantages, I personally would have a
small a preference for (a), keeping Alpine simple and lean as is, but
adding glibc support in case it makes sense.

Best regards,

Nico


Tomas Kolda <koldat@gmail.com> writes:

> Trying to send email again from my personal email as my previous post
> was marked as invalid.
>
> Hi,
>
> I know that this topic is very sensitive in Alpine, but please let me
> describe first what and why we would like to achieve.
>
> 1. We love to use Alpine for our containers. Alpine has very fresh
> versions packages, it is very easy to use, versatile, customizable and
> clean. Basically only superlatives.
> 2. Our applications are mostly Java. But many of our jars are JNIs and
> are compiled against glibc.
> 3. We were using Alpine with Glibc hacks like
> https://github.com/sgerrand/alpine-pkg-glibc It was working nice until
> we ran into a problem where some other library was trying to load
> other native library and it started crashing.
> 4. We have switched to native MUSL based everywhere (Java, libs,
> etc.). I was happy that it was possible, but I was surprised that the
> performance of our applications degraded a lot. Our application is
> heavily multithreaded. After long investigation I have found that
> degradation is because of poor implementation of pthread locks in
> muslc. Probably the advantage of glibc  is the adaptive locking
> feature and maybe other types of tunings. In our example application
> spends with muslc build 600% CPU  in kernel and 200% in user space,
> but on glibc 100% CPU in kernel and 700% in user space (where 100%
> means 1 CPU core). Yes it is that huge difference.
>
> At this point I had many options of what to do and one I would like to describe.
>
> - I started thinking that I would like to have a variant of Alpine
> with glibc. Everything else is the same and great as usual.
> - After the initial painful bootstrap to get bare minimum to be able
> to compile I got the first 300 packages built (enough for us) with
> only a few tweaks in some packages that are explicitly applying MUSL
> patches. I simply disabled these.
> - Now our application is running fast again with native builds and no tweaks.
> - I know glibc is maybe a little bit, but we simply need performance
> and compatibility.
>
> Example of change in procps. There is a tweak to make it work in MUSL
> so I simply conditionally disable it:
> if [ "$CLIBC" != "glibc" ]; then
> export LIBS="$LIBS -lintl"
> fi
>
> I would love to have the patches part of the main branch so I do not
> need to do merges from time to time. I can only dream that the Alpine
> community will provide the official build as well. I think that more
> people would like that when I look at a number of projects around
> hacking glibc in Alpine.
>
> Please do not take it as a heretical thought. It came as a need.
>
> What do you think about the project? I can share my changes in some
> branch or in github. I can also share initial 3.16 binaries to have an
> easier bootstrap. It was really a pain.
>
> Thank you very much for your project and the hard work you put into it.
> Tomas


--
Sustainable and modern Infrastructures by ungleich.ch
Thomas Lamprecht <thomas@lamprecht.org>
Details
Message ID
<e5703d3f-4953-6797-2a93-324ea72810b9@lamprecht.org>
In-Reply-To
<CAGFcrMmddidhEknrSUtydvocghN2pm30pUQ8eGsuqEd9eNpLSg@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Am 21/06/2022 um 12:20 schrieb Tomas Kolda:
> - Alpine serves very fresh packages. Other distros go too slow.
> - Alpine has very little vulnerabilities reported compared to other distros.
> - Alpine is so easy to customize compared to other distros.
> - Alpine is perfect for docker use case (mine)

Basically all of above is also true for Arch Linux which supports glibc, so
why not use that? I'm using it in a hosted VM for some personal important
services since over six years, without any issue whatsoever and quite good
experience - packaging is also simple and straight forward, so not hard to
re-package one's application for Arch Linux.

FWIW, if your working at a company that would profit from such a thing why
not try to implement glibc integration and build CI yourself, you wouldn't
even need to cover the whole archive available in Alpine Linux, just a java
build and the external library dependencies of your app. Basically doing the
approach of Nico's (a) proposal yourself.

That would serve as good proof of concept and give a much better idea about
practicability and maintenance effort required, it also could proof that
this change alone was really responsible for the performance hit your app
took and you wouldn't need to wait for any such hypothetical change being
released in upstream Alpine Linux.

IMO, and that really is an outside view, as while I'm dabbling with Alpine
Linux and its inner workings on and off since years I'm really not active in
the community in any form, it feels rather misguided to add glibc into
Alpine Linux itself in general as complete musl alternative, just feels like
going against core characteristics and the reasoning for doing such really
doesn't outweigh Natanael's arguments at all.

cheers,
Thomas
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrM=YLSnEwOWsRc0krE68HdQL2QvUALs1-PDZaCPyrkYoDw@mail.gmail.com>
In-Reply-To
<87bkum1div.fsf@ungleich.ch> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 13:07 odesílatel Nico Schottelius
<nico.schottelius@ungleich.ch> napsal:
>
> - a) Adding glibc support as an add-on/additionally to musl
>      While some things need to be clarified (which is libc.so...), most
>      things could stay the same and there would simply be an additional
>      libc installed and available.
>
>      Using this approach there might be *some* packages which are
>      available as -glibc variants (like a JVM), but most of Alpine would
>      stay as-is.
>
I am not sure it is possible to have it in a stable way. There are
more projects trying to achieve it. Some bring libraries from other
distros (sgerrand), some does compatibility layer (gcompat). In both
cases it is not 100% same so all applications does not work. First
does not work when you dynamically want to use other library from OS
that is musl. The other does not work when it relies on exact
functionality as it is only proxy. Some functionality also relies on
"headers" as well that affects it during compile time. That is why I
tried (b).
>
> - b) Replacing musl with glibc
>      This would be more the scenario of a "fork" or "glibc version" of
>      Alpine. All packages would need to be compiled, similar to adding a
>      new architecture, but potentially doubling the builds.
>
>      This might be more coherent and easier to understand and
>      potentially less confusing, if you only have software linked
>      against *one* libc.
>
Yes that is what I propose. To have a new "architecture" build that
will simply have everything with glibc enabled. That is what we do. We
have Alpine build with CLIBC=glibc. We have around 300 packages (what
we need) built with a full set of build tools (build-base + abuild +
apk-tools + git).
>
> Both variants come with their disadvantages, I personally would have a
> small a preference for (a), keeping Alpine simple and lean as is, but
> adding glibc support in case it makes sense.
>
I would prefer (a) as well in case it can work for 100% cases. I do
not know how to achieve it. Any try to mix these libraries end up with
a crash.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmshmDEo=Bh4ooduO0+16RrqF2CKtWkc=AZWtKrry3a9Q@mail.gmail.com>
In-Reply-To
<e5703d3f-4953-6797-2a93-324ea72810b9@lamprecht.org> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 14:00 odesílatel Thomas Lamprecht
<thomas@lamprecht.org> napsal:
>
> Basically all of above is also true for Arch Linux which supports glibc, so
> why not use that? I'm using it in a hosted VM for some personal important
> services since over six years, without any issue whatsoever and quite good
> experience - packaging is also simple and straight forward, so not hard to
> re-package one's application for Arch Linux.
>
Maybe ... :)
>
> FWIW, if your working at a company that would profit from such a thing why
> not try to implement glibc integration and build CI yourself, you wouldn't
> even need to cover the whole archive available in Alpine Linux, just a java
> build and the external library dependencies of your app. Basically doing the
> approach of Nico's (a) proposal yourself.
>
Yes that is what we have and what we use already. We are building it
based on incoming changes. I am just merging it with my changes. Any
difference to community is not good so I am just trying to have it
official. Maybe more people needs something like this. So it is not
just "idea". It is working project.
>
> That would serve as good proof of concept and give a much better idea about
> practicability and maintenance effort required, it also could proof that
> this change alone was really responsible for the performance hit your app
> took and you wouldn't need to wait for any such hypothetical change being
> released in upstream Alpine Linux.
>
- This introduce some changes in some of the ports (mostly only
disable MUSL tweaks) - so far like 5% of packages require change
- This introduce one more build as it behaves like new architecture
- I agree that having build of all is hard to estimate. But after
initial bootstrap (nightmare at least for me) I was able to "fix" that
300 packages in two days.
Jakub Panek <me@panekj.dev>
Details
Message ID
<vTcnWREDRbSp_m2UXDa_2LCTqGU6XH8o9aBAPLeCyJv0MqPQF5dQ2GNQ-tVLrqiuRCrVsuJRTTz1F6RuKHry-bCvRHyMW3GSOTgjutxKURI=@panekj.dev>
In-Reply-To
<e5703d3f-4953-6797-2a93-324ea72810b9@lamprecht.org> (view parent)
DKIM signature
missing
Download raw message
> IMO, and that really is an outside view, as while I'm dabbling with Alpine
> Linux and its inner workings on and off since years I'm really not active in
> the community in any form, it feels rather misguided to add glibc into
> Alpine Linux itself in general as complete musl alternative, just feels like
> going against core characteristics and the reasoning for doing such really
> doesn't outweigh Natanael's arguments at all.

As a person that dabbles with Alpine and is somewhat active in community,
I agree that glibc doesn't feel right with Alpine. It's not a component
that you can simply replace in ecosystem as everything has to be rebuilt
against it. IMO, libc and package manager are base components which make
OS distinct so calling it Alpine with glibc would be a disservice as
Alpine focuses on size, simplicity and security, which for glibc I
would say neither are a focus.

> What do you think about the project? I can share my changes in some
> branch or in github. I can also share initial 3.16 binaries to have an
> easier bootstrap. It was really a pain.

I would be happy to contribute towards apk-tools + glibc OS (option b))
and it's ports as much as I can but would really recommend for it
to define itself as separate project from Alpine (e.g. different name).
I could also provide some compute and storage for it.

Regards,
Jakub P.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmYUyfuUzBr32CwHtZf-+n-mHBBXvpArUcuNzVCCNRynQ@mail.gmail.com>
In-Reply-To
<vTcnWREDRbSp_m2UXDa_2LCTqGU6XH8o9aBAPLeCyJv0MqPQF5dQ2GNQ-tVLrqiuRCrVsuJRTTz1F6RuKHry-bCvRHyMW3GSOTgjutxKURI=@panekj.dev> (view parent)
DKIM signature
missing
Download raw message
út 21. 6. 2022 v 15:05 odesílatel Jakub Panek <me@panekj.dev> napsal:
>
> As a person that dabbles with Alpine and is somewhat active in community,
> I agree that glibc doesn't feel right with Alpine. It's not a component
> that you can simply replace in ecosystem as everything has to be rebuilt
> against it. IMO, libc and package manager are base components which make
> OS distinct so calling it Alpine with glibc would be a disservice as
> Alpine focuses on size, simplicity and security, which for glibc I
> would say neither are a focus.
>
So the Alpine can be first distro that supports two libc libraries! ;-)
>
> I would be happy to contribute towards apk-tools + glibc OS (option b))
> and it's ports as much as I can but would really recommend for it
> to define itself as separate project from Alpine (e.g. different name).
> I could also provide some compute and storage for it.
>
As Natanael said it would increase the resource needs (people) even
more. I think... Changing one library does not need to generate
another distro. Maybe I should share it somehow so you can try it in
chroot or docker to see there is really actually almost no change
which I like most.

Maybe Jakube we can setup a repository of packages somewhere on your
web server? I can copy it there. Then it is easy to start to play with
that as you have build toolchain ready.
Details
Message ID
<CKVUTPCNCCJ5.MRXA8K04UO72@sumire>
In-Reply-To
<CAGFcrMmYUyfuUzBr32CwHtZf-+n-mHBBXvpArUcuNzVCCNRynQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Tue Jun 21, 2022 at 3:39 PM CEST, Tomas Kolda wrote:
> út 21. 6. 2022 v 15:05 odesílatel Jakub Panek <me@panekj.dev> napsal:
> >
> > As a person that dabbles with Alpine and is somewhat active in community,
> > I agree that glibc doesn't feel right with Alpine. It's not a component
> > that you can simply replace in ecosystem as everything has to be rebuilt
> > against it. IMO, libc and package manager are base components which make
> > OS distinct so calling it Alpine with glibc would be a disservice as
> > Alpine focuses on size, simplicity and security, which for glibc I
> > would say neither are a focus.
> >
> So the Alpine can be first distro that supports two libc libraries! ;-)
void linux already does, unless you specifically mean multilib'd at the
same time.

and gentoo, of course :)
> >
> > I would be happy to contribute towards apk-tools + glibc OS (option b))
> > and it's ports as much as I can but would really recommend for it
> > to define itself as separate project from Alpine (e.g. different name).
> > I could also provide some compute and storage for it.
> >
> As Natanael said it would increase the resource needs (people) even
> more. I think... Changing one library does not need to generate
> another distro. Maybe I should share it somehow so you can try it in
> chroot or docker to see there is really actually almost no change
> which I like most.
>
> Maybe Jakube we can setup a repository of packages somewhere on your
> web server? I can copy it there. Then it is easy to start to play with
> that as you have build toolchain ready.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMk_4-WALsR88e4Xi5DAf8y5-a=kT=2413C1rHspXH_Gfw@mail.gmail.com>
In-Reply-To
<CKVUTPCNCCJ5.MRXA8K04UO72@sumire> (view parent)
DKIM signature
missing
Download raw message
> void linux already does, unless you specifically mean multilib'd at the
> same time.
>
> and gentoo, of course :)
>
Nice I did not know that. From what I see there they are also not
combining. Both are having it as separate flavor builds (one or the
other).
Jakub Panek <me@panekj.dev>
Details
Message ID
<2d9mplT0SfCec-0RrOe07ywj3_XWJjggRYxMKVIGYmXVv7RJDr65BgVD-F6Dgxo0u4y5gF3reFXYN5QIVmjZ_hiUY1F6LgkgbCk6Kg9Y5bA=@panekj.dev>
In-Reply-To
<CAGFcrMmYUyfuUzBr32CwHtZf-+n-mHBBXvpArUcuNzVCCNRynQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
> út 21. 6. 2022 v 15:05 odesílatel Jakub Panek me@panekj.dev napsal:
>
> > As a person that dabbles with Alpine and is somewhat active in community,
> > I agree that glibc doesn't feel right with Alpine. It's not a component
> > that you can simply replace in ecosystem as everything has to be rebuilt
> > against it. IMO, libc and package manager are base components which make
> > OS distinct so calling it Alpine with glibc would be a disservice as
> > Alpine focuses on size, simplicity and security, which for glibc I
> > would say neither are a focus.
>
> So the Alpine can be first distro that supports two libc libraries! ;-)

Void Linux and Gentoo as mentioned by alice have done that before so
not really "first" :)
As for simultaneous multilibc on host it would be a terrible idea
as you have to monitor/maintain both libcs at the same time with (IMO)
very little benefit.

> > I would be happy to contribute towards apk-tools + glibc OS (option b))
> > and it's ports as much as I can but would really recommend for it
> > to define itself as separate project from Alpine (e.g. different name).
> > I could also provide some compute and storage for it.
>
> As Natanael said it would increase the resource needs (people) even
> more. I think... Changing one library does not need to generate
> another distro. Maybe I should share it somehow so you can try it in
> chroot or docker to see there is really actually almost no change
> which I like most.
>
> Maybe Jakube we can setup a repository of packages somewhere on your
> web server? I can copy it there. Then it is easy to start to play with
> that as you have build toolchain ready.

Sure, if you can provide a public endpoint (HTTP/SFTP/FTP(S)/RSYNC/other)
I can grab and host right it now. If not possible, it will take me
some time to set up it for multi-user purpose so you can throw it there
yourself :)

Whichever the way, you can send me an email directly (as to not clutter
mailing list even more) but it would be good to see the modified
ports source available publicly so the bootstrap could be verified as
well. I'm fine with GitHub, GitLab or wherever it would be hosted with
preference for something that could be easily integrated for
collaboration and CI/CD.

Regards,
Jakub P.
Laurent Bercot <ska-devel@skarnet.org>
Details
Message ID
<emdea05cbc-c1e3-405b-a5ba-f75b4eaedcee@95047c56.com>
In-Reply-To
<CAGFcrMk_4-WALsR88e4Xi5DAf8y5-a=kT=2413C1rHspXH_Gfw@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
  I find it interesting that when thinking about such things (making
a distro supporting two libcs, or supporting two init systems, or
patching some software to work with the better-known alternative, or...)
the frame of reference is always "adding the support of $biggerthing
to a project that has $smallthing as a characteristic", and never the
opposite.

  It always ends up meaning more burden for the project that typically
has fewer resources, less manpower, and that is only kept afloat by
the superhuman good will and effort of its maintainers.

  Why does nobody ever say "ohey, let's add musl support to Fedora!" ?

  No, it has to be Alpine, whose defining characteristic is to be a
musl-based distribution, which is the main reason why it's small and
efficient and has the success it has, that needs to grow glibc support,
i.e. diluting its meager resources away from its central vision and
into something *every other distro under the sun* already does, which
means decreasing its value in the software ecosystem.

  I see this pattern happening over, and over, and over.
  I don't get it.
  That's a lie. I kinda get it, but I really don't like it.

  I would prefer more energy to be spent in understanding why the
musl pthread thing is so inefficient for your use case, and having a
discussion in musl about how to improve it, and whether the use case
is common enough to be worth trade-offs, if necessary.
  I would prefer technical issues to be solved rather than translated
into organizational burden and vision dilution put onto projects.

--
  Laurent
Jakub Panek <PanekJ@pm.me>
Details
Message ID
<kaVXbcNp5pJ1Dcrm5QSmfV17xdlpMrXsDO1knxwhw2GQBRRaIU2zAu-y2lsmiEoHOcQWHk5pFES0WmrYSXUJ0bm3FNK2-ov-Sppqra_TZiI=@pm.me>
In-Reply-To
<emdea05cbc-c1e3-405b-a5ba-f75b4eaedcee@95047c56.com> (view parent)
DKIM signature
missing
Download raw message
> Why does nobody ever say "ohey, let's add musl support to Fedora!" ?
>
> No, it has to be Alpine

https://github.com/NixOS/nixpkgs/issues/90147 ?

R, Jakub. P
Details
Message ID
<a3367d7-d3c5-4755-289e-37507033405f@dereferenced.org>
In-Reply-To
<CAGFcrM=KPg5AZie-ux055SVAZu08EmSJ3s-Q_Map5j5CmGaaKA@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Hi,

On Tue, 21 Jun 2022, Tomas Kolda wrote:

> Trying to send email again from my personal email as my previous post
> was marked as invalid.
>
> Hi,
>
> I know that this topic is very sensitive in Alpine, but please let me
> describe first what and why we would like to achieve.
>
> 1. We love to use Alpine for our containers. Alpine has very fresh
> versions packages, it is very easy to use, versatile, customizable and
> clean. Basically only superlatives.
> 2. Our applications are mostly Java. But many of our jars are JNIs and
> are compiled against glibc.
> 3. We were using Alpine with Glibc hacks like
> https://github.com/sgerrand/alpine-pkg-glibc It was working nice until
> we ran into a problem where some other library was trying to load
> other native library and it started crashing.

Yes, I have written about this before[0].  Mixing glibc and musl
environments will lead to instability -- this can be overcome with proper 
multi-arch support, but we have not implemented this in Alpine yet (and 
apk-tools would need some minor changes to the solver to become 
multi-arch aware).

> 4. We have switched to native MUSL based everywhere (Java, libs,
> etc.). I was happy that it was possible, but I was surprised that the
> performance of our applications degraded a lot. Our application is
> heavily multithreaded. After long investigation I have found that
> degradation is because of poor implementation of pthread locks in
> muslc.

Locks in musl are done with atomics.  The advantage is probably due to the 
fact that glibc has lock elision, which musl does not yet have. 
Investigating the addition of lock elision to musl is on my list of things 
to do eventually.

Which JVM are you using?  OpenJDK 17 and later have their own internal 
implementation of lock elision for Java/JNI locks, which should improve 
performance.

> Probably the advantage of glibc  is the adaptive locking
> feature and maybe other types of tunings. In our example application
> spends with muslc build 600% CPU  in kernel and 200% in user space,
> but on glibc 100% CPU in kernel and 700% in user space (where 100%
> means 1 CPU core). Yes it is that huge difference.

These statistics do not actually mean anything useful.  The increased 
kernel wait load is simply due to futex contention, which is already known 
due to the lack of lock elision support in musl.

> At this point I had many options of what to do and one I would like to describe.
>
> - I started thinking that I would like to have a variant of Alpine
> with glibc. Everything else is the same and great as usual.
> - After the initial painful bootstrap to get bare minimum to be able
> to compile I got the first 300 packages built (enough for us) with
> only a few tweaks in some packages that are explicitly applying MUSL
> patches. I simply disabled these.
> - Now our application is running fast again with native builds and no tweaks.
> - I know glibc is maybe a little bit, but we simply need performance
> and compatibility.
>
> Example of change in procps. There is a tweak to make it work in MUSL
> so I simply conditionally disable it:
> if [ "$CLIBC" != "glibc" ]; then
> export LIBS="$LIBS -lintl"
> fi
>
> I would love to have the patches part of the main branch so I do not
> need to do merges from time to time. I can only dream that the Alpine
> community will provide the official build as well. I think that more
> people would like that when I look at a number of projects around
> hacking glibc in Alpine.

If you believe Alpine should offer a glibc build, you should propose it as 
a system change proposal to the Alpine TSC.  This is because you will 
either need to:

- change abuild to make variation-specific changes easier to do (e.g. 
restricting musl/glibc-specific patches to their own variants)

- customize hundreds of APKBUILDs which will require the cooperation of 
package maintainers (who may be skeptical)

Speaking in personal capacity, I do not think the Alpine TSC will approve 
this proposal because the project is already overwhelmed with work to do.

Ariadne

[0]: https://ariadne.space/2021/08/26/there-is-no-such-thing-as-a-glibc-based-alpine-image/
Details
Message ID
<6ead6b41-72b3-969a-51cd-ebc102b215c@dereferenced.org>
In-Reply-To
<emdea05cbc-c1e3-405b-a5ba-f75b4eaedcee@95047c56.com> (view parent)
DKIM signature
missing
Download raw message
Hi,

On Tue, 21 Jun 2022, Laurent Bercot wrote:

>
> I find it interesting that when thinking about such things (making
> a distro supporting two libcs, or supporting two init systems, or
> patching some software to work with the better-known alternative, or...)
> the frame of reference is always "adding the support of $biggerthing
> to a project that has $smallthing as a characteristic", and never the
> opposite.
>
> It always ends up meaning more burden for the project that typically
> has fewer resources, less manpower, and that is only kept afloat by
> the superhuman good will and effort of its maintainers.
>
> Why does nobody ever say "ohey, let's add musl support to Fedora!" ?
>
> No, it has to be Alpine, whose defining characteristic is to be a
> musl-based distribution, which is the main reason why it's small and
> efficient and has the success it has, that needs to grow glibc support,
> i.e. diluting its meager resources away from its central vision and
> into something *every other distro under the sun* already does, which
> means decreasing its value in the software ecosystem.
>
> I see this pattern happening over, and over, and over.
> I don't get it.
> That's a lie. I kinda get it, but I really don't like it.
>
> I would prefer more energy to be spent in understanding why the
> musl pthread thing is so inefficient for your use case, and having a
> discussion in musl about how to improve it, and whether the use case
> is common enough to be worth trade-offs, if necessary.

+1.  The solution is to add support for lock elision to musl, as glibc 
have done.  Or to use a newer version of the JDK which adds lock elision 
at the language level.

But the problem with lock elision in libc is that it is very difficult to 
prove that an implementation is correct.  When it is implemented at the 
language level, it is easy to prove.  This is why it has not happened in 
musl yet, in my opinion.

Ariadne
Details
Message ID
<aad724e-eddb-cd18-3f6c-53475a40f2eb@dereferenced.org>
In-Reply-To
<20220621114559.4d38d98e@ncopa-desktop.lan> (view parent)
DKIM signature
missing
Download raw message
Hi,

On Tue, 21 Jun 2022, Natanael Copa wrote:

> On Tue, 21 Jun 2022 10:49:07 +0200
> Tomas Kolda <koldat@gmail.com> wrote:
>
>
>>> I would rather spend the resources on fixing the issues with musl.
>>>
>> I completely agree and understand your view. Unfortunately I am not in
>> a position to offer or fund the resources.
>
> This is what I was afraid of.
>
>> Yes it can bring users and yes it can generate additional issues.
>
> So more work, but not not more resources.
>
>> My intention really was to have a pure Alpine built against glibc. It
>> is a C library as a MUSL. Everything else is exactly same.
>
> I actually don't believe this, even if that is how it would start.
>
> Adding glibc support will open new possibilities, like adding support
> for systemd (see
> https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/33329).
>
> I believe this would trigger big discussions and possible conflicts.
> After months of conflicts and 3-5 (or more) developers leaving the
> project systemd will be merged. It would be kind of stupid to not do
> so.
>
> Now we will have to maintain both openrc and systemd services. systemd
> will add its own set of new dependencies (together with many awesome
> features of course). The initramfs scripts will need work to handle
> this, the installer script will need to be adapted to this.
>
> We would now effectively be supporting two different OSes in one
> project and at some point we will realize that we simply don't have
> resources for supporting both. The most realistic outcome would be that
> we'd need to drop openrc/busybox and musl libc support. How many other
> distros that have added systemd support has *not* ended up with it as
> the only supported init?
>
> At this point there would not really be any big difference from
> CentOS/Redhat/Debian/Ubuntu/etc.
>
> So why not just use CentOS/Redhat/Debian/Ubuntu in the first place? I
> believe there is where we'd end up anyway eventually.

Most likely he is looking for a "componentized" Linux distribution. 
Alpine, for example, generally avoids pre-declared dependencies in favor 
of automatically discovered ones.  This allows the packages to be treated 
as distinct components, and is one of the key reasons why Alpine 
containers are small.

For cases like these, the most likely alternative at the moment is probably 
Microsoft's CBL-Mariner.  My team at work is building a glibc-based 
companion distribution to Alpine, but it is not in a state where I can 
recommend it yet.

Ariadne
Details
Message ID
<e3841a2c-1681-245d-2bfc-6079e93b471@dereferenced.org>
In-Reply-To
<CAGFcrMk_4-WALsR88e4Xi5DAf8y5-a=kT=2413C1rHspXH_Gfw@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Hi,

On Tue, 21 Jun 2022, Tomas Kolda wrote:

>> void linux already does, unless you specifically mean multilib'd at the
>> same time.
>>
>> and gentoo, of course :)
>>
> Nice I did not know that. From what I see there they are also not
> combining. Both are having it as separate flavor builds (one or the
> other).
>

Combining two runtime environments without appropriate namespacing (as 
Debian has done for example) results in a lot of stability problems.

In general, I don't think this idea is very good.  I would avoid it, and 
instead use the appropriate runtime environment for the workload. 
Sometimes, that is glibc, other times it is musl.

Ariadne
Olliver Schinagl <oliver@schinagl.nl>
Details
Message ID
<3b2c5b5e-5d05-5de1-88ef-9d44a3d1418c@schinagl.nl>
In-Reply-To
<CAGFcrM=RUOpE+NfBS0k-My-Sk81okHJHBF0cnO4Z9_VLzuyYeQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Hey Tomas,

On 21-06-2022 10:54, Tomas Kolda wrote:
> út 21. 6. 2022 v 10:43 odesílatel Dominique Martinet
> <asmadeus@codewreck.org> napsal:
>>
>> (With that said, I agree improving musl might be better, but.. any
>> reason to stick to alpine for this particular usecase? I don't know what
>> you need to rebuild your JNI dependencies but it might be even less work
>> to slim down another distro's container)
>>
> We are using Redhat UBI minimal 8 as well. But there is so many
> packages with many vulnerabilities so security scans are showing many
> issues. Even if it is sometimes false positives it still does not look
> good to customers.
> 
> I tried many distros, but Alpine install to different root (-p
> /newroot) allows me to have just a tiny bit of packages (only what I
> need) with zero issues.
> 
> So yes we can do that. I just like the simplicity that is already available.

I have also thought of a glibc-based alpine (like). For most things, 
muslc is great; for some things, glibc is needed/desired. So why stick 
with alpine?

TBH, because alpine is really amazing. I'm migrated from 
gentoo/ubuntu/debian to Arch Linux on my desktop/laptop; and while Arch 
shares quite a few similarities with Alpine, yet, strangely enough 
Alpine is still far superior. How?

Well for one, the team is amazing.

But technically also, security updates are often quickly addressed, 
plenty of package, ease of packaging. There's just so many things Alpine 
does 'better' then all of the distro's it has learned from. So if not 
alpine, what distro then? That's a good question, one I haven't been 
able to answer.

Void seems to kind of offer however what we'd be after, musl OR glibc at 
least. But void is even more understaffed, slower, 'smaller', so 
stepping that direction is also not helpful, and lacks the alpine 
awesomeness :)

Just some thoughts ...

Olliver
Olliver Schinagl <oliver@schinagl.nl>
Details
Message ID
<1452d156-96ec-6306-6175-a430ef61e4aa@schinagl.nl>
In-Reply-To
<CAGFcrMmYUyfuUzBr32CwHtZf-+n-mHBBXvpArUcuNzVCCNRynQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On 21-06-2022 15:39, Tomas Kolda wrote:
> út 21. 6. 2022 v 15:05 odesílatel Jakub Panek <me@panekj.dev> napsal:
>>
>> As a person that dabbles with Alpine and is somewhat active in community,
>> I agree that glibc doesn't feel right with Alpine. It's not a component
>> that you can simply replace in ecosystem as everything has to be rebuilt
>> against it. IMO, libc and package manager are base components which make
>> OS distinct so calling it Alpine with glibc would be a disservice as
>> Alpine focuses on size, simplicity and security, which for glibc I
>> would say neither are a focus.
>>
> So the Alpine can be first distro that supports two libc libraries! ;-)
>>
>> I would be happy to contribute towards apk-tools + glibc OS (option b))
>> and it's ports as much as I can but would really recommend for it
>> to define itself as separate project from Alpine (e.g. different name).
>> I could also provide some compute and storage for it.
>>
> As Natanael said it would increase the resource needs (people) even
> more. I think... Changing one library does not need to generate
> another distro. Maybe I should share it somehow so you can try it in
> chroot or docker to see there is really actually almost no change
> which I like most.

That is often the problem anyway, resources, be it $work or foss. The 
sad thing about open source in general (which I know, is also its 
strength), is the duplication of effort. Oh I don't like how you do XYZ, 
lets fork/re-write. Just look at the number of distro's/applications in 
C or C++ flavor, KDE or Gnome flavor. It kind of is sad that we waste so 
much resources here. Sometimes, a new gem is born of course.

On topic, it would have been nice, to add glibc support ot the APKBUILD 
files, keep alpine an musl project, but share the package base with a 
second 'project'.

The previous effort of glibc alpine failed, because it was a one man 
show, lack of resources. Getting something of the ground would be great; 
but then we'd have something that Void tries to do; but struggling even 
more for resources, though sharing the package base, kind of like we can 
use the same package for both x86 and s390x.

It does make me wonder, what is the 'raison d'etre' of void? Why does 
the world need a 'pretty similar distro' and can't we all work together 
on a common/better distro together :) One can dream ...

Olliver

> 
> Maybe Jakube we can setup a repository of packages somewhere on your
> web server? I can copy it there. Then it is easy to start to play with
> that as you have build toolchain ready.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmgnR0E0hrSNheOCROMhvkpWUkW59B-RuamDYXnntjL0Q@mail.gmail.com>
In-Reply-To
<1452d156-96ec-6306-6175-a430ef61e4aa@schinagl.nl> (view parent)
DKIM signature
missing
Download raw message
I completely agree with all you said Oliver. Very well written. It
should be set in stone.
>
> On topic, it would have been nice, to add glibc support ot the APKBUILD
> files, keep alpine an musl project, but share the package base with a
> second 'project'.
>
Yes please, this would be great. Maybe after some time people will see
it just works it can be reconsidered to have it as another platform.
This will simplify merges a lot.

Anyway, I uploaded sources to github. Here is the PR of all my changes
so far to be able to build that 300 packages (build stack + good base
we need to build our dockers).
- https://github.com/koldat/alpine/pull/1
- It is based on 3.16 branch
- I will work with Jakub to upload first compiled binaries so when you
are interested you can just "apk add" it to some root.
- When base is installed abuild works as usual for most packages.
- I have docker builder as well so one can easily compile/test musl vs
glibc without any external dependencies (ok only docker is needed).
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmSG6RbG_O+NrHZ0Cys=6stLy59JjBsqd=BnfncLXdiCQ@mail.gmail.com>
In-Reply-To
<a3367d7-d3c5-4755-289e-37507033405f@dereferenced.org> (view parent)
DKIM signature
missing
Download raw message
> Yes, I have written about this before[0].  Mixing glibc and musl
> environments will lead to instability -- this can be overcome with proper
> multi-arch support, but we have not implemented this in Alpine yet (and
> apk-tools would need some minor changes to the solver to become
> multi-arch aware).
>
I know. That is why I want to have glibc build. Many JNI dependencies
in Java is glibc based for example. It is really hard to keep up.
>
> Locks in musl are done with atomics.  The advantage is probably due to the
> fact that glibc has lock elision, which musl does not yet have.
> Investigating the addition of lock elision to musl is on my list of things
> to do eventually.
>
> Which JVM are you using?  OpenJDK 17 and later have their own internal
> implementation of lock elision for Java/JNI locks, which should improve
> performance.
>
I tried JDK 8, 11, 17. Latest versions. No difference. Yes it is in
futex as you said. From user perspective it matters, because it is 5
times slower. As I said I do not believe MUSL can dramatically change
pthread implementation as it is most sensitive part and very hard to
test. It is very long run.

> - customize hundreds of APKBUILDs which will require the cooperation of
> package maintainers (who may be skeptical)
I do not agree with this. Most changes in that 300 set was in the
basic packages that was requiring MUSL tweaks. There will be less
tweaks after we will go further as more complex applications are using
libraries that are solved already. Anyway even from statistical
perspective it is 17 changes for 300 packages -> 5-6%. I am not sure
how many packages are in Alpine, but we can extrapolate.

I also do not agree that someone would just for one "if" be pushing
change out. There are many patches for all other reasons. I am
positive that people tend to help others. And this change does not
generate any work for them if they will not need to support it. They
will just try to keep it in working way.
>
> Speaking in personal capacity, I do not think the Alpine TSC will approve
> this proposal because the project is already overwhelmed with work to do.
>
Agree. I think that for start it would be great to at least allow "if"
patches to be able to merge it. It will be noop for Alpine.

See example changes here:
https://github.com/koldat/alpine/pull/1
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmFtO2xS-LUmyXA4R7u+cVx8kTwo+O-KWV_+EBth_YzKw@mail.gmail.com>
In-Reply-To
<emdea05cbc-c1e3-405b-a5ba-f75b4eaedcee@95047c56.com> (view parent)
DKIM signature
missing
Download raw message
>   No, it has to be Alpine, whose defining characteristic is to be a
> musl-based distribution, which is the main reason why it's small and
> efficient and has the success it has, that needs to grow glibc support,
> i.e. diluting its meager resources away from its central vision and
> into something *every other distro under the sun* already does, which
> means decreasing its value in the software ecosystem.
>
You are right. Alpine can definitely pick their way. I do not want to
create yet another distro. I was asking for cooperation that was
outlined by Oliver as well. And maybe after some time Alpine will see
market for this change. There is a good reason main distros are using
glibc. I am trying to be pragmatic and not just say MUSL is great.
There are so many variables in the world. And maybe that is why we see
this type of request time to time.
>
>   I would prefer more energy to be spent in understanding why the
> musl pthread thing is so inefficient for your use case, and having a
> discussion in musl about how to improve it, and whether the use case
> is common enough to be worth trade-offs, if necessary.
>   I would prefer technical issues to be solved rather than translated
> into organizational burden and vision dilution put onto projects.
>
Yes I would like that as well. This would potentially solve this one
particular issue (time frame is questionable). It does not solve the
others.
Details
Message ID
<CKWT7Z7BBSUI.2FVAM05ZUS5UO@sumire>
In-Reply-To
<CAGFcrMmSG6RbG_O+NrHZ0Cys=6stLy59JjBsqd=BnfncLXdiCQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Wed Jun 22, 2022 at 2:04 PM CEST, Tomas Kolda wrote:
> > Yes, I have written about this before[0].  Mixing glibc and musl
> > environments will lead to instability -- this can be overcome with proper
> > multi-arch support, but we have not implemented this in Alpine yet (and
> > apk-tools would need some minor changes to the solver to become
> > multi-arch aware).
> >
> I know. That is why I want to have glibc build. Many JNI dependencies
> in Java is glibc based for example. It is really hard to keep up.
> >
> > Locks in musl are done with atomics.  The advantage is probably due to the
> > fact that glibc has lock elision, which musl does not yet have.
> > Investigating the addition of lock elision to musl is on my list of things
> > to do eventually.
> >
> > Which JVM are you using?  OpenJDK 17 and later have their own internal
> > implementation of lock elision for Java/JNI locks, which should improve
> > performance.
> >
> I tried JDK 8, 11, 17. Latest versions. No difference. Yes it is in
> futex as you said. From user perspective it matters, because it is 5
> times slower. As I said I do not believe MUSL can dramatically change
> pthread implementation as it is most sensitive part and very hard to
> test. It is very long run.
>
> > - customize hundreds of APKBUILDs which will require the cooperation of
> > package maintainers (who may be skeptical)
> I do not agree with this. Most changes in that 300 set was in the
> basic packages that was requiring MUSL tweaks. There will be less
> tweaks after we will go further as more complex applications are using
> libraries that are solved already. Anyway even from statistical
> perspective it is 17 changes for 300 packages -> 5-6%. I am not sure
> how many packages are in Alpine, but we can extrapolate.
>
> I also do not agree that someone would just for one "if" be pushing
> change out. There are many patches for all other reasons. I am
> positive that people tend to help others. And this change does not
> generate any work for them if they will not need to support it. They
> will just try to keep it in working way.
> >
> > Speaking in personal capacity, I do not think the Alpine TSC will approve
> > this proposal because the project is already overwhelmed with work to do.
> >
> Agree. I think that for start it would be great to at least allow "if"
> patches to be able to merge it. It will be noop for Alpine.
>
> See example changes here:
> https://github.com/koldat/alpine/pull/1
Details
Message ID
<CKWTBZV0CKNR.2S5ZYWZVZ5KDP@sumire>
In-Reply-To
<CAGFcrMmSG6RbG_O+NrHZ0Cys=6stLy59JjBsqd=BnfncLXdiCQ@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Wed Jun 22, 2022 at 2:04 PM CEST, Tomas Kolda wrote:
> > Yes, I have written about this before[0].  Mixing glibc and musl
> > environments will lead to instability -- this can be overcome with proper
> > multi-arch support, but we have not implemented this in Alpine yet (and
> > apk-tools would need some minor changes to the solver to become
> > multi-arch aware).
> >
> I know. That is why I want to have glibc build. Many JNI dependencies
> in Java is glibc based for example. It is really hard to keep up.
> >
> > Locks in musl are done with atomics.  The advantage is probably due to the
> > fact that glibc has lock elision, which musl does not yet have.
> > Investigating the addition of lock elision to musl is on my list of things
> > to do eventually.
> >
> > Which JVM are you using?  OpenJDK 17 and later have their own internal
> > implementation of lock elision for Java/JNI locks, which should improve
> > performance.
> >
> I tried JDK 8, 11, 17. Latest versions. No difference. Yes it is in
> futex as you said. From user perspective it matters, because it is 5
> times slower. As I said I do not believe MUSL can dramatically change
> pthread implementation as it is most sensitive part and very hard to
> test. It is very long run.
>
> > - customize hundreds of APKBUILDs which will require the cooperation of
> > package maintainers (who may be skeptical)
> I do not agree with this. Most changes in that 300 set was in the
> basic packages that was requiring MUSL tweaks. There will be less
> tweaks after we will go further as more complex applications are using
> libraries that are solved already. Anyway even from statistical
> perspective it is 17 changes for 300 packages -> 5-6%. I am not sure
> how many packages are in Alpine, but we can extrapolate.
>
> I also do not agree that someone would just for one "if" be pushing
> change out. There are many patches for all other reasons. I am
> positive that people tend to help others. And this change does not
> generate any work for them if they will not need to support it. They
> will just try to keep it in working way.
> >
> > Speaking in personal capacity, I do not think the Alpine TSC will approve
> > this proposal because the project is already overwhelmed with work to do.
> >
> Agree. I think that for start it would be great to at least allow "if"
> patches to be able to merge it. It will be noop for Alpine.
the example patches you have posted on github are very far from no-ops
at all, they require constant vigilance on how to maintain the
musl-specific patches to be separate and still work. i don't think
anyone wants to add musl specific patches as musl && msg && patch -p1,
and they were all also removed from checksums.

if they were part of aports, then it becomes the responsibility of the
people that work on aports to ensure that everything still works. but
since aports itself does not feed an actual glibc apk system.. that
doesn't really make sense. they only make sense as an entire separate
project where you either keep specific patches to apply to get aports to
build, or as a fork of aports (named something else) where you rebase
your changes on top every once in a while.

is it a lot of work? sure. it's not any more work than making them in
the first place, though, and it gives zero additional work to anyone
that maintains aports that they didn't ask for.

>
> See example changes here:
> https://github.com/koldat/alpine/pull/1

apologies- i accidentally sent an empty reply before.
Details
Message ID
<CKWTLJ45RZNF.9JNZVIKSGFKO@sumire>
In-Reply-To
<CAGFcrMmFtO2xS-LUmyXA4R7u+cVx8kTwo+O-KWV_+EBth_YzKw@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
On Wed Jun 22, 2022 at 2:12 PM CEST, Tomas Kolda wrote:
> >   No, it has to be Alpine, whose defining characteristic is to be a
> > musl-based distribution, which is the main reason why it's small and
> > efficient and has the success it has, that needs to grow glibc support,
> > i.e. diluting its meager resources away from its central vision and
> > into something *every other distro under the sun* already does, which
> > means decreasing its value in the software ecosystem.
> >
> You are right. Alpine can definitely pick their way. I do not want to
> create yet another distro. I was asking for cooperation that was
> outlined by Oliver as well.
you don't want to make another distro incorporating $hugesetofchanges,
so you just want others to work on it for free incorporating them
instead?
> And maybe after some time Alpine will see
> market for this change.
last i checked nobody working on alpine was working for 'a market' to
make money out of (with alpine)- personally i do it for free, but maybe
we are all supposed to be getting paid? certainly, if someone was
throwing free money my way maybe this would be an actual interesting
project to work towards.
> There is a good reason main distros are using
> glibc. I am trying to be pragmatic and not just say MUSL is great.
then use a glibc distro? i don't get it, again, as Laurent has said,
nothing whatsoever is forcing you to use musl or alpine.

i understand you believe you are not asking for much, just a 'small' set
of compatibility patches to easily allow someone else to 'build alpine
but with glibc'.. but if they are as easy as you have demonstrated and
claim, then.. you're done? you successfully made all the changes, and
can now build your forked+patched aports whenever you wish with glibc,
and can keep it that way. sounds like you already found the solution,
and can keep working on it there, as your own project. sadly though,
this is indeed 'an entire other distro'.. though you are basically
already done making it.

> There are so many variables in the world. And maybe that is why we see
> this type of request time to time.
> >
> >   I would prefer more energy to be spent in understanding why the
> > musl pthread thing is so inefficient for your use case, and having a
> > discussion in musl about how to improve it, and whether the use case
> > is common enough to be worth trade-offs, if necessary.
> >   I would prefer technical issues to be solved rather than translated
> > into organizational burden and vision dilution put onto projects.
> >
> Yes I would like that as well. This would potentially solve this one
> particular issue (time frame is questionable). It does not solve the
> others.
a very easy solution (for all the others): use literally any other
glibc-based distribution.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrM=wAegGwYTH5ruGcKX=X2AgYnktG=p+rtgxQCs_w9qdPA@mail.gmail.com>
In-Reply-To
<CKWTBZV0CKNR.2S5ZYWZVZ5KDP@sumire> (view parent)
DKIM signature
missing
Download raw message
st 22. 6. 2022 v 18:44 odesílatel alice <alice@ayaya.dev> napsal:
> the example patches you have posted on github are very far from no-ops
> at all, they require constant vigilance on how to maintain the
> musl-specific patches to be separate and still work. i don't think
> anyone wants to add musl specific patches as musl && msg && patch -p1,
> and they were all also removed from checksums.
>
Main reason is that abuild does not allow to conditionally exclude
patch as it requires it (build fails as checksum exists for something
not used). So I used this approach (current functionality). Otherwise
a build script would need a tweak like:

sources_exclude=.....
or even without condition
sources_exclude_glibc=.....

Ofcourse there are many ways of having it nice and current POC is not
final. I first tested if it is possible (it is). Then I tried to ask
if community even would want to do something like that (in
anyacceptable  way).
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrMmqX_UC4KgH-cih2v96w5Bsfk+-RLCfnbY=F8aN9JpdMg@mail.gmail.com>
In-Reply-To
<CKWTLJ45RZNF.9JNZVIKSGFKO@sumire> (view parent)
DKIM signature
missing
Download raw message
> you don't want to make another distro incorporating $hugesetofchanges,
> so you just want others to work on it for free incorporating them
> instead?

I wanted to help other people that would like to have glibc in Alpine.
If there is none then it is the answer as well. That is why I ask
here.

> last i checked nobody working on alpine was working for 'a market' to
> make money out of (with alpine)- personally i do it for free, but maybe
> we are all supposed to be getting paid? certainly, if someone was
> throwing free money my way maybe this would be an actual interesting
> project to work towards.

I am not sure I understand. I also send patches to open source in my
time. We are doing it for fun. And it is more fun when there are more
users.

> then use a glibc distro? i don't get it, again, as Laurent has said,
> nothing whatsoever is forcing you to use musl or alpine.

I know and I mentioned. Please do not take it offensive. I am just
asking if Alpine users would like it.

> i understand you believe you are not asking for much, just a 'small' set
> of compatibility patches to easily allow someone else to 'build alpine
> but with glibc'.. but if they are as easy as you have demonstrated and
> claim, then.. you're done? you successfully made all the changes, and
> can now build your forked+patched aports whenever you wish with glibc,
> and can keep it that way. sounds like you already found the solution,
> and can keep working on it there, as your own project. sadly though,
> this is indeed 'an entire other distro'.. though you are basically
> already done making it.

I am not sure if I was rude. I hope I was not. Yes I can do all of
that. But there is nothing wrong with trying to find better options.
If that options are not viable I understand it.

> a very easy solution (for all the others): use literally any other
> glibc-based distribution.

Yes it is definitely one of the options.
Tomas Kolda <koldat@gmail.com>
Details
Message ID
<CAGFcrM=m1m4NFLrwj1JZPjDg4oUC7eygo3AKDMxQ6y+tr2QKYg@mail.gmail.com>
In-Reply-To
<CAGFcrMmqX_UC4KgH-cih2v96w5Bsfk+-RLCfnbY=F8aN9JpdMg@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Thank you all for your patience with giving me the feedback. Based on
the answers of biggest contributors I do not see this attempt as the
one that will be accepted. I was actually expecting that, but you know
who does not try will not achieve anything :-)

Anyway thank you for this great distro that I like very much even
without glibc. Your build system rocks (apk, abuild....).

I wil keep the github for a while. I also posted binaries to Jakub in
case someone will want to take a look.

Thanks again!
Tomas
Reply to thread Export thread (mbox)