Hello!
I'm Nadia and I'm a long-since, happy user of alpine for containerized
workloads.
I have recently stumbled across a problem related to multiarch
container images, version pinning, and alpine's package retention
policy (where only the latest version of a package is available for a
given alpine release).
I have a Dockerfile that looks like this:
```Dockerfile
FROM --platform=$TARGETPLATFORM alpine:3.20.3
RUN apk --no-cache add --repository community chromium-
swiftshader=130.0.6723.116-r0
# ...
```
This Dockerfile is currently (2024-11-18) unbuildable in a multiarch
context, because the chromium-swiftshader package has a version drift
across architectures:
For x86_64, the last (and only available) version is 131.0.6778.69-r0
https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=x86_64&origin=&flagged=&maintainer=
For aarch64, it is 130.0.6723.116-r0
https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=aarch64&origin=&flagged=&maintainer=
I'm sure the issue with chromium-swiftshader is being worked on, and
that is not what I want to surface here.
I would like to bring attention to the problem that when and if this
happens for a given package, container build pipelines that try to
build for multiple architectures will break without a fix: Updating to
the latest version is not possible, as it is not present in the
outdated architecture, and keeping the outdated version is not possible
either, as it is no longer installable in the updated architecture.
I believe version pinning is established as a good practice in the
industry, and thus I think this use case needs some consideration from
the Alpine side. Even if not very common, packages breaking for certain
architecture is something that can definitely happen, and I think it
would be great for alpine to handle this more gracefully.
I'd love to hear if this has happened to someone else, and how they are
handling it, and/or what's the point of view of alpine developers on
this issue.
BR,
- N
On Mon Nov 18, 2024 at 6:21 PM CET, Nadia Santalla wrote:
> Hello!
Hi!
>> I'm Nadia and I'm a long-since, happy user of alpine for containerized> workloads.
That's nice!
>> I have recently stumbled across a problem related to multiarch> container images, version pinning, and alpine's package retention> policy (where only the latest version of a package is available for a> given alpine release).>> I have a Dockerfile that looks like this:>> ```Dockerfile> FROM --platform=$TARGETPLATFORM alpine:3.20.3>> RUN apk --no-cache add --repository community chromium-> swiftshader=130.0.6723.116-r0> # ...> ```>> This Dockerfile is currently (2024-11-18) unbuildable in a multiarch> context, because the chromium-swiftshader package has a version drift> across architectures:> For x86_64, the last (and only available) version is 131.0.6778.69-r0 > https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=x86_64&origin=&flagged=&maintainer=> For aarch64, it is 130.0.6723.116-r0> https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=aarch64&origin=&flagged=&maintainer=>
Usually all architectures have the same version. Of course some
inconsistencies may happen because builder some are faster than others.
Particulary the last days the arm builders were failing, see this
incident: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10832/> I'm sure the issue with chromium-swiftshader is being worked on, and> that is not what I want to surface here.>> I would like to bring attention to the problem that when and if this> happens for a given package, container build pipelines that try to> build for multiple architectures will break without a fix: Updating to> the latest version is not possible, as it is not present in the> outdated architecture, and keeping the outdated version is not possible> either, as it is no longer installable in the updated architecture.>> I believe version pinning is established as a good practice in the> industry, and thus I think this use case needs some consideration from> the Alpine side. Even if not very common, packages breaking for certain> architecture is something that can definitely happen, and I think it> would be great for alpine to handle this more gracefully.
If a version breaks, forcing a specific version isn't a fix, it's more
like a temporary workaround. I wouldn't use this in production, most
importantly you must remind you again to remove the constraint.
Otherwise you end up with way more problems than you probably had
before.
Partial upgrades are always tricky and only lead to problems. It's a lot
easier if we can safely assume for new commits that all systems are on
that tree.
For example if we update two packages in a single MR, and the former
package depends on a new version of the latter package, we need to add
the version constraint to the package dependency. This effort, plus
reviewing bad constraints and removing old constraints, is A LOT and is
not worth the effort.
Addionally adding support for old versions in our repositories doubles
(or more?) the disk size of the mirrors which is not doable for most
mirrors. (we already disable most -dbg packages due to size issues).
Arch Linux has a archive for old packages
(https://wiki.archlinux.org/title/Arch_Linux_Archive). Someone could do
the same, but honestly this is just promoting bad behaviour and nobody
should rely on it.
>> I'd love to hear if this has happened to someone else, and how they are> handling it, and/or what's the point of view of alpine developers on> this issue.>> BR,> - N
Hi
> Arch Linux has a archive for old packages> (https://wiki.archlinux.org/title/Arch_Linux_Archive). Someone could do> the same,
I’m doing weekly snapshots on https://mirror.fel.cvut.cz/alpine/snapshots (for future scientific use), but only for x86_64.
> but honestly this is just promoting bad behaviour and nobody should rely on it.
Exactly. Your approach of pinning versions of packages in docker files is just not compatible with the release model of Linux distros. We have stable branches where you can rely on backward compatibility of package upgrades and get security fixes.
Jakub
Sent from mobile phone
> On 19. 11. 2024, at 0:20, fossdd <fossdd@pwned.life> wrote:> > On Mon Nov 18, 2024 at 6:21 PM CET, Nadia Santalla wrote:>> Hello!> > Hi!> >> >> I'm Nadia and I'm a long-since, happy user of alpine for containerized>> workloads.> > That's nice!> >> >> I have recently stumbled across a problem related to multiarch>> container images, version pinning, and alpine's package retention>> policy (where only the latest version of a package is available for a>> given alpine release).>> >> I have a Dockerfile that looks like this:>> >> ```Dockerfile>> FROM --platform=$TARGETPLATFORM alpine:3.20.3>> >> RUN apk --no-cache add --repository community chromium->> swiftshader=130.0.6723.116-r0>> # ...>> ```>> >> This Dockerfile is currently (2024-11-18) unbuildable in a multiarch>> context, because the chromium-swiftshader package has a version drift>> across architectures:>> For x86_64, the last (and only available) version is 131.0.6778.69-r0>> https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=x86_64&origin=&flagged=&maintainer=>> For aarch64, it is 130.0.6723.116-r0>> https://pkgs.alpinelinux.org/packages?name=chromium-swiftshader&branch=v3.20&repo=&arch=aarch64&origin=&flagged=&maintainer=>> > > Usually all architectures have the same version. Of course some> inconsistencies may happen because builder some are faster than others.> > Particulary the last days the arm builders were failing, see this> incident: https://gitlab.alpinelinux.org/alpine/infra/infra/-/issues/10832/> >> I'm sure the issue with chromium-swiftshader is being worked on, and>> that is not what I want to surface here.>> >> I would like to bring attention to the problem that when and if this>> happens for a given package, container build pipelines that try to>> build for multiple architectures will break without a fix: Updating to>> the latest version is not possible, as it is not present in the>> outdated architecture, and keeping the outdated version is not possible>> either, as it is no longer installable in the updated architecture.>> >> I believe version pinning is established as a good practice in the>> industry, and thus I think this use case needs some consideration from>> the Alpine side. Even if not very common, packages breaking for certain>> architecture is something that can definitely happen, and I think it>> would be great for alpine to handle this more gracefully.> > If a version breaks, forcing a specific version isn't a fix, it's more> like a temporary workaround. I wouldn't use this in production, most> importantly you must remind you again to remove the constraint.> Otherwise you end up with way more problems than you probably had> before.> > Partial upgrades are always tricky and only lead to problems. It's a lot> easier if we can safely assume for new commits that all systems are on> that tree.> > For example if we update two packages in a single MR, and the former> package depends on a new version of the latter package, we need to add> the version constraint to the package dependency. This effort, plus> reviewing bad constraints and removing old constraints, is A LOT and is> not worth the effort.> > Addionally adding support for old versions in our repositories doubles> (or more?) the disk size of the mirrors which is not doable for most> mirrors. (we already disable most -dbg packages due to size issues).> > Arch Linux has a archive for old packages> (https://wiki.archlinux.org/title/Arch_Linux_Archive). Someone could do> the same, but honestly this is just promoting bad behaviour and nobody> should rely on it.> >> >> I'd love to hear if this has happened to someone else, and how they are>> handling it, and/or what's the point of view of alpine developers on>> this issue.>> >> BR,>> - N
Hi Jakub, Fossdd,
Thanks a lot for sharing your insights!
I think it might be useful for me to elaborate more on why the
container and non-container (so-called baremetal, but I'd rather not
call it that) use case differs.
In the container world, we care *a lot* about images being
reproducible: This is one of the reasons why containers were born,
after all. Allegedly, we're have still a lot of strides to make
regarding byte-to-byte reproducibility, but I believe we'll eventually
get there. Regardless of that, what is well-established and even
assumed in the container world is the following: If someone builds a
container image from a Dockerfile (+ a given set of source files)
today, and someone else builds a container image using the same
Dockerfile (and same set of source files) next week, the resulting
image should _behave the same_.
This assumption does not hold if in your Dockerfile, you install (or
download, for that matter) binaries or other components without
specified version. `apk` (or `apt`) `install` are very much likely to
install one thing today and a different one next week.
This assumption is heavily ingrained in the container world. To give
some examples, Dockerfile liners that are shipped by default in the
most popular editors, have rules that explicitly tell you to pin
versions in `apk add`:
https://github.com/hadolint/hadolint/blob/4ab5d7809325c8de23956a44ca5a1f3c25907faf/src/Hadolint/Rule/DL3018.hs#L20
This is popping up by default on the most popular editors across the
world.
That being said, I understand this is definitely container-specific. I
run arch linux (btw) on my workstations and servers, which has a
similar policy as alpine, and I very rarely have had to pin version for
a package. I think the "partial upgrades are bad and problematic" is
very true for full system installations, but I see it being definitely
at odds with container best practices.
I think it is good if alpine kept using the same practices it is
currently doing, and assume no partial upgrades to keep the maintainer
effort required into reasonable levels. But I also think that the
container community would appreciate a best-effort approach on helping
towards reproducible container builds.
> Addionally adding support for old versions in our repositories> doubles (or more?) the disk size of the mirrors which is not> doable for most mirrors. (we already disable most -dbg packages> due to size issues).
I definitely see this cost problem and, while I do not have any idea to
magically solve it, maybe I can try with a suggestion to contain it: If
old versions are provided for specific, non-general use cases
(containers), then perhaps not all mirrors need to support it, and
perhaps a single one (like archlinux's archive) could suffice. People
who are willing to trade reproducibility for potential dependency
breakages in their containers would be able to use it.
This email came out way longer than what I was planning, but at least
I'm hoping I did a decent job explaining about why we pin version
numbers in the container world and how I hope alpine could support that
use case better.
- Nad
On Tue Nov 19, 2024 at 2:04 PM CET, Nadia Santalla wrote:
> Hi Jakub, Fossdd,>> Thanks a lot for sharing your insights!>> I think it might be useful for me to elaborate more on why the> container and non-container (so-called baremetal, but I'd rather not> call it that) use case differs.>> In the container world, we care *a lot* about images being> reproducible: This is one of the reasons why containers were born,> after all. Allegedly, we're have still a lot of strides to make> regarding byte-to-byte reproducibility, but I believe we'll eventually> get there. Regardless of that, what is well-established and even> assumed in the container world is the following: If someone builds a> container image from a Dockerfile (+ a given set of source files)> today, and someone else builds a container image using theFuxels same> Dockerfile (and same set of source files) next week, the resulting> image should _behave the same_.
That's why we have stable releases. Stable releases only contain
security (and bug) fixes.
>> This assumption does not hold if in your Dockerfile, you install (or> download, for that matter) binaries or other components without> specified version. `apk` (or `apt`) `install` are very much likely to> install one thing today and a different one next week.>> This assumption is heavily ingrained in the container world. To give> some examples, Dockerfile liners that are shipped by default in the> most popular editors, have rules that explicitly tell you to pin> versions in `apk add`:> https://github.com/hadolint/hadolint/blob/4ab5d7809325c8de23956a44ca5a1f3c25907faf/src/Hadolint/Rule/DL3018.hs#L20> This is popping up by default on the most popular editors across the> world.
I never experienced such behaviour, and I do not agree to it. For
example, pinning versions does not fix security issues.
You could use the tilde operator instead of the equal sign (see
apk-world(5)). But often we fast-forward to a another stable version, which
include bug and security fixes and still behaves the same. So again,
use stable releases and everything should behave the same. That's
our policy and it works pretty good imo!
>> That being said, I understand this is definitely container-specific. I> run arch linux (btw) on my workstations and servers, which has a> similar policy as alpine, and I very rarely have had to pin version for> a package. I think the "partial upgrades are bad and problematic" is> very true for full system installations, but I see it being definitely> at odds with container best practices.
This also applies to every other Alpine installation, even containers.
You still have linked libraries in Alpine containers, which WILL break.
For example, I receive a lot of bug reports, because people are using
containers from docker.io/library/docker, which uses Alpine as a base
system. And they install system Python packages. However because Python
fiddles their own python executable with a different version in the
container, most (all?) system python packages will fail.
That's also why we dont allow system pip package installations.
>> I think it is good if alpine kept using the same practices it is> currently doing, and assume no partial upgrades to keep the maintainer> effort required into reasonable levels. But I also think that the> container community would appreciate a best-effort approach on helping> towards reproducible container builds.
We have a (currently mostly inactive) group which targets bit-per-bit
reproduciblity for Alpine at #alpine-reproducible on OFTC.
>> > Addionally adding support for old versions in our repositories> > doubles (or more?) the disk size of the mirrors which is not> > doable for most mirrors. (we already disable most -dbg packages> > due to size issues).>> I definitely see this cost problem and, while I do not have any idea to> magically solve it, maybe I can try with a suggestion to contain it: If> old versions are provided for specific, non-general use cases> (containers), then perhaps not all mirrors need to support it, and> perhaps a single one (like archlinux's archive) could suffice. People> who are willing to trade reproducibility for potential dependency> breakages in their containers would be able to use it.>> This email came out way longer than what I was planning, but at least> I'm hoping I did a decent job explaining about why we pin version > numbers in the container world and how I hope alpine could support that> use case better.>> - Nad
Hi, thanks againf or your insights! Answering inline below:
On Tue, 2024-11-19 at 16:13 +0100, fossdd wrote:
> That's why we have stable releases. Stable releases only contain> security (and bug) fixes. > I never experienced such behaviour, and I do not agree to it. For> example, pinning versions does not fix security issues.
I think there might be a misunderstanding here. We don't keep pinned
versions forever, or for a long time. The container ecosystem has
toolin that aids with that. An example of this can be:
https://github.com/roobre/renovate-alpine/pull/13
I think getting updates like this is highly beneficial for containers,
for a number of reasons:
- Security. Maintainers *notice* when packages they depend on release
updates and security fixes, and can react by updating and releasing a
new version of their image containing the fix.
- Testing. Some versions break stuff. Receiving updates in PRs allows
for a testing suite to run and ensure the updated dependency still
works. This also applies to new alpine releases, which may contain
breaking changes.
- Reproducibility. What I said earlier, the same container image will
contain the same version. If a maintainer needs to reproduce a problem
that happened one version ago, they can.
> This also applies to every other Alpine installation, even> containers.> You still have linked libraries in Alpine containers, which WILL> break.
I think this is reasonably understood in the container world, while
definitely not on desktop. It is possible that I let versions drift, or
that two packages depend on one another. If that happens, I will adapt
my rules so the bot bumps those dependencies together.
If something breaks badly, the test suite (or even the build) will fail
and I'll know that I need to update the base image first.
That's what I think that retaining old versions for a while (on a
dedicated repo potentially, to save costs and to avoid unaware users
using it) can likely work. There are definitely tradeoffs but I don't
think it's something that people who work with containers daily cannot
expect or work around, with the benefits outweighting that effort.
- N