MIME-Version: 1.0
References: <20191230145542.1a7ca9cf@vostro>
In-Reply-To: <20191230145542.1a7ca9cf@vostro>
From: Ariadne Conill <ariadne@dereferenced.org>
Date: Mon, 30 Dec 2019 14:47:38 -0600
Message-ID: <CAAOiGNz+TN0nvWMB5YLnvFVk4jEGxN3s8VbnzJNGV_UU8FNSfw@mail.gmail.com>
Subject: Re: new package format and repository layout changes
To: Timo Teras <timo.teras@iki.fi>
Cc: ~alpine/devel@lists.alpinelinux.org
Content-Type: text/plain; charset="UTF-8"

Hello,

On Mon, Dec 30, 2019 at 6:56 AM Timo Teras <timo.teras@iki.fi> wrote:
>
> Hi all,
>
> I am currently going through the list of data that goes to a package
> and a repository index, as well as the installed db. And trying to
> draft the first schema version of what data goes where. So I am now
> having some issues I'd like to discuss here.
>
> 1. Repository pinning
>
> There's one fundamental issue in the current installed-db that causes
> pain - especially when we do strong signing. This is how the package
> pinning is done (the "@edge" tagging to enable specific repositories
> for specific dependencies only).
>
> Main problem is that the package origin repository needs to be tracked
> for detecting pinning changes and it's not in the package meta data
> currently. The current workaround is to have the origin tag in
> installed-db which means there's data that cannot be signed ahead of
> time. There's also some other subtle issues.
>
> My thinking is to start putting the repository meta data (distribution
> name, branch, component etc.) in the package. This way the package
> origin is known and signed.

The problem with this is that it makes it difficult to pack a new repo
with apk fetch.  I would prefer to retain that capability, as I need
it for work.  While it is possible that we could repack the fetched
packages with new metadata, it seems wasteful to me.

> 2. repositories list format
>
> If the above happens, we might need to do some changes how the tags are
> specified.
>
> There was some discussion earlier if we should support more debian
> style definition of listing the distro repositories. E.g.
>   http://dl-cdn.alpinelinux.org/alpine edge main community

I think this is only worth it if we add support for multiple types of
repository (for example deb-src).  Otherwise, we should keep things
simple.

>
> Where the first word is the base URL (or perhaps even some $MIRROR
> variable). The second word the distribution branch. And remaining words
> would be the list of enabled repositories.
>
> I think the package naming could then be:
>   $base_url/$branch/$repo/$arch/$pkgname-$pkgrel.$uniqueid.$arch.apk
> and automatically constructed from the package metadata.

I don't follow.  There's no guarantee that a generated URI will
actually point to a package that still exists.  Packages are added and
removed from repos all the time.

> (Also wondering if the $uniqueid should be just random generated uuid,
> or some sort of hash calculated from the package metadata and contents.
> The requirement is that it can be used to identify if two packages are
> the same or not.)

If we really need a uniqueid (I'm skeptical), then I would suggest
using a truncated hash for it, or perhaps simply a CRC32 or similar.

> 3. 'noarch' handling
>
> When implementing the above, I would finally like to properly implement
> the 'noarch'. Currently the sources set 'noarch' and build subpackage
> properly. But they are put to the target architecture's storage and
> when creating index the arch is rewritten to the target arch always.
> The plan is to start creating real 'noarch' repository and put the
> built package there. I'm wonder if we'd put separate index there, or
> include the noarch packages also in the target arch index.

We should use a separate index for noarch.  I would prefer to see it
work in a way where we could move Alpine to being a fully multi-arch
distro in the future, where we can use qemu-user to run binaries for
other archs.  This mostly solves cross-compiling in a clean way, too.

> 4. version handling
>
> Sort of unrelated, but something I'd like to also bring up once again.
> Since now that if we do proper distribution / branch tracking. And the
> package downgrades happen at times. I'm wondering if we should make the
> package version "informative" only. And use the build_time to decide
> which package is the "preferred version". In most cases it is the
> latest built package from the repository we want to be using.

If a package (or admin) declares a specific version dependency, we
should prefer it unless we supply --available.  A problem with using
build_time as the automatic preference occurs in the case where a user
mixes repositories without correctly pinning them.  In the present
case, the highest version matching the declared dependencies will
always be preferred.  If we switch to build_time, this case may result
in versions being mixed if a security update happens in an old release
that isn't simply a version bump.

> Alternative would be introduce some sort of concept similar to
> debian/pacman package "epoch".
>
> Though another way to look at it that the buildtime is the
> automatically generated epoch number. :)

I suspect the epoch concept is actually *not* the right way to go,
which is why I have never been in favor of it.  I suspect what we need
to do is have repository weighting, and use the weightings to
determine which version should be selected instead of blindly taking
the highest version.  That way we can say edge has highest preference,
but 3.11 has moderate preference, when calculating the upgrade
transaction(s).

Ariadne