MIME-Version: 1.0
References: <20191203180717.0016af8a@vostro> <20191229181150.7a0dcace@vostro>
In-Reply-To: <20191229181150.7a0dcace@vostro>
From: Ariadne Conill <ariadne@dereferenced.org>
Date: Sun, 29 Dec 2019 17:03:22 -0600
Message-ID: <CAAOiGNz4ZqpCVX6GODPta84nYDvuc1J83VE1_mYTbeKkQooEvw@mail.gmail.com>
Subject: Re: apk-tools plans
To: Timo Teras <timo.teras@iki.fi>
Cc: ~alpine/devel@lists.alpinelinux.org
Content-Type: text/plain; charset="UTF-8"

Hello,

On Sun, Dec 29, 2019 at 10:13 AM Timo Teras <timo.teras@iki.fi> wrote:
>
> Hi,
>
> On Tue, 3 Dec 2019 18:07:17 +0200
> Timo Teras <timo.teras@iki.fi> wrote:
>
> > Another thing, I really want to improve is the security of 'apk audit'
> > and the system integrity checking. The concept is to create and store
> > signed file manifests in the DB that can be used to establish strong
> > trust in a system. (The current 'apk audit' was designed for 'lbu
> > commit'.)
> >
> > Slightly related is also changing the file formats so that signature
> > checking can be done first without much parsing to make the attack
> > surface smaller. (The old design was motivated by restrictions from
> > the original shell script based Alpine package manager; the signatures
> > were not considered as first class citizen back then.)
> >[snip]
> > Unfortunately the above changes cannot be fixed easily without
> > changing to binary file formats.
> >
> > The primary target for me is to redo the binary apk package and index
> > formats for next Alpine release. We can discuss the exact details in
> > this or another thread later in the coming weeks. Main design being
> > security and speed. The idea is that index will be mmap:able; and the
> > structures from .apk can be directly copied to installed-db.
>
> So I've made quite a bit progress with this. The work-in-progress
> branch is now available at:
> https://gitlab.alpinelinux.org/alpine/apk-tools/tree/v3.0-wip
>
> This is still early preview code. The intent is to show at this point
> on how I'm planning the file format to be like and how the signing is
> to work. There's still open ends and the "schemas" are still volatile.
>
> So far the code contains the basic encoding and decoding of the file
> format as well as signing and verification of it. It's made pretty
> generic and the signing can already do RSA/ECC/etc based on what type of
> private key is given.

This is a great improvement over what we have in 2.10, and I can't
wait for it to land in Alpine.  However, I think we should continue to
support apk-tools 2.10 for the spring release, just to make sure the
new design is fully proven, as it is a complete redesign of things.

> From technical point of view the format is first a container layer
> basically Tag-Length-Value blobs. The main blocks are to be the
> "database", "signatures", and for packages the "files" section.
>
> The "database" section is mostly resembling flat buffers format.
> Basically it's a hierarchical object tree. The intent is to have enough
> information to make deep copies without schema, but to pretty-print it
> you'd need the schema. This is a trade of chosen to keep the repetitive
> field type encodings out, but to allow some generic functionality to be
> written without schema knowledge.
>
> As said, the main motivate for this work is to allow mmap() access, and
> fairly trivial signature verification code. Additionally the format is
> designed so that the packages' signed data dump can be just trivially
> copied into the installed database. So there will be support to copy
> "signed database" blobs to be inside another database. This is the key
> to strong audit trail so that installed-db is just copy of these blobs.
> And to do all this without much parsing so that accesses are fast
> enough.
>
> I will be working to finish up the schemas of the "package", "index"
> and "installed database" formats next. And then start working on the
> new format tools to create packages.
>
> Plan is to move from 'abuild' to 'apk-tools' the intelligence to
> construct packages, and manage repositories. So in future 'abuild'
> would just call 'apk mkpkg' or similar with a description file what
> needs to go in. This should be helpful if some other distributions
> choose to use apk but want to integrate it to their build scripts.

I suspect that this will be useful to distributions outside the
Alpine/abuild ecosystem.  There's a few distributions which want to
use APK, but want to build their own packages with their own tools.

> Additionally, we are hoping to put all the repository management code
> to apk-tools. There it will be more simpler to implement features such
> as "keep superceded packages in repository for at least X days before
> deleting them".
>
> Feedback welcome. Though, better to concentrate to the architectural
> and overview of how things - instead of nitpicking style and/or minor
> issues on the commit as it's still pretty volatile.

I'll go over it all this week, but I think it pretty much is in
agreement with what we have discussed in IRC in the past.

Ariadne