Date: Sun, 29 Dec 2019 18:11:50 +0200
From: Timo Teras <timo.teras@iki.fi>
To: ~alpine/devel@lists.alpinelinux.org
Subject: Re: apk-tools plans
Message-ID: <20191229181150.7a0dcace@vostro>
In-Reply-To: <20191203180717.0016af8a@vostro>
References: <20191203180717.0016af8a@vostro>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable

Hi,

On Tue, 3 Dec 2019 18:07:17 +0200
Timo Teras <timo.teras@iki.fi> wrote:

> Another thing, I really want to improve is the security of 'apk audit'
> and the system integrity checking. The concept is to create and store
> signed file manifests in the DB that can be used to establish strong
> trust in a system. (The current 'apk audit' was designed for 'lbu
> commit'.)
>=20
> Slightly related is also changing the file formats so that signature
> checking can be done first without much parsing to make the attack
> surface smaller. (The old design was motivated by restrictions from
> the original shell script based Alpine package manager; the signatures
> were not considered as first class citizen back then.)
>[snip]
> Unfortunately the above changes cannot be fixed easily without
> changing to binary file formats.
>=20
> The primary target for me is to redo the binary apk package and index
> formats for next Alpine release. We can discuss the exact details in
> this or another thread later in the coming weeks. Main design being
> security and speed. The idea is that index will be mmap:able; and the
> structures from .apk can be directly copied to installed-db.

So I've made quite a bit progress with this. The work-in-progress
branch is now available at:
https://gitlab.alpinelinux.org/alpine/apk-tools/tree/v3.0-wip

This is still early preview code. The intent is to show at this point
on how I'm planning the file format to be like and how the signing is
to work. There's still open ends and the "schemas" are still volatile.

So far the code contains the basic encoding and decoding of the file
format as well as signing and verification of it. It's made pretty
generic and the signing can already do RSA/ECC/etc based on what type of
private key is given.

=46rom technical point of view the format is first a container layer
basically Tag-Length-Value blobs. The main blocks are to be the
"database", "signatures", and for packages the "files" section.

The "database" section is mostly resembling flat buffers format.
Basically it's a hierarchical object tree. The intent is to have enough
information to make deep copies without schema, but to pretty-print it
you'd need the schema. This is a trade of chosen to keep the repetitive
field type encodings out, but to allow some generic functionality to be
written without schema knowledge.

As said, the main motivate for this work is to allow mmap() access, and
fairly trivial signature verification code. Additionally the format is
designed so that the packages' signed data dump can be just trivially
copied into the installed database. So there will be support to copy
"signed database" blobs to be inside another database. This is the key
to strong audit trail so that installed-db is just copy of these blobs.
And to do all this without much parsing so that accesses are fast
enough.

I will be working to finish up the schemas of the "package", "index"
and "installed database" formats next. And then start working on the
new format tools to create packages.

Plan is to move from 'abuild' to 'apk-tools' the intelligence to
construct packages, and manage repositories. So in future 'abuild'
would just call 'apk mkpkg' or similar with a description file what
needs to go in. This should be helpful if some other distributions
choose to use apk but want to integrate it to their build scripts.

Additionally, we are hoping to put all the repository management code
to apk-tools. There it will be more simpler to implement features such
as "keep superceded packages in repository for at least X days before
deleting them".

Feedback welcome. Though, better to concentrate to the architectural
and overview of how things - instead of nitpicking style and/or minor
issues on the commit as it's still pretty volatile.

Timo