Date: Tue, 3 Dec 2019 18:07:17 +0200
From: Timo Teras <timo.teras@iki.fi>
To: ~alpine/devel@lists.alpinelinux.org
Subject: apk-tools plans
Message-ID: <20191203180717.0016af8a@vostro>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hi Alpiners,

I am writing to let you know, and discuss here some items related to
apk-tools development.

For many years already, we have had discussion in the IRC on certain
apk-tools related design changes, and feature improvements. I have
already twice or so started working on some of it, yet there's been
always something else that came up. But I'm finally committing to doing
these. Hopefully, we already get some of the changes done by late
February / early March, and get next spring Alpine release to use them.

As background.. it has come obvious that the Alpine package base has
increased a lot. And low end embedded systems are starting to struggle
with the amount of packages in our DB. There's also a number of bugs
filed related to this. Majority of the apk-tools startup time goes to
just uncompressing and parsing the index files and installed db.

Another thing, I really want to improve is the security of 'apk audit'
and the system integrity checking. The concept is to create and store
signed file manifests in the DB that can be used to establish strong
trust in a system. (The current 'apk audit' was designed for 'lbu
commit'.)

Slightly related is also changing the file formats so that signature
checking can be done first without much parsing to make the attack
surface smaller. (The old design was motivated by restrictions from
the original shell script based Alpine package manager; the signatures
were not considered as first class citizen back then.)

In connection with the work, I'm hoping also to do additional minor
improvements such as implementing some multicore support, and
supporting better handling of duplicate files in simultaneously
installed packages.

Unfortunately the above changes cannot be fixed easily without changing
to binary file formats.

The primary target for me is to redo the binary apk package and index
formats for next Alpine release. We can discuss the exact details in
this or another thread later in the coming weeks. Main design being
security and speed. The idea is that index will be mmap:able; and the
structures from .apk can be directly copied to installed-db.

For later features there's ideas such as supporting TPM or hostkey
signed system DB (e.g. 'world', set of config options, trusted
package keys, etc.). So the whole system state would be signed and
validateable.

Due to packages and indexes being binary formats in future, the plan is
to build into apk the feature to create these files. So some of the
related functionality from 'abuild' would be moved to apk. This also
simplifies things if apk-tools is used in other distributions.

Please, feel free to comment or present additional questions regarding
the concept or roadmap. I'll publish more technical details of the
formats later. And try to publish my work-in-progress git tree of apk
sooner than later.

For those needing/wanting to parse the old style text databases, we
could add applets to dump current state in the old style format.

Cheers,
Timo