Date: Thu, 23 Jan 2020 18:49:34 +0200
From: Timo Teras <timo.teras@iki.fi>
To: "Drew DeVault" <sir@cmpwn.com>
Cc: "Ariadne Conill" <ariadne@dereferenced.org>,
 <~alpine/devel@lists.alpinelinux.org>
Subject: Re: Lets talk about apk-tools 3, and apk-tools in 2020 in general
Message-ID: <20200123184934.0009c19a@vostro>
In-Reply-To: <C03A5J4BZ2TN.2VO1P2UDDOI6Q@homura>
References: <1c4796e0cda2248c2de159d4d467421c@dereferenced.org>
	<C03A5J4BZ2TN.2VO1P2UDDOI6Q@homura>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Thu, 23 Jan 2020 10:36:10 -0500
"Drew DeVault" <sir@cmpwn.com> wrote:

> Thanks for writing this up, Ariadne. I agree with everything you said.
> I'd like to call attention to a few points:
> 
> On Thu Jan 23, 2020 at 3:13 PM, Ariadne Conill wrote:
> > * Changing the format in such a radical way brings significant
> > risk. The tar streams code has already been audited and a
> > few CVEs have been fixed over the years. Throwing that out
> > means we start over again, possibly reintroducing variations
> > of bugs we have already fixed. Many stakeholders have said
> > privately that they would rather not have exposure to this
> > risk and would prefer a more conservative approach.  
> 
> We ought to be careful to avoid fixing things which aren't broken,
> noting that unoptimal != broken. I do NOT see APKv3 as an opportunity
> to scratch a bunch of itches. Let's keep the changes as small as
> possible and avoid throwing out lots of code, or introducing lots of
> new code.

The current index format needs to go. It has become broken by design.
Rather than add bandaid. I'd like to fix it properly at this point. It
may need lot of code (and a lot of additional tests in the testsuite).
That's software development.

> We should structure these changes as slow, incremental improvements,
> where one change can be built and shipped and then fully battle tested
> before the next one lands. apk is used in many production systems
> today and has production-tier expectations of stability. The
> importance of a conservative approach is probably the most relevant
> take-away from this discussion.

One option to consider then is to add increased transition time.
Perhaps building both package/index formats for a time to allow the new
formats to gain time proven merits. Of course all non-Alpine users are
free to make their own transition schedule, or choose to not upgrade,
or to migrate something else they prefer instead.

> > * Usage of a unified container format for package data and
> > database data removes transparency from the current package
> > format. Right now, an APK package can be manipulated with
> > the tar command if a user wishes to know its contents. Using
> > the package manager is not even required.  
> 
> Just to lend validation this point: I use /bin/tar to study apk files
> all the time. Tar beats homegrown 10 times out of 10.

We can do "apk2tar" or similar applet to give yous till tars. The
benefit here is that the above would check signatures etc. Just using
tar on raw .apk is not secure as it does not check the signatures.

> > There are other changes that people are concerned about, such
> > as being able to compose new repositories from pre-existing
> > ones. While those are important discussions to have as well,
> > we are not really discussing them here, as those concerns can
> > easily be overcome. Overall governance of the apk-tools
> > project itself is also not necessarily being discussed here,
> > while we need to have that discussion as there are many
> > non-Alpine stakeholders at this point, we can do that later.  
> 
> I was one of the people bringing these concerns forward. I think that
> tightening the scope and focusing on reuse of existing designs and
> code makes this easier - if we can avoid overhauling the container
> format then we don't have to bikeshed about what goes into it.

This is something we can talk about. But I'm not sure what benefit it
gives. Because even if we keep TAR format, APK will not honor the tar
headers anymore - it would use the database blob for all file metadata.
And the metadata needs to go to the database so we can get strong secure
auditing via system database.

The main problem is that in TAR files the file metadata, directory
structure and file checksums are scattered and not easily signable.
This needs to change to support the increased security requirements we
have for apk.

> Finally, in terms of general principles, I want to emphasize that we
> shouldn't turn ourselves into paperclip maximizers. Performance isn't
> the singluar metric we need to optimize - maintainability,
> reliability, and usability are all _more_ important. Any
> optimizations which cannot be proven to directly improve known
> bottlenecks in apk are a NACK from me. apk is already one of the
> fastest, if not _the_ fastest, package managers around.

We have bugs saying otherwise. And I personally think the
index parsing performance is not enough on embedded. But what comes to
package installation speeds it's very good currently. The suggestion to
dropping TAR format is not due to performance, but due to security
concerns.

Timo