Date: Fri, 24 Jan 2020 12:21:02 +0100
From: Natanael Copa <ncopa@alpinelinux.org>
To: Timo Teras <timo.teras@iki.fi>
Cc: "Drew DeVault" <sir@cmpwn.com>, "Ariadne Conill"
 <ariadne@dereferenced.org>, <~alpine/devel@lists.alpinelinux.org>
Subject: Re: Lets talk about apk-tools 3, and apk-tools in 2020 in general
Message-ID: <20200124122102.5a92a498@ncopa-desktop.copa.dup.pw>
In-Reply-To: <20200123184934.0009c19a@vostro>
References: <1c4796e0cda2248c2de159d4d467421c@dereferenced.org>
	<C03A5J4BZ2TN.2VO1P2UDDOI6Q@homura>
	<20200123184934.0009c19a@vostro>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Thu, 23 Jan 2020 18:49:34 +0200
Timo Teras <timo.teras@iki.fi> wrote:

> On Thu, 23 Jan 2020 10:36:10 -0500
> "Drew DeVault" <sir@cmpwn.com> wrote:
> 
> > Thanks for writing this up, Ariadne. I agree with everything you said.
> > I'd like to call attention to a few points:
> > 
> > On Thu Jan 23, 2020 at 3:13 PM, Ariadne Conill wrote:  
> > > * Changing the format in such a radical way brings significant
> > > risk. The tar streams code has already been audited and a
> > > few CVEs have been fixed over the years. Throwing that out
> > > means we start over again, possibly reintroducing variations
> > > of bugs we have already fixed. Many stakeholders have said
> > > privately that they would rather not have exposure to this
> > > risk and would prefer a more conservative approach.    
> > 
> > We ought to be careful to avoid fixing things which aren't broken,
> > noting that unoptimal != broken. I do NOT see APKv3 as an opportunity
> > to scratch a bunch of itches. Let's keep the changes as small as
> > possible and avoid throwing out lots of code, or introducing lots of
> > new code.  
> 
> The current index format needs to go. It has become broken by design.
> Rather than add bandaid. I'd like to fix it properly at this point. It
> may need lot of code (and a lot of additional tests in the testsuite).
> That's software development.
> 
> > We should structure these changes as slow, incremental improvements,
> > where one change can be built and shipped and then fully battle tested
> > before the next one lands. apk is used in many production systems
> > today and has production-tier expectations of stability. The
> > importance of a conservative approach is probably the most relevant
> > take-away from this discussion.  
> 
> One option to consider then is to add increased transition time.
> Perhaps building both package/index formats for a time to allow the new
> formats to gain time proven merits. Of course all non-Alpine users are
> free to make their own transition schedule, or choose to not upgrade,
> or to migrate something else they prefer instead.

I guess the biggest challenge is when we change package format and edge
builders start build new format. How do we do that? Should we rebuild
everything? Or should we have the old tar format mixed with the new? do
we generate double, both old and new?

When we set up the builders for the new stable release we will build
everything from scratch, so at one point we will have a stable release
with all packages in new format. Question is what we do with edge.

I don't think we need to answer this now, but it is something we need
to plan for. (Which I think was Ariadnes point here)

> > > * Usage of a unified container format for package data and
> > > database data removes transparency from the current package
> > > format. Right now, an APK package can be manipulated with
> > > the tar command if a user wishes to know its contents. Using
> > > the package manager is not even required.    
> > 
> > Just to lend validation this point: I use /bin/tar to study apk files
> > all the time. Tar beats homegrown 10 times out of 10.  
> 
> We can do "apk2tar" or similar applet to give yous till tars. The
> benefit here is that the above would check signatures etc. Just using
> tar on raw .apk is not secure as it does not check the signatures.

I admit that I like the raw tar format. It is very handy, but I
understand why it needs to go away (it took me a while). We will need a
convenient multiplatform way to extract contents or list contents in
apk files. 

a tool that can do a raw extraction, like `apk2tar` or extract-apk or
whatever will be sufficient. I think it would be nice if it was a
single file, either statically compiled (inclusive for MacOS and maybe
windows), or it could be a single file python script or similar.

In any case, I think its a problem that can be solved.

... 

> We have bugs saying otherwise. And I personally think the
> index parsing performance is not enough on embedded. But what comes to
> package installation speeds it's very good currently. The suggestion to
> dropping TAR format is not due to performance, but due to security
> concerns.

Also, as Ariadne pointed out. Even if its fast enough for now, we need
it to scale for the future, and I think it is good that we address this
before it becomes a problem on more than embedded.

Thank you for working on this!

-nc