Subject: [alpine-devel] apk-tools work-in-progress

From: Timo Teras <[email-masked]>
Date: Wed, 8 Oct 2014 13:17:21 +0300

Hi,

It's been awhile since my last spurt on apk-tools. In fact, a bit too
long time. Fortunately, I'm working on apk actively this month.

I earlier today pushed apk-tools-2.4.5 to edge which contains various
bug fixes.

The observant may have also noticed that apk-tools.git already contains
code to use libfetch instead of fork+exec of wget to get http(s)/ftp
URIs. This seems to be working and apparently provides a nice speed
improvement in most cases (due to less forking, having connection
caching/re-use, and using if-modified-since headers).

I'm also planning to fix:
#2251 - option to restart init.d services on package upgrades
#3027 - support xattr (at least to the extent PaX/grsec needs it)

We also had the patch series, various ideas and other suggestions on
the mailing list that would affect compatibility. One to name are the
command line argument parsing patches/suggestions from Dubiousjim,
Isaac Dunham and ncopa. I will probably implement the 'command argument
grouping' stuff. And try to write a follow up mail on suggested new
arguments / applet names based on the old emails on the subject.

And there was also the logging patchset from Jim. I'll probably apply
it mostly as-is - perhaps just making it disabled by default. And allow
it to be enabled by 'touch /etc/apk/logging' or similar.

I'm thinking to get all that in within next week or two. Hopefully
having all to new stuff in time before alpine-3.1 branching.

If you have new patchsets, old ones that are rebased, something I missed
or just other ideas for apk-tools, please reply to this mail.

/Timo

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Carlo Landmeter <[email-masked]>
Date: Fri, 10 Oct 2014 10:32:09 +0200

> or just other ideas for apk-tools, please reply to this mail.
One of the missing features in apk-tools (I've had this question multiple
times on IRC) is the ability of query packages file list when not installed.
I think we have discussed this before, but I though I would just bump it
again to see what you think about it.
Maybe add an switch to generate an alternative database which we can query
after it has been generated locally?

-carlo

---

From: Timo Teras <[email-masked]>
Date: Fri, 10 Oct 2014 11:48:03 +0300

On Fri, 10 Oct 2014 10:32:09 +0200
Carlo Landmeter <[email-masked]> wrote:

> > If you have new patchsets, old ones that are rebased, something I
> > missed or just other ideas for apk-tools, please reply to this mail.
>
> One of the missing features in apk-tools (I've had this question
> multiple times on IRC) is the ability of query packages file list
> when not installed. I think we have discussed this before, but I
> though I would just bump it again to see what you think about it.
> Maybe add an switch to generate an alternative database which we can
> query after it has been generated locally?

This would be http://bugs.alpinelinux.org/issues/2623

I'm not sure how this could be done nicely using existing code.
Probably the "fetch | tar" pipeline is what I'd end up implementing in
APK.

This does tie up a bit with the 'info' / 'search' and flags
simplification. There should be applet that deals with raw 'packages'
like 'apk verify' does. Printing meta-data from a .apk could be one
option.

So we basically have queries of:
 - installed
 - index
 - package

Now the info in 'index' is a subset of stuff in 'installed'; and
package contains all that is in 'installed' though, it'll be in
different format.

But yes, this should be taken into account when considering changes to
applet names.

Thanks,
Timo

---

---

From: Roger Newman <[email-masked]>
Date: Fri, 10 Oct 2014 10:54:10 +0200

> after it has been generated locally?

I like the idea of getting a list of files without installing the pkg.
Was asking that this morning on IRC.

+1 from my side for that feature.

- Roger

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 10 Oct 2014 12:10:28 +0200

> after it has been generated locally?

I have some ideas related new 'package browser'. Basically a server
with access to the apk repository with a Makefile that will generate a
$pkgname.filelist file. eg:

%.filelist: %.apk
	tar -ztf $< > $@

we could also prefix each line with "$pkgname " and finally cat them
all to a single file and make it available via http so you could
basically search it with:

  curl $URL | grep filename

and we could have a tiny wrapper script too (like we have for sprunge)

This all could be done without any additional coding in apk-tools.

More ideas for package browser (statically generated html files) are:

We could also fish out the .PKGINFO and generate a $pkgname.meta.yaml
and generate static html pages for packages.

It all could be regenerated with mqtt trigger after packages have been
uploaded to mirror.

Since we use 'make' for it, only newly uploaded packages will be
re-generated.

-nc

---

---

From: Isaac Dunham <[email-masked]>
Date: Fri, 10 Oct 2014 08:14:41 -0700

> This all could be done without any additional coding in apk-tools.

+1 for this plan.
Debian does something kind of similar; uninstalled packages need
a separate database and program to query it.

> Since we use 'make' for it, only newly uploaded packages will be

Sounds good.

Isaac Dunham

---

---

From: Timo Teras <[email-masked]>
Date: Wed, 8 Oct 2014 16:20:28 +0300

On Wed, 8 Oct 2014 13:17:21 +0300
Timo Teras <[email-masked]> wrote:

> Isaac Dunham and ncopa. I will probably implement the 'command
> argument grouping' stuff. And try to write a follow up mail on
> suggested new arguments / applet names based on the old emails on the
> subject.

Ok. option groups are now implemented - but the organization needs clean
up still.

So the tricky tasks are:

- potentially merge info and search applets, and unify their output
  (or make it a 'ls' / 'list' and remove info/search)

- unify short options -R/-D/etc for all applets recursive/dependency
  handling policy

Insightful mails from ncopa about this are:
http://lists.alpinelinux.org/alpine-devel/3042.html
http://lists.alpinelinux.org/alpine-devel/3064.html

Need to make a plan about how to do this.

Additional compatibility breaking changes to consider are:

- rename 'index' to 'mkindex'

- potentially make 'apk -U' a shortcut to 'update' 

- remove 'version' applets:
  move default functionality to "upgrade --analyze" or "index --version"
  -I should go to "index --version", and/or generic --version
  -t and -c probably deserve a new 'verstr' applet

And it seems this can be done in simple way now:

- short form for --simulate; -s suggested but overlaps with fetch
  --stdout and info --size currently; though if we move --simulate
  option to commit group it'd work nice

Did I miss something?

---

---

From: Jeff Pohlmeyer <[email-masked]>
Date: Wed, 8 Oct 2014 09:06:41 -0500

> or just other ideas for apk-tools, please reply to this mail.

This works as expected:

# apk add no-such-package
ERROR: unsatisfiable constraints:
  no-such-package (missing):
    required by: world[no-such-package]

# echo $?
1

But what about this?

# apk del no-such-package
OK: 1264 MiB in 519 packages

# echo $?
0

 - Jeff

---

---

From: Timo Teras <[email-masked]>
Date: Thu, 9 Oct 2014 08:12:54 +0300

On Wed, 8 Oct 2014 09:06:41 -0500

> OK: 1264 MiB in 519 packages

Should be simple to fix. Though, should we differentiate the following
scenarios when a package is not removed?

1) no-such-package or package-not-installed are trivial and should flag
error

2) package can be implicitly installed as dependency. apk del does not
do anything with it.

3) package is in 'world' but also a dependency. apk del updates world
but still does not remove the package.

Are these all error conditions, or not?

One option would be to return the number of packages names specified
that were not removed.

Any comments?

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 10 Oct 2014 09:14:25 +0200

On Thu, 9 Oct 2014 08:12:54 +0300

> 1) no-such-package or package-not-installed are trivial and should flag

I'd like this to be error. You thought it was installed but it was not,
or you spelled wrong something.

> 2) package can be implicitly installed as dependency. apk del does not

I'd like this to be error.
 
> but still does not remove the package.

This is the only candidate for not returning error. I think what would
be nice is if this situation also returns error, unless a flag for
'i-am-just-interested-in-removing-it from-world' was specified.

maybe:

 apk del --world pkg	# returns success if it was removed from
			# world. no error message since the desired
			# operation was sucess.

 apk del pkg		# remove from work but return error. give
			# current error msg

> Are these all error conditions, or not?

I'd say yes, all 3 are error conditions, unless if --world is specified
for case #3.

You asked for it to be removed but it was not, so the problem you tried
to solve by removing it is probably still there.

For example, you have a package that is causing problems, lets say a
shell called 'bash' have ever returning sec issues so you want it
remove. The stressed sysadmin types: 'apk del bach' and does not notice
he spelled it wrong. He just sees 'OK' and goes on to next box...

> One option would be to return the number of packages names specified

We could also have specific error code for the different causes of
error.

---