~alpine/devel

9 6

[alpine-devel] apk-tools work-in-progress

Timo Teras
Details
Message ID
<20141008131721.747fcc9d@vostro>
Sender timestamp
1412763441
DKIM signature
missing
Download raw message
Hi,

It's been awhile since my last spurt on apk-tools. In fact, a bit too
long time. Fortunately, I'm working on apk actively this month.

I earlier today pushed apk-tools-2.4.5 to edge which contains various
bug fixes.

The observant may have also noticed that apk-tools.git already contains
code to use libfetch instead of fork+exec of wget to get http(s)/ftp
URIs. This seems to be working and apparently provides a nice speed
improvement in most cases (due to less forking, having connection
caching/re-use, and using if-modified-since headers).

I'm also planning to fix:
#2251 - option to restart init.d services on package upgrades
#3027 - support xattr (at least to the extent PaX/grsec needs it)

We also had the patch series, various ideas and other suggestions on
the mailing list that would affect compatibility. One to name are the
command line argument parsing patches/suggestions from Dubiousjim,
Isaac Dunham and ncopa. I will probably implement the 'command argument
grouping' stuff. And try to write a follow up mail on suggested new
arguments / applet names based on the old emails on the subject.

And there was also the logging patchset from Jim. I'll probably apply
it mostly as-is - perhaps just making it disabled by default. And allow
it to be enabled by 'touch /etc/apk/logging' or similar.

I'm thinking to get all that in within next week or two. Hopefully
having all to new stuff in time before alpine-3.1 branching.

If you have new patchsets, old ones that are rebased, something I missed
or just other ideas for apk-tools, please reply to this mail.

/Timo


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Jeff Pohlmeyer
Details
Message ID
<CAEuRe+3vca1BiCVCCCHqA6pbXJCRpQt=XhqYY+cztt+MdOtqVg@mail.gmail.com>
In-Reply-To
<20141008131721.747fcc9d@vostro> (view parent)
Sender timestamp
1412777201
DKIM signature
missing
Download raw message
On Wed, Oct 8, 2014 at 5:17 AM, Timo Teras <timo.teras@iki.fi> wrote:

> If you have new patchsets, old ones that are rebased, something I missed
> or just other ideas for apk-tools, please reply to this mail.


This works as expected:

# apk add no-such-package
ERROR: unsatisfiable constraints:
  no-such-package (missing):
    required by: world[no-such-package]

# echo $?
1


But what about this?

# apk del no-such-package
OK: 1264 MiB in 519 packages

# echo $?
0


 - Jeff


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

[alpine-devel] apk-tools command line arguments

Timo Teras
Details
Message ID
<20141008162028.39cc7b66@vostro>
In-Reply-To
<20141008131721.747fcc9d@vostro> (view parent)
Sender timestamp
1412774428
DKIM signature
missing
Download raw message
On Wed, 8 Oct 2014 13:17:21 +0300
Timo Teras <timo.teras@iki.fi> wrote:

> We also had the patch series, various ideas and other suggestions on
> the mailing list that would affect compatibility. One to name are the
> command line argument parsing patches/suggestions from Dubiousjim,
> Isaac Dunham and ncopa. I will probably implement the 'command
> argument grouping' stuff. And try to write a follow up mail on
> suggested new arguments / applet names based on the old emails on the
> subject.

Ok. option groups are now implemented - but the organization needs clean
up still.

So the tricky tasks are:

- potentially merge info and search applets, and unify their output
  (or make it a 'ls' / 'list' and remove info/search)

- unify short options -R/-D/etc for all applets recursive/dependency
  handling policy

Insightful mails from ncopa about this are:
http://lists.alpinelinux.org/alpine-devel/3042.html
http://lists.alpinelinux.org/alpine-devel/3064.html

Need to make a plan about how to do this.


Additional compatibility breaking changes to consider are:

- rename 'index' to 'mkindex'

- potentially make 'apk -U' a shortcut to 'update' 

- remove 'version' applets:
  move default functionality to "upgrade --analyze" or "index --version"
  -I should go to "index --version", and/or generic --version
  -t and -c probably deserve a new 'verstr' applet

And it seems this can be done in simple way now:

- short form for --simulate; -s suggested but overlaps with fetch
  --stdout and info --size currently; though if we move --simulate
  option to commit group it'd work nice

Did I miss something?


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Timo Teras
Details
Message ID
<20141009081254.2a09944c@vostro>
In-Reply-To
<CAEuRe+3vca1BiCVCCCHqA6pbXJCRpQt=XhqYY+cztt+MdOtqVg@mail.gmail.com> (view parent)
Sender timestamp
1412831574
DKIM signature
missing
Download raw message
On Wed, 8 Oct 2014 09:06:41 -0500
Jeff Pohlmeyer <yetanothergeek@gmail.com> wrote:

> On Wed, Oct 8, 2014 at 5:17 AM, Timo Teras <timo.teras@iki.fi> wrote:
> 
> > If you have new patchsets, old ones that are rebased, something I
> > missed or just other ideas for apk-tools, please reply to this mail.
> 
> But what about this?
> 
> # apk del no-such-package
> OK: 1264 MiB in 519 packages
> 
> # echo $?
> 0

Should be simple to fix. Though, should we differentiate the following
scenarios when a package is not removed?

1) no-such-package or package-not-installed are trivial and should flag
error

2) package can be implicitly installed as dependency. apk del does not
do anything with it.

3) package is in 'world' but also a dependency. apk del updates world
but still does not remove the package.

Are these all error conditions, or not?

One option would be to return the number of packages names specified
that were not removed.

Any comments?


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Isaac Dunham
Details
Message ID
<20141010151440.GA1805@newbook>
In-Reply-To
<20141010121028.62d1e3f0@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1412954081
DKIM signature
missing
Download raw message
On Fri, Oct 10, 2014 at 12:10:28PM +0200, Natanael Copa wrote:
> On Fri, 10 Oct 2014 10:32:09 +0200
> Carlo Landmeter <clandmeter@gmail.com> wrote:
> 
> > >
> > >
> > > If you have new patchsets, old ones that are rebased, something I missed
> > > or just other ideas for apk-tools, please reply to this mail.
> > >
> > >
> > One of the missing features in apk-tools (I've had this question multiple
> > times on IRC) is the ability of query packages file list when not installed.
> > I think we have discussed this before, but I though I would just bump it
> > again to see what you think about it.
> > Maybe add an switch to generate an alternative database which we can query
> > after it has been generated locally?
> 
> I have some ideas related new 'package browser'. Basically a server
> with access to the apk repository with a Makefile that will generate a
> $pkgname.filelist file. eg:
> 
> 
> %.filelist: %.apk
> 	tar -ztf $< > $@
> 
> 
> we could also prefix each line with "$pkgname " and finally cat them
> all to a single file and make it available via http so you could
> basically search it with:
> 
>   curl $URL | grep filename
> 
> and we could have a tiny wrapper script too (like we have for sprunge)
> 
> This all could be done without any additional coding in apk-tools.

+1 for this plan.
Debian does something kind of similar; uninstalled packages need
a separate database and program to query it.

> More ideas for package browser (statically generated html files) are:
> 
> We could also fish out the .PKGINFO and generate a $pkgname.meta.yaml
> and generate static html pages for packages.
> 
> It all could be regenerated with mqtt trigger after packages have been
> uploaded to mirror.
> 
> Since we use 'make' for it, only newly uploaded packages will be
> re-generated.

Sounds good.

Isaac Dunham

> -nc
> 
> > 
> > -carlo
> 


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Natanael Copa
Details
Message ID
<20141010091425.571b93b2@ncopa-desktop.alpinelinux.org>
In-Reply-To
<20141009081254.2a09944c@vostro> (view parent)
Sender timestamp
1412925265
DKIM signature
missing
Download raw message
On Thu, 9 Oct 2014 08:12:54 +0300
Timo Teras <timo.teras@iki.fi> wrote:

> On Wed, 8 Oct 2014 09:06:41 -0500
> Jeff Pohlmeyer <yetanothergeek@gmail.com> wrote:
> 
> > On Wed, Oct 8, 2014 at 5:17 AM, Timo Teras <timo.teras@iki.fi> wrote:
> > 
> > > If you have new patchsets, old ones that are rebased, something I
> > > missed or just other ideas for apk-tools, please reply to this mail.
> > 
> > But what about this?
> > 
> > # apk del no-such-package
> > OK: 1264 MiB in 519 packages
> > 
> > # echo $?
> > 0
> 
> Should be simple to fix. Though, should we differentiate the following
> scenarios when a package is not removed?
> 
> 1) no-such-package or package-not-installed are trivial and should flag
> error

I'd like this to be error. You thought it was installed but it was not,
or you spelled wrong something.

> 2) package can be implicitly installed as dependency. apk del does not
> do anything with it.

I'd like this to be error.
 
> 3) package is in 'world' but also a dependency. apk del updates world
> but still does not remove the package.

This is the only candidate for not returning error. I think what would
be nice is if this situation also returns error, unless a flag for
'i-am-just-interested-in-removing-it from-world' was specified.

maybe:

 apk del --world pkg	# returns success if it was removed from
			# world. no error message since the desired
			# operation was sucess.

 apk del pkg		# remove from work but return error. give
			# current error msg

 
> Are these all error conditions, or not?

I'd say yes, all 3 are error conditions, unless if --world is specified
for case #3.

You asked for it to be removed but it was not, so the problem you tried
to solve by removing it is probably still there.

For example, you have a package that is causing problems, lets say a
shell called 'bash' have ever returning sec issues so you want it
remove. The stressed sysadmin types: 'apk del bach' and does not notice
he spelled it wrong. He just sees 'OK' and goes on to next box...

> One option would be to return the number of packages names specified
> that were not removed.

We could also have specific error code for the different causes of
error.

> Any comments?
> 
> 
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
> 



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Carlo Landmeter
Details
Message ID
<CA+cSEmOhmw_cnVWN+OBOcU46Z1_qP9+dMbYVC+NpVqiOhNYN8Q@mail.gmail.com>
In-Reply-To
<20141008131721.747fcc9d@vostro> (view parent)
Sender timestamp
1412929929
DKIM signature
missing
Download raw message
>
>
> If you have new patchsets, old ones that are rebased, something I missed
> or just other ideas for apk-tools, please reply to this mail.
>
>
One of the missing features in apk-tools (I've had this question multiple
times on IRC) is the ability of query packages file list when not installed.
I think we have discussed this before, but I though I would just bump it
again to see what you think about it.
Maybe add an switch to generate an alternative database which we can query
after it has been generated locally?

-carlo
Roger Newman
Details
Message ID
<54379EB2.7000101@riseup.net>
In-Reply-To
<CA+cSEmOhmw_cnVWN+OBOcU46Z1_qP9+dMbYVC+NpVqiOhNYN8Q@mail.gmail.com> (view parent)
Sender timestamp
1412931250
DKIM signature
missing
Download raw message
>>
>>
>> If you have new patchsets, old ones that are rebased, something I missed
>> or just other ideas for apk-tools, please reply to this mail.
>>
>>
> One of the missing features in apk-tools (I've had this question multiple
> times on IRC) is the ability of query packages file list when not installed.
> I think we have discussed this before, but I though I would just bump it
> again to see what you think about it.
> Maybe add an switch to generate an alternative database which we can query
> after it has been generated locally?
> 
> -carlo
> 

I like the idea of getting a list of files without installing the pkg.
Was asking that this morning on IRC.

+1 from my side for that feature.

- Roger


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Timo Teras
Details
Message ID
<20141010114803.121e68fb@vostro>
In-Reply-To
<CA+cSEmOhmw_cnVWN+OBOcU46Z1_qP9+dMbYVC+NpVqiOhNYN8Q@mail.gmail.com> (view parent)
Sender timestamp
1412930883
DKIM signature
missing
Download raw message
On Fri, 10 Oct 2014 10:32:09 +0200
Carlo Landmeter <clandmeter@gmail.com> wrote:

> > If you have new patchsets, old ones that are rebased, something I
> > missed or just other ideas for apk-tools, please reply to this mail.
>
> One of the missing features in apk-tools (I've had this question
> multiple times on IRC) is the ability of query packages file list
> when not installed. I think we have discussed this before, but I
> though I would just bump it again to see what you think about it.
> Maybe add an switch to generate an alternative database which we can
> query after it has been generated locally?

This would be http://bugs.alpinelinux.org/issues/2623

I'm not sure how this could be done nicely using existing code.
Probably the "fetch | tar" pipeline is what I'd end up implementing in
APK.

This does tie up a bit with the 'info' / 'search' and flags
simplification. There should be applet that deals with raw 'packages'
like 'apk verify' does. Printing meta-data from a .apk could be one
option.

So we basically have queries of:
 - installed
 - index
 - package

Now the info in 'index' is a subset of stuff in 'installed'; and
package contains all that is in 'installed' though, it'll be in
different format.

But yes, this should be taken into account when considering changes to
applet names.

Thanks,
Timo


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Natanael Copa
Details
Message ID
<20141010121028.62d1e3f0@ncopa-desktop.alpinelinux.org>
In-Reply-To
<CA+cSEmOhmw_cnVWN+OBOcU46Z1_qP9+dMbYVC+NpVqiOhNYN8Q@mail.gmail.com> (view parent)
Sender timestamp
1412935828
DKIM signature
missing
Download raw message
On Fri, 10 Oct 2014 10:32:09 +0200
Carlo Landmeter <clandmeter@gmail.com> wrote:

> >
> >
> > If you have new patchsets, old ones that are rebased, something I missed
> > or just other ideas for apk-tools, please reply to this mail.
> >
> >
> One of the missing features in apk-tools (I've had this question multiple
> times on IRC) is the ability of query packages file list when not installed.
> I think we have discussed this before, but I though I would just bump it
> again to see what you think about it.
> Maybe add an switch to generate an alternative database which we can query
> after it has been generated locally?

I have some ideas related new 'package browser'. Basically a server
with access to the apk repository with a Makefile that will generate a
$pkgname.filelist file. eg:


%.filelist: %.apk
	tar -ztf $< > $@


we could also prefix each line with "$pkgname " and finally cat them
all to a single file and make it available via http so you could
basically search it with:

  curl $URL | grep filename

and we could have a tiny wrapper script too (like we have for sprunge)

This all could be done without any additional coding in apk-tools.


More ideas for package browser (statically generated html files) are:

We could also fish out the .PKGINFO and generate a $pkgname.meta.yaml
and generate static html pages for packages.

It all could be regenerated with mqtt trigger after packages have been
uploaded to mirror.

Since we use 'make' for it, only newly uploaded packages will be
re-generated.

-nc

> 
> -carlo



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---