Hi!

I have been thinking of (re)writing or refactor the tool to scan the
aports tree. Currently we have lua-aports which can scan the tree and
list build order of given packages, but the interface is not great.

I would like to have some feedback on how you would like that the
interface should be?


The things that it will need to do that current `ap` can do is:

- list the build order of given packages (ap builddirs)
- recursively print all the dependencies for given packages (ap recursdeps)
- dump all aports to json (ap dump-json)

I am not sure those are needed:

- list the .apk files generated from given aport (ap apk-list)
- give a list of packages that needs to be built (ap build-list. not
  sure if its needed but may be handy)
- print the reverse build time dependencies for a given package (ap
  revdep)

Things that the tools needs to be able to help us to do:

- detect if there are any invalid dependencies (eg. packages in main
  that depends on packages in community or testing)

- detect missing build time dependencies (due to arch being disabled)

- detect circular build time dependencies

- clean source cache (delete downloaded sources older than X that is
  not currently used by any aports)

- implement relgroups (check that all aports with same relgroup has
  same pkgver - this is not in use in aports yet)

- print reverse build deps. If you for example disable/enable a library
  for an architecture, what other packages needs also to be
  disabled/enabled.


Are there other things it needs to be able to do? Maybe parse the
secfixes sections? look for duplicates? verify that upstream sources
are still available? search in makedepends/sources/arch/options? (eg
list all aports that has !check in options, or all aports that has a
given arch explicitly disabled)

This could be implemented either by a single tool similar to current
`ap` or it could be implemented as multiple tools.

How do you think it should work? Any suggestions for what the --help
output should be? Or maybe even a suggestions for a simple manpage.

I'd like to work on this after 3.12 is out, but want to start the
discussion on how it should work.

Thanks!

-nc

On Thu, 2020-05-21 at 15:41 +0200, Natanael Copa wrote:
> I would like to have some feedback on how you would like that the
> interface should be?

One thing I'd really love that tool to have would be more usability for
developers, see below.

> I am not sure those are needed:
> 
> [...]
> - print the reverse build time dependencies for a given package (ap
>   revdep)

That one might be useful for developers to rebuild things on soname
bumps. Right now it's kind of a pain to rebuild everything in the right
order when soname bumps occur, it'd be lovely if the new tool had
something git-bisect-ish. That is if I could tell it "package X
changed, run script Y for all packages that (directly) depend on it".

> Things that the tools needs to be able to help us to do:
> 
> - detect if there are any invalid dependencies (eg. packages in main
>   that depends on packages in community or testing)

I think that is something that CI catches anyway these days - but it
might be useful when demoting a packge from main to community (or
community -> testing).

> - detect missing build time dependencies (due to arch being disabled)

That sounds a bit redundant, you'd notice that when running abuild.

> - detect circular build time dependencies

That would be something that'd be really nice to have for CI so we
don't run into these in the first place.

> - clean source cache (delete downloaded sources older than X that is
>   not currently used by any aports)

That sounds nice, I usually do `rm -rf */*/src` in my aports tree every
month or so when my disk starts filling up once again. It'd be nice if
this could be run automatically somehow so I..err users don't forget to
run this before things start to blow up because there's no space left
in / once again ;)

> - implement relgroups (check that all aports with same relgroup has
>   same pkgver - this is not in use in aports yet)

Ah, that'd for things like sqlite-tcl and sqlite to make sure they're
the same version? That'd be very nice to have for CI.

> - print reverse build deps. If you for example disable/enable a
> library
>   for an architecture, what other packages needs also to be
>   disabled/enabled.

I think it'd be nice if this also had the ability to run a script for
each affected package. Maybe functionality that enables/disables the
affected packages on all arches the affected package is disabled on
would be good to have too (and very helpful when bootstrapping new
arches), or when enabling core packages on a certain arch (e.g. when
Rust was made available for more arches).

Thanks for working on this,

Rasmus

On Thu, 21 May 2020 16:56:31 +0200
Rasmus Thomsen <oss@cogitri.dev> wrote:

> On Thu, 2020-05-21 at 15:41 +0200, Natanael Copa wrote:
> > I would like to have some feedback on how you would like that the
> > interface should be?  
> 
> One thing I'd really love that tool to have would be more usability for
> developers, see below.
> 
> > I am not sure those are needed:
> > 
> > [...]
> > - print the reverse build time dependencies for a given package (ap
> >   revdep)  
> 
> That one might be useful for developers to rebuild things on soname
> bumps. Right now it's kind of a pain to rebuild everything in the right
> order when soname bumps occur, it'd be lovely if the new tool had
> something git-bisect-ish. That is if I could tell it "package X
> changed, run script Y for all packages that (directly) depend on it".

I use `apk list --depends so:libname.so.$old` to generate a list of
packages that needs to be rebuilt, which I feed to a script
'rebuild-since' which will bump pkgrel, rebuild and git commit unless
there are commits since specified commit hash (the commit which has the
ABI breaking library update). `ap builddirs` is used to calculate the
build order.

not sure if there is any value print one level of reverse dependencies.
 
> > Things that the tools needs to be able to help us to do:
> > 
> > - detect if there are any invalid dependencies (eg. packages in main
> >   that depends on packages in community or testing)  
> 
> I think that is something that CI catches anyway these days - but it
> might be useful when demoting a packge from main to community (or
> community -> testing).

May be nice to have the answer of: "if I move this from testing to
community, what other packages needs to also move?"

> > - detect missing build time dependencies (due to arch being
> > disabled)  
> 
> That sounds a bit redundant, you'd notice that when running abuild.

No. You don't. When qt5-qtwebkit was disabled for armhf there was a lot
of packages that broke. It was not detected by abuild until next
rebuild of those. Would be nice to solve that at the same time as it
gets disabled rather than wait til the builders fails during next
upgrade.

> > - detect circular build time dependencies  
> 
> That would be something that'd be really nice to have for CI so we
> don't run into these in the first place.

exactly.

> 
> > - clean source cache (delete downloaded sources older than X that is
> >   not currently used by any aports)  
> 
> That sounds nice, I usually do `rm -rf */*/src` in my aports tree
> every month or so when my disk starts filling up once again. It'd be
> nice if this could be run automatically somehow so I..err users don't
> forget to run this before things start to blow up because there's no
> space left in / once again ;)

This is not for the */*/src. Its for /var/cache/distfiles. If you have those files in /var/cache/distfiles, which ones are candidates to delete?

strongswan-5.8.1.tar.bz2
strongswan-5.8.4.tar.bz2
subversion-1.12.2.tar.bz2
subversion-1.13.0.tar.bz2
sudo-1.8.23.tar.gz
sudo-1.8.31.tar.gz
sweeper-20.04.0.tar.xz
swig-3.0.12.tar.gz
swig-4.0.1.tar.gz
...


You could either look for duplicates and have some nasty algorithm that
finds the latest version and delete the oldest, or you could whitelist
all the remote sources specified in the aports and delete the rest.

> > - implement relgroups (check that all aports with same relgroup has
> >   same pkgver - this is not in use in aports yet)  
> 
> Ah, that'd for things like sqlite-tcl and sqlite to make sure they're
> the same version? That'd be very nice to have for CI.

exactly. also for kernel 3rd party modules, poppler and poppler-qt and
similar. There are various packages that needs this.
> 
> > - print reverse build deps. If you for example disable/enable a
> > library
> >   for an architecture, what other packages needs also to be
> >   disabled/enabled.  
> 
> I think it'd be nice if this also had the ability to run a script for
> each affected package. Maybe functionality that enables/disables the
> affected packages on all arches the affected package is disabled on
> would be good to have too (and very helpful when bootstrapping new
> arches), or when enabling core packages on a certain arch (e.g. when
> Rust was made available for more arches).

Good idea. I guess that is what you meant with git-bisec-ish, that you
can pass a script that is executed for each. That is something we may
want have to most of the operations. Alternatively you can `... | xargs
./myscript.sh ` so you get the full list and can loop over the args.

> 
> Thanks for working on this,
> 
> Rasmus
>