I'm pretty familiar with Arch's makepkg, which seems to be the
basis/inspiration of Alpine's abuild. And I've read the (thin)
documentation available on abuild. Despite this, though, there was a lot
of specific details about abuild's operation that I was unclear on. I
figured the most effective way to answer these questions was to
familiarize myself with the source, so that's what I'm doing. Have about
1/2 of it digested at this point.
Now I'm already a lot clearer on the relation between `abuild build
package`, `abuild rootpkg`, and plain `abuild`, and so on. I'll expand the wiki
documentation of abuild when I have a chance.
I also found some quirks and questionable things in the source, which I
will also pass on when I've collected my thoughts.
But there is one large design issue that had me puzzled. It's the
treatment of the repo tree underneath ~/.cache/abuild versus the package
tree underneath REPODEST. If REPODEST is unset, then PKGDEST defaults to
$startdir---the directory holding the APKGBUILD you're building---and
the treatment of ~/.cache/abuild in this case makes perfect sense. It's
a tree holding a bunch of links to the actual packages, where they sit
interspersed in the aports tree, and ~/.cache/abuild additionally holds
APKINDEX files that are automatically updated and signed when one does
plain `abuild`. This is just the natural behavior I'd want or expect.
However, if I've set REPODEST, then packages don't stay in the aports
tree, but rather get moved to the tree underneath REPODEST. However,
~/.cache/abuild is handled the same way: links to the packages
underneath REPODEST, plus actual APKINDEX files that are automatically
updated. Nothing is done to update APKINDEX files that may or may not
exist under REPODEST.
Now, when one needs to install makedepends when abuilding, both the
repos listed in your /etc/apk/repositories AND the ~/.cache/aports
repository are consulted. However, when just using plain `apk add ...`,
only the former is consulted. So if you want apk to also use your local
custom packages, you should include a local repo path in your
/etc/apk/repositories. But that repo need to have APKINDEX files that
are refreshed when packages are added. One _could_ just add
/home/YOU/.cache/abuild/* here. I expect that would work fine. (I expect
you need to expand the glob manually though.) But then what's the point
of having abuild move all your packages under REPODEST? If one instead
adds $REPODEST/* to /etc/apk/repositories (again, expanding the
pathnames by hand), you'd have a repository with no indexes; or indexes
that you had to manually update before using.
What I've done instead is this:
* add $REPODEST/* to /etc/apk/repositories (In my case, these come
before the official repositories, so that my custom builds are
used preferentially whenever they are present. I also have the
official repositories listed as "@main ..." and "@testing ..."
entries, so that I can override my custom builds when needed.)
* REPODEST is set in /etc/abuild.conf to point to a tree. At
$REPODEST/{main,testing,local}/$CARCH I have symlinks pointing
to ~/.cache/abuild/{main,testing,local}/$CARCH/APKINDEX.tar.gz
This tree then gets populated with real package files when I
use abuild.
* ~/.cache/abuild/{main,testing,local}/$CARCH is as described above.
This tree gets populated with symlinks to the real package files
and with real and up-to-date APKINDEX files when I use abuild.
* The "local" repository is a new folder I created in aports. I have
a private git branch of the aports repo that tracks that new
folder, as well as local changes I make to package scripts under
main and testing.
This setup seems to fit decently with the existing behavior and design
of abuild and the aports repo. I wish less trial-and-error had been
needed to arrive at it, but as I said, I can work on improving the
existing documentation. (Also, I didn't come here asking questions; I
was trying to figure out what I could on my own.)
But it is puzzling how REPODEST has real package files but has to have
symlinks to APKINDEX files under ~/.cache/abuild, whereas the latter has
real APKINDEX files but symlinks to package files under REPODEST. I get
the impression that abuild wasn't designed with the use of REPODEST
centrally in mind? Regardless, is there really some good reason now to
have separate package trees under REPODEST and ~/.cache/abuild?
Here is an alternative, just off the top of my head:
* if REPODEST is unset, handle ~/.cache/abuild as presently.
* if REPODEST is set and ~/.cache/abuild doesn't exist, then
use REPODEST in place of ~/.cache/abuild
* if REPODEST is set and ~/.cache/abuild exists (and points to a
separate location than REPODEST), handle ~/.cache/abuild as
presently.
* if REPODEST is set and ~/.cache/abuild points to it (either is a
symlink to REPODEST, or REPODEST resolves to ~/.cache/abuild),
then use REPODEST in place of ~/.cache/abuild. (Instead of trying
to place the packages under REPODEST and symlinks to them in what
is ultimately the same location.)
--
Dubiousjim
dubiousjim@gmail.com
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, 27 Nov 2012 08:28:11 -0500
Dubiousjim <lists+alpine-devel@jimpryor.net> wrote:
> I'm pretty familiar with Arch's makepkg, which seems to be the> basis/inspiration of Alpine's abuild.
I just looked at the first git commit of abuild:
+# script to build apk packages (light version og makepkg)
So yes. makepkg was inspiration. At that point we thought of using
pacman, or at least beeing compatible with pacman. We gave that up.
> And I've read the (thin)> documentation available on abuild. Despite this, though, there was a lot> of specific details about abuild's operation that I was unclear on. I> figured the most effective way to answer these questions was to> familiarize myself with the source, so that's what I'm doing. Have about> 1/2 of it digested at this point.
Sorry for not beeing good at documenting things...
> Now I'm already a lot clearer on the relation between `abuild build> package`, `abuild rootpkg`, and plain `abuild`, and so on. I'll expand the wiki> documentation of abuild when I have a chance.
That'd be great!
> I also found some quirks and questionable things in the source, which I> will also pass on when I've collected my thoughts.> > But there is one large design issue that had me puzzled. It's the> treatment of the repo tree underneath ~/.cache/abuild versus the package> tree underneath REPODEST.
Congrats! You have found one of my hidden skeletons :)
> If REPODEST is unset, then PKGDEST defaults to> $startdir---the directory holding the APKGBUILD you're building---and> the treatment of ~/.cache/abuild in this case makes perfect sense. It's> a tree holding a bunch of links to the actual packages, where they sit> interspersed in the aports tree, and ~/.cache/abuild additionally holds> APKINDEX files that are automatically updated and signed when one does> plain `abuild`. This is just the natural behavior I'd want or expect.> > However, if I've set REPODEST, then packages don't stay in the aports> tree, but rather get moved to the tree underneath REPODEST. However,> ~/.cache/abuild is handled the same way: links to the packages> underneath REPODEST, plus actual APKINDEX files that are automatically> updated. Nothing is done to update APKINDEX files that may or may not> exist under REPODEST.> Now, when one needs to install makedepends when abuilding, both the> repos listed in your /etc/apk/repositories AND the ~/.cache/aports> repository are consulted. However, when just using plain `apk add ...`,> only the former is consulted. So if you want apk to also use your local> custom packages, you should include a local repo path in your> /etc/apk/repositories. But that repo need to have APKINDEX files that> are refreshed when packages are added. One _could_ just add> /home/YOU/.cache/abuild/* here. I expect that would work fine. (I expect> you need to expand the glob manually though.)
That is actually what I do on my development boxes.
> But then what's the point> of having abuild move all your packages under REPODEST? If one instead> adds $REPODEST/* to /etc/apk/repositories (again, expanding the> pathnames by hand), you'd have a repository with no indexes; or indexes> that you had to manually update before using.
I ran a git log and searched for REPODEST. Last time that things that
had to do with REPODEST was around april 2011. What happened then was
that apk-tools got support for multi arch when we ported Alpine Linux
to x86_64. At that time I only wanted the arch stuff work.
Next time REPODEST is touched is Oct 24 2009, and this is only the
usage help text.
Next time again is a rename of abuild to abuild.in, May 8, 2009.
Next time is Feb 11, 2009, which was last time REPODEST appeared in
commit message. This was the last time I did anything that had to do
with REPODEST. First commit was Oct 18, 2008. In other words, the
REPODEST stuff was onlhy thought of early in the development process,
and then forgotten - or at least not though of at all.
The ~/.cache/abuild symlinks came later.
> What I've done instead is this:> > * add $REPODEST/* to /etc/apk/repositories (In my case, these come> before the official repositories, so that my custom builds are> used preferentially whenever they are present. I also have the> official repositories listed as "@main ..." and "@testing ..."> entries, so that I can override my custom builds when needed.)> > * REPODEST is set in /etc/abuild.conf to point to a tree. At > $REPODEST/{main,testing,local}/$CARCH I have symlinks pointing> to ~/.cache/abuild/{main,testing,local}/$CARCH/APKINDEX.tar.gz> This tree then gets populated with real package files when I> use abuild.> > * ~/.cache/abuild/{main,testing,local}/$CARCH is as described above.> This tree gets populated with symlinks to the real package files> and with real and up-to-date APKINDEX files when I use abuild.> > * The "local" repository is a new folder I created in aports. I have> a private git branch of the aports repo that tracks that new> folder, as well as local changes I make to package scripts under> main and testing.> > This setup seems to fit decently with the existing behavior and design> of abuild and the aports repo. I wish less trial-and-error had been> needed to arrive at it, but as I said, I can work on improving the> existing documentation. (Also, I didn't come here asking questions; I> was trying to figure out what I could on my own.)
I'm sorry for your struggling with this.
> But it is puzzling how REPODEST has real package files but has to have> symlinks to APKINDEX files under ~/.cache/abuild, whereas the latter has> real APKINDEX files but symlinks to package files under REPODEST. I get> the impression that abuild wasn't designed with the use of REPODEST> centrally in mind?
You are correct about this. I had ideas of allowing users to store
their built packages in other directories. PKGDEST was originally
thought for this. Then was there a need to separate main and testing
repos, so I could not just set PKGDEST in abuild.conf. So REPODEST was
invented. The main need for this was the build scripts that builds
the entire repository and uploads to master mirror.
The buildrepo[1] script is the main (and only?) reason for REPODEST.
> Regardless, is there really some good reason now to> have separate package trees under REPODEST and ~/.cache/abuild?
Not really. Well, I think I wanted avoid rebuilding the index for every
package in case everything was built from scratch. I suppose that is
wrong since you need an INDEX to get recently built dependencies.
> Here is an alternative, just off the top of my head:> > * if REPODEST is unset, handle ~/.cache/abuild as presently.> * if REPODEST is set and ~/.cache/abuild doesn't exist, then> use REPODEST in place of ~/.cache/abuild> * if REPODEST is set and ~/.cache/abuild exists (and points to a> separate location than REPODEST), handle ~/.cache/abuild as> presently.> * if REPODEST is set and ~/.cache/abuild points to it (either is a> symlink to REPODEST, or REPODEST resolves to ~/.cache/abuild),> then use REPODEST in place of ~/.cache/abuild. (Instead of trying> to place the packages under REPODEST and symlinks to them in what> is ultimately the same location.)
Looks sane to me. I will think of it. I agree REPODEST needs some love.
buildrepo has not been touched since that time and could have use of
some attention too.
Thanks!
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, 27 Nov 2012 08:28:11 -0500
Dubiousjim <lists+alpine-devel@jimpryor.net> wrote:
> Here is an alternative, just off the top of my head:> > * if REPODEST is unset, handle ~/.cache/abuild as presently.
This sounds good.
> * if REPODEST is set and ~/.cache/abuild doesn't exist,then> use REPODEST in place of ~/.cache/abuild> * if REPODEST is set and ~/.cache/abuild exists (and points to a> separate location than REPODEST), handle ~/.cache/abuild as> presently.
I wonder if this would be simpler:
* if REPODEST is set and does not resolve to same location as
~/.cache/abuild, then use REPODEST instead of ~/.cache/abuild
Worst case for this would be that scripts depending on current
behaviour would generate index twice? I think we should try improve
buildrepo while here.
If you still prefer what you suggested, then I'm ok with that too.
> * if REPODEST is set and ~/.cache/abuild points to it (either is a> symlink to REPODEST, or REPODEST resolves to ~/.cache/abuild),> then use REPODEST in place of ~/.cache/abuild. (Instead of trying> to place the packages under REPODEST and symlinks to them in what> is ultimately the same location.)
Yes, this sounds good.
Thanks you very much for your feedback!
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Wed, 28 Nov 2012 11:38:42 +0100
Natanael Copa <ncopa@alpinelinux.org> wrote:
> On Tue, 27 Nov 2012 08:28:11 -0500> Dubiousjim <lists+alpine-devel@jimpryor.net> wrote:> > > Here is an alternative, just off the top of my head:> > > > * if REPODEST is unset, handle ~/.cache/abuild as presently.> > This sounds good.> > > * if REPODEST is set and ~/.cache/abuild doesn't exist,then> > use REPODEST in place of ~/.cache/abuild> > * if REPODEST is set and ~/.cache/abuild exists (and points to a> > separate location than REPODEST), handle ~/.cache/abuild as> > presently.> > I wonder if this would be simpler:> * if REPODEST is set and does not resolve to same location as> ~/.cache/abuild, then use REPODEST instead of ~/.cache/abuild> > Worst case for this would be that scripts depending on current> behaviour would generate index twice? I think we should try improve> buildrepo while here.
I have implemented this now. It was infact very very simple.
It is in git:
http://git.alpinelinux.org/cgit/abuild/commit/?id=5f453f1cdec8bc2c4174878fdd818b5e9ad13505
I don't think it will break anything either.
Please test how it works.
Thanks!
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---