Subject: [alpine-devel] REPODEST vs ~/.cache/abuild

From: Dubiousjim <[email-masked]>
Date: Tue, 27 Nov 2012 08:28:11 -0500

I'm pretty familiar with Arch's makepkg, which seems to be the
basis/inspiration of Alpine's abuild. And I've read the (thin)
documentation available on abuild. Despite this, though, there was a lot
of specific details about abuild's operation that I was unclear on. I
figured the most effective way to answer these questions was to
familiarize myself with the source, so that's what I'm doing. Have about
1/2 of it digested at this point.

Now I'm already a lot clearer on the relation between `abuild build
package`, `abuild rootpkg`, and plain `abuild`, and so on. I'll expand the wiki
documentation of abuild when I have a chance.

I also found some quirks and questionable things in the source, which I
will also pass on when I've collected my thoughts.

But there is one large design issue that had me puzzled. It's the
treatment of the repo tree underneath ~/.cache/abuild versus the package
tree underneath REPODEST. If REPODEST is unset, then PKGDEST defaults to
$startdir---the directory holding the APKGBUILD you're building---and
the treatment of ~/.cache/abuild in this case makes perfect sense. It's
a tree holding a bunch of links to the actual packages, where they sit
interspersed in the aports tree, and ~/.cache/abuild additionally holds
APKINDEX files that are automatically updated and signed when one does
plain `abuild`. This is just the natural behavior I'd want or expect.

However, if I've set REPODEST, then packages don't stay in the aports
tree, but rather get moved to the tree underneath REPODEST. However,
~/.cache/abuild is handled the same way: links to the packages
underneath REPODEST, plus actual APKINDEX files that are automatically
updated. Nothing is done to update APKINDEX files that may or may not
exist under REPODEST.

Now, when one needs to install makedepends when abuilding, both the
repos listed in your /etc/apk/repositories AND the ~/.cache/aports
repository are consulted. However, when just using plain `apk add ...`,
only the former is consulted. So if you want apk to also use your local
custom packages, you should include a local repo path in your
/etc/apk/repositories. But that repo need to have APKINDEX files that
are refreshed when packages are added. One _could_ just add
/home/YOU/.cache/abuild/* here. I expect that would work fine. (I expect
you need to expand the glob manually though.) But then what's the point
of having abuild move all your packages under REPODEST? If one instead
adds $REPODEST/* to /etc/apk/repositories (again, expanding the
pathnames by hand), you'd have a repository with no indexes; or indexes
that you had to manually update before using.

What I've done instead is this:

    * add $REPODEST/* to /etc/apk/repositories (In my case, these come
      before the official repositories, so that my custom builds are
      used preferentially whenever they are present. I also have the
      official repositories listed as "@main ..." and "@testing ..."
      entries, so that I can override my custom builds when needed.)

    * REPODEST is set in /etc/abuild.conf to point to a tree. At 
      $REPODEST/{main,testing,local}/$CARCH I have symlinks pointing
      to ~/.cache/abuild/{main,testing,local}/$CARCH/APKINDEX.tar.gz
      This tree then gets populated with real package files when I
      use abuild.

    * ~/.cache/abuild/{main,testing,local}/$CARCH is as described above.
      This tree gets populated with symlinks to the real package files
      and with real and up-to-date APKINDEX files when I use abuild.

    * The "local" repository is a new folder I created in aports. I have
      a private git branch of the aports repo that tracks that new
      folder, as well as local changes I make to package scripts under
      main and testing.

This setup seems to fit decently with the existing behavior and design
of abuild and the aports repo. I wish less trial-and-error had been
needed to arrive at it, but as I said, I can work on improving the
existing documentation. (Also, I didn't come here asking questions; I
was trying to figure out what I could on my own.)

But it is puzzling how REPODEST has real package files but has to have
symlinks to APKINDEX files under ~/.cache/abuild, whereas the latter has
real APKINDEX files but symlinks to package files under REPODEST. I get
the impression that abuild wasn't designed with the use of REPODEST
centrally in mind? Regardless, is there really some good reason now to
have separate package trees under REPODEST and ~/.cache/abuild?

Here is an alternative, just off the top of my head:

    * if REPODEST is unset, handle ~/.cache/abuild as presently.
    * if REPODEST is set and ~/.cache/abuild doesn't exist, then
      use REPODEST in place of ~/.cache/abuild
    * if REPODEST is set and ~/.cache/abuild exists (and points to a
      separate location than REPODEST), handle ~/.cache/abuild as
      presently.
    * if REPODEST is set and ~/.cache/abuild points to it (either is a
      symlink to REPODEST, or REPODEST resolves to ~/.cache/abuild),
      then use REPODEST in place of ~/.cache/abuild. (Instead of trying
      to place the packages under REPODEST and symlinks to them in what
      is ultimately the same location.)

-- 
Dubiousjim
[email-masked]

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Natanael Copa <[email-masked]>
Date: Tue, 27 Nov 2012 17:05:51 +0100

On Tue, 27 Nov 2012 08:28:11 -0500
Dubiousjim <[email-masked]> wrote:

> basis/inspiration of Alpine's abuild.

I just looked at the first git commit of abuild:

+# script to build apk packages (light version og makepkg)

So yes. makepkg was inspiration. At that point we thought of using
pacman, or at least beeing compatible with pacman. We gave that up.

> And I've read the (thin)

Sorry for not beeing good at documenting things...
 
> documentation of abuild when I have a chance.

That'd be great!
 
> tree underneath REPODEST.

Congrats! You have found one of my hidden skeletons :)

> If REPODEST is unset, then PKGDEST defaults to
 
> you need to expand the glob manually though.)

That is actually what I do on my development boxes.

> But then what's the point

I ran a git log and searched for REPODEST. Last time that things that
had to do with REPODEST was around april 2011. What happened then was
that apk-tools got support for multi arch when we ported Alpine Linux
to x86_64. At that time I only wanted the arch stuff work.

Next time REPODEST is touched is Oct 24 2009, and this is only the
usage help text.

Next time again is a rename of abuild to abuild.in, May 8, 2009.

Next time is Feb 11, 2009, which was last time REPODEST appeared in
commit message. This was the last time I did anything that had to do
with REPODEST. First commit was Oct 18, 2008. In other words, the
REPODEST stuff was onlhy thought of early in the development process,
and then forgotten - or at least not though of at all.

The ~/.cache/abuild symlinks came later.

> was trying to figure out what I could on my own.)

I'm sorry for your struggling with this.

> centrally in mind?

You are correct about this. I had ideas of allowing users to store
their built packages in other directories. PKGDEST was originally
thought for this. Then was there a need to separate main and testing
repos, so I could not just set PKGDEST in abuild.conf. So REPODEST was
invented. The main need for this was the build scripts that builds
the entire repository and uploads to master mirror.

The buildrepo[1] script is the main (and only?) reason for REPODEST.

> Regardless, is there really some good reason now to

Not really. Well, I think I wanted avoid rebuilding the index for every
package in case everything was built from scratch. I suppose that is
wrong since you need an INDEX to get recently built dependencies.

>       is ultimately the same location.)
 
Looks sane to me. I will think of it. I agree REPODEST needs some love.
buildrepo has not been touched since that time and could have use of
some attention too.

Thanks!

-nc

---

---

From: Dubiousjim <[email-masked]>
Date: Tue, 27 Nov 2012 16:22:42 -0500

Hi Natanael, thanks for your quick reply. I'm glad I basically
understood what's going on. Let me know how you feel about the
suggestion I made at the end after thinking it over. If it seems like a
good plan, I'll submit patches.

-- 
Dubiousjim
[email-masked]

---

---

From: Natanael Copa <[email-masked]>
Date: Wed, 28 Nov 2012 11:38:42 +0100

>     * if REPODEST is unset, handle ~/.cache/abuild as presently.

This sounds good.

>     * if REPODEST is set and ~/.cache/abuild doesn't exist,then

I wonder if this would be simpler:
      * if REPODEST is set and does not resolve to same location as
        ~/.cache/abuild, then use REPODEST instead of ~/.cache/abuild

Worst case for this would be that scripts depending on current
behaviour would generate index twice? I think we should try improve
buildrepo while here.

If you still prefer what you suggested, then I'm ok with that too.

>       is ultimately the same location.)

Yes, this sounds good.

Thanks you very much for your feedback!

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 7 Feb 2013 17:11:40 +0100

On Wed, 28 Nov 2012 11:38:42 +0100

> behaviour would generate index twice? I think we should try improve

I have implemented this now. It was infact very very simple.

It is in git:
http://git.alpinelinux.org/cgit/abuild/commit/?id=5f453f1cdec8bc2c4174878fdd818b5e9ad13505

I don't think it will break anything either.

Please test how it works.

Thanks!

-nc

---