X-Original-To: alpine-devel@lists.alpinelinux.org Delivered-To: alpine-devel@mail.alpinelinux.org Received: from ncopa-desktop.alpinelinux.org (3.203.202.84.customer.cdi.no [84.202.203.3]) (using SSLv3 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: nc@alpinelinux.org) by mail.alpinelinux.org (Postfix) with ESMTPSA id 3E8C6DC00EC; Tue, 27 Nov 2012 16:05:54 +0000 (UTC) Date: Tue, 27 Nov 2012 17:05:51 +0100 From: Natanael Copa To: Dubiousjim Cc: alpine-devel@lists.alpinelinux.org Subject: Re: [alpine-devel] REPODEST vs ~/.cache/abuild Message-ID: <20121127170551.5eba1ba2@ncopa-desktop.alpinelinux.org> In-Reply-To: <20121127132811.GA69074@vaio.jimpryor.net> References: <20121127132811.GA69074@vaio.jimpryor.net> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.13; x86_64-unknown-linux-gnu) X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Tue, 27 Nov 2012 08:28:11 -0500 Dubiousjim wrote: > I'm pretty familiar with Arch's makepkg, which seems to be the > basis/inspiration of Alpine's abuild. I just looked at the first git commit of abuild: +# script to build apk packages (light version og makepkg) So yes. makepkg was inspiration. At that point we thought of using pacman, or at least beeing compatible with pacman. We gave that up. > And I've read the (thin) > documentation available on abuild. Despite this, though, there was a lot > of specific details about abuild's operation that I was unclear on. I > figured the most effective way to answer these questions was to > familiarize myself with the source, so that's what I'm doing. Have about > 1/2 of it digested at this point. Sorry for not beeing good at documenting things... > Now I'm already a lot clearer on the relation between `abuild build > package`, `abuild rootpkg`, and plain `abuild`, and so on. I'll expand the wiki > documentation of abuild when I have a chance. That'd be great! > I also found some quirks and questionable things in the source, which I > will also pass on when I've collected my thoughts. > > But there is one large design issue that had me puzzled. It's the > treatment of the repo tree underneath ~/.cache/abuild versus the package > tree underneath REPODEST. Congrats! You have found one of my hidden skeletons :) > If REPODEST is unset, then PKGDEST defaults to > $startdir---the directory holding the APKGBUILD you're building---and > the treatment of ~/.cache/abuild in this case makes perfect sense. It's > a tree holding a bunch of links to the actual packages, where they sit > interspersed in the aports tree, and ~/.cache/abuild additionally holds > APKINDEX files that are automatically updated and signed when one does > plain `abuild`. This is just the natural behavior I'd want or expect. > > However, if I've set REPODEST, then packages don't stay in the aports > tree, but rather get moved to the tree underneath REPODEST. However, > ~/.cache/abuild is handled the same way: links to the packages > underneath REPODEST, plus actual APKINDEX files that are automatically > updated. Nothing is done to update APKINDEX files that may or may not > exist under REPODEST. > Now, when one needs to install makedepends when abuilding, both the > repos listed in your /etc/apk/repositories AND the ~/.cache/aports > repository are consulted. However, when just using plain `apk add ...`, > only the former is consulted. So if you want apk to also use your local > custom packages, you should include a local repo path in your > /etc/apk/repositories. But that repo need to have APKINDEX files that > are refreshed when packages are added. One _could_ just add > /home/YOU/.cache/abuild/* here. I expect that would work fine. (I expect > you need to expand the glob manually though.) That is actually what I do on my development boxes. > But then what's the point > of having abuild move all your packages under REPODEST? If one instead > adds $REPODEST/* to /etc/apk/repositories (again, expanding the > pathnames by hand), you'd have a repository with no indexes; or indexes > that you had to manually update before using. I ran a git log and searched for REPODEST. Last time that things that had to do with REPODEST was around april 2011. What happened then was that apk-tools got support for multi arch when we ported Alpine Linux to x86_64. At that time I only wanted the arch stuff work. Next time REPODEST is touched is Oct 24 2009, and this is only the usage help text. Next time again is a rename of abuild to abuild.in, May 8, 2009. Next time is Feb 11, 2009, which was last time REPODEST appeared in commit message. This was the last time I did anything that had to do with REPODEST. First commit was Oct 18, 2008. In other words, the REPODEST stuff was onlhy thought of early in the development process, and then forgotten - or at least not though of at all. The ~/.cache/abuild symlinks came later. > What I've done instead is this: > > * add $REPODEST/* to /etc/apk/repositories (In my case, these come > before the official repositories, so that my custom builds are > used preferentially whenever they are present. I also have the > official repositories listed as "@main ..." and "@testing ..." > entries, so that I can override my custom builds when needed.) > > * REPODEST is set in /etc/abuild.conf to point to a tree. At > $REPODEST/{main,testing,local}/$CARCH I have symlinks pointing > to ~/.cache/abuild/{main,testing,local}/$CARCH/APKINDEX.tar.gz > This tree then gets populated with real package files when I > use abuild. > > * ~/.cache/abuild/{main,testing,local}/$CARCH is as described above. > This tree gets populated with symlinks to the real package files > and with real and up-to-date APKINDEX files when I use abuild. > > * The "local" repository is a new folder I created in aports. I have > a private git branch of the aports repo that tracks that new > folder, as well as local changes I make to package scripts under > main and testing. > > This setup seems to fit decently with the existing behavior and design > of abuild and the aports repo. I wish less trial-and-error had been > needed to arrive at it, but as I said, I can work on improving the > existing documentation. (Also, I didn't come here asking questions; I > was trying to figure out what I could on my own.) I'm sorry for your struggling with this. > But it is puzzling how REPODEST has real package files but has to have > symlinks to APKINDEX files under ~/.cache/abuild, whereas the latter has > real APKINDEX files but symlinks to package files under REPODEST. I get > the impression that abuild wasn't designed with the use of REPODEST > centrally in mind? You are correct about this. I had ideas of allowing users to store their built packages in other directories. PKGDEST was originally thought for this. Then was there a need to separate main and testing repos, so I could not just set PKGDEST in abuild.conf. So REPODEST was invented. The main need for this was the build scripts that builds the entire repository and uploads to master mirror. The buildrepo[1] script is the main (and only?) reason for REPODEST. > Regardless, is there really some good reason now to > have separate package trees under REPODEST and ~/.cache/abuild? Not really. Well, I think I wanted avoid rebuilding the index for every package in case everything was built from scratch. I suppose that is wrong since you need an INDEX to get recently built dependencies. > Here is an alternative, just off the top of my head: > > * if REPODEST is unset, handle ~/.cache/abuild as presently. > * if REPODEST is set and ~/.cache/abuild doesn't exist, then > use REPODEST in place of ~/.cache/abuild > * if REPODEST is set and ~/.cache/abuild exists (and points to a > separate location than REPODEST), handle ~/.cache/abuild as > presently. > * if REPODEST is set and ~/.cache/abuild points to it (either is a > symlink to REPODEST, or REPODEST resolves to ~/.cache/abuild), > then use REPODEST in place of ~/.cache/abuild. (Instead of trying > to place the packages under REPODEST and symlinks to them in what > is ultimately the same location.) Looks sane to me. I will think of it. I agree REPODEST needs some love. buildrepo has not been touched since that time and could have use of some attention too. Thanks! -nc --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org ---