Subject: [alpine-devel] Managing source packages

From: Natanael Copa <[email-masked]>
Date: Wed, 29 Dec 2010 11:12:33 +0100

The latest saveas-http:// idea hack brings up another thing we need to
handle.

How do we deal with source packages that disappears from upstream?

Problem: we add a source URL in APKBUILD, 4 weeks later the project is
gone for some reason or they have moved the source file or their server
is just down.

When bootstrapping, this is a major pain. According the GPL license we
also have to provide the sources so we need to do something about this.

The current workaround is http://distfiles.alpinelinux.org which
basically is the shared /var/cache/distfiles for the build boxes. In
abuild.conf you can add
DISTFILES_MIRROR=http://distfiles.alpinelinux.org/distfiles and abuild
will use this instead of downloading from upstream.

I still wonder though, how and when do we delete files from distfiles?
I dont think we should delete source files that are used for v1.10
releases, so how do we do that? git checkout v1.10, make a list of all
sourcefiles and mark them as "protected"? then finally delete all files
that are not marked as "protected"?

Or do we switch to debian style src packages or srpms like?

I have also some ideas of a script that parses current aports and
downloads headers only, and then reports all 404 or non-responding
servers and notifies the maintainer.

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Andrew Manison <[email-masked]>
Date: Wed, 29 Dec 2010 11:34:28 -0500

This is an ongoing and persistent problem, as I found when developing 
the "rebuild-alpine" script. I think the only correct solution that 
preserves the ability to rebuild all aports, as well as keeping us in 
compliance with the GPL, is to have our own source repository.

The existing distfiles workaround provides this, but it is something of 
afterthought, rather than part of the whole packaging scheme. I would 
favour a source package format (.sapk?) with its location configurable 
as part of the apk setup, just as most distributions have a standard 
source repository that can be added to the list of sources for package 
management.

This will require an additional step in the package development process, 
that of building the source package. This should be easily added as an 
abuild subcommand.

Andrew

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 30 Dec 2010 11:30:15 +0100

On Wed, 29 Dec 2010 11:34:28 -0500

> compliance with the GPL, is to have our own source repository.

This requires some big changes in the building infrastruc.

> The existing distfiles workaround provides this, but it is something
> of afterthought, rather than part of the whole packaging scheme. I
> would favour a source package format (.sapk?) with its location
> configurable as part of the apk setup, just as most distributions
> have a standard source repository that can be added to the list of
> sources for package management.
> 
> This will require an additional step in the package development
> process, that of building the source package. This should be easily
> added as an abuild subcommand.

We already have abuild srcpkg, but its broken.

The problem is, that source packages should include:
 * buildrecipy (APKBUILD)
 * all patches
 * all scripts (pre/post/triggers)

So, since we have an aports tree, bsd/archlinux/gentoo style, we
basically have all APKBUILS in one place. If we move to source packages
the APKBUILD in source package will clash with the APKBUILD in aports
tree, which basically means, we cannot build directly in the APKBUILD
tree.

Well, the aports tree would only build sourcepackages. Nothing more.

Then we would need an index for the source packages and some tools to
handle build dependencies. (you have a bunch of *.sapk, how do
you know what order they should be built?)

So, my question is: How do we implement source packages without making
things more complicated than they already are?

-nc

---

---

From: Jeff Bilyk <[email-masked]>
Date: Sat, 1 Jan 2011 12:52:49 -0500

Personally, I like the simplicity of not having to redo the build infrastructure to accommodate source packages.  If abuild srcpkg was working and a script existed to parse aports tree and alert when a source package was no longer available, I think that the current setup is ok.

As far as cleaning out the distfiles cache, a script that runs on distfiles.alpinelinux.org and clears out any pre 1.10 source packages would be good enough imho.  When 1.10 becomes deprecated, the script could then be run to clear out pre2.0 files.

Thoughts?

Jeff

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 3 Jan 2011 13:53:43 +0100

On Sat, 1 Jan 2011 12:52:49 -0500
Jeff Bilyk <[email-masked]> wrote:

> Personally, I like the simplicity of not having to redo the build
> infrastructure to accommodate source packages.  If abuild srcpkg was
> working and a script existed to parse aports tree and alert when a
> source package was no longer available, I think that the current
> setup is ok.

Yeah, i tend to agree here.

> As far as cleaning out the distfiles cache, a script that runs on
> distfiles.alpinelinux.org and clears out any pre 1.10 source packages
> would be good enough imho.  When 1.10 becomes deprecated, the script
> could then be run to clear out pre2.0 files.

The difficult task is: how do you get a list of source files to
delete that are exclusively pre 1.10?

-nc

---

---

From: Jeff Bilyk <[email-masked]>
Date: Mon, 3 Jan 2011 12:18:10 -0500

> delete that are exclusively pre 1.10?
I don't mind taking a crack at it, but the script will be a bit(read:
a lot) kludgy and in need of cleanup after.  That being, I should be
able to get something workable.

-- 
Jeff

---

---

From: "Matt Smith" <[email-masked]>
Date: Sat, 1 Jan 2011 21:36:19 -0600 (CST)

On Saturday, January 1, 2011 11:52am, "Jeff Bilyk" <[email-masked]> said:
> Personally, I like the simplicity of not having to redo the build infrastructure
> to accommodate source packages.  If abuild srcpkg was working and a script existed
> to parse aports tree and alert when a source package was no longer available, I
> think that the current setup is ok.
> 
> distfiles.alpinelinux.org and clears out any pre 1.10 source packages would be
> good enough imho.  When 1.10 becomes deprecated, the script could then be run to
> clear out pre2.0 files.
> 
> Thoughts?
> 
> Jeff
> 
...snip...
>> servers and notifies the maintainer.

I want to point out the potential for filename clashes, especially with patch files, etc., all going into one directory like that.  It might be better to have the distfiles be saved to $distfiles_dir/$package/<package files>.

Aside from that, what we if did like Ubuntu and had Long-Term Support (LTS) releases, where bug-fixes are backported and distfiles+APKBUILDs are kept for the life of the release (1+ years, for instance).  Then keep the dist-files+APKBUILDs for the previous two releases; the next release, be it an LTS release or not, would mark the End-of-Life (EOL) for the oldest release still kept.

So at any time, we're supporting one LTS release and 2 update releases.

Something like that anyway.

At any rate, I think it'd be a good idea to keep older distfiles+APKBUILDs around for a little while, so in case someone should need to downgrade a package, they atleast have the option in one form or another.  Perhaps even keeping the APKs.

Unfortunately, that will eat up a good bit of disk space, I presume.  And although source packages sound like a good idea, people using embedded systems or otherwise wouldn't necessarily have the alpine-sdk installed.

Matt

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 3 Jan 2011 14:20:59 +0100

On Sat, 1 Jan 2011 21:36:19 -0600 (CST)
"Matt Smith" <[email-masked]> wrote:

> > infrastructure to accommodate source packages.  If abuild srcpkg
> > was working and a script existed to parse aports tree and alert
> > when a source package was no longer available, I think that the
> > current setup is ok.
> > 
> > distfiles.alpinelinux.org and clears out any pre 1.10 source
> > packages would be good enough imho.  When 1.10 becomes deprecated,
> > the script could then be run to clear out pre2.0 files.
> > 
> > Thoughts?
> > 
> > Jeff
> > 
> >> I still wonder though, how and when do we delete files from
> >> distfiles? I dont think we should delete source files that are
> >> used for v1.10 releases, so how do we do that? git checkout v1.10,
> >> make a list of all sourcefiles and mark them as "protected"? then
> >> finally delete all files that are not marked as "protected"?
> >>
> I want to point out the potential for filename clashes, especially
> with patch files, etc., all going into one directory like that.

I dont think that is a big deal. It is only the http:// and ftp://
files we keep in distfiles. The patches are kept in git.

>  It
> might be better to have the distfiles be saved to
> $distfiles_dir/$package/<package files>.

This might actually be a good idea anyways. It will avoid one directory
with tons of files (check how long time it takes to open
http://distfiles.gentoo.org/distfiles and you'll see whta I mean)

It will also make cleaning maintenence easier since it will be clear
what apkbuild that pulled in that particular source.

The downside is that aports cannot share source package so things like
linuk kernel source will be downloaded * N times (where N is number of
flavours). Same with for example poppler.

One thing I thought was that we could create snapshot dirs for
each git tag (v1.10.0, v1.10.1 -> v2.1.x) where we create
hardlinks to the distfiles dir. So then you can just delete all files
in distfiles that are not referenced to in current git master. The
hardlink in snapshot dir will make sure its kept as long as needed.

If you would like to rebuild v1.10.x you could just checkout that git
tag, point your DISTFILES_MIRROR to that specific dir and you had
everythin you needed for rebuilding it.

The only drawback I can see with hardlinks is if we want rsync the
distfiles to mirrors. That is ofcourse aslo  possible to solve
different ways and we could wait with it til there actually are more
distfiles mirrors.

> Aside from that, what we if did like Ubuntu and had Long-Term Support
> (LTS) releases, where bug-fixes are backported and
> distfiles+APKBUILDs are kept for the life of the release (1+ years,
> for instance).  Then keep the dist-files+APKBUILDs for the previous
> two releases; the next release, be it an LTS release or not, would
> mark the End-of-Life (EOL) for the oldest release still kept.

I have got requests for this before and I do think its a good idea.
The problem is that it would require someone to actually take the
responsability do the job. I don't have capacity for that.

That said, I try maintain older releases as good as possible and I will
backport fixes fro 1.10 (and even 1.9) on request.

> Something like that anyway.

I like it. Anyone willing to do the real job backporting bugfixes?
Actually, only keeping an eye on the changes and decide what should be
backported would be helpful. We could start with bugfixes only.

> At any rate, I think it'd be a good idea to keep older
> distfiles+APKBUILDs around for a little while, so in case someone
> should need to downgrade a package, they atleast have the option in
> one form or another.  Perhaps even keeping the APKs.

Yes. We could create source packages (with 'abuild srcpkg') for
archival purposes. But this won't be very useful when rebuilding from
scratch, which was Andrew's initial frustration.

> Unfortunately, that will eat up a good bit of disk space, I presume.
> And although source packages sound like a good idea, people using
> embedded systems or otherwise wouldn't necessarily have the
> alpine-sdk installed.

Yeah. tradeoffs...

---

---

From: "Matt Smith" <[email-masked]>
Date: Fri, 7 Jan 2011 20:47:38 -0600 (CST)

> files we keep in distfiles. The patches are kept in git.

Ah, good point.

> flavours). Same with for example poppler.

We could eliminate the need to download common source files more than
once by implementing a method to grab the md5sums from all APKBUILDs,
checking which md5sum lines are identical and occur atleast twice or more.

The determined common source files would then be moved/copied to a
'common_files' directory.  Then when abuild searches for the file
under the package's directory and doesn't find it, have it also
search the 'common_files' directory.
 
> everythin you needed for rebuilding it.

Sounds good to me.

..snip..
> backport fixes fro 1.10 (and even 1.9) on request.

Yeah, that's fair enough.  Unfortunately, I'm not looking to take on such
a task at this time, either.  Unless, of course, we formed a small team
to take care of it.  Then I'd be willing to take a crack at it.

Anyone else?

> backported would be helpful. We could start with bugfixes only.

Yes, bug fixes would be a good start.

> scratch, which was Andrew's initial frustration.

I thought I might double-check and ask: Wouldn't the distfiles server work
for source packages too?

>> embedded systems or otherwise wouldn't necessarily have the

What do you think about keeping say, the last 5 versions of the APK
with the latest pkgrel (ie, package 1.20-1.24, and say all versions
were at pkgrel 0, except one: 1.20-r0, 1.21-r2, 1.22-r0, etc)?

Then implement a way to downgrade to a previously available version
through apk-tools.  (ie, $ apk del php; apk add php-5.3.3, without
adding the pkgrel - unless that becomes a wanted feature later on.)

We could then implement a 'downgrade' command to apk-tools to achieve
the same thing in one command: $ apk downgrade php-5.3.3

Also, on a side note, I think we should implement some sort of 'hold'
feature or sub-command to apk-tools, in order to keep a package at
it's current version.  This would certainly aid some users of edge,
but might also be used by others who are not following edge.

Matt

---