There will soon be a spate of emails from AlpineBuildBot regarding the 
building of various modules. I have developed a script, intended to be 
run at regular intervals, which goes through the aports tree and builds 
all packages. The current run has failures and the build logs from them 
are being mailed to the maintainers, as well as to the list. Many of the 
failures are due to missing source archives. At present, the distfiles 
mirror (distfiles.alpinelinux.org/distfiles) is quite out of date. If 
the primary source goes offline we have no archived copy of it. There a 
two remedies to this: either make sure that the source tarball is on the 
distfiles mirror, or update the package to the current release. If the 
latter course is taken, the source tarball still need to find its way 
onto the distfiles mirror to preserve it for future builds.

At one point I had the build down to just two failing packages. Now it's 
up to 30 in main and 8 in testing. I am available to work with the 
maintainers to get the builds working. Sometimes it's a matter of 
missing dependencies in the APKBUILD script, that don't show up because 
the packages are installed on the maintainers system. Others are due to 
mutually exclusive packages. For example, libiconv is remarkably 
'sticky' and once installed for a build sometimes fails to uninstall.

My goal is to make it so that it is easy to setup a build server, point 
it at the git repository, and build all the packages for edge or a given 
release. I'm not there yet, but getting closer. In harmony with 
Nataneal's earlier email, this would fit well with the proposed team 
structure, and provide some redundancy to the build and release process. 
This periodic total build would belong to the 'stable maintainers' 
sub-team of the developers team, while the total build is a useful step 
as part of the release process.

Andrew


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

I remember there was a discussion a month or so ago about making official
archives of the source packages so that builds wouldn't later fail when the
original upstream disappeared.  What ever came of that?  Do we have a
solution?

Jeremy Thomerson

On Thu, Feb 24, 2011 at 5:20 PM, Andrew Manison
<amanison@anselsystems.com>wrote:

>  There will soon be a spate of emails from AlpineBuildBot regarding the
> building of various modules. I have developed a script, intended to be run
> at regular intervals, which goes through the aports tree and builds all
> packages. The current run has failures and the build logs from them are
> being mailed to the maintainers, as well as to the list. Many of the
> failures are due to missing source archives. At present, the distfiles
> mirror (distfiles.alpinelinux.org/distfiles) is quite out of date. If the
> primary source goes offline we have no archived copy of it. There a two
> remedies to this: either make sure that the source tarball is on the
> distfiles mirror, or update the package to the current release. If the
> latter course is taken, the source tarball still need to find its way onto
> the distfiles mirror to preserve it for future builds.
>
> At one point I had the build down to just two failing packages. Now it's up
> to 30 in main and 8 in testing. I am available to work with the maintainers
> to get the builds working. Sometimes it's a matter of missing dependencies
> in the APKBUILD script, that don't show up because the packages are
> installed on the maintainers system. Others are due to mutually exclusive
> packages. For example, libiconv is remarkably 'sticky' and once installed
> for a build sometimes fails to uninstall.
>
> My goal is to make it so that it is easy to setup a build server, point it
> at the git repository, and build all the packages for edge or a given
> release. I'm not there yet, but getting closer. In harmony with Nataneal's
> earlier email, this would fit well with the proposed team structure, and
> provide some redundancy to the build and release process. This periodic
> total build would belong to the 'stable maintainers' sub-team of the
> developers team, while the total build is a useful step as part of the
> release process.
>
> Andrew
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
>
>

On Thu, 24 Feb 2011 18:20:27 -0500
Andrew Manison <amanison@anselsystems.com> wrote:

> I have developed a script, intended
> to be run at regular intervals, which goes through the aports tree
> and builds all packages.

Nice!

> The current run has failures and the build
> logs from them are being mailed to the maintainers, as well as to the
> list. 

I wonder if we want direct those emails to a second list.

> Many of the failures are due to missing source archives. At
> present, the distfiles mirror (distfiles.alpinelinux.org/distfiles)
> is quite out of date.

Shouldn't really be out of date. The distfiles dir is shared among my
ncdev verserver and the build-* vservers. So far nothing gets deleted
from there.

> If the primary source goes offline we have no
> archived copy of it. There a two remedies to this: either make sure
> that the source tarball is on the distfiles mirror, or update the
> package to the current release. If the latter course is taken, the
> source tarball still need to find its way onto the distfiles mirror
> to preserve it for future builds.

We need archive for all git tags yes. I have an idea how to solve this
which I will come back to.

I also thik we could split the task in 2:
1. verify source URL is still valid.
2. verify that sources builds.

For checking source url I have implemented 'abuild sourcecheck' I think
we could have a separate run of sourcecheck on all packages from a cron
job.

> At one point I had the build down to just two failing packages. Now
> it's up to 30 in main and 8 in testing. I am available to work with
> the maintainers to get the builds working.

Good. Lets get those packages fixed.

> Sometimes it's a matter of 
> missing dependencies in the APKBUILD script, that don't show up
> because the packages are installed on the maintainers system.

This should be detected when the build-edge autobuilds it.

> Others
> are due to mutually exclusive packages. For example, libiconv is
> remarkably 'sticky' and once installed for a build sometimes fails to
> uninstall.

libiconv is an evil package and for that reason blacklisted in gentoo
embedded (this was one of the reason we left gentoo for building alpine)

The problem is that when some of your packages was built, libiconv was
auto detected and pulled in. Those packages shouldn't have the libiconv
dependency in the first place but once they are there the system is
messed up.

> My goal is to make it so that it is easy to setup a build server,
> point it at the git repository, and build all the packages for edge
> or a given release. I'm not there yet, but getting closer.

Ok, let us help you to get there.

> In harmony
> with Nataneal's earlier email, this would fit well with the proposed
> team structure, and provide some redundancy to the build and release
> process. This periodic total build would belong to the 'stable
> maintainers' sub-team of the developers team, while the total build
> is a useful step as part of the release process.

Or we could have a "quality assurance team" or "testing team" that does
regression testing and such.

Thanks alot!

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

On Thu, 24 Feb 2011 22:46:17 -0600
Jeremy Thomerson <jeremy@thomersonfamily.com> wrote:

> I remember there was a discussion a month or so ago about making
> official archives of the source packages so that builds wouldn't
> later fail when the original upstream disappeared.  What ever came of
> that?  Do we have a solution?

Here is what I'd like to do:

for each git tag, create a distfiles/$tag dir. go through all the
APKBUILDs for that tag and create a hardlink to the source archives
in ../ (or download if missing)

We can then delete all source packages in distfiles that are not
reffered to from any APKBUILD in current git master (and optionally
scan for old -atimes) The hardlinks makes sure we have archive for
previous releases, without duplicating the data on disk.

After that we need to add the hardlinks routine when creating new tags
and a cleanup cron job.

The only thing we need to be careful with is if we create distfiles
mirrors or backups. We just need to be aware of that there are lots of
hardlinks. (or we will waste lots of space on the mirrors)

-nc


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

Hi Andrew,

On Thu, 24 Feb 2011 18:20:27 -0500
Andrew Manison <amanison@anselsystems.com> wrote:

>   There will soon be a spate of emails from AlpineBuildBot regarding
> the building of various modules. I have developed a script, intended
> to be run at regular intervals, which goes through the aports tree
> and builds all packages. The current run has failures and the build
> logs from them are being mailed to the maintainers, as well as to the
> list. Many of the failures are due to missing source archives. At
> present, the distfiles mirror (distfiles.alpinelinux.org/distfiles)
> is quite out of date. If the primary source goes offline we have no
> archived copy of it. There a two remedies to this: either make sure
> that the source tarball is on the distfiles mirror, or update the
> package to the current release. If the latter course is taken, the
> source tarball still need to find its way onto the distfiles mirror
> to preserve it for future builds.

Tinderbox-style testing is a great idea.  What would be useful on the
reports, though, is if the build architecture is listed.  The reason
why I say this, is because it is possible that a build may succeed on
x86 and x86_64, but fail on powerpc or arm which are both slated to be
included as an official port post-2.2.

Would you be able to add this to the report?

> 
> At one point I had the build down to just two failing packages. Now
> it's up to 30 in main and 8 in testing. I am available to work with
> the maintainers to get the builds working. Sometimes it's a matter of 
> missing dependencies in the APKBUILD script, that don't show up
> because the packages are installed on the maintainers system. Others
> are due to mutually exclusive packages. For example, libiconv is
> remarkably 'sticky' and once installed for a build sometimes fails to
> uninstall.
> 
> My goal is to make it so that it is easy to setup a build server,
> point it at the git repository, and build all the packages for edge
> or a given release. I'm not there yet, but getting closer. In harmony
> with Nataneal's earlier email, this would fit well with the proposed
> team structure, and provide some redundancy to the build and release
> process. This periodic total build would belong to the 'stable
> maintainers' sub-team of the developers team, while the total build
> is a useful step as part of the release process.

Great!  I think doing archive-wide rebuilds will improve the QA process
of edge and keep us fixing bugs quickly.  Which fits into my goal of
keeping edge in a more releasable state always.

William


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---

On Fri, Feb 25, 2011 at 4:24 AM, Natanael Copa <ncopa@alpinelinux.org> wrote:
> On Thu, 24 Feb 2011 18:20:27 -0500
> Andrew Manison <amanison@anselsystems.com> wrote:
>
>> I have developed a script, intended
>> to be run at regular intervals, which goes through the aports tree
>> and builds all packages.
>
> Nice!
>
>> The current run has failures and the build
>> logs from them are being mailed to the maintainers, as well as to the
>> list.
>
> I wonder if we want direct those emails to a second list.

Alpine-build@lists.alpinelinux.org is now set up if that's a direction
that we would want to take.  Same subscription process as with the
other lists.  Since there seems to be small bursts of traffic, i'd
suggest that we start using a 2nd list for the build reports, and have
anyone who has git commit access or has contributed to packages
subscribe to the list.

>
>> Many of the failures are due to missing source archives. At
>> present, the distfiles mirror (distfiles.alpinelinux.org/distfiles)
>> is quite out of date.
>
> Shouldn't really be out of date. The distfiles dir is shared among my
> ncdev verserver and the build-* vservers. So far nothing gets deleted
> from there.
>
>> If the primary source goes offline we have no
>> archived copy of it. There a two remedies to this: either make sure
>> that the source tarball is on the distfiles mirror, or update the
>> package to the current release. If the latter course is taken, the
>> source tarball still need to find its way onto the distfiles mirror
>> to preserve it for future builds.
>
> We need archive for all git tags yes. I have an idea how to solve this
> which I will come back to.
>
> I also thik we could split the task in 2:
> 1. verify source URL is still valid.
> 2. verify that sources builds.
>
> For checking source url I have implemented 'abuild sourcecheck' I think
> we could have a separate run of sourcecheck on all packages from a cron
> job.
>
>> At one point I had the build down to just two failing packages. Now
>> it's up to 30 in main and 8 in testing. I am available to work with
>> the maintainers to get the builds working.
>
> Good. Lets get those packages fixed.
>
>> Sometimes it's a matter of
>> missing dependencies in the APKBUILD script, that don't show up
>> because the packages are installed on the maintainers system.
>
> This should be detected when the build-edge autobuilds it.
>
>> Others
>> are due to mutually exclusive packages. For example, libiconv is
>> remarkably 'sticky' and once installed for a build sometimes fails to
>> uninstall.
>
> libiconv is an evil package and for that reason blacklisted in gentoo
> embedded (this was one of the reason we left gentoo for building alpine)
>
> The problem is that when some of your packages was built, libiconv was
> auto detected and pulled in. Those packages shouldn't have the libiconv
> dependency in the first place but once they are there the system is
> messed up.
>
>> My goal is to make it so that it is easy to setup a build server,
>> point it at the git repository, and build all the packages for edge
>> or a given release. I'm not there yet, but getting closer.
>
> Ok, let us help you to get there.
>
>> In harmony
>> with Nataneal's earlier email, this would fit well with the proposed
>> team structure, and provide some redundancy to the build and release
>> process. This periodic total build would belong to the 'stable
>> maintainers' sub-team of the developers team, while the total build
>> is a useful step as part of the release process.
>
> Or we could have a "quality assurance team" or "testing team" that does
> regression testing and such.
>
> Thanks alot!
>
> -nc
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
>
>



-- 
Jeff


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---