Subject: Re: [alpine-devel] Alpine 2.2
From: Natanael Copa <ncopa@alpinelinux.org>
To: Jeff Bilyk <jbilyk@gmail.com>
Cc: Alpine Development <alpine-devel@lists.alpinelinux.org>
In-Reply-To: <AANLkTimyzPyuy=_-=zJSHo_VKu60zNSY5qsUhME6jhcY@mail.gmail.com>
References: <1289383750.18266.23.camel@ncopa-desktop.nor.wtbts.net>
	 <AANLkTimyzPyuy=_-=zJSHo_VKu60zNSY5qsUhME6jhcY@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Date: Wed, 24 Nov 2010 10:00:08 +0100
Message-ID: <1290589208.6958.30.camel@ncopa-desktop.nor.wtbts.net>
Precedence: list
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit

sorry for late reply.

On Wed, 2010-11-17 at 22:08 -0500, Jeff Bilyk wrote:
> Re: testing and QA, I took a look at APTS (thanks to ncopa pointing it
> out...).  Taking the initial idea but simplifying a bit, I figured
> taking the following approach might work:
> 
> Take the .mk files from svn://svn.alpinelinux.org/apts, strip the
> first line and sed apk_delete for apk_del.
> Run the following script (a gutted version of run-tests from APTS):
> ----------
> #!/bin/sh -e
> 
> # Get file to output to
> OUTFILE="$@"
> if [ -z "$OUTFILE" ];
>  then echo "Usage: $0 outfile" && exit 0;
> fi
> 
> # Clear file if text
> echo "" > $OUTFILE
> 
> # Add package, run basic test then remove package
> for package in `ls ./packages`;
> do sh -e ./packages/$package >> $OUTFILE 2>&1 && echo "$package passed
> tests" >> $OUTFILE;
> done
> ----------
> 
> A sample of a .mk file that's in SVN that we could adapt: sed.mk:
> 
> ----------
> sed:
> 	apk_add $@
> 	echo "hello" | sed 's/hello/world/g' | grep 'world'
> 	apk_delete $@
> 	[ `readlink /bin/sed` = /bin/busybox ]
> ----------
> 
> The output can be redirected and parsed (or maybe just take stderr?)
> so that once edge ISOs are autobuilt, they can have some basic tests
> done on most packages (or just the important ones?).  Assuming the ISO
> passes this basic QA upload the ISO to the mirrors.  If it fails, add
> the output from the script to sircbot's output on alpine-devel so that
> the package can be fixed.
> 
> Thoughts?

sounds good. But i think we should split the testing in 2 parts:
1. testing packages
2. testing iso images

Packages needs to be tested before the releases. We should test the
packages while they are in edge. (and we should make edge snapshots)

I also think that we should try make some regression tests. For example,
when we upgraded kamailio to 3.1 there was a problem with config files
ending up in /usr. I didnt detect that since kamailio started just fine
anyways. So we could have a test that verifies that config is in correct
place.

Ideally, when there is a problem with a package, we should make a test
script that triggers the error so we can make sure the same problem
never comes back again.

When it comes to testing iso images, I think some (half)automated way to
test the installer would be good, verify that lbu commit works, verify
that encrypted apkovls work etc. Edge snapshots should be tested too.


Now, we need to start some place, so I think what you do is good.

-nc



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---