X-Original-To: alpine-devel@lists.alpinelinux.org Delivered-To: alpine-devel@lists.alpinelinux.org Received: from mail-qy0-f183.google.com (mail-qy0-f183.google.com [209.85.221.183]) by lists.alpinelinux.org (Postfix) with ESMTP id D2BF54DF073A for ; Thu, 29 Oct 2009 13:46:07 +0000 (UTC) Received: by qyk13 with SMTP id 13so1335177qyk.27 for ; Thu, 29 Oct 2009 06:46:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=FU9S8C6sm8FunTghOqCv8BFqiVQYj3zy2yQt0WSbWvo=; b=dsD3cTet8mZZt1jQ4wv4NF2l4iT/iMiHvh70HtdFCqkmF/bHmzZ/jXiKOVtbUadI0U rS95oNFiv0Wb+S/q3rrjA7plIDELly5fvhOjOdBVtLZIvzm6BOL2NZf3Mw/J3CVCQ1eR Ck3sQBNEWhgOg8whA4Vv8oSE99ssV/l3LdTzk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=wUiWq/FwHTuFZnCsk3LkRTUoupD2Wsp9swGeq/7sq4GYvmgr+3ns8u/WjlPmzvJz/Z h2t+6FADrwaRCZ+tx0rZIOrpzHIkD8nkPXvHKmfBupn8Cun0d8h+N8XbfYwR8yw3S4b1 nE5/mzALwDIwU7l6NWO8pH9FyYLhUTNZxSb4M= X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Sender: jeremythomerson@gmail.com Reply-To: jeremy@thomersonfamily.com Received: by 10.229.13.213 with SMTP id d21mr8963qca.45.1256823966650; Thu, 29 Oct 2009 06:46:06 -0700 (PDT) In-Reply-To: <1256801075.20939.2767.camel@ncopa-desktop.nor.wtbts.net> References: <1256801075.20939.2767.camel@ncopa-desktop.nor.wtbts.net> Date: Thu, 29 Oct 2009 08:46:06 -0500 X-Google-Sender-Auth: d3c611d97a9fbda6 Message-ID: <6602aeca0910290646nfb59f7et4acfec11314953f@mail.gmail.com> Subject: Re: [alpine-devel] new build server From: Jeremy Thomerson To: Natanael Copa Cc: alpine-devel@lists.alpinelinux.org Content-Type: multipart/alternative; boundary=00151759031e13d22604771322a9 --00151759031e13d22604771322a9 Content-Type: text/plain; charset=ISO-8859-1 Wow - you must never sleep! Congratulations. This is a great improvement. PS - you probably considered this, but in the future if you want an easy way to communicate a queue or event triggering between multiple servers, try ActiveMQ. It's an enterprise messaging bus and has clients in pretty much any language. You can even set it up to receive or broadcast events through simple HTTP if you need to. Anyway, I've used it on a bunch of projects and it's super easy to set up and a great piece of software. Using something like that would even give you the ability to have multiple build servers subscribing to the channel and either building all packages, or dividing up the packages, each building one until the queue is done. Just a thought anyway.... Jeremy Thomerson On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa wrote: > Hi, > > We have got a new build server. I have spent last days on setting it up. > The idea is that we want a git hook to trigger building updated packages > and uploading them to master mirror. Since git server and build server > are 2 different we could have solved it with a cron job. Not good enough > I thought... > > So, what we do is we create a fifo for each git branch on the git > server. Then we have a git hook that writes to the fifo when someone > pushes changes. > > Then on build server, we run a daemon that ssh to the git server and > read from the fifo. For each read line, it rebuilds and uploads new > packages. That way we avoid race conditions when 2 different developers > pushes stuff at the same time and we avoid the delays we would have when > using cron jobs. > > Then I wanted this to run as a daemon, that starts/stops from an init.d > script so it starts up automatically after a reboot. Having it start was > easy. I wrote a daemon in shell for that. Having it stop was not so > easy, since when the daeomn was killed the ssh process was left running. > I could not solve that in shell, so i rewrite it in lua. Lua lacks > signal handling so I ended up rewrite it in C. After a bug that > forkbombed my vserver host to death (!!?) and a bug that killed all the > packages on main mirror we are finally back and things are working. > > The new build server have 8 horses that kicks so if a developer wants a > build environment for kernel hacking or similar, just let me know. > > -nc > > > > --- > Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org > Help: alpine-devel+help@lists.alpinelinux.org > --- > > --00151759031e13d22604771322a9 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Wow - you must never sleep!=A0 Congratulations. This is a great improvement= .

PS - you probably considered this, but in the future if you want a= n easy way to communicate a queue or event triggering between multiple serv= ers, try ActiveMQ.=A0 It's an enterprise messaging bus and has clients = in pretty much any language.=A0 You can even set it up to receive or broadc= ast events through simple HTTP if you need to.=A0 Anyway, I've used it = on a bunch of projects and it's super easy to set up and a great piece = of software.=A0 Using something like that would even give you the ability t= o have multiple build servers subscribing to the channel and either buildin= g all packages, or dividing up the packages, each building one until the qu= eue is done.

Just a thought anyway....

Jeremy Thomerson

On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa <natanael.copa@gmail.com> wrote:
Hi,

We have got a new build server. I have spent last days on setting it up. The idea is that we want a git hook to trigger building updated packages and uploading them to master mirror. Since git server and build server
are 2 different we could have solved it with a cron job. Not good enough I thought...

So, what we do is we create a fifo for each git branch on the git
server. Then we have a git hook that writes to the fifo when someone
pushes changes.

Then on build server, we run a daemon that ssh to the git server and
read from the fifo. For each read line, it rebuilds and uploads new
packages. That way we avoid race conditions when 2 different developers
pushes stuff at the same time and we avoid the delays we would have when using cron jobs.

Then I wanted this to run as a daemon, that starts/stops from an init.d
script so it starts up automatically after a reboot. Having it start was easy. I wrote a daemon in shell for that. Having it stop was not so
easy, since when the daeomn was killed the ssh process was left running. I could not solve that in shell, so i rewrite it in lua. Lua lacks
signal handling so I ended up rewrite it in C. After a bug that
forkbombed my vserver host to death (!!?) and a bug that killed all the
packages on main mirror we are finally back and things are working.

The new build server have 8 horses that kicks so if a developer wants a
build environment for kernel hacking or similar, just let me know.

-nc



---
Unsubscribe: =A0alpine-devel+unsubscribe@lists.alpinelinux.org
Help: =A0 =A0 =A0 =A0 alpine-devel+help@lists.alpinelinux.org
---


--00151759031e13d22604771322a9-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org ---