X-Original-To: alpine-devel@lists.alpinelinux.org Delivered-To: alpine-devel@lists.alpinelinux.org Received: from qw-out-2122.google.com (qw-out-2122.google.com [74.125.92.26]) by lists.alpinelinux.org (Postfix) with ESMTP id 588044DF073A for ; Thu, 29 Oct 2009 21:35:41 +0000 (UTC) Received: by qw-out-2122.google.com with SMTP id 9so584734qwb.13 for ; Thu, 29 Oct 2009 14:35:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:reply-to:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=BUFV8dU0iVaSkRrpBrbA/S8SsJI+4IDc0iIKL92Ce8M=; b=dbj+fLjypx+VWIMkAuCfWdMurmIWbqm1nWQr44sLtroPQgY36lTcPEMl+p1W0uc5xj D3BE2tu9CucSxWwtDpOZ/V+Ya9fuL0i3mqwAXM0FiyafMPCcFlMyTaTBIsSiNQo6OafI VH6lQLrRajGvigDoxvKdMhzoG/LSAjrR7KMtQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:reply-to:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=vs0U7CFQMlnvIvS/B2VMoIQC8cFREaFfMvUWBJCTG1csiykkwje1X9aDwMeoe64gIA 1UhFAZwgVg8v0VMsmRr3PppOD25BvZWRdZSfJ1etHfd+es/g+v09mDKFi+vM092rr9Br x57doHh8bUmymCZMAg/6LOjTp1Hm9e+KrLPnA= X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Sender: jeremythomerson@gmail.com Reply-To: jeremy@thomersonfamily.com Received: by 10.229.69.85 with SMTP id y21mr79180qci.92.1256852140230; Thu, 29 Oct 2009 14:35:40 -0700 (PDT) In-Reply-To: <1256835607.20939.3220.camel@ncopa-desktop.nor.wtbts.net> References: <1256801075.20939.2767.camel@ncopa-desktop.nor.wtbts.net> <6602aeca0910290646nfb59f7et4acfec11314953f@mail.gmail.com> <1256835607.20939.3220.camel@ncopa-desktop.nor.wtbts.net> Date: Thu, 29 Oct 2009 16:35:40 -0500 X-Google-Sender-Auth: 29833cc9e27cc806 Message-ID: <6602aeca0910291435geb90e6erb2d295242beb96a3@mail.gmail.com> Subject: Re: [alpine-devel] new build server From: Jeremy Thomerson To: Natanael Copa Cc: alpine-devel@lists.alpinelinux.org Content-Type: multipart/alternative; boundary=00032557b4d25a8bd2047719b1cc --00032557b4d25a8bd2047719b1cc Content-Type: text/plain; charset=ISO-8859-1 Well, I think there would be several alternatives. I suppose you could even use XMPP, but that would require writing a lot of the plumbing yourself. Here are a couple alternatives, although I have not used them: - RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems to be much smaller (around 1MB?) install. Have not seen exactly what the requirements are for running it, but seems simple. I have not found documentation on how to connect to it and communicate through simple telnet commands, but there are integrations in many languages. For instance, here's one in Perl that uses Stomp (mentioned below): http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl - Stomp [http://stomp.codehaus.org/] - easy to implement the clients, although you will need to couple it with a message broker (RabbitMQ / ActiveMQ / etc). It has a lightweight broker that's written in Ruby - not sure if that will fit your desired footprint. Jeremy On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa wrote: > On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote: > > Wow - you must never sleep! Congratulations. This is a great > > improvement. > > > > PS - you probably considered this, but in the future if you want an > > easy way to communicate a queue or event triggering between multiple > > servers, try ActiveMQ. It's an enterprise messaging bus and has > > clients in pretty much any language. You can even set it up to > > receive or broadcast events through simple HTTP if you need to. > > Anyway, I've used it on a bunch of projects and it's super easy to set > > up and a great piece of software. Using something like that would > > even give you the ability to have multiple build servers subscribing > > to the channel and either building all packages, or dividing up the > > packages, each building one until the queue is done. > > > > Just a thought anyway.... > > Yeah, I would actually like/need something like that. What I have still > dont work perfectly. ssh seesions are not terminated on server. > > I'd need a: > * git hook that sends a single line to the message server > * clinents/buildserver(s) that just sits idle til they get the message > * something that does nto require java and is alot lighter than 33MB. > > in other words, is there anything simpler/lighter than ActiveMQ? > > Thanks! > > -nc > > > Jeremy Thomerson > > > > On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa > > wrote: > > Hi, > > > > We have got a new build server. I have spent last days on > > setting it up. > > The idea is that we want a git hook to trigger building > > updated packages > > and uploading them to master mirror. Since git server and > > build server > > are 2 different we could have solved it with a cron job. Not > > good enough > > I thought... > > > > So, what we do is we create a fifo for each git branch on the > > git > > server. Then we have a git hook that writes to the fifo when > > someone > > pushes changes. > > > > Then on build server, we run a daemon that ssh to the git > > server and > > read from the fifo. For each read line, it rebuilds and > > uploads new > > packages. That way we avoid race conditions when 2 different > > developers > > pushes stuff at the same time and we avoid the delays we would > > have when > > using cron jobs. > > > > Then I wanted this to run as a daemon, that starts/stops from > > an init.d > > script so it starts up automatically after a reboot. Having it > > start was > > easy. I wrote a daemon in shell for that. Having it stop was > > not so > > easy, since when the daeomn was killed the ssh process was > > left running. > > I could not solve that in shell, so i rewrite it in lua. Lua > > lacks > > signal handling so I ended up rewrite it in C. After a bug > > that > > forkbombed my vserver host to death (!!?) and a bug that > > killed all the > > packages on main mirror we are finally back and things are > > working. > > > > The new build server have 8 horses that kicks so if a > > developer wants a > > build environment for kernel hacking or similar, just let me > > know. > > > > -nc > > > > > > > > --- > > Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org > > Help: alpine-devel+help@lists.alpinelinux.org > > --- > > > > > > > --00032557b4d25a8bd2047719b1cc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Well, I think there would be several alternatives.=A0 I suppose you could e= ven use XMPP, but that would require writing a lot of the plumbing yourself= .=A0

Here are a couple alternatives, although I have not used them:=

- RabbitMQ - [http://www.rabbitmq.c= om] - a messaging broker that seems to be much smaller (around 1MB?) in= stall.=A0 Have not seen exactly what the requirements are for running it, b= ut seems simple.

I have not found documentation on how to connect to it and communicate = through simple telnet commands, but there are integrations in many language= s.=A0 For instance, here's one in Perl that uses Stomp (mentioned below= ):
http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/example= s/perl/rabbitmq_stomp_send.pl

- Stomp [http://stomp.codehaus.o= rg/] - easy to implement the clients, although you will need to couple = it with a message broker (RabbitMQ / ActiveMQ / etc).=A0 It has a lightweig= ht broker that's written in Ruby - not sure if that will fit your desir= ed footprint.

Jeremy

On Thu, Oct 29, 2009 at 12:00 = PM, Natanael Copa <natanael.copa@gmail.com> wrote:
On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote= :
> Wow - you must never sleep! =A0Congratulations. This is a great
> improvement.
>
> PS - you probably considered this, but in the future if you want an > easy way to communicate a queue or event triggering between multiple > servers, try ActiveMQ. =A0It's an enterprise messaging bus and has=
> clients in pretty much any language. =A0You can even set it up to
> receive or broadcast events through simple HTTP if you need to.
> Anyway, I've used it on a bunch of projects and it's super eas= y to set
> up and a great piece of software. =A0Using something like that would > even give you the ability to have multiple build servers subscribing > to the channel and either building all packages, or dividing up the > packages, each building one until the queue is done.
>
> Just a thought anyway....

Yeah, I would actually like/need something like that. What I have sti= ll
dont work perfectly. ssh seesions are not terminated on server.

I'd need a:
* git hook that sends a single line to the message server
* clinents/buildserver(s) that just sits idle til they get the message
* something that does nto require java and is alot lighter than 33MB.

in other words, is there anything simpler/lighter than ActiveMQ?

Thanks!

-nc

> Jeremy Thomerson
>
> On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
> <natanael.copa@gmail.com= > wrote:
> =A0 =A0 =A0 =A0 Hi,
>
> =A0 =A0 =A0 =A0 We have got a new build server. I have spent last days= on
> =A0 =A0 =A0 =A0 setting it up.
> =A0 =A0 =A0 =A0 The idea is that we want a git hook to trigger buildin= g
> =A0 =A0 =A0 =A0 updated packages
> =A0 =A0 =A0 =A0 and uploading them to master mirror. Since git server = and
> =A0 =A0 =A0 =A0 build server
> =A0 =A0 =A0 =A0 are 2 different we could have solved it with a cron jo= b. Not
> =A0 =A0 =A0 =A0 good enough
> =A0 =A0 =A0 =A0 I thought...
>
> =A0 =A0 =A0 =A0 So, what we do is we create a fifo for each git branch= on the
> =A0 =A0 =A0 =A0 git
> =A0 =A0 =A0 =A0 server. Then we have a git hook that writes to the fif= o when
> =A0 =A0 =A0 =A0 someone
> =A0 =A0 =A0 =A0 pushes changes.
>
> =A0 =A0 =A0 =A0 Then on build server, we run a daemon that ssh to the = git
> =A0 =A0 =A0 =A0 server and
> =A0 =A0 =A0 =A0 read from the fifo. For each read line, it rebuilds an= d
> =A0 =A0 =A0 =A0 uploads new
> =A0 =A0 =A0 =A0 packages. That way we avoid race conditions when 2 dif= ferent
> =A0 =A0 =A0 =A0 developers
> =A0 =A0 =A0 =A0 pushes stuff at the same time and we avoid the delays = we would
> =A0 =A0 =A0 =A0 have when
> =A0 =A0 =A0 =A0 using cron jobs.
>
> =A0 =A0 =A0 =A0 Then I wanted this to run as a daemon, that starts/sto= ps from
> =A0 =A0 =A0 =A0 an init.d
> =A0 =A0 =A0 =A0 script so it starts up automatically after a reboot. H= aving it
> =A0 =A0 =A0 =A0 start was
> =A0 =A0 =A0 =A0 easy. I wrote a daemon in shell for that. Having it st= op was
> =A0 =A0 =A0 =A0 not so
> =A0 =A0 =A0 =A0 easy, since when the daeomn was killed the ssh process= was
> =A0 =A0 =A0 =A0 left running.
> =A0 =A0 =A0 =A0 I could not solve that in shell, so i rewrite it in lu= a. Lua
> =A0 =A0 =A0 =A0 lacks
> =A0 =A0 =A0 =A0 signal handling so I ended up rewrite it in C. After a= bug
> =A0 =A0 =A0 =A0 that
> =A0 =A0 =A0 =A0 forkbombed my vserver host to death (!!?) and a bug th= at
> =A0 =A0 =A0 =A0 killed all the
> =A0 =A0 =A0 =A0 packages on main mirror we are finally back and things= are
> =A0 =A0 =A0 =A0 working.
>
> =A0 =A0 =A0 =A0 The new build server have 8 horses that kicks so if a<= br> > =A0 =A0 =A0 =A0 developer wants a
> =A0 =A0 =A0 =A0 build environment for kernel hacking or similar, just = let me
> =A0 =A0 =A0 =A0 know.
>
> =A0 =A0 =A0 =A0 -nc
>
>
>
> =A0 =A0 =A0 =A0 ---
> =A0 =A0 =A0 =A0 Unsubscribe: =A0alpine-devel+unsubscribe@lists.alpinelinux.or= g
> =A0 =A0 =A0 =A0 Help: =A0 =A0 =A0 =A0 alpine-devel+help@lists.alpinelinux.org > =A0 =A0 =A0 =A0 ---
>
>



--00032557b4d25a8bd2047719b1cc-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org ---