Subject: [alpine-devel] new build server

From: Natanael Copa <[email-masked]>
Date: Thu, 29 Oct 2009 08:24:35 +0100

Hi,

We have got a new build server. I have spent last days on setting it up.
The idea is that we want a git hook to trigger building updated packages
and uploading them to master mirror. Since git server and build server
are 2 different we could have solved it with a cron job. Not good enough
I thought...

So, what we do is we create a fifo for each git branch on the git
server. Then we have a git hook that writes to the fifo when someone
pushes changes.

Then on build server, we run a daemon that ssh to the git server and
read from the fifo. For each read line, it rebuilds and uploads new
packages. That way we avoid race conditions when 2 different developers
pushes stuff at the same time and we avoid the delays we would have when
using cron jobs.

Then I wanted this to run as a daemon, that starts/stops from an init.d
script so it starts up automatically after a reboot. Having it start was
easy. I wrote a daemon in shell for that. Having it stop was not so
easy, since when the daeomn was killed the ssh process was left running.
I could not solve that in shell, so i rewrite it in lua. Lua lacks
signal handling so I ended up rewrite it in C. After a bug that
forkbombed my vserver host to death (!!?) and a bug that killed all the
packages on main mirror we are finally back and things are working.

The new build server have 8 horses that kicks so if a developer wants a
build environment for kernel hacking or similar, just let me know.

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Jeremy Thomerson <[email-masked]>
Date: Thu, 29 Oct 2009 08:46:06 -0500

Wow - you must never sleep!  Congratulations. This is a great improvement.

PS - you probably considered this, but in the future if you want an easy way
to communicate a queue or event triggering between multiple servers, try
ActiveMQ.  It's an enterprise messaging bus and has clients in pretty much
any language.  You can even set it up to receive or broadcast events through
simple HTTP if you need to.  Anyway, I've used it on a bunch of projects and
it's super easy to set up and a great piece of software.  Using something
like that would even give you the ability to have multiple build servers
subscribing to the channel and either building all packages, or dividing up
the packages, each building one until the queue is done.

Just a thought anyway....

Jeremy Thomerson

On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa <[email-masked]>wrote:

> Unsubscribe:  [email-masked]<[email-masked]>
> Help:         [email-masked]<[email-masked]>
> ---
>
>

---

From: Natanael Copa <[email-masked]>
Date: Thu, 29 Oct 2009 18:00:07 +0100

On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
> PS - you probably considered this, but in the future if you want an
> easy way to communicate a queue or event triggering between multiple
> servers, try ActiveMQ.  It's an enterprise messaging bus and has
> clients in pretty much any language.  You can even set it up to
> receive or broadcast events through simple HTTP if you need to.
> Anyway, I've used it on a bunch of projects and it's super easy to set
> up and a great piece of software.  Using something like that would
> even give you the ability to have multiple build servers subscribing
> to the channel and either building all packages, or dividing up the
> packages, each building one until the queue is done.
> 

Yeah, I would actually like/need something like that. What I have still
dont work perfectly. ssh seesions are not terminated on server.

I'd need a:
* git hook that sends a single line to the message server
* clinents/buildserver(s) that just sits idle til they get the message
* something that does nto require java and is alot lighter than 33MB.

in other words, is there anything simpler/lighter than ActiveMQ?

Thanks!

-nc

> Jeremy Thomerson
> 
> On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
> <[email-masked]> wrote:
>         Hi,
>         

---

---

From: Jeremy Thomerson <[email-masked]>
Date: Thu, 29 Oct 2009 16:35:40 -0500

Well, I think there would be several alternatives.  I suppose you could even
use XMPP, but that would require writing a lot of the plumbing yourself.

Here are a couple alternatives, although I have not used them:

- RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems to be
much smaller (around 1MB?) install.  Have not seen exactly what the
requirements are for running it, but seems simple.

I have not found documentation on how to connect to it and communicate
through simple telnet commands, but there are integrations in many
languages.  For instance, here's one in Perl that uses Stomp (mentioned
below):
http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl

- Stomp [http://stomp.codehaus.org/] - easy to implement the clients,
although you will need to couple it with a message broker (RabbitMQ /
ActiveMQ / etc).  It has a lightweight broker that's written in Ruby - not
sure if that will fit your desired footprint.

Jeremy

---

From: Mika Havela <[email-masked]>
Date: Fri, 30 Oct 2009 21:44:15 +0100

Hi!
Could prosody (http://prosody.im) be used for these needs (it's lightweight).
It supports "Bidirectional-streams Over Synchronous HTTP (BOSH)".

<<mhavela>>

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 02 Nov 2009 10:44:20 +0100

> It supports "Bidirectional-streams Over Synchronous HTTP (BOSH)".

I think it could, but I was even thinking of using IRC. ngircd running
as a messag server. You "subscribe" to a channel. sic or ii (or a
modified version of sic) could be used as the client. I think that would
be even lighter and no deps more than uclibc.

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 02 Nov 2009 10:37:13 +0100

On Thu, 2009-10-29 at 16:35 -0500, Jeremy Thomerson wrote:
> Well, I think there would be several alternatives.  I suppose you
> could even use XMPP, but that would require writing a lot of the
> plumbing yourself.  
> 
> - RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems
> to be much smaller (around 1MB?) install.  Have not seen exactly what
> the requirements are for running it, but seems simple.

looks kinda cool. seems to be written in erlang.

> languages.  For instance, here's one in Perl that uses Stomp
> (mentioned below):
> ActiveMQ / etc).  It has a lightweight broker that's written in Ruby -
> not sure if that will fit your desired footprint.

I think we could use IRC actually. ngirc is smallish and you have tiny
IRC clients like sic and ii which could be used with a script or
modified to do what we want.

> Jeremy
> 
> On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa
>         > Wow - you must never sleep!  Congratulations. This is a
>         great
>         > improvement.
>         >
>         >         We have got a new build server. I have spent last
>         days on
>         >         setting it up.
>         >         The idea is that we want a git hook to trigger
>         building
>         >         updated packages
>         >         and uploading them to master mirror. Since git
>         server and
>         >         build server
>         >         are 2 different we could have solved it with a cron
>         job. Not
>         >         good enough
>         >         I thought...
>         >
>         >         So, what we do is we create a fifo for each git
>         branch on the
>         >         git
>         >         server. Then we have a git hook that writes to the
>         fifo when
>         >         someone
>         >         pushes changes.
>         >
>         >         Then on build server, we run a daemon that ssh to
>         the git
>         >         server and
>         >         read from the fifo. For each read line, it rebuilds
>         and
>         >         uploads new
>         >         packages. That way we avoid race conditions when 2
>         different
>         >         developers
>         >         pushes stuff at the same time and we avoid the
>         delays we would
>         >         have when
>         >         using cron jobs.
>         >
>         >         Then I wanted this to run as a daemon, that
>         starts/stops from
>         >         an init.d
>         >         script so it starts up automatically after a reboot.
>         Having it
>         >         start was
>         >         easy. I wrote a daemon in shell for that. Having it
>         stop was
>         >         not so
>         >         easy, since when the daeomn was killed the ssh
>         process was
>         >         left running.
>         >         I could not solve that in shell, so i rewrite it in
>         lua. Lua
>         >         lacks
>         >         signal handling so I ended up rewrite it in C. After
>         a bug
>         >         that
>         >         forkbombed my vserver host to death (!!?) and a bug
>         that
>         >         killed all the
>         >         packages on main mirror we are finally back and
>         things are
>         >         working.
>         >
>         >         The new build server have 8 horses that kicks so if
>         a
>         >         developer wants a
>         >         build environment for kernel hacking or similar,
>         just let me
>         >         know.
>         >
>         >         -nc
>         >
>         >
>         >
>         >         ---

---