~alpine/devel

6 3

[alpine-devel] new build server

Details
Message ID
<1256801075.20939.2767.camel@ncopa-desktop.nor.wtbts.net>
Sender timestamp
1256801075
DKIM signature
missing
Download raw message
Hi,

We have got a new build server. I have spent last days on setting it up.
The idea is that we want a git hook to trigger building updated packages
and uploading them to master mirror. Since git server and build server
are 2 different we could have solved it with a cron job. Not good enough
I thought...

So, what we do is we create a fifo for each git branch on the git
server. Then we have a git hook that writes to the fifo when someone
pushes changes.

Then on build server, we run a daemon that ssh to the git server and
read from the fifo. For each read line, it rebuilds and uploads new
packages. That way we avoid race conditions when 2 different developers
pushes stuff at the same time and we avoid the delays we would have when
using cron jobs.

Then I wanted this to run as a daemon, that starts/stops from an init.d
script so it starts up automatically after a reboot. Having it start was
easy. I wrote a daemon in shell for that. Having it stop was not so
easy, since when the daeomn was killed the ssh process was left running.
I could not solve that in shell, so i rewrite it in lua. Lua lacks
signal handling so I ended up rewrite it in C. After a bug that
forkbombed my vserver host to death (!!?) and a bug that killed all the
packages on main mirror we are finally back and things are working.

The new build server have 8 horses that kicks so if a developer wants a
build environment for kernel hacking or similar, just let me know.

-nc



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Jeremy Thomerson <jeremy@thomersonfamily.com>
Details
Message ID
<6602aeca0910290646nfb59f7et4acfec11314953f@mail.gmail.com>
In-Reply-To
<1256801075.20939.2767.camel@ncopa-desktop.nor.wtbts.net> (view parent)
Sender timestamp
1256823966
DKIM signature
missing
Download raw message
Wow - you must never sleep!  Congratulations. This is a great improvement.

PS - you probably considered this, but in the future if you want an easy way
to communicate a queue or event triggering between multiple servers, try
ActiveMQ.  It's an enterprise messaging bus and has clients in pretty much
any language.  You can even set it up to receive or broadcast events through
simple HTTP if you need to.  Anyway, I've used it on a bunch of projects and
it's super easy to set up and a great piece of software.  Using something
like that would even give you the ability to have multiple build servers
subscribing to the channel and either building all packages, or dividing up
the packages, each building one until the queue is done.

Just a thought anyway....

Jeremy Thomerson

On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa <natanael.copa@gmail.com>wrote:

> Hi,
>
> We have got a new build server. I have spent last days on setting it up.
> The idea is that we want a git hook to trigger building updated packages
> and uploading them to master mirror. Since git server and build server
> are 2 different we could have solved it with a cron job. Not good enough
> I thought...
>
> So, what we do is we create a fifo for each git branch on the git
> server. Then we have a git hook that writes to the fifo when someone
> pushes changes.
>
> Then on build server, we run a daemon that ssh to the git server and
> read from the fifo. For each read line, it rebuilds and uploads new
> packages. That way we avoid race conditions when 2 different developers
> pushes stuff at the same time and we avoid the delays we would have when
> using cron jobs.
>
> Then I wanted this to run as a daemon, that starts/stops from an init.d
> script so it starts up automatically after a reboot. Having it start was
> easy. I wrote a daemon in shell for that. Having it stop was not so
> easy, since when the daeomn was killed the ssh process was left running.
> I could not solve that in shell, so i rewrite it in lua. Lua lacks
> signal handling so I ended up rewrite it in C. After a bug that
> forkbombed my vserver host to death (!!?) and a bug that killed all the
> packages on main mirror we are finally back and things are working.
>
> The new build server have 8 horses that kicks so if a developer wants a
> build environment for kernel hacking or similar, just let me know.
>
> -nc
>
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org<alpine-devel%2Bunsubscribe@lists.alpinelinux.org>
> Help:         alpine-devel+help@lists.alpinelinux.org<alpine-devel%2Bhelp@lists.alpinelinux.org>
> ---
>
>
Jeremy Thomerson <jeremy@thomersonfamily.com>
Details
Message ID
<6602aeca0910291435geb90e6erb2d295242beb96a3@mail.gmail.com>
In-Reply-To
<1256835607.20939.3220.camel@ncopa-desktop.nor.wtbts.net> (view parent)
Sender timestamp
1256852140
DKIM signature
missing
Download raw message
Well, I think there would be several alternatives.  I suppose you could even
use XMPP, but that would require writing a lot of the plumbing yourself.

Here are a couple alternatives, although I have not used them:

- RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems to be
much smaller (around 1MB?) install.  Have not seen exactly what the
requirements are for running it, but seems simple.

I have not found documentation on how to connect to it and communicate
through simple telnet commands, but there are integrations in many
languages.  For instance, here's one in Perl that uses Stomp (mentioned
below):
http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl

- Stomp [http://stomp.codehaus.org/] - easy to implement the clients,
although you will need to couple it with a message broker (RabbitMQ /
ActiveMQ / etc).  It has a lightweight broker that's written in Ruby - not
sure if that will fit your desired footprint.

Jeremy

On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa <natanael.copa@gmail.com>wrote:

> On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
> > Wow - you must never sleep!  Congratulations. This is a great
> > improvement.
> >
> > PS - you probably considered this, but in the future if you want an
> > easy way to communicate a queue or event triggering between multiple
> > servers, try ActiveMQ.  It's an enterprise messaging bus and has
> > clients in pretty much any language.  You can even set it up to
> > receive or broadcast events through simple HTTP if you need to.
> > Anyway, I've used it on a bunch of projects and it's super easy to set
> > up and a great piece of software.  Using something like that would
> > even give you the ability to have multiple build servers subscribing
> > to the channel and either building all packages, or dividing up the
> > packages, each building one until the queue is done.
> >
> > Just a thought anyway....
>
> Yeah, I would actually like/need something like that. What I have still
> dont work perfectly. ssh seesions are not terminated on server.
>
> I'd need a:
> * git hook that sends a single line to the message server
> * clinents/buildserver(s) that just sits idle til they get the message
> * something that does nto require java and is alot lighter than 33MB.
>
> in other words, is there anything simpler/lighter than ActiveMQ?
>
> Thanks!
>
> -nc
>
> > Jeremy Thomerson
> >
> > On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
> > <natanael.copa@gmail.com> wrote:
> >         Hi,
> >
> >         We have got a new build server. I have spent last days on
> >         setting it up.
> >         The idea is that we want a git hook to trigger building
> >         updated packages
> >         and uploading them to master mirror. Since git server and
> >         build server
> >         are 2 different we could have solved it with a cron job. Not
> >         good enough
> >         I thought...
> >
> >         So, what we do is we create a fifo for each git branch on the
> >         git
> >         server. Then we have a git hook that writes to the fifo when
> >         someone
> >         pushes changes.
> >
> >         Then on build server, we run a daemon that ssh to the git
> >         server and
> >         read from the fifo. For each read line, it rebuilds and
> >         uploads new
> >         packages. That way we avoid race conditions when 2 different
> >         developers
> >         pushes stuff at the same time and we avoid the delays we would
> >         have when
> >         using cron jobs.
> >
> >         Then I wanted this to run as a daemon, that starts/stops from
> >         an init.d
> >         script so it starts up automatically after a reboot. Having it
> >         start was
> >         easy. I wrote a daemon in shell for that. Having it stop was
> >         not so
> >         easy, since when the daeomn was killed the ssh process was
> >         left running.
> >         I could not solve that in shell, so i rewrite it in lua. Lua
> >         lacks
> >         signal handling so I ended up rewrite it in C. After a bug
> >         that
> >         forkbombed my vserver host to death (!!?) and a bug that
> >         killed all the
> >         packages on main mirror we are finally back and things are
> >         working.
> >
> >         The new build server have 8 horses that kicks so if a
> >         developer wants a
> >         build environment for kernel hacking or similar, just let me
> >         know.
> >
> >         -nc
> >
> >
> >
> >         ---
> >         Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org<alpine-devel%2Bunsubscribe@lists.alpinelinux.org>
> >         Help:         alpine-devel+help@lists.alpinelinux.org<alpine-devel%2Bhelp@lists.alpinelinux.org>
> >         ---
> >
> >
>
>
>
Details
Message ID
<1256835607.20939.3220.camel@ncopa-desktop.nor.wtbts.net>
In-Reply-To
<6602aeca0910290646nfb59f7et4acfec11314953f@mail.gmail.com> (view parent)
Sender timestamp
1256835607
DKIM signature
missing
Download raw message
On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
> Wow - you must never sleep!  Congratulations. This is a great
> improvement.
> 
> PS - you probably considered this, but in the future if you want an
> easy way to communicate a queue or event triggering between multiple
> servers, try ActiveMQ.  It's an enterprise messaging bus and has
> clients in pretty much any language.  You can even set it up to
> receive or broadcast events through simple HTTP if you need to.
> Anyway, I've used it on a bunch of projects and it's super easy to set
> up and a great piece of software.  Using something like that would
> even give you the ability to have multiple build servers subscribing
> to the channel and either building all packages, or dividing up the
> packages, each building one until the queue is done.
> 
> Just a thought anyway....

Yeah, I would actually like/need something like that. What I have still
dont work perfectly. ssh seesions are not terminated on server.

I'd need a:
* git hook that sends a single line to the message server
* clinents/buildserver(s) that just sits idle til they get the message
* something that does nto require java and is alot lighter than 33MB.

in other words, is there anything simpler/lighter than ActiveMQ?

Thanks!

-nc

> Jeremy Thomerson
> 
> On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
> <natanael.copa@gmail.com> wrote:
>         Hi,
>         
>         We have got a new build server. I have spent last days on
>         setting it up.
>         The idea is that we want a git hook to trigger building
>         updated packages
>         and uploading them to master mirror. Since git server and
>         build server
>         are 2 different we could have solved it with a cron job. Not
>         good enough
>         I thought...
>         
>         So, what we do is we create a fifo for each git branch on the
>         git
>         server. Then we have a git hook that writes to the fifo when
>         someone
>         pushes changes.
>         
>         Then on build server, we run a daemon that ssh to the git
>         server and
>         read from the fifo. For each read line, it rebuilds and
>         uploads new
>         packages. That way we avoid race conditions when 2 different
>         developers
>         pushes stuff at the same time and we avoid the delays we would
>         have when
>         using cron jobs.
>         
>         Then I wanted this to run as a daemon, that starts/stops from
>         an init.d
>         script so it starts up automatically after a reboot. Having it
>         start was
>         easy. I wrote a daemon in shell for that. Having it stop was
>         not so
>         easy, since when the daeomn was killed the ssh process was
>         left running.
>         I could not solve that in shell, so i rewrite it in lua. Lua
>         lacks
>         signal handling so I ended up rewrite it in C. After a bug
>         that
>         forkbombed my vserver host to death (!!?) and a bug that
>         killed all the
>         packages on main mirror we are finally back and things are
>         working.
>         
>         The new build server have 8 horses that kicks so if a
>         developer wants a
>         build environment for kernel hacking or similar, just let me
>         know.
>         
>         -nc
>         
>         
>         
>         ---
>         Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
>         Help:         alpine-devel+help@lists.alpinelinux.org
>         ---
>         
> 




---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Details
Message ID
<96a6f50a0910301344n61c5e55btf188997b490805b3@mail.gmail.com>
In-Reply-To
<6602aeca0910291435geb90e6erb2d295242beb96a3@mail.gmail.com> (view parent)
Sender timestamp
1256935455
DKIM signature
missing
Download raw message
Hi!
Could prosody (http://prosody.im) be used for these needs (it's lightweight).
It supports "Bidirectional-streams Over Synchronous HTTP (BOSH)".

<<mhavela>>


On Thu, Oct 29, 2009 at 10:35 PM, Jeremy Thomerson
<jeremy@thomersonfamily.com> wrote:
> Well, I think there would be several alternatives.  I suppose you could even
> use XMPP, but that would require writing a lot of the plumbing yourself.
>
> Here are a couple alternatives, although I have not used them:
>
> - RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems to be
> much smaller (around 1MB?) install.  Have not seen exactly what the
> requirements are for running it, but seems simple.
>
> I have not found documentation on how to connect to it and communicate
> through simple telnet commands, but there are integrations in many
> languages.  For instance, here's one in Perl that uses Stomp (mentioned
> below):
> http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl
>
> - Stomp [http://stomp.codehaus.org/] - easy to implement the clients,
> although you will need to couple it with a message broker (RabbitMQ /
> ActiveMQ / etc).  It has a lightweight broker that's written in Ruby - not
> sure if that will fit your desired footprint.
>
> Jeremy
>
> On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa <natanael.copa@gmail.com>
> wrote:
>>
>> On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
>> > Wow - you must never sleep!  Congratulations. This is a great
>> > improvement.
>> >
>> > PS - you probably considered this, but in the future if you want an
>> > easy way to communicate a queue or event triggering between multiple
>> > servers, try ActiveMQ.  It's an enterprise messaging bus and has
>> > clients in pretty much any language.  You can even set it up to
>> > receive or broadcast events through simple HTTP if you need to.
>> > Anyway, I've used it on a bunch of projects and it's super easy to set
>> > up and a great piece of software.  Using something like that would
>> > even give you the ability to have multiple build servers subscribing
>> > to the channel and either building all packages, or dividing up the
>> > packages, each building one until the queue is done.
>> >
>> > Just a thought anyway....
>>
>> Yeah, I would actually like/need something like that. What I have still
>> dont work perfectly. ssh seesions are not terminated on server.
>>
>> I'd need a:
>> * git hook that sends a single line to the message server
>> * clinents/buildserver(s) that just sits idle til they get the message
>> * something that does nto require java and is alot lighter than 33MB.
>>
>> in other words, is there anything simpler/lighter than ActiveMQ?
>>
>> Thanks!
>>
>> -nc
>>
>> > Jeremy Thomerson
>> >
>> > On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
>> > <natanael.copa@gmail.com> wrote:
>> >         Hi,
>> >
>> >         We have got a new build server. I have spent last days on
>> >         setting it up.
>> >         The idea is that we want a git hook to trigger building
>> >         updated packages
>> >         and uploading them to master mirror. Since git server and
>> >         build server
>> >         are 2 different we could have solved it with a cron job. Not
>> >         good enough
>> >         I thought...
>> >
>> >         So, what we do is we create a fifo for each git branch on the
>> >         git
>> >         server. Then we have a git hook that writes to the fifo when
>> >         someone
>> >         pushes changes.
>> >
>> >         Then on build server, we run a daemon that ssh to the git
>> >         server and
>> >         read from the fifo. For each read line, it rebuilds and
>> >         uploads new
>> >         packages. That way we avoid race conditions when 2 different
>> >         developers
>> >         pushes stuff at the same time and we avoid the delays we would
>> >         have when
>> >         using cron jobs.
>> >
>> >         Then I wanted this to run as a daemon, that starts/stops from
>> >         an init.d
>> >         script so it starts up automatically after a reboot. Having it
>> >         start was
>> >         easy. I wrote a daemon in shell for that. Having it stop was
>> >         not so
>> >         easy, since when the daeomn was killed the ssh process was
>> >         left running.
>> >         I could not solve that in shell, so i rewrite it in lua. Lua
>> >         lacks
>> >         signal handling so I ended up rewrite it in C. After a bug
>> >         that
>> >         forkbombed my vserver host to death (!!?) and a bug that
>> >         killed all the
>> >         packages on main mirror we are finally back and things are
>> >         working.
>> >
>> >         The new build server have 8 horses that kicks so if a
>> >         developer wants a
>> >         build environment for kernel hacking or similar, just let me
>> >         know.
>> >
>> >         -nc
>> >
>> >
>> >
>> >         ---
>> >         Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
>> >         Help:         alpine-devel+help@lists.alpinelinux.org
>> >         ---
>> >
>> >
>>
>>
>
>


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Details
Message ID
<1257154633.7454.202.camel@ncopa-desktop.nor.wtbts.net>
In-Reply-To
<6602aeca0910291435geb90e6erb2d295242beb96a3@mail.gmail.com> (view parent)
Sender timestamp
1257154633
DKIM signature
missing
Download raw message
On Thu, 2009-10-29 at 16:35 -0500, Jeremy Thomerson wrote:
> Well, I think there would be several alternatives.  I suppose you
> could even use XMPP, but that would require writing a lot of the
> plumbing yourself.  
> 
> Here are a couple alternatives, although I have not used them:
> 
> - RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems
> to be much smaller (around 1MB?) install.  Have not seen exactly what
> the requirements are for running it, but seems simple.

looks kinda cool. seems to be written in erlang.

> I have not found documentation on how to connect to it and communicate
> through simple telnet commands, but there are integrations in many
> languages.  For instance, here's one in Perl that uses Stomp
> (mentioned below):
> http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl
> 
> - Stomp [http://stomp.codehaus.org/] - easy to implement the clients,
> although you will need to couple it with a message broker (RabbitMQ /
> ActiveMQ / etc).  It has a lightweight broker that's written in Ruby -
> not sure if that will fit your desired footprint.

I think we could use IRC actually. ngirc is smallish and you have tiny
IRC clients like sic and ii which could be used with a script or
modified to do what we want.


> Jeremy
> 
> On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa
> <natanael.copa@gmail.com> wrote:
>         On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
>         > Wow - you must never sleep!  Congratulations. This is a
>         great
>         > improvement.
>         >
>         > PS - you probably considered this, but in the future if you
>         want an
>         > easy way to communicate a queue or event triggering between
>         multiple
>         > servers, try ActiveMQ.  It's an enterprise messaging bus and
>         has
>         > clients in pretty much any language.  You can even set it up
>         to
>         > receive or broadcast events through simple HTTP if you need
>         to.
>         > Anyway, I've used it on a bunch of projects and it's super
>         easy to set
>         > up and a great piece of software.  Using something like that
>         would
>         > even give you the ability to have multiple build servers
>         subscribing
>         > to the channel and either building all packages, or dividing
>         up the
>         > packages, each building one until the queue is done.
>         >
>         > Just a thought anyway....
>         
>         
>         Yeah, I would actually like/need something like that. What I
>         have still
>         dont work perfectly. ssh seesions are not terminated on
>         server.
>         
>         I'd need a:
>         * git hook that sends a single line to the message server
>         * clinents/buildserver(s) that just sits idle til they get the
>         message
>         * something that does nto require java and is alot lighter
>         than 33MB.
>         
>         in other words, is there anything simpler/lighter than
>         ActiveMQ?
>         
>         Thanks!
>         
>         -nc
>         
>         
>         > Jeremy Thomerson
>         >
>         > On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
>         > <natanael.copa@gmail.com> wrote:
>         >         Hi,
>         >
>         >         We have got a new build server. I have spent last
>         days on
>         >         setting it up.
>         >         The idea is that we want a git hook to trigger
>         building
>         >         updated packages
>         >         and uploading them to master mirror. Since git
>         server and
>         >         build server
>         >         are 2 different we could have solved it with a cron
>         job. Not
>         >         good enough
>         >         I thought...
>         >
>         >         So, what we do is we create a fifo for each git
>         branch on the
>         >         git
>         >         server. Then we have a git hook that writes to the
>         fifo when
>         >         someone
>         >         pushes changes.
>         >
>         >         Then on build server, we run a daemon that ssh to
>         the git
>         >         server and
>         >         read from the fifo. For each read line, it rebuilds
>         and
>         >         uploads new
>         >         packages. That way we avoid race conditions when 2
>         different
>         >         developers
>         >         pushes stuff at the same time and we avoid the
>         delays we would
>         >         have when
>         >         using cron jobs.
>         >
>         >         Then I wanted this to run as a daemon, that
>         starts/stops from
>         >         an init.d
>         >         script so it starts up automatically after a reboot.
>         Having it
>         >         start was
>         >         easy. I wrote a daemon in shell for that. Having it
>         stop was
>         >         not so
>         >         easy, since when the daeomn was killed the ssh
>         process was
>         >         left running.
>         >         I could not solve that in shell, so i rewrite it in
>         lua. Lua
>         >         lacks
>         >         signal handling so I ended up rewrite it in C. After
>         a bug
>         >         that
>         >         forkbombed my vserver host to death (!!?) and a bug
>         that
>         >         killed all the
>         >         packages on main mirror we are finally back and
>         things are
>         >         working.
>         >
>         >         The new build server have 8 horses that kicks so if
>         a
>         >         developer wants a
>         >         build environment for kernel hacking or similar,
>         just let me
>         >         know.
>         >
>         >         -nc
>         >
>         >
>         >
>         >         ---
>         >         Unsubscribe:  alpine-devel
>         +unsubscribe@lists.alpinelinux.org
>         >         Help:         alpine-devel
>         +help@lists.alpinelinux.org
>         >         ---
>         >
>         >
>         
>         
>         
> 




---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Details
Message ID
<1257155060.7454.217.camel@ncopa-desktop.nor.wtbts.net>
In-Reply-To
<96a6f50a0910301344n61c5e55btf188997b490805b3@mail.gmail.com> (view parent)
Sender timestamp
1257155060
DKIM signature
missing
Download raw message
On Fri, 2009-10-30 at 21:44 +0100, Mika Havela wrote:
> Hi!
> Could prosody (http://prosody.im) be used for these needs (it's lightweight).
> It supports "Bidirectional-streams Over Synchronous HTTP (BOSH)".

I think it could, but I was even thinking of using IRC. ngircd running
as a messag server. You "subscribe" to a channel. sic or ii (or a
modified version of sic) could be used as the client. I think that would
be even lighter and no deps more than uclibc.

> <<mhavela>>
> 
> 
> On Thu, Oct 29, 2009 at 10:35 PM, Jeremy Thomerson
> <jeremy@thomersonfamily.com> wrote:
> > Well, I think there would be several alternatives.  I suppose you could even
> > use XMPP, but that would require writing a lot of the plumbing yourself.
> >
> > Here are a couple alternatives, although I have not used them:
> >
> > - RabbitMQ - [http://www.rabbitmq.com] - a messaging broker that seems to be
> > much smaller (around 1MB?) install.  Have not seen exactly what the
> > requirements are for running it, but seems simple.
> >
> > I have not found documentation on how to connect to it and communicate
> > through simple telnet commands, but there are integrations in many
> > languages.  For instance, here's one in Perl that uses Stomp (mentioned
> > below):
> > http://hg.rabbitmq.com/rabbitmq-stomp/file/tip/examples/perl/rabbitmq_stomp_send.pl
> >
> > - Stomp [http://stomp.codehaus.org/] - easy to implement the clients,
> > although you will need to couple it with a message broker (RabbitMQ /
> > ActiveMQ / etc).  It has a lightweight broker that's written in Ruby - not
> > sure if that will fit your desired footprint.
> >
> > Jeremy
> >
> > On Thu, Oct 29, 2009 at 12:00 PM, Natanael Copa <natanael.copa@gmail.com>
> > wrote:
> >>
> >> On Thu, 2009-10-29 at 08:46 -0500, Jeremy Thomerson wrote:
> >> > Wow - you must never sleep!  Congratulations. This is a great
> >> > improvement.
> >> >
> >> > PS - you probably considered this, but in the future if you want an
> >> > easy way to communicate a queue or event triggering between multiple
> >> > servers, try ActiveMQ.  It's an enterprise messaging bus and has
> >> > clients in pretty much any language.  You can even set it up to
> >> > receive or broadcast events through simple HTTP if you need to.
> >> > Anyway, I've used it on a bunch of projects and it's super easy to set
> >> > up and a great piece of software.  Using something like that would
> >> > even give you the ability to have multiple build servers subscribing
> >> > to the channel and either building all packages, or dividing up the
> >> > packages, each building one until the queue is done.
> >> >
> >> > Just a thought anyway....
> >>
> >> Yeah, I would actually like/need something like that. What I have still
> >> dont work perfectly. ssh seesions are not terminated on server.
> >>
> >> I'd need a:
> >> * git hook that sends a single line to the message server
> >> * clinents/buildserver(s) that just sits idle til they get the message
> >> * something that does nto require java and is alot lighter than 33MB.
> >>
> >> in other words, is there anything simpler/lighter than ActiveMQ?
> >>
> >> Thanks!
> >>
> >> -nc
> >>
> >> > Jeremy Thomerson
> >> >
> >> > On Thu, Oct 29, 2009 at 2:24 AM, Natanael Copa
> >> > <natanael.copa@gmail.com> wrote:
> >> >         Hi,
> >> >
> >> >         We have got a new build server. I have spent last days on
> >> >         setting it up.
> >> >         The idea is that we want a git hook to trigger building
> >> >         updated packages
> >> >         and uploading them to master mirror. Since git server and
> >> >         build server
> >> >         are 2 different we could have solved it with a cron job. Not
> >> >         good enough
> >> >         I thought...
> >> >
> >> >         So, what we do is we create a fifo for each git branch on the
> >> >         git
> >> >         server. Then we have a git hook that writes to the fifo when
> >> >         someone
> >> >         pushes changes.
> >> >
> >> >         Then on build server, we run a daemon that ssh to the git
> >> >         server and
> >> >         read from the fifo. For each read line, it rebuilds and
> >> >         uploads new
> >> >         packages. That way we avoid race conditions when 2 different
> >> >         developers
> >> >         pushes stuff at the same time and we avoid the delays we would
> >> >         have when
> >> >         using cron jobs.
> >> >
> >> >         Then I wanted this to run as a daemon, that starts/stops from
> >> >         an init.d
> >> >         script so it starts up automatically after a reboot. Having it
> >> >         start was
> >> >         easy. I wrote a daemon in shell for that. Having it stop was
> >> >         not so
> >> >         easy, since when the daeomn was killed the ssh process was
> >> >         left running.
> >> >         I could not solve that in shell, so i rewrite it in lua. Lua
> >> >         lacks
> >> >         signal handling so I ended up rewrite it in C. After a bug
> >> >         that
> >> >         forkbombed my vserver host to death (!!?) and a bug that
> >> >         killed all the
> >> >         packages on main mirror we are finally back and things are
> >> >         working.
> >> >
> >> >         The new build server have 8 horses that kicks so if a
> >> >         developer wants a
> >> >         build environment for kernel hacking or similar, just let me
> >> >         know.
> >> >
> >> >         -nc
> >> >
> >> >
> >> >
> >> >         ---
> >> >         Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> >> >         Help:         alpine-devel+help@lists.alpinelinux.org
> >> >         ---
> >> >
> >> >
> >>
> >>
> >
> >
> 
> 
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
> 




---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Reply to thread Export thread (mbox)