Subject: [alpine-devel] Understanding alpine-setup

From: [email-masked]
Date: Sun, 04 Mar 2012 04:07:40 -0500

Hi, I recently discovered Alpine, and it looks great! --- like most of
what I'd been hankering for when dissatisfied with Arch or FreeBSD. I've
tried it out on a test system and am eager to start using it in earnest.

I'm trying to do a somewhat custom install (onto a pre-partitioned
LVM-over-luks drive), and thought I should read through the setup
scripts to see what's going on. There's only the thinnest of information
available on the Wiki. (Only for LVM, not for luks, and not for LVM
during installation.)

But it looks like the installation scripts are either using some very
subtle tricks or are very much works in progress. I thought I had better
start asking questions.

In setup-alpine:

1. The "-q" option is parsed but not documented in usage. Intentionally?
2. The "-a" option is bewildering. If I supply it, we create a directory
under /tmp and save its name in ROOT. However we set a trap on exit to
remove this directory, and never cancel the trap. From what I've seen so
far, that ROOT directory is never copied anywhere else, right? So this
functionality doesn't yet work? Also, the ROOT variable isn't exported,
so I don't see how the subroutine scripts like
setup-{hostname,interfaces,keymap} expect to have it populated. Is all
of this just under construction?
3. And if I don't supply the "-a" option, then setup-alpine sets
PKGADD="apk add"; but then so far as I can see, PKGADD is never again
referenced in any of the setup-* scripts.

That's what puzzles me so far. Still reading...
-- 
Jim Pryor
[email-masked]

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Paul Onyschuk <[email-masked]>
Date: Sun, 4 Mar 2012 10:48:35 +0100

On Sun, 04 Mar 2012 04:07:40 -0500
jimpryor.net wrote:

> scripts to see what's going on. There's only the thinnest of
> information available on the Wiki. (Only for LVM, not for luks, and
> not for LVM during installation.)
> 

You are looking for "setup-disk" probably.  You can invoke this
manually and this part can be skipped in "setup-alpine" (I don't
remember anymore if there was a option to say no or you just provided
empty hdd name).

Just look at coLinux tutorial to get idea [1].  You will need to edit
fstab and so on by hand.  Alternatively you could use chroot-like
installation - just check wiki.

Be noticed that I ain't using Alpine for some time, so maybe someone
else will provide better answer.

[1] http://wiki.alpinelinux.org/wiki/Install_Alpine_on_coLinux#Setup

---

---

From: [email-masked]
Date: Sun, 04 Mar 2012 05:19:31 -0500

Thanks Paul for the reply; that helps give me some guidance.

I'm still reading the setup scripts to get a general overview though.
Some more questions...

* In setup-interfaces, in various places, the response "abort" is parsed
but undocumented. Intentional?

* Perhaps not a problem, but I notice that "none" is only accepted by
setup-lbu interactively. You can't pass it in via the LBUOPTS variable
in an ANSWERFILE read by alpine-setup. On the other hand, you can pass
"none" to setup-apkcache in that way. Just verifying that this
difference is intended. Is the idea that running setup-lbu isn't
optional if the diskmode isn't "sys"? (But then why permit "none" as an
interactive answer?)

* setup-cryptswap tries to apk add cryptsetup-luks, but this package
doesn't exist (anymore)? Should it just be cryptsetup?

About to dive into setup-bootable and setup-disk now. It looks like
setup-bootable (like setup-cryptswap and some others) is only invoked
manually, correct?

-- 
Jim Pryor
[email-masked]

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 5 Mar 2012 10:06:00 +0100

On Sun, 04 Mar 2012 05:19:31 -0500
[email-masked] wrote:

> * In setup-interfaces, in various places, the response "abort" is
> parsed but undocumented. Intentional?

Not intentional.

I had at some point the idea of having "abort" as a standard way to get
out. That idea never "took off" and have not been consequently
implemented.

Things like this should probably be documented *before* we start a new
setup script implementation.

> optional if the diskmode isn't "sys"? (But then why permit "none" as
> an interactive answer?)

No this is not intentional. This is a result of different contributors
having different ideas of how things should work. The unattended stuff
is something that I think needs to be re-implemented.
 
> doesn't exist (anymore)? Should it just be cryptsetup?

This is another example of a script that was written once many years
ago, but never used. When things moved on (switching to openrc) we
never verified that this still works. I probably does not and should
most likely be removed.
 
> setup-bootable (like setup-cryptswap and some others) is only invoked
 
Yes.

Setup-bootable is not really related the other setup-* scirpts. Its a
tool that helps you get the contents over from the .iso image to an
USB or CF. So its a tool to create new bootables from an already
installed system.

When it comes to setup-bootable, I am not 100% of how to handle its
future because it is supposed to solve various different (but still
simlar) problems. I'll summarize them up here:

* First time install. A user discovers the existance of Alpine Linux
  and want create a bootable USB for first time boot. He/she have only
  read about cdrom in the history books but have never seen any cd
  player. We distribute .iso only.

  Theoretically, its very simple. Just treat the .iso as an archive,
  extract the files to a FAT formatted usb and run syslinux. (and
  make sure that the MBR is ok). You can do this from windows with 7zip
  and syslinux.exe (and fdisk /mbr?) But it would be handy to have
  tools for it. (yeah I know about unetbootin and pendrive linux but i
  think those are clumsy)

* A long time Alpine Linux user wants create yet another boot USB. It
  could be for upgrading an existing install or setting up a new box.
  This is what setup-bootable solves.

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 5 Mar 2012 09:45:54 +0100

On Sun, 04 Mar 2012 04:07:40 -0500
[email-masked] wrote:

> what I'd been hankering for when dissatisfied with Arch or FreeBSD.
> I've tried it out on a test system and am eager to start using it in
> earnest.
> 

The setup-disk script does the install-to-disk part. It was meant as a
quick way to install the system on disk. It does not yet give all the
flexibility you would expect from a modern distro. It does allow you
create the partitions manually though.

Run setup-alpine and on quesion on disk you selece "none"/"no" etc.

Create the raid,LVM devices you want and mount up the dir structure on
some temp dir. For example /myroot.

Finally you run setup-disk with the mounted root as arg. For example:

  setup-disk /myroot

It should be able to figure out raid and lvm. How it handles luks i
don't know.

> subtle tricks or are very much works in progress. I thought I had
> better start asking questions.

The scripts grew out from "let-me-get this sytem run from ram
quickly" and have organically grown out from there.

> 1. The "-q" option is parsed but not documented in usage.

No.

I have always wanted it to give me a minimal running system asap rather
than ask about all kinds of different things the user might or might not
want.

People sent patches to setup-alpine that asked lots more questions so I
added the -q option. I just forgot add it the the usage output.

This is now fixed. (in bb0ceedbbc533e87ae9832ca107065186ad9fae1)

> 2. The "-a" option is bewildering. If I supply it, we
> create a directory under /tmp and save its name in ROOT. However we
> set a trap on exit to remove this directory, and never cancel the
> trap. From what I've seen so far, that ROOT directory is never copied
> anywhere else, right? So this functionality doesn't yet work? Also,
> the ROOT variable isn't exported, so I don't see how the subroutine
> scripts like setup-{hostname,interfaces,keymap} expect to have it
> populated. Is all of this just under construction?

heh...

I look at the commits. from 2006...

The idea was that you could run setup-alpine on any box and spit out a
configuration file (.apkovl.tar.gz) that you used other places.

Seems like I never completed that to work fully and, actually nobody
has ever asked for that functionality.

I wonder if we can just remove that since I think it will never be used
anyways?

> referenced in any of the setup-* scripts.

I wonder if the PKGADD variable has its origins from the time when we
didn't even have decided what package manager we would land on. I think
the idea was that we could test different package managers.

This is something that could go away.

> That's what puzzles me so far. Still reading...

Thanks for your feedback!

I have been thinking of rewriting the setup-* scripts in lua. There
have been need for unattended installs and better setup-disk script.
There was some unattended patches that i let in but I'm not 100% happy
with that implementation (and have personally neer used it).

The idea is that you could provide a file with all or parts of the info
and only get prompted of the missing stuff. I even had a proof of
concept for ACF at some point but ACF development took other direction.
Currently it appears clumsy to use ACF backend as its heavily web
centric.

Anyways... The reason that I haven't put more effort in cleaning up the
setup-* scripts is that I think the long time solution is to make
something in Lua. I just have not had the time to actually implement
it yet.

-nc

---

---

From: Jim Pryor <[email-masked]>
Date: Mon, 05 Mar 2012 05:03:34 -0500

> with that implementation (and have personally neer used it).
...
> something in Lua. I just have not had the time to actually implement

Thanks for your replies Natanael. Can you tell me where to look for any
earlier discussion of your dissatisfaction with the current setup
scripts?

Lua *is* a terrific and small-footprint language to use if something
stronger than ash is needed. But to my novice eyes the existing scripts
already look like a pretty good system. Maybe needing some tweaks, but
also quite flexible. Doesn't the "answerfile" functionality already do
this, for instance:

> and only get prompted of the missing stuff.

My main difficulty was the thin documentation, and not having a good
overview of how all the pieces fit together. I've now made a variety of
small wiki commits to help improve this for others. And the dense new
page I already mentioned,
<http://wiki.alpinelinux.org/wiki/Alpine_Setup_Scripts>.

-- 
Jim Pryor
[email-masked]

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 5 Mar 2012 15:21:03 +0100

On Mon, 05 Mar 2012 05:03:34 -0500
Jim Pryor <[email-masked]> wrote:

> > There was some unattended patches that i let in but I'm not 100%
> > happy with that implementation (and have personally neer used it).
> ...
> > Anyways... The reason that I haven't put more effort in cleaning up
> > the setup-* scripts is that I think the long time solution is to
> > make something in Lua. I just have not had the time to actually
> > implement it yet.
> 
> Thanks for your replies Natanael. Can you tell me where to look for
> any earlier discussion of your dissatisfaction with the current setup
> scripts?

It is probably who are mostly dissatisfied so this is something I have
been thinking for a while.

I searched the email archive and there have various discussions on how
to implement unattended installs. I mention Lua here:
http://lists.alpinelinux.org/alpine-devel/0957.html

I think it has been mentioned in IRC some time that it would have been
nice to integrate the setup-* scripts with ACF better. After all they
do partially same job.
 
> stronger than ash is needed. But to my novice eyes the existing
> scripts already look like a pretty good system.

Ok, thanks for this feedback. Very nice to hear this from someone who
have read the code and actually understands it :)

Yes, it does the job. We could improve the framework (I cloned some of
the functions from the openbsd installer functions and used them in
setup-vs-guest script:

http://git.alpinelinux.org/cgit/aports/tree/main/util-vserver/setup-vs-guest

But while working on all those scripts I have various times thought
that the shell things are a bit clumsy, and I miss arrays/hashtables.
Specially for things like setup-apkrepos, setup-interfaces and
setup-disk. I already have a lua moule in progress for parsing /sys and
fishing out disk and partition info in convenient data structures. Bash
is way to big be considered (it doesnt help that its slow either).

I must admit that using shell if possible is a good thing. I just feel
that we are at the limit where something stronger is wanted.

> Maybe needing some
> tweaks, but also quite flexible. Doesn't the "answerfile"
> functionality already do this, for instance:
> 
> > The idea is that you could provide a file with all or parts of the
> > info and only get prompted of the missing stuff.

It does. But I'm not too happy with the implementation. It was a bolted
on as an "afterthought".

It could of course be a "not-invented-by-me" syndrome :)

What I had in mind was something like a JSON file that holds the
answers:

{
  "hostname": "myalpine",
  "keyboard": "en",
  "timezone": "UTC",
  "interfaces": {
     "eth0": { "method": "static", .... }.
     "eth1": { "method": "dhcp" }
  },
  ...
}

This is very convenient with Lua.

Also, the idea is also to at some point have a GTK ui for this. Would
be nice to use same backend.

I guess this is was what ACF was originally supposed to do but ended up
as way too web (and html) centric.

Actually, ACF grew out from "we need to manage Alpine boxes via web and
we need to do it *now*". There was a thought that it should be a
general purpose configuration framework but ended up as a more or less
web only thing.

The setup-* scripts on the other hand grew out from "i want my alpine
box up and run as quick as possible" for development.

> overview of how all the pieces fit together. I've now made a variety
> of small wiki commits to help improve this for others. And the dense
> new page I already mentioned,
 
Very nice! Even if we switch to Lua, we need to be backward
compatible with the current script, at least to certain degree. So
those docs are definitively a keeper - and they were badly needed.

Now, I don't think we need decide what to do with setup-* scripts right
now. We have a feature freeze around 1 april and there is no chance that
we will reach implement setup-* in Lua before that so we need to polish
our current scripts and get the thing out. In other words: same as last
N release iterations...

Again, thanks!

-nc

---

---

From: Jim Pryor <[email-masked]>
Date: Mon, 05 Mar 2012 09:57:12 -0500

> that we are at the limit where something stronger is wanted.

Hi Natanael, thanks for the explanations. I haven't yet gotten to know
ACF, so have no appreciation of what better integration with that
framework might call for.

For things like arrays/hashtables, though, mightn't awk suffice? It's
not sh, but it is still part of BusyBox.

I'm not at all anti-Lua; I've written a number of Lua libraries. And I'm
just beginning to learn your system, so can at best dimly understand the
pressures you feel. But I've always liked lightweight solutions, that's
what draws me to your distro. So I just thought I'd ask and try to
understand this better.
-- 
Jim Pryor
[email-masked]

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 5 Mar 2012 17:00:12 +0100

On Mon, 05 Mar 2012 09:57:12 -0500

> > that the shell things are a bit clumsy, and I miss
> > arrays/hashtables. Specially for things like setup-apkrepos,
> > setup-interfaces and setup-disk. I already have a lua moule in
> > progress for parsing /sys and fishing out disk and partition info
> > in convenient data structures. Bash is way to big be considered (it
> > doesnt help that its slow either).
> > 
> > I must admit that using shell if possible is a good thing. I just
> > feel that we are at the limit where something stronger is wanted.
> 

Yes, awk could be used. I still feel it a bit clumsy compared to write
it all in Lua.

> I'm not at all anti-Lua; I've written a number of Lua libraries. And
> I'm just beginning to learn your system, so can at best dimly
> understand the pressures you feel. But I've always liked lightweight
> solutions, that's what draws me to your distro. So I just thought I'd
> ask and try to understand this better.

I agree with you and I want be as lightweight as possible. In this case
I have considered Lua as an acceptable middle way. I will think of what
you say though. I am not married with Lua in this case :)

-nc

---

---

From: Ted Trask <[email-masked]>
Date: Mon, 5 Mar 2012 08:22:36 -0800 (PST)

Comments on ACF:

On Monday, March 5, 2012 9:21 AM

> nice to integrate the setup-* scripts with ACF better. After all they

<snip>

>  general purpose configuration framework but ended up as a more or less

I agree that ACF became too web centric. I'm in the middle of significant changes to ACF to make it more generic. So, you should be able to use the ACF functionality from a generic Lua application and/or from the acf-cli application. I've made most of the changes to acf-core, but there are corresponding changes needed to EVERY other ACF package to be compatible. Considering that I've started a new job and am in the process of moving, my ACF development time is minimal right now. So, it's extremely doubtful that I will have it done by April 1 for Alpine Linux 2.4 as was my goal. We shall see...

Ted

---

From: Natanael Copa <[email-masked]>
Date: Mon, 5 Mar 2012 22:04:27 +0100

On Mon, 5 Mar 2012 08:22:36 -0800 (PST)
Ted Trask <[email-masked]> wrote:

> > Also, the idea is also to at some point have a GTK ui for this.
> >Would be nice to use same backend.
> 
> >  I guess this is was what ACF was originally supposed to do but
> >ended up as way too web (and html) centric.
> 
> >  Actually, ACF grew out from "we need to manage Alpine boxes via
> >web and we need to do it *now*". There was a thought that it should
> >be a general purpose configuration framework but ended up as a more
> >or less web only thing.
> 
> I agree that ACF became too web centric.

It was the time factor that pushed that. We need the web management
fast.

> I'm in the middle of
> significant changes to ACF to make it more generic. So, you should be
> able to use the ACF functionality from a generic Lua application
> and/or from the acf-cli application.

This is one of the motivating factors for wanting move the setup-*
scripts to Lua. Instead of having 2 different frameworks to modify the
config files we have share stuff.

> I've made most of the changes to
> acf-core, but there are corresponding changes needed to EVERY other
> ACF package to be compatible. Considering that I've started a new job
> and am in the process of moving, my ACF development time is minimal
> right now. So, it's extremely doubtful that I will have it done by
> April 1 for Alpine Linux 2.4 as was my goal. We shall see...

Don't stress with it. Take care of your life. If we make it for v2.4, we
make it. Otherwise we take it next release.

Thanks!

-nc

---