Subject: [alpine-devel] Soekris 4801, alpine 2.0, kernel trouble

From: Jon Clausen <[email-masked]>
Date: Thu, 26 Aug 2010 14:03:30 +0200

Hi

I'm in the process of setting up Alpine 2.0 as my main router/firewall.

It's my first hike into "The Alps", so there's a lot to get used to, but
overall I do like it :)

Fortunately for me, I have two 4801's so I'm in a position to test stuff
before I commit the system on the actual firewall hardware... which turns
out to be very lucky indeed...

The long story short:

Booting the 2.6.32.18 kernel with the default parameters on the soekris
results in a kernel panic.

After trying out the kernels of alpine 2 beta-* and rc-* I found that:

2.6.32.15 boots ok
2.6.32.16 upwards 'panic'

After searching the web I found a hint that 'noreplace-paravirt' had worked
for someone else. So I tried adding that to the append line in syslinux.cfg
- and lo and behold: Now the kernel boots :)

So far so good, except now I get:

pci 0000:00:07.0: BAR 6: address space collision on of device
[0x3f000000-0x3f00ffff]
pci 0000:00:08.0: BAR 6: address space collision on of device
[0x3f000000-0x3f00ffff]

- which I'm somewhat unhappy about, since those are two of the three onboard
NICs :(
(Incidentally the 2.6.32.15 kernel also has these collisions)

The thing is that this system is going to be pretty 'complicated', with a
bunch of vlans, and I'll be needing the NICs.

I'm not too sure how to proceed from here?

Do any of you have insights/experience about this kind of issue?

>From what I've read so far, it may be related to:
https://patchwork.kernel.org/patch/39988/ and symptoms range from 'more or
less cosmetic, no real problems' to 'complete system lockup'.

Obviously I'm not too thrilled by the prospect of the latter potentially
happening to my firewall... :P

I have a serial console on the box, so it's pretty esay for me to 'grab' the
output. In fact I've saved the output from booting the various kernels (with
various kernel parameters) but I've nowhere public to put them ATM.

They're about 7K-9K each. If anyone's interested, let me know?

tia
/jon
-- 
YMMV

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 26 Aug 2010 17:08:08 +0200

> It's my first hike into "The Alps", so there's a lot to get used to, but

nice!

> results in a kernel panic.

not good.

> 2.6.32.16 upwards 'panic'

This is very useful information.

form the git log, the only thing that changed really was support for EFI.

http://git.alpinelinux.org/cgit/aports/log/main/linux-grsec?h=2.0-stable

> - and lo and behold: Now the kernel boots :)

Thats interesting. I wonder how that could be?

Could you try the vserver iso image too? It has a different kernel,
without the grsecurity patches. The syslinux.cfg is broken so you'd
need enter 'vserver' at the boot prompt where it complains about
missing grsec.

> (Incidentally the 2.6.32.15 kernel also has these collisions)

I dont think those causes any problems.

> I'm not too sure how to proceed from here?

I just pushed 2.6.32.19 kernel. i doubt it makes any difference

> Do any of you have insights/experience about this kind of issue?

I have seen the address collision messages several places but it has
never caused any probles for me (that i am aware of at least)

> They're about 7K-9K each. If anyone's interested, let me know?

The ones that panic would be interesting to see. Not sure how much i
can help there though.

-- 
Natanael Copa

---

---

From: Jon Clausen <[email-masked]>
Date: Thu, 26 Aug 2010 19:08:49 +0200

> > It's my first hike into "The Alps", so there's a lot to get used to, but

:)

> > results in a kernel panic.

Agreed

> This is very useful information.

Good :)

> Thats interesting. I wonder how that could be?

Frankly, I have no idea - except I did notice a kernel message:

"Booting paravirtualized kernel on bare hardware"

But my first thought was that noreplace-paravirt would make matters worse.

> need enter 'vserver' at the boot prompt where it complains about

Will do. Probably not tonight, but yeah it makes sense to test that.

> I dont think those causes any problems.

Well... I guess the only way to be sure is to set it up and see what
happens... it's just that *some* of the posts I've seen were talking about:

'seems to run ok for a bit, then bombs out <mumble> <mumble> memory leak?'

or somesuch

> I just pushed 2.6.32.19 kernel. i doubt it makes any difference

I'll have to check that as well...

> The ones that panic would be interesting to see. Not sure how much i

OK, I'll tar 'em up and send them in a PM.

- speaking of which;

What's the 'listiquette'?

Attachment size limits, etc?

I see in the archives that people generally respond to both the poster and
the list. Is that the norm? (This message is going to the list only)

/jon
-- 
YMMV

---

---

From: Jon Clausen <[email-masked]>
Date: Fri, 27 Aug 2010 10:06:57 +0200

> Will do. Probably not tonight, but yeah it makes sense to test that.

OK, I just tested the vserver kernels (beta2, rc3 and 'final') and they boot
fine without the 'noreplace-paravirt' parameter. (And yes, they also have
the BAR 6 collisions)

So I guess this points towards the grsec patches?

Console output available.

/jon
-- 
YMMV

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 27 Aug 2010 11:38:00 +0200

> > > pci 0000:00:08.0: BAR 6: address space collision on of device

This is a different issue. I'd prefer keep this separated. Lets fix the
kernel panic first.

> OK, I'll tar 'em up and send them in a PM.

got them. I talked with pipacs on #grsecurity on OFTC on irc. He is the
author of the PAX patches. This will be difficult to debug since its not
reproduceable on qemu and he nor me has this hardware.

Basicly, we will need to help him to help us fix it. Would be nice if
you had a chance to join on irc. so we both could talk directly with
pipacs on how to do this.

> Attachment size limits, etc?

I dint we have any written rules for that really. I'd think attachment
size up to atleast 1MB is ok or til someone complains.

> the list. Is that the norm? (This message is going to the list only)

I normally just hit reply-all. Just in case sender has not joined the
list.

> /jon
-nc

---

---

From: Jon Clausen <[email-masked]>
Date: Fri, 27 Aug 2010 13:02:36 +0200

> This is a different issue. I'd prefer keep this separated. Lets fix the

OK.

> reproduceable on qemu and he nor me has this hardware.

Right. I'm happy to help anyone help me ;)

Before we go further with the grsec trail, however, I was thinking something
else might help narrow down the issue:

Looking at the config-<kernel-version> files that come with the *.iso'es, I
noticed a number of *VIRT* options are set.
Now, I'm not too familiar with those, but I was thinking it might be worth a
shot with a kernel:

* with grsec patches
* without the "virtualization" options

The reasoning being, that 'noreplace-paravirt' might conceivably change the
behaviour of parts of the kernel affected by options such as:

CONFIG_PARAVIRT_GUEST=y
CONFIG_PARAVIRT=y

Should be interesting to see what happens with a 'bare metal' kernel with
the grsec patches.

I don't have the build environment, though, so... is it possible for you to
build a 'non paravirtualized' kernel for me to test?

> pipacs on how to do this.

Yeah... That's unlikely this side of the weekend... I've got a pretty tight
schedule... :P

> size up to atleast 1MB is ok or til someone complains.

Right.

> I normally just hit reply-all. Just in case sender has not joined the

Hmm... I'm gonna have to play around with sieve, and see if I can get it
to match the list and PM 'duplicates' reliably, then...

Oh well, that's another issue entirely.

regards
/jon
-- 
YMMV

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 27 Aug 2010 13:44:45 +0200

> else might help narrow down the issue:

pipacs gave me something he thought would fix it. I'll just verify that
it boots in qemu and then I'll push it to edge. It should be available
from edge as linux-grsec-2.6.32.19-r1 soonish.

We should create an issue on the bugtracking system so it does not get
forgotten. I'd like the fix for this in alpine-2.0.1. Could you please?
Also paste the log of the kernel backtrace part.

thanks!

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 27 Aug 2010 14:18:51 +0200

> from edge as linux-grsec-2.6.32.19-r1 soonish.

linux-kernel-2.6.32.19-r1 is pushed.

Please let me know if you need an entire .iso image. I can upload
something someplace if needed.

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 27 Aug 2010 16:15:16 +0200

> something someplace if needed.

I built an alpine-edge.iso for you with the patched kernel:

http://dev.alpinelinux.org/~ncopa/alpine/alpine-edge-100827-x86.iso
(35MB over slow connection)

Please let me know if that boots on soekris without the
noreplace-paravirt

Thanks!

-nc

---

---

From: Jon Clausen <[email-masked]>
Date: Fri, 27 Aug 2010 17:40:59 +0200

> Please let me know if that boots on soekris without the

Just a quick feedback:

Tested the kernel from the above iso, and it boots :D

I've attached the console output. There are some errors, which are caused by
the way this test was performed, but in the grand scheme of things they
don't matter.

What I do is copy the kernel & initrd from the iso to my tftp server, and
make a corresponding section in pxelinux.cfg/<file>

But the CF-card has the *.cmg from alpine 2.0 - and so anything kernel
specific in there is 'not found'. The reason I'm using tftp is that it's
much quicker to just copy the files around, than messing about with the CF
card, readers and stuff...

Is there a way to tell the system to pull the *.cmg from tftp, nfs, or
something?

> Thanks!

No, I'm the one thanking here !)

About the bugreport: If it's still needed I'll do it, but like I said I'm
pretty busy over the weekend...

thanks,
jon

-- 
YMMV

---

From: Jon Clausen <[email-masked]>
Date: Thu, 2 Sep 2010 09:53:30 +0200

> Also paste the log of the kernel backtrace part.

OK: http://redmine.alpinelinux.org/issues/422

This is just a heads up, since I don't know who gets notified about new
bugs...

I included a representative selection of boot logs, but not all of them.

Let me know if there is anything missing?

Sorry for the long 'turnaround' in getting the bug reported.

regards
/jon
-- 
YMMV

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 02 Sep 2010 15:11:26 +0200

> OK: http://redmine.alpinelinux.org/issues/422

...

> Let me know if there is anything missing?

Its more than enough.

> Sorry for the long 'turnaround' in getting the bug reported.

np! Thanks for taking time doing it! now our "whats changed since last
release" will look nice :)

Thanks!
-nc

---

---

From: Jon Clausen <[email-masked]>
Date: Thu, 2 Sep 2010 20:10:47 +0200

> release" will look nice :)

:D

> Thanks!

And thanks to you for the time *you* took to work on the issue!

/jon
-- 
YMMV

---