Subject: [alpine-devel] Cannot mount root filesystem with latest vanilla kernel

From: Isaac Dunham <[email-masked]>
Date: Sun, 14 Sep 2014 13:17:03 -0700

Hello,
Posting from a different distro, since I can't boot Alpine.

I've got a bit of a problem with the latest linux-vanilla kernel.
Yesterday I did an update and upgrade, and today I'm getting an
initramfs shell whenever I boot.
I have /dev/sda14 as root, formatted as ext2.
I can mount it from another distro, as ext2.

When I try mounting /dev/sda14 on /sysroot without specifying fstype,
I get this errormessage:
mount: cannot mount /dev/sda14 on /sysroot: no such file or directory
and there are no new messages in dmesg.

If I specify -t ext2 (or ext3 or ext4), I get 
mount: cannot mount /dev/sda14 on /sysroot: no such device
and dmesg has a large number of complaints about symbol version mismatches
in the filesystem driver (ext2 or ext3 or ext4).

Any ideas how to fix it?

Thanks,
Isaac Dunham

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Isaac Dunham <[email-masked]>
Date: Sun, 14 Sep 2014 14:05:08 -0700

> in the filesystem driver (ext2 or ext3 or ext4).

apk fix linux-vanilla
solved it.

No idea what caused this, though.

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 15 Sep 2014 11:31:05 +0200

On Sun, 14 Sep 2014 14:05:08 -0700

> No idea what caused this, though.

i think update-extlinux or mkinitfs trigger failed for some reason so
the initramfs image got broken. I have no idea why though.

-nc

---

---

From: Isaac Dunham <[email-masked]>
Date: Mon, 15 Sep 2014 12:35:09 -0700

> the initramfs image got broken. I have no idea why though.

Well, ran into more problems:
- on the second boot, the problem reappeared.
- I chrooted in, purged linux-vanilla, exited, umounted the root
  partition, ran e2fsck -p (which fixed several isssues with "deleted
  inodes"), ran e2fsck -cckv, which reported no bad blocks or other
  issues, remounted & chrooted, installed linux-vanilla, ran
 apk fix mkinitfs,
  exited, umounted, ran sync; sync; sync;
  and rebooted to find that the issue was still there.

Most modules are giving me complaints about missing symbols or bad
symbol versions.

I am using grub1 (installed on /dev/sda8) to boot; the consistent naming
of Alpine kernels means that I can use a static entry.
grub continues to work fine with my other installs, and has not been
updated.

Could someone else test whether in Qemu, the latest linux-vanilla has
any such issues (after module loading, does
dmesg |grep symbol
show any errors?)

Thanks,
Isaac Dunham

---

---

From: Isaac Dunham <[email-masked]>
Date: Mon, 15 Sep 2014 21:57:16 -0700

> Most modules are giving me complaints about missing symbols or bad

Well, I installed the latest from the 3.0 repo, which is to say I
downgraded from kernel 3.14.18 (in edge) to 3.14.17, and everything worked.

So I'm fairly conviced that there's a bug or miscompile in 3.14.18.

Thanks,
Isaac Dunham

---

---

From: Isaac Dunham <[email-masked]>
Date: Tue, 16 Sep 2014 16:42:19 -0700

> So I'm fairly conviced that there's a bug or miscompile in 3.14.18.

Definitely bug in 3.14.18:
alpine build of 3.14.17:	works
local build of 3.14.17: 	works
alpine build of 3.14.18: 	does not work
local build of 3.14.18: 	does not work

I still don't know what the cause is, though.

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Wed, 17 Sep 2014 16:06:13 +0200

On Tue, 16 Sep 2014 16:42:19 -0700

> I still don't know what the cause is, though.

which arch is it? I have 3.14.18 working on my x86_64 laptop.

The config was modified from 3.14.17 -> .18:

@@ -484,7 +484,10 @@ CONFIG_SCHED_HRTICK=y
 # CONFIG_KEXEC is not set
 # CONFIG_CRASH_DUMP is not set
 CONFIG_PHYSICAL_START=0x1000000
-# CONFIG_RELOCATABLE is not set
+CONFIG_RELOCATABLE=y
+CONFIG_RANDOMIZE_BASE=y
+CONFIG_RANDOMIZE_BASE_MAX_OFFSET=0x20000000
+CONFIG_X86_NEED_RELOCS=y
 CONFIG_PHYSICAL_ALIGN=0x1000000
 CONFIG_HOTPLUG_CPU=y
 # CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set

Mostly due to be similar x86_64, which recently got requirement of
CONFIG_RELOCATABLE for EFI (and it looked like a good thing to have
anyways).

-nc

---

---

From: Isaac Dunham <[email-masked]>
Date: Wed, 17 Sep 2014 08:44:46 -0700

> CONFIG_RELOCATABLE for EFI (and it looked like a good thing to have

i386 on an Atom N270-based laptop (Aspire One 150).
(It uses a weird BIOS that may actually be EFI or similar underneath,
but only exposes the BIOS api.)

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 18 Sep 2014 16:53:27 +0200

On Wed, 17 Sep 2014 08:44:46 -0700

> but only exposes the BIOS api.)

some more debug info:
this is booting vanilla edge iso and entering initramfs:

/ # insmod /lib/modules/3.14.19/kernel/fs/ext2/ext2.ko                          
insmod: can't insert '/lib/modules/3.14.19/kernel/fs/ext2/ext2.ko': unknown symbol in module, or unknown parameter    

/ # modprobe ext2                                                               
modprobe: can't load module ext2 (kernel/fs/ext2/ext2.ko): Invalid argument 

/ # modprobe squashfs                                                           
modprobe: can't load module squashfs (kernel/fs/squashfs/squashfs.ko): Invalid argument    

the modules are there but they don't load for some reason.

It does not happen with x86_64.

I wonder what is going on. Maybe the CONFIG_RELOCATABLE change
mentioned above is related? I have no idea...

---

---

From: Isaac Dunham <[email-masked]>
Date: Thu, 18 Sep 2014 09:04:15 -0700

> > but only exposes the BIOS api.)
<snip>

> mentioned above is related? I have no idea...

I'm suspecting that might be it.

It's not a bug in the build environment, since the exact same system
could build a working 3.14.17 and a non-working 3.14.18.
That's what the test matrix was about.

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 19 Sep 2014 11:23:54 +0200

On Thu, 18 Sep 2014 09:04:15 -0700
Isaac Dunham <[email-masked]> wrote:

> > > > Mostly due to be similar x86_64, which recently got requirement
> > > > of CONFIG_RELOCATABLE for EFI (and it looked like a good thing
> > > > to have anyways).
> > > > 
> > > 
> > > (It uses a weird BIOS that may actually be EFI or similar
> > > underneath, but only exposes the BIOS api.)
> <snip>

It is CONFIG_RANDOMIZE_BASE=y that triggers it.

I have disabled that again on x86 vanilla kernel (and will do so on
grsec kernel too which also seems to be affected)

Thanks you for reporting it, for your help finding the cause and for
your patience.

-nc

---

---

From: Isaac Dunham <[email-masked]>
Date: Fri, 19 Sep 2014 07:11:24 -0700

> Thanks you for reporting it, for your help finding the cause and for

Does this occur with kmod (ie, is Busybox being overly simplistic,
or does this flat-out break the kernel?)

I suppose that it should be reported to either kernel.org or Busybox,
depending which it is.

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 22 Sep 2014 11:39:00 +0200

On Fri, 19 Sep 2014 07:11:24 -0700

> or does this flat-out break the kernel?)

I didn't have time to test that out.

> I suppose that it should be reported to either kernel.org or Busybox,

I suspect it is a bug in kernel actually, but am not sure.

I don't have capacity to follow it up at the moment. sorry :-/

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 18 Sep 2014 16:41:43 +0200

> but only exposes the BIOS api.)

I think something got mis-compiled for x86. not sure how or why.

The compile itself seems to be ok:
http://bld1.alpinelinux.org/buildlogs/build-edge-x86/main/linux-vanilla/linux-vanilla-3.14.18-r0.log

But I get errors when trying to boot the cdrom in qemu on x86. It boots
but the squashfs is not getting loaded in initramfs causing the modloop
fail to mount.

I wonder if the script that generates the kernel module deps is broken
or maybe depmod or insmod.

-nc

---