Subject: Diskless systems do not boot with latest apk versions from cache

From: MBB <[email-masked]>
Date: Sat, 01 Feb 2025 09:24:11 +0100

Hi all!

I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).

All have the same issue. 

I'm doing: 
apk update; apk upgrade

Checking the version of an upgraded binary before reboot, e.g. named, looks ok.

After reboot, it loads an old version even though a newer package is available in cache.

gate [/etc/apk]$ ls -al cache
lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache

gate [/etc/apk/cache]$ ls -altr bind-9*
-rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
-rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
-rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk

gate [/etc/apk/cache]$ named -V
BIND 9.18.32 (Extended Support Version) <id:d1f1392>
running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024

Is there something I'm doing wrong?

BR
Norbert

---

From: "Sertonix" <[email-masked]>
Date: Sat, 01 Feb 2025 12:12:29 +0000

> Is there something I'm doing wrong?

It looks correct what you are doing. Could you try if running this
commands returns any error?

$ apk --simulate add bind=9.18.33-r0 --no-network

---

From: MBB <[email-masked]>
Date: Sat, 01 Feb 2025 14:39:34 +0100

> $ apk --simulate add bind=9.18.33-r0 --no-network

No errors with this command.

*gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network 
doas (norbert@gate) password:  
(1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0) 
(2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0) 
(3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0) 
(4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0) 
(5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0) 
OK: 157 MiB in 215 packages

---

From: MBB <[email-masked]>
Date: Sun, 02 Feb 2025 09:11:22 +0100

> OK: 157 MiB in 215 packages

I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.

It seems that something's pointing to a specific version.

---

From: MBB <[email-masked]>
Date: Sun, 02 Feb 2025 16:16:03 +0100

> It seems that something's pointing to a specific version.

`apk cache sync` breaks the entire system. 

This cleans the cache from outdated APKINDEX* and *.apk files. 
But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.

Restoring the old cache directory, brings the system to life again.

Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.

Restoring the old APKINDEX* files from a backup fixes this.

---

From: Kevin Garman <[email-masked]>
Date: Mon, 3 Feb 2025 15:07:18 +0000

I have long been struggling with similar issues on my diskless installs.  I run..

 apk cache -v sync

...and sometimes it breaks my system(s) and sometimes it doesn't.  I've read where cache sync should be used with care, but I believe I'm using it as intended.  It seems like it should be straight forward as all it's suppose to do (as I understand it) is make the cache match what the system needs during boot.  It seems like running cache sync should be the solution to problem it's causing!

I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.

________________________________
From: ~alpine/users <~alpine/[email-masked]> on behalf of MBB <[email-masked]>
Sent: Sunday, February 2, 2025 09:16
To: ~alpine/[email-masked] <~alpine/[email-masked]>; Sertonix <[email-masked]>
Subject: Re: Diskless systems do not boot with latest apk versions from cache

---

From: MBB <[email-masked]>
Date: Sat, 08 Feb 2025 09:25:37 +0100

> I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.

I fixed it for all 3 of my systems.
Key is to update the /media/mmcblk0p1/boot/initramfs-rpi

I created a new one following this guide: https://wiki.alpinelinux.org/wiki/Initramfs_init[1]
I activated these features to get an initramfs which matches the original 

features="base dhcp keymap kms mmc squashfs usb"

When I compared the old an new initramfs, only these binaries showed differences:

Files initramfs/bin/busybox and mkinitfs/bin/busybox differ 
Files initramfs/lib/libapk.so.2.14.0 and mkinitfs/lib/libapk.so.2.14.0 differ 
Files initramfs/lib/libcrypto.so.3 and mkinitfs/lib/libcrypto.so.3 differ 
Files initramfs/lib/libssl.so.3 and mkinitfs/lib/libssl.so.3 differ 
Files initramfs/sbin/apk and mkinitfs/sbin/apk differ

Because creating a new initramfs is part of the update-kernel script, updating the kernel fixes the issue as well.
https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System[2]

> > > > > Norbert

--------
[1] https://wiki.alpinelinux.org/wiki/Initramfs_init
[2] https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System

---

From: Prabu Anand Kalivaradhan <[email-masked]>
Date: Sat, 8 Feb 2025 14:31:44 +0530

Hi MBB,

Please confirm if the steps given in the page
https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
is complete.

If something is missing, please highlight here or in the wiki. Most of
the content on the above page has been consolidated from other parts
of wiki. Since i don't use diskless i'm not 100% sure about the
correctness of the content.

I am still hesitating to touch those wiki pages related to Raspberry.
Anyone willing to offering to help test content changes, please reply
me here in the list or personally.

Regards,
Prabu

On Sat, 8 Feb 2025 at 13:56, MBB <[email-masked]> wrote:
> I created a new one following this guide: https://wiki.alpinelinux.org/wiki/Initramfs_init
>
> https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
>
>
> > > > >
>
> > > > > > BR
>
> > > > > > Norbert
>
> > > > >
>
> > > > >
>
> > > >
>
> > > >
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>

---

From: MBB <[email-masked]>
Date: Sun, 09 Feb 2025 10:20:44 +0100

> me here in the list or personally.
These steps are basically what I did to update the kernel.

To my understanding, the update is not memory heavy. It succeeded on a RPi2 with 1GB RAM. 
But there must be enough space in /tmp for the update.
Since SBCs don't have much RAM for a large enough tmpfs, the update fails, but it didn't result in a broken modloop-image in my case.
To work around, I used the environment variable TMPDIR to point to a directory on a physical device, like a SD-Card or USB-Stick.

TMPDIR=/media/sdc1/tmp update-kernel /media/mmcblk0p1/boot/

This must be a *nix file system, FAT won't work.