~alpine/users

8 4

Diskless systems do not boot with latest apk versions from cache

Details
Message ID
<9410103.CDJkKcVGEf@ntb-norbert>
DKIM signature
missing
Download raw message
Hi all!

I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).

All have the same issue. 

I'm doing: 
apk update; apk upgrade

Checking the version of an upgraded binary before reboot, e.g. named, looks ok.

After reboot, it loads an old version even though a newer package is available in cache.

gate [/etc/apk]$ ls -al cache
lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache

gate [/etc/apk/cache]$ ls -altr bind-9*
-rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
-rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
-rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk

gate [/etc/apk/cache]$ named -V
BIND 9.18.32 (Extended Support Version) <id:d1f1392>
running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024

Is there something I'm doing wrong?

BR
Norbert
Details
Message ID
<D7H3LSPZS6EG.2TEXOF210O39I@posteo.net>
In-Reply-To
<9410103.CDJkKcVGEf@ntb-norbert> (view parent)
DKIM signature
missing
Download raw message
On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> Hi all!
>
> I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
>
> All have the same issue. 
>
> I'm doing: 
> apk update; apk upgrade
>
> Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
>
> After reboot, it loads an old version even though a newer package is available in cache.
>
> gate [/etc/apk]$ ls -al cache
> lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
>
> gate [/etc/apk/cache]$ ls -altr bind-9*
> -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
>
> gate [/etc/apk/cache]$ named -V
> BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
>
> Is there something I'm doing wrong?

It looks correct what you are doing. Could you try if running this
commands returns any error?

$ apk --simulate add bind=9.18.33-r0 --no-network

> BR
> Norbert
Details
Message ID
<2700244.k3LOHGUjKi@ntb-norbert>
In-Reply-To
<D7H3LSPZS6EG.2TEXOF210O39I@posteo.net> (view parent)
DKIM signature
missing
Download raw message
Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> > Hi all!
> >
> > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> >
> > All have the same issue. 
> >
> > I'm doing: 
> > apk update; apk upgrade
> >
> > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> >
> > After reboot, it loads an old version even though a newer package is available in cache.
> >
> > gate [/etc/apk]$ ls -al cache
> > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> >
> > gate [/etc/apk/cache]$ ls -altr bind-9*
> > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> >
> > gate [/etc/apk/cache]$ named -V
> > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> >
> > Is there something I'm doing wrong?
> 
> It looks correct what you are doing. Could you try if running this
> commands returns any error?
> 
> $ apk --simulate add bind=9.18.33-r0 --no-network

No errors with this command.

*gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network 
doas (norbert@gate) password:  
(1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0) 
(2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0) 
(3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0) 
(4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0) 
(5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0) 
OK: 157 MiB in 215 packages
> 
> > BR
> > Norbert
> 
> 
Details
Message ID
<2324387.irdbgypaU6@ntb-norbert>
In-Reply-To
<2700244.k3LOHGUjKi@ntb-norbert> (view parent)
DKIM signature
missing
Download raw message
Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
> Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> > > Hi all!
> > >
> > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> > >
> > > All have the same issue. 
> > >
> > > I'm doing: 
> > > apk update; apk upgrade
> > >
> > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> > >
> > > After reboot, it loads an old version even though a newer package is available in cache.
> > >
> > > gate [/etc/apk]$ ls -al cache
> > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> > >
> > > gate [/etc/apk/cache]$ ls -altr bind-9*
> > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> > >
> > > gate [/etc/apk/cache]$ named -V
> > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> > >
> > > Is there something I'm doing wrong?
> > 
> > It looks correct what you are doing. Could you try if running this
> > commands returns any error?
> > 
> > $ apk --simulate add bind=9.18.33-r0 --no-network
> 
> No errors with this command.
> 
> *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network 
> doas (norbert@gate) password:  
> (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0) 
> (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0) 
> (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0) 
> (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0) 
> (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0) 
> OK: 157 MiB in 215 packages

I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.

It seems that something's pointing to a specific version.

> > 
> > > BR
> > > Norbert
> > 
> > 
> 
> 
Details
Message ID
<2408933.iZASKD2KPV@ntb-norbert>
In-Reply-To
<2324387.irdbgypaU6@ntb-norbert> (view parent)
DKIM signature
missing
Download raw message
Am Sonntag, 2. Februar 2025, 09:11:22 Mitteleuropäische Normalzeit schrieb MBB:
> Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
> > Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> > > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> > > > Hi all!
> > > >
> > > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> > > >
> > > > All have the same issue. 
> > > >
> > > > I'm doing: 
> > > > apk update; apk upgrade
> > > >
> > > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> > > >
> > > > After reboot, it loads an old version even though a newer package is available in cache.
> > > >
> > > > gate [/etc/apk]$ ls -al cache
> > > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> > > >
> > > > gate [/etc/apk/cache]$ ls -altr bind-9*
> > > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> > > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> > > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> > > >
> > > > gate [/etc/apk/cache]$ named -V
> > > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> > > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> > > >
> > > > Is there something I'm doing wrong?
> > > 
> > > It looks correct what you are doing. Could you try if running this
> > > commands returns any error?
> > > 
> > > $ apk --simulate add bind=9.18.33-r0 --no-network
> > 
> > No errors with this command.
> > 
> > *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network 
> > doas (norbert@gate) password:  
> > (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0) 
> > (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0) 
> > (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0) 
> > (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0) 
> > (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0) 
> > OK: 157 MiB in 215 packages
> 
> I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
> After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.
> 
> It seems that something's pointing to a specific version.
> 

`apk cache sync` breaks the entire system. 

This cleans the cache from outdated APKINDEX* and *.apk files. 
But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.

Restoring the old cache directory, brings the system to life again.

Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.

Restoring the old APKINDEX* files from a backup fixes this.

> > > 
> > > > BR
> > > > Norbert
> > > 
> > > 
> > 
> > 
> 
Kevin Garman <Garman@scadaware.com>
Details
Message ID
<MW5PR22MB3060F6ABB3D979F10878C9F7D6F52@MW5PR22MB3060.namprd22.prod.outlook.com>
In-Reply-To
<2408933.iZASKD2KPV@ntb-norbert> (view parent)
DKIM signature
missing
Download raw message
I have long been struggling with similar issues on my diskless installs.  I run..

 apk cache -v sync

...and sometimes it breaks my system(s) and sometimes it doesn't.  I've read where cache sync should be used with care, but I believe I'm using it as intended.  It seems like it should be straight forward as all it's suppose to do (as I understand it) is make the cache match what the system needs during boot.  It seems like running cache sync should be the solution to problem it's causing!

I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.




________________________________
From: ~alpine/users <~alpine/users@lists.alpinelinux.org> on behalf of MBB <mbbert@gmail.com>
Sent: Sunday, February 2, 2025 09:16
To: ~alpine/users@lists.alpinelinux.org <~alpine/users@lists.alpinelinux.org>; Sertonix <sertonix@posteo.net>
Subject: Re: Diskless systems do not boot with latest apk versions from cache


Am Sonntag, 2. Februar 2025, 09:11:22 Mitteleuropäische Normalzeit schrieb MBB:
> Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
> > Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> > > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> > > > Hi all!
> > > >
> > > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> > > >
> > > > All have the same issue.
> > > >
> > > > I'm doing:
> > > > apk update; apk upgrade
> > > >
> > > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> > > >
> > > > After reboot, it loads an old version even though a newer package is available in cache.
> > > >
> > > > gate [/etc/apk]$ ls -al cache
> > > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> > > >
> > > > gate [/etc/apk/cache]$ ls -altr bind-9*
> > > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> > > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> > > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> > > >
> > > > gate [/etc/apk/cache]$ named -V
> > > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> > > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> > > >
> > > > Is there something I'm doing wrong?
> > >
> > > It looks correct what you are doing. Could you try if running this
> > > commands returns any error?
> > >
> > > $ apk --simulate add bind=9.18.33-r0 --no-network
> >
> > No errors with this command.
> >
> > *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network
> > doas (norbert@gate) password:
> > (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0)
> > (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0)
> > (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0)
> > (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0)
> > (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0)
> > OK: 157 MiB in 215 packages
>
> I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
> After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.
>
> It seems that something's pointing to a specific version.
>

`apk cache sync` breaks the entire system.

This cleans the cache from outdated APKINDEX* and *.apk files.
But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.

Restoring the old cache directory, brings the system to life again.

Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.

Restoring the old APKINDEX* files from a backup fixes this.

> > >
> > > > BR
> > > > Norbert
> > >
> > >
> >
> >
>
Details
Message ID
<23824424.6Emhk5qWAg@ntb-norbert>
In-Reply-To
<MW5PR22MB3060F6ABB3D979F10878C9F7D6F52@MW5PR22MB3060.namprd22.prod.outlook.com> (view parent)
DKIM signature
missing
Download raw message
Am Montag, 3. Februar 2025, 16:07:18 Mitteleuropäische Normalzeit schrieb Kevin Garman:
> I have long been struggling with similar issues on my diskless installs.  I run..
> 
>  apk cache -v sync
> 
> ...and sometimes it breaks my system(s) and sometimes it doesn't.  I've read where cache sync should be used with care, but I believe I'm using it as intended.  It seems like it should be straight forward as all it's suppose to do (as I understand it) is make the cache match what the system needs during boot.  It seems like running cache sync should be the solution to problem it's causing!
> 
> I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.
> 
> 
> 
> 
> ________________________________
> From: ~alpine/users <~alpine/users@lists.alpinelinux.org> on behalf of MBB <mbbert@gmail.com>
> Sent: Sunday, February 2, 2025 09:16
> To: ~alpine/users@lists.alpinelinux.org <~alpine/users@lists.alpinelinux.org>; Sertonix <sertonix@posteo.net>
> Subject: Re: Diskless systems do not boot with latest apk versions from cache
> 
> 
> Am Sonntag, 2. Februar 2025, 09:11:22 Mitteleuropäische Normalzeit schrieb MBB:
> > Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
> > > Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> > > > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> > > > > Hi all!
> > > > >
> > > > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> > > > >
> > > > > All have the same issue.
> > > > >
> > > > > I'm doing:
> > > > > apk update; apk upgrade
> > > > >
> > > > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> > > > >
> > > > > After reboot, it loads an old version even though a newer package is available in cache.
> > > > >
> > > > > gate [/etc/apk]$ ls -al cache
> > > > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> > > > >
> > > > > gate [/etc/apk/cache]$ ls -altr bind-9*
> > > > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> > > > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> > > > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> > > > >
> > > > > gate [/etc/apk/cache]$ named -V
> > > > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> > > > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> > > > >
> > > > > Is there something I'm doing wrong?
> > > >
> > > > It looks correct what you are doing. Could you try if running this
> > > > commands returns any error?
> > > >
> > > > $ apk --simulate add bind=9.18.33-r0 --no-network
> > >
> > > No errors with this command.
> > >
> > > *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network
> > > doas (norbert@gate) password:
> > > (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0)
> > > (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0)
> > > (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0)
> > > (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0)
> > > (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0)
> > > OK: 157 MiB in 215 packages
> >
> > I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
> > After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.
> >
> > It seems that something's pointing to a specific version.
> >
> 
> `apk cache sync` breaks the entire system.
> 
> This cleans the cache from outdated APKINDEX* and *.apk files.
> But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.
> 
> Restoring the old cache directory, brings the system to life again.
> 
> Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.
> 
> Restoring the old APKINDEX* files from a backup fixes this.
> 


I fixed it for all 3 of my systems.
Key is to update the /media/mmcblk0p1/boot/initramfs-rpi

I created a new one following this guide: https://wiki.alpinelinux.org/wiki/Initramfs_init[1]
I activated these features to get an initramfs which matches the original 

features="base dhcp keymap kms mmc squashfs usb"


When I compared the old an new initramfs, only these binaries showed differences:

Files initramfs/bin/busybox and mkinitfs/bin/busybox differ 
Files initramfs/lib/libapk.so.2.14.0 and mkinitfs/lib/libapk.so.2.14.0 differ 
Files initramfs/lib/libcrypto.so.3 and mkinitfs/lib/libcrypto.so.3 differ 
Files initramfs/lib/libssl.so.3 and mkinitfs/lib/libssl.so.3 differ 
Files initramfs/sbin/apk and mkinitfs/sbin/apk differ

Because creating a new initramfs is part of the update-kernel script, updating the kernel fixes the issue as well.
https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System[2]


> > > >
> > > > > BR
> > > > > Norbert
> > > >
> > > >
> > >
> > >
> >
> 
> 
> 
> 



--------
[1] https://wiki.alpinelinux.org/wiki/Initramfs_init
[2] https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
Prabu Anand Kalivaradhan <kprabuanand@gmail.com>
Details
Message ID
<CAAt6f0aTgdLSr+QC-gbZeEB8u_8H41wqtc-qa5jFoQ+q0_Y0nw@mail.gmail.com>
In-Reply-To
<23824424.6Emhk5qWAg@ntb-norbert> (view parent)
DKIM signature
missing
Download raw message
Hi MBB,

Please confirm if the steps given in the page
https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
is complete.

If something is missing, please highlight here or in the wiki. Most of
the content on the above page has been consolidated from other parts
of wiki. Since i don't use diskless i'm not 100% sure about the
correctness of the content.

I am still hesitating to touch those wiki pages related to Raspberry.
Anyone willing to offering to help test content changes, please reply
me here in the list or personally.

Regards,
Prabu

On Sat, 8 Feb 2025 at 13:56, MBB <mbbert@gmail.com> wrote:
>
> Am Montag, 3. Februar 2025, 16:07:18 Mitteleuropäische Normalzeit schrieb Kevin Garman:
>
> > I have long been struggling with similar issues on my diskless installs.  I run..
>
> >
>
> >  apk cache -v sync
>
> >
>
> > ...and sometimes it breaks my system(s) and sometimes it doesn't.  I've read where cache sync should be used with care, but I believe I'm using it as intended.  It seems like it should be straight forward as all it's suppose to do (as I understand it) is make the cache match what the system needs during boot.  It seems like running cache sync should be the solution to problem it's causing!
>
> >
>
> > I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.
>
> >
>
> >
>
> >
>
> >
>
> > ________________________________
>
> > From: ~alpine/users <~alpine/users@lists.alpinelinux.org> on behalf of MBB <mbbert@gmail.com>
>
> > Sent: Sunday, February 2, 2025 09:16
>
> > To: ~alpine/users@lists.alpinelinux.org <~alpine/users@lists.alpinelinux.org>; Sertonix <sertonix@posteo.net>
>
> > Subject: Re: Diskless systems do not boot with latest apk versions from cache
>
> >
>
> >
>
> > Am Sonntag, 2. Februar 2025, 09:11:22 Mitteleuropäische Normalzeit schrieb MBB:
>
> > > Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
>
> > > > Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
>
> > > > > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
>
> > > > > > Hi all!
>
> > > > > >
>
> > > > > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
>
> > > > > >
>
> > > > > > All have the same issue.
>
> > > > > >
>
> > > > > > I'm doing:
>
> > > > > > apk update; apk upgrade
>
> > > > > >
>
> > > > > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
>
> > > > > >
>
> > > > > > After reboot, it loads an old version even though a newer package is available in cache.
>
> > > > > >
>
> > > > > > gate [/etc/apk]$ ls -al cache
>
> > > > > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
>
> > > > > >
>
> > > > > > gate [/etc/apk/cache]$ ls -altr bind-9*
>
> > > > > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
>
> > > > > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
>
> > > > > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
>
> > > > > >
>
> > > > > > gate [/etc/apk/cache]$ named -V
>
> > > > > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
>
> > > > > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
>
> > > > > >
>
> > > > > > Is there something I'm doing wrong?
>
> > > > >
>
> > > > > It looks correct what you are doing. Could you try if running this
>
> > > > > commands returns any error?
>
> > > > >
>
> > > > > $ apk --simulate add bind=9.18.33-r0 --no-network
>
> > > >
>
> > > > No errors with this command.
>
> > > >
>
> > > > *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network
>
> > > > doas (norbert@gate) password:
>
> > > > (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0)
>
> > > > (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0)
>
> > > > (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0)
>
> > > > (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0)
>
> > > > (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0)
>
> > > > OK: 157 MiB in 215 packages
>
> > >
>
> > > I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
>
> > > After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.
>
> > >
>
> > > It seems that something's pointing to a specific version.
>
> > >
>
> >
>
> > `apk cache sync` breaks the entire system.
>
> >
>
> > This cleans the cache from outdated APKINDEX* and *.apk files.
>
> > But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.
>
> >
>
> > Restoring the old cache directory, brings the system to life again.
>
> >
>
> > Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.
>
> >
>
> > Restoring the old APKINDEX* files from a backup fixes this.
>
> >
>
>
> I fixed it for all 3 of my systems.
>
> Key is to update the /media/mmcblk0p1/boot/initramfs-rpi
>
>
> I created a new one following this guide: https://wiki.alpinelinux.org/wiki/Initramfs_init
>
> I activated these features to get an initramfs which matches the original
>
>
> features="base dhcp keymap kms mmc squashfs usb"
>
>
>
> When I compared the old an new initramfs, only these binaries showed differences:
>
>
> Files initramfs/bin/busybox and mkinitfs/bin/busybox differ
> Files initramfs/lib/libapk.so.2.14.0 and mkinitfs/lib/libapk.so.2.14.0 differ
> Files initramfs/lib/libcrypto.so.3 and mkinitfs/lib/libcrypto.so.3 differ
> Files initramfs/lib/libssl.so.3 and mkinitfs/lib/libssl.so.3 differ
> Files initramfs/sbin/apk and mkinitfs/sbin/apk differ
>
>
>
> Because creating a new initramfs is part of the update-kernel script, updating the kernel fixes the issue as well.
>
> https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
>
>
> > > > >
>
> > > > > > BR
>
> > > > > > Norbert
>
> > > > >
>
> > > > >
>
> > > >
>
> > > >
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
Details
Message ID
<4971180.GXAFRqVoOG@ntb-norbert>
In-Reply-To
<CAAt6f0aTgdLSr+QC-gbZeEB8u_8H41wqtc-qa5jFoQ+q0_Y0nw@mail.gmail.com> (view parent)
DKIM signature
missing
Download raw message
Am Samstag, 8. Februar 2025, 10:01:44 Mitteleuropäische Normalzeit schrieb Prabu Anand Kalivaradhan:
> Hi MBB,
> 
> Please confirm if the steps given in the page
> https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
> is complete.
> 
> If something is missing, please highlight here or in the wiki. Most of
> the content on the above page has been consolidated from other parts
> of wiki. Since i don't use diskless i'm not 100% sure about the
> correctness of the content.
> 
> I am still hesitating to touch those wiki pages related to Raspberry.
> Anyone willing to offering to help test content changes, please reply
> me here in the list or personally.
> 
> Regards,
> Prabu
> 
These steps are basically what I did to update the kernel.

To my understanding, the update is not memory heavy. It succeeded on a RPi2 with 1GB RAM. 
But there must be enough space in /tmp for the update.
Since SBCs don't have much RAM for a large enough tmpfs, the update fails, but it didn't result in a broken modloop-image in my case.
To work around, I used the environment variable TMPDIR to point to a directory on a physical device, like a SD-Card or USB-Stick.


TMPDIR=/media/sdc1/tmp update-kernel /media/mmcblk0p1/boot/

This must be a *nix file system, FAT won't work.

> On Sat, 8 Feb 2025 at 13:56, MBB <mbbert@gmail.com> wrote:
> >
> > Am Montag, 3. Februar 2025, 16:07:18 Mitteleuropäische Normalzeit schrieb Kevin Garman:
> >
> > > I have long been struggling with similar issues on my diskless installs.  I run..
> >
> > >
> >
> > >  apk cache -v sync
> >
> > >
> >
> > > ...and sometimes it breaks my system(s) and sometimes it doesn't.  I've read where cache sync should be used with care, but I believe I'm using it as intended.  It seems like it should be straight forward as all it's suppose to do (as I understand it) is make the cache match what the system needs during boot.  It seems like running cache sync should be the solution to problem it's causing!
> >
> > >
> >
> > > I've been working to figure out a simple/repeatable set of steps to reproduce the problem, but so far I haven't been able to get there.
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> > > ________________________________
> >
> > > From: ~alpine/users <~alpine/users@lists.alpinelinux.org> on behalf of MBB <mbbert@gmail.com>
> >
> > > Sent: Sunday, February 2, 2025 09:16
> >
> > > To: ~alpine/users@lists.alpinelinux.org <~alpine/users@lists.alpinelinux.org>; Sertonix <sertonix@posteo.net>
> >
> > > Subject: Re: Diskless systems do not boot with latest apk versions from cache
> >
> > >
> >
> > >
> >
> > > Am Sonntag, 2. Februar 2025, 09:11:22 Mitteleuropäische Normalzeit schrieb MBB:
> >
> > > > Am Samstag, 1. Februar 2025, 14:39:34 Mitteleuropäische Normalzeit schrieb MBB:
> >
> > > > > Am Samstag, 1. Februar 2025, 13:12:29 Mitteleuropäische Normalzeit schrieb Sertonix:
> >
> > > > > > On Sat Feb 1, 2025 at 9:24 AM CET, MBB wrote:
> >
> > > > > > > Hi all!
> >
> > > > > > >
> >
> > > > > > > I have 3 Raspberry Pis, different PI versions and different Alpine versions (3.18.9, 3.19.1, 3.19.4).
> >
> > > > > > >
> >
> > > > > > > All have the same issue.
> >
> > > > > > >
> >
> > > > > > > I'm doing:
> >
> > > > > > > apk update; apk upgrade
> >
> > > > > > >
> >
> > > > > > > Checking the version of an upgraded binary before reboot, e.g. named, looks ok.
> >
> > > > > > >
> >
> > > > > > > After reboot, it loads an old version even though a newer package is available in cache.
> >
> > > > > > >
> >
> > > > > > > gate [/etc/apk]$ ls -al cache
> >
> > > > > > > lrwxrwxrwx    1 root     root            22 Feb  1 08:50 cache -> /media/mmcblk0p1/cache
> >
> > > > > > >
> >
> > > > > > > gate [/etc/apk/cache]$ ls -altr bind-9*
> >
> > > > > > > -rwxr-xr-x    1 root     root        207297 Mar 15  2024 bind-9.18.24-r1.5aca84cc.apk
> >
> > > > > > > -rwxr-xr-x    1 root     root        208996 Dec 17 16:37 bind-9.18.32-r0.1254446d.apk
> >
> > > > > > > -rwxr-xr-x    1 root     root        209078 Jan 29 19:44 bind-9.18.33-r0.b6697cb2.apk
> >
> > > > > > >
> >
> > > > > > > gate [/etc/apk/cache]$ named -V
> >
> > > > > > > BIND 9.18.32 (Extended Support Version) <id:d1f1392>
> >
> > > > > > > running on Linux armv7l 6.6.14-0-rpi #1-Alpine SMP Fri Jan 26 13:40:47 UTC 2024
> >
> > > > > > >
> >
> > > > > > > Is there something I'm doing wrong?
> >
> > > > > >
> >
> > > > > > It looks correct what you are doing. Could you try if running this
> >
> > > > > > commands returns any error?
> >
> > > > > >
> >
> > > > > > $ apk --simulate add bind=9.18.33-r0 --no-network
> >
> > > > >
> >
> > > > > No errors with this command.
> >
> > > > >
> >
> > > > > *gate [*~*]$ *doas apk --simulate add bind=9.18.33-r0 --no-network
> >
> > > > > doas (norbert@gate) password:
> >
> > > > > (1/5) Upgrading bind-libs (9.18.32-r0 -> 9.18.33-r0)
> >
> > > > > (2/5) Upgrading bind-tools (9.18.32-r0 -> 9.18.33-r0)
> >
> > > > > (3/5) Upgrading bind-dnssec-root (9.18.32-r0 -> 9.18.33-r0)
> >
> > > > > (4/5) Upgrading bind (9.18.32-r0 -> 9.18.33-r0)
> >
> > > > > (5/5) Upgrading bind-openrc (9.18.32-r0 -> 9.18.33-r0)
> >
> > > > > OK: 157 MiB in 215 packages
> >
> > > >
> >
> > > > I removed all outdated bind packages from the cache with the result, that bind is no longer available after reboot.
> >
> > > > After moving the bind*9.18.32-r0*.apk files back into the cache directory, the service gets restored and starts up without issues after reboot.
> >
> > > >
> >
> > > > It seems that something's pointing to a specific version.
> >
> > > >
> >
> > >
> >
> > > `apk cache sync` breaks the entire system.
> >
> > >
> >
> > > This cleans the cache from outdated APKINDEX* and *.apk files.
> >
> > > But after reboot nothing is loaded and the boot process ends with a `(none) local:` prompt.
> >
> > >
> >
> > > Restoring the old cache directory, brings the system to life again.
> >
> > >
> >
> > > Same issue when I manually delete only the APKINDEX* files, and update the local DB with `apk update` to download fresh APKINDEX.
> >
> > >
> >
> > > Restoring the old APKINDEX* files from a backup fixes this.
> >
> > >
> >
> >
> > I fixed it for all 3 of my systems.
> >
> > Key is to update the /media/mmcblk0p1/boot/initramfs-rpi
> >
> >
> > I created a new one following this guide: https://wiki.alpinelinux.org/wiki/Initramfs_init
> >
> > I activated these features to get an initramfs which matches the original
> >
> >
> > features="base dhcp keymap kms mmc squashfs usb"
> >
> >
> >
> > When I compared the old an new initramfs, only these binaries showed differences:
> >
> >
> > Files initramfs/bin/busybox and mkinitfs/bin/busybox differ
> > Files initramfs/lib/libapk.so.2.14.0 and mkinitfs/lib/libapk.so.2.14.0 differ
> > Files initramfs/lib/libcrypto.so.3 and mkinitfs/lib/libcrypto.so.3 differ
> > Files initramfs/lib/libssl.so.3 and mkinitfs/lib/libssl.so.3 differ
> > Files initramfs/sbin/apk and mkinitfs/sbin/apk differ
> >
> >
> >
> > Because creating a new initramfs is part of the update-kernel script, updating the kernel fixes the issue as well.
> >
> > https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_Diskless_System
> >
> >
> > > > > >
> >
> > > > > > > BR
> >
> > > > > > > Norbert
> >
> > > > > >
> >
> > > > > >
> >
> > > > >
> >
> > > > >
> >
> > > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
> 
Reply to thread Export thread (mbox)