Hi All,
I'm trying to run vaultwarden (tried 1.33.2-r0 and 1.34.1-r1) on a
RaspberryPi 4 (aarch64). This used to work fine, but as of upgrading
from alpine 3.21 to 3.22, I get an 'Illegal instruction' error whenever
I try to execute the main binary:
```
$ /usr/bin/vaultwarden
Illegal instruction
```
I initially thought this might be a packaging bug but I'm certain I
used to run successfully the older 1.33.2-r0 prior to recently
upgrading to alpine 3.22. Instead then I'm inclined to believe that
something else *weird* is going on on this system. Nothing sinister,
but I haven't been running alpine long and this was my first attempt at
upgrading in place. Maybe I missed a step? I'm fairly sure I followed
the regular upgrade instructions, followed by upgrading the kernel and
initramfs:
https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_diskless_system
Does anyone have any thoughts for how to begin to debug this? I would
think that building vaultwarden from source might be the way to go
initially, but this is a rather resource constrained system. I have in
fact tried building it, but even with generous swap space and single
threaded building (cargo build -j 1) it gets oomkilled.
This whole thing is written in rust of course, I know very little about
rust. Maybe this is a glaringly obvious issue to someone else.
I would massively appreciate any ideas, sucks to be without your
password manager!
Thanks
Max
On Fri, 25 Jul 2025 11:33:20 +0100
Ian Smith <m4r35n357@gmx.co.uk> wrote:
> You _might_ get some more information from strace or ltrace.
Trying this! Not gonna spam the whole output here but the last few
lines:
```
openat(AT_FDCWD, "/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=133008, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 200704, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x7faf4e9000
mmap(0x7faf518000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1f000) = 0x7faf518000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fafc4d000
mprotect(0x7fafc52000, 4096, PROT_READ) = 0
mprotect(0x7fafb85000, 40960, PROT_READ) = 0
mprotect(0x7fafa38000, 462848, PROT_READ) = 0
mprotect(0x7faf63d000, 12288, PROT_READ) = 0
mprotect(0x7faf5f4000, 32768, PROT_READ) = 0
mprotect(0x7faf5a7000, 16384, PROT_READ) = 0
mprotect(0x7faf549000, 4096, PROT_READ) = 0
mprotect(0x7faf518000, 4096, PROT_READ) = 0
mprotect(0x55848e4000, 1753088, PROT_READ) = 0
set_tid_address(0x7fafc55e80) = 11689
getrandom("\x4d\xb3\xb9\xbf\x87\xe3\xec\x20\xc1\xac\x31\x55\xc5\xf9\x13\x68\x03\x5d\x6f\x1b\x05\x3c\x7c\xc5\x03\x5e\xc8\xba\x09\xd1\x35\xbd", 32, GRND_NONBLOCK) = 32
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x7faf612968} ---
+++ killed by SIGILL +++
Illegal instruction
```
I'm none the wiser to be honest! I did try gdb also:
```
gdb vaultwarden
[noise]
Reading symbols from vaultwarden...
(No debugging symbols found in vaultwarden)
(gdb) run
Starting program: /usr/bin/vaultwarden
Program received signal SIGILL, Illegal instruction.
0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2
(gdb) bt
#0 0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2
#1 0x0000007ff79ba7f8 in ?? () from /usr/lib/libmimalloc-secure.so.2
#2 0x0000007ff7fa8368 in do_init_fini (queue=<optimized out>)
at ldso/dynlink.c:1610
#3 0x0000007ff7fa9240 in __libc_start_init () at ldso/dynlink.c:1623
#4 0x0000007ff7f5d03c in libc_start_main_stage2 (main=0x5555c46fe8,
argc=1, argv=0x7ffffffc08) at src/env/__libc_start_main.c:92
#5 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```
Based on that it seems maybe a problem in mimalloc2? I tried
re-installing that, maybe it had become corrupted or something, but no
change in behaviour. `vaultwarden` is the only package on this system
depending on mimalloc2, so a bug/problem in that library doesn't seem
inherently insane.
Max
On Fri, Jul 25, 2025 at 7:11 AM Maximilian Friedersdorff <
max@friedersdorff.com> wrote:
> On Fri, 25 Jul 2025 11:33:20 +0100> Ian Smith <m4r35n357@gmx.co.uk> wrote:>> > You _might_ get some more information from strace or ltrace.>> [...]> gdb vaultwarden> [noise]>> Reading symbols from vaultwarden...> (No debugging symbols found in vaultwarden)> (gdb) run> Starting program: /usr/bin/vaultwarden>> Program received signal SIGILL, Illegal instruction.> 0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2> (gdb) bt> #0 0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2> #1 0x0000007ff79ba7f8 in ?? () from /usr/lib/libmimalloc-secure.so.2> #2 0x0000007ff7fa8368 in do_init_fini (queue=<optimized out>)> at ldso/dynlink.c:1610> #3 0x0000007ff7fa9240 in __libc_start_init () at ldso/dynlink.c:1623> #4 0x0000007ff7f5d03c in libc_start_main_stage2 (main=0x5555c46fe8,> argc=1, argv=0x7ffffffc08) at src/env/__libc_start_main.c:92> #5 0x0000000000000000 in ?? ()> Backtrace stopped: previous frame identical to this frame (corrupt stack?)>> Based on that it seems maybe a problem in mimalloc2?
It looks like a Microsoft library. See <
https://microsoft.github.io/mimalloc/build.html>.
You will probably get more information if you install the debug symbols. I
don't know how to do it, though.
Jeff
On 25 July 2025 12:09:57 BST, Maximilian Friedersdorff <max@friedersdorff.com> wrote:
>On Fri, 25 Jul 2025 11:33:20 +0100>Ian Smith <m4r35n357@gmx.co.uk> wrote:>>> You _might_ get some more information from strace or ltrace.>>Trying this! Not gonna spam the whole output here but the last few>lines:>>```>openat(AT_FDCWD, "/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)>openat(AT_FDCWD, "/usr/local/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)>openat(AT_FDCWD, "/usr/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3>fcntl(3, F_SETFD, FD_CLOEXEC) = 0>fstat(3, {st_mode=S_IFREG|0755, st_size=133008, ...}) = 0>read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960>mmap(NULL, 200704, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x7faf4e9000>mmap(0x7faf518000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1f000) = 0x7faf518000>close(3) = 0>mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fafc4d000>mprotect(0x7fafc52000, 4096, PROT_READ) = 0>mprotect(0x7fafb85000, 40960, PROT_READ) = 0>mprotect(0x7fafa38000, 462848, PROT_READ) = 0>mprotect(0x7faf63d000, 12288, PROT_READ) = 0>mprotect(0x7faf5f4000, 32768, PROT_READ) = 0>mprotect(0x7faf5a7000, 16384, PROT_READ) = 0>mprotect(0x7faf549000, 4096, PROT_READ) = 0>mprotect(0x7faf518000, 4096, PROT_READ) = 0>mprotect(0x55848e4000, 1753088, PROT_READ) = 0>set_tid_address(0x7fafc55e80) = 11689>getrandom("\x4d\xb3\xb9\xbf\x87\xe3\xec\x20\xc1\xac\x31\x55\xc5\xf9\x13\x68\x03\x5d\x6f\x1b\x05\x3c\x7c\xc5\x03\x5e\xc8\xba\x09\xd1\x35\xbd", 32, GRND_NONBLOCK) = 32>--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x7faf612968} --->+++ killed by SIGILL +++>Illegal instruction>```>>I'm none the wiser to be honest! I did try gdb also:>>```>gdb vaultwarden>[noise]>>Reading symbols from vaultwarden...>(No debugging symbols found in vaultwarden)>(gdb) run>Starting program: /usr/bin/vaultwarden>>Program received signal SIGILL, Illegal instruction.>0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2>(gdb) bt>#0 0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2>#1 0x0000007ff79ba7f8 in ?? () from /usr/lib/libmimalloc-secure.so.2>#2 0x0000007ff7fa8368 in do_init_fini (queue=<optimized out>)> at ldso/dynlink.c:1610>#3 0x0000007ff7fa9240 in __libc_start_init () at ldso/dynlink.c:1623>#4 0x0000007ff7f5d03c in libc_start_main_stage2 (main=0x5555c46fe8,> argc=1, argv=0x7ffffffc08) at src/env/__libc_start_main.c:92>#5 0x0000000000000000 in ?? ()>Backtrace stopped: previous frame identical to this frame (corrupt stack?)>```>>Based on that it seems maybe a problem in mimalloc2? I tried>re-installing that, maybe it had become corrupted or something, but no>change in behaviour. `vaultwarden` is the only package on this system>depending on mimalloc2, so a bug/problem in that library doesn't seem>inherently insane.>>Max
Are you able to try rebuilding the mimalloc2 with -DMI_NO_OPT_ARCH=ON added to the cmake arguments?
In this section:
https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.22-stable/community/mimalloc2/APKBUILD?ref_type=heads#L48
If not, I should have time to test tomorrow.
> Are you able to try rebuilding the mimalloc2 with -DMI_NO_OPT_ARCH=ON > added to the cmake arguments?
I have successfully built mimalloc2 with that option. I also had to
disable checks, but I think only because the timeout set in the checks
was too aggressive for the anemic Raspberry Pi 4.
This has solved this issue at though, my password manager is back up!
Thanks for you help!
Question then, does that represent a bug in the APKBUILD for the
mimalloc2 package? Should I raise a Merge Request to add this compile
time option?
Thanks all
Max
On 27 July 2025 12:59:21 BST, max@friedersdorff.com wrote:
>> Are you able to try rebuilding the mimalloc2 with -DMI_NO_OPT_ARCH=ON added to the cmake arguments?>>I have successfully built mimalloc2 with that option. I also had to disable checks, but I think only because the timeout set in the checks was too aggressive for the anemic Raspberry Pi 4.>>This has solved this issue at though, my password manager is back up!>>Thanks for you help!>>Question then, does that represent a bug in the APKBUILD for the mimalloc2 package? Should I raise a Merge Request to add this compile time option?>>Thanks all>Max
Good to hear!
The optimizations should definitely be disabled in the aport, a lot of our supported ARM devices don't support those instructions.