Subject: Illegal instruction on running vaultwarden on RPI (aarch64)

From: Maximilian Friedersdorff <[email-masked]>
Date: Fri, 25 Jul 2025 09:49:29 +0100

Hi All,

I'm trying to run vaultwarden (tried 1.33.2-r0 and 1.34.1-r1) on a
RaspberryPi 4 (aarch64).  This used to work fine, but as of upgrading
from alpine 3.21 to 3.22, I get an 'Illegal instruction' error whenever
I try to execute the main binary:

```
$ /usr/bin/vaultwarden
Illegal instruction
```

I initially thought this might be a packaging bug but I'm certain I
used to run successfully the older 1.33.2-r0 prior to recently
upgrading to alpine 3.22.  Instead then I'm inclined to believe that
something else *weird* is going on on this system.  Nothing sinister,
but I haven't been running alpine long and this was my first attempt at
upgrading in place. Maybe I missed a step?  I'm fairly sure I followed
the regular upgrade instructions, followed by upgrading the kernel and
initramfs:

https://wiki.alpinelinux.org/wiki/Diskless_Mode#Upgrading_a_diskless_system

Does anyone have any thoughts for how to begin to debug this?  I would
think that building vaultwarden from source might be the way to go
initially, but this is a rather resource constrained system.  I have in
fact tried building it, but even with generous swap space and single
threaded building (cargo build -j 1) it gets oomkilled.

This whole thing is written in rust of course, I know very little about
rust. Maybe this is a glaringly obvious issue to someone else.

I would massively appreciate any ideas, sucks to be without your
password manager!

Thanks
Max

---

From: Ian Smith <[email-masked]>
Date: Fri, 25 Jul 2025 11:33:20 +0100

You _might_ get some more information from strace or ltrace.

---

From: Maximilian Friedersdorff <[email-masked]>
Date: Fri, 25 Jul 2025 12:09:57 +0100

On Fri, 25 Jul 2025 11:33:20 +0100

> You _might_ get some more information from strace or ltrace.

Trying this! Not gonna spam the whole output here but the last few
lines:

```
openat(AT_FDCWD, "/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
fcntl(3, F_SETFD, FD_CLOEXEC)           = 0
fstat(3, {st_mode=S_IFREG|0755, st_size=133008, ...}) = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 960) = 960
mmap(NULL, 200704, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x7faf4e9000
mmap(0x7faf518000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x1f000) = 0x7faf518000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fafc4d000
mprotect(0x7fafc52000, 4096, PROT_READ) = 0
mprotect(0x7fafb85000, 40960, PROT_READ) = 0
mprotect(0x7fafa38000, 462848, PROT_READ) = 0
mprotect(0x7faf63d000, 12288, PROT_READ) = 0
mprotect(0x7faf5f4000, 32768, PROT_READ) = 0
mprotect(0x7faf5a7000, 16384, PROT_READ) = 0
mprotect(0x7faf549000, 4096, PROT_READ) = 0
mprotect(0x7faf518000, 4096, PROT_READ) = 0
mprotect(0x55848e4000, 1753088, PROT_READ) = 0
set_tid_address(0x7fafc55e80)           = 11689
getrandom("\x4d\xb3\xb9\xbf\x87\xe3\xec\x20\xc1\xac\x31\x55\xc5\xf9\x13\x68\x03\x5d\x6f\x1b\x05\x3c\x7c\xc5\x03\x5e\xc8\xba\x09\xd1\x35\xbd", 32, GRND_NONBLOCK) = 32
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0x7faf612968} ---
+++ killed by SIGILL +++
Illegal instruction
```

I'm none the wiser to be honest!  I did try gdb also:

```
gdb vaultwarden
[noise]

Reading symbols from vaultwarden...
(No debugging symbols found in vaultwarden)
(gdb) run
Starting program: /usr/bin/vaultwarden

Program received signal SIGILL, Illegal instruction.
0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2
(gdb) bt
#0  0x0000007ff79bc968 in ?? () from /usr/lib/libmimalloc-secure.so.2
#1  0x0000007ff79ba7f8 in ?? () from /usr/lib/libmimalloc-secure.so.2
#2  0x0000007ff7fa8368 in do_init_fini (queue=<optimized out>)
    at ldso/dynlink.c:1610
#3  0x0000007ff7fa9240 in __libc_start_init () at ldso/dynlink.c:1623
#4  0x0000007ff7f5d03c in libc_start_main_stage2 (main=0x5555c46fe8,
    argc=1, argv=0x7ffffffc08) at src/env/__libc_start_main.c:92
#5  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

Based on that it seems maybe a problem in mimalloc2?  I tried
re-installing that, maybe it had become corrupted or something, but no
change in behaviour.  `vaultwarden` is the only package on this system
depending on mimalloc2, so a bug/problem in that library doesn't seem
inherently insane.

Max

---

From: Naomi Rennie-Waldock <[email-masked]>
Date: Sat, 26 Jul 2025 15:58:18 +0100

>depending on mimalloc2, so a bug/problem in that library doesn't seem

Are you able to try rebuilding the mimalloc2 with -DMI_NO_OPT_ARCH=ON added to the cmake arguments?

In this section:
https://gitlab.alpinelinux.org/alpine/aports/-/blob/3.22-stable/community/mimalloc2/APKBUILD?ref_type=heads#L48

If not, I should have time to test tomorrow.

---

From: [email-masked]
Date: Sun, 27 Jul 2025 12:59:21 +0100

> added to the cmake arguments?

I have successfully built mimalloc2 with that option.  I also had to 
disable checks, but I think only because the timeout set in the checks 
was too aggressive for the anemic Raspberry Pi 4.

This has solved this issue at though, my password manager is back up!

Thanks for you help!

Question then, does that represent a bug in the APKBUILD for the 
mimalloc2 package? Should I raise a Merge Request to add this compile 
time option?

Thanks all
Max

---

From: Naomi Rennie-Waldock <[email-masked]>
Date: Tue, 29 Jul 2025 01:02:25 +0100

On 27 July 2025 12:59:21 BST, [email-masked] wrote:
>I have successfully built mimalloc2 with that option.  I also had to disable checks, but I think only because the timeout set in the checks was too aggressive for the anemic Raspberry Pi 4.
>
>Question then, does that represent a bug in the APKBUILD for the mimalloc2 package? Should I raise a Merge Request to add this compile time option?
>
>Thanks all
>Max

Good to hear!

The optimizations should definitely be disabled in the aport, a lot of our supported ARM devices don't support those instructions.

---

From: Jeffrey Walton <[email-masked]>
Date: Fri, 25 Jul 2025 08:27:47 -0400

On Fri, Jul 25, 2025 at 7:11 AM Maximilian Friedersdorff <
[email-masked]> wrote:

> [...]
> Based on that it seems maybe a problem in mimalloc2?

It looks like a Microsoft library. See <
https://microsoft.github.io/mimalloc/build.html>.

You will probably get more information if you install the debug symbols. I
don't know how to do it, though.

Jeff