Date: Thu, 12 Aug 2021 16:59:57 -0400
From: "Alex Xu (Hello71)" <alex_y_xu@yahoo.ca>
Subject: [RFC] Enable -fno-plt for x86 and x86_64
To: ~alpine/devel@lists.alpinelinux.org
MIME-Version: 1.0
Message-Id: <1628515011.zujvcn248v.none@localhost>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
References: <1628515011.zujvcn248v.none.ref@localhost>

## Summary

-fno-plt is a gcc flag that disables the use of PLT stubs for function=20
calls. I suggest we enable it for x86 and x86_64. The code size bloat is=20
substantial on aarch64, and (as far as I could tell) it is silently=20
ignored on arm32, mips64, s390x, riscv64, and ppc64le. On mips64 and=20
riscv64, there is -mno-plt, but it significantly bloats code size and=20
instruction count.

## Benefits

Each PLT thunk adds at least 32 bytes to the binary size on x86 and=20
x86_64. The PLT also occupies valuable instruction cache space and, on=20
x86, indirect jump predictor space (the actual function call occupies=20
one slot, plus PLT accesses on x86 use indirect calls so occupy another=20
slot).

I did a quick test, and building apk-tools on x86_64 with -fno-plt (plus=20
the standard abuild.conf flags) reduces package size from 119480 to=20
115686 (-3794) and install size from 253648 to 245384 (-8264).

## Downsides

As far as I know, the main reason why -fno-plt is not used by default in=20
gcc is because it is incompatible with lazy linking. This is not=20
relevant to Alpine Linux, as musl does not support lazy linking anyways.

There is another, less documented issue: -fno-plt increases code size=20
for non-interposable ELF function calls on x86 and x86_64 from 5 bytes=20
to 6 bytes. On aarch64, calls double from 8 bytes to 16 bytes, plus 8=20
more bytes per function to get the GOT address. On rv64gc, -mno-plt=20
inflates calls from 4 bytes to 10 bytes. Note that these numbers are=20
with -fPIE; -fno-PIE numbers differ.

Certain packages may not compile or run properly with -fno-plt, but=20
these are likely to be minimal. See the next section for more=20
information.

## Other distros

Arch Linux has enabled -fno-plt by default (and supports only x86_64)=20
since 2017: https://github.com/archlinux/svntogit-packages/commit/72de4d337=
e02e5626598038b801517e47988b4c8.

It is disabled for xorg, glibc, valgrind, and openjdk:
https://github.com/archlinux/svntogit-packages/search?q=3D%22-fno-plt%22

Gentoo does not use -fno-plt but a third-party overlay enables -fno-plt=20
along with other optimization flags such as LTO. There are minimal=20
issues reported with -fno-plt, mostly around the usual suspects doing=20
sketchy things like wine, glibc, and xorg-server:
https://github.com/InBetweenNames/gentooLTO/blob/master/sys-config/ltoize/f=
iles/package.cflags/no-plt.conf

I strongly suspect the remaining packages are false positives, since=20
exceptions are applied for basically anybody that reports an issue, and=20
they ave almost never removed.

OpenWRT has enabled -fno-plt since 2016:=20
https://github.com/openwrt/openwrt/commit/fb713ddd4dd49fb60ee4ab732071abf2c=
3ad5fc5.

As far as I could tell, Debian, Fedora, and openSUSE do not use=20
-fno-plt by default. I believe for Debian, this is because BIND_NOW is=20
still not enabled by default. For Fedora and openSUSE, I think binary=20
size is not a priority.

## Conclusion

I think -fno-plt has moderate benefits, and minimal costs for x86 and=20
x86_64. There is some risk of breaking some packages, but I think the=20
risk is minimal as long as the known-broken packages (xorg, glibc,=20
valgrind, and openjdk) have -fno-plt fully tested and disabled if=20
necessary, and the flag overall is given enough time to test in edge=20
before a stable release.

I am primarily concerned with and have primarily analyzed x86 and=20
x86_64, but am not opposed to enabling it also for other architectures=20
if someone does better analysis and finds a better cost-benefit ratio.