Subject: [alpine-devel] force compile flag for musl?

From: Natanael Copa <[email-masked]>
Date: Wed, 25 Oct 2017 16:46:14 +0200

Hi,

I wonder what you think about overriding the -Os compile flag for musl,
and hardcode it to -O2.

I think this makes sense since the functions in libc are so often used
that we want trade better performance at the cost of slightly bigger
binary.

This means that we override whatever user as set CFLAGS to
in /etc/abuild.conf

We already do this with zlib.

What do you think?

diff --git a/main/musl/APKBUILD b/main/musl/APKBUILD
index 1938bbb3ca..193002186d 100644
--- a/main/musl/APKBUILD
+++ b/main/musl/APKBUILD
@@ -54,6 +54,8 @@ build() {
        fi
 
        # note: not autotools
+       # force -O2 compile flag for better performance
+       CFLAGS="-O2" \
        LDFLAGS="$LDFLAGS -Wl,-soname,libc.musl-${CARCH}.so.1" \
        ./configure \
                --build=$CBUILD \

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: "A. Wilcox" <[email-masked]>
Date: Wed, 25 Oct 2017 14:17:28 -0500

On 25/10/17 09:46, Natanael Copa wrote:
> Hi,
> 
> 
> I wonder what you think about overriding the -Os compile flag for 
> musl, and hardcode it to -O2.

Possibly.  Is there some benchmarks available, maybe using libc-test
or such?

> What do you think?
> 
> +       # force -O2 compile flag for better performance +
> CFLAGS="-O2" \

No.  Stuff in abuild.conf needs to be preserved and -O2 tacked on the
end (GCC will only honour last -O flag passed, so this is what you want).

This is INCREDIBLY important to us at Adélie because for instance
ppc64 BE requires -fno-inline-small-functions due to GCC bug elsewise
it will cause ABI issues with long double.  We also use -march / -mcpu
which would be discarded here as well, which would cause it to be
slower on some platforms (-O2 won't help as much on x86_32 as it would
on -march=pentium4 x86_32, for instance).

I agree with the idea of using -O2 but not with the implementation of
blowing away all other CFLAGS.

Best,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org

---

From: Natanael Copa <[email-masked]>
Date: Wed, 25 Oct 2017 21:54:22 +0200

On Wed, 25 Oct 2017 14:17:28 -0500

> on -march=pentium4 x86_32, for instance).

Thanks. This is useful information.
 
> blowing away all other CFLAGS.

I was thinking about this, yes. I think something like:

  CFLAGS="$CFLAGS -O2"

would be better. Alternatively we could have an optional performance opt, something like:

  CFLAGS="${CFLAGS_OPT_PERFORMACE:-$CFLAGS}"

But I think we just keep it simple as it is, because I measured it and
-O2 was actually slightly slower for the specific use case.

Thanks!

-nc

---

---

From: William Pitcock <[email-masked]>
Date: Wed, 25 Oct 2017 18:36:40 -0500

Hi,

> on -march=pentium4 x86_32, for instance).

Maybe we should fix the specfiles to bring in
-fno-inline-small-functions on ppc64/ppc64le?

This seems like something that could hurt if you run into it on a ppc64 machine.

William

---

---

From: "A. Wilcox" <[email-masked]>
Date: Wed, 25 Oct 2017 19:51:08 -0500

On 25/10/17 18:36, William Pitcock wrote:
>> On 25/10/17 09:46, Natanael Copa wrote: This is INCREDIBLY
>> important to us at Adélie because for instance ppc64 BE requires
>> -fno-inline-small-functions due to GCC bug elsewise it will cause
>> ABI issues with long double.
> 

I plan on doing more regression testing with different versions and
builds of GCC (at least 5.x, 6.3, and 7.2 in addition to 6.4) to see
if this is a regression or where the bug was introduced.  It would be
much better to find the root cause of this breakage and include a
patch to gcc than to work around it with a specfile.

Also, I have so far not run into any issues with ppc64 BE with
- -fno-inline-small-functions but that could be masking a more subtle
compiler bug.  It's all quite scary, to be blunt... but such is life
with very untested code... don't think anyone has ever used
- -mabi=elfv2 with ppc64 before.

Best,
- --arw

- -- 
A. Wilcox (awilfox)

---

---

From: Przemysław Pawełczyk <[email-masked]>
Date: Wed, 25 Oct 2017 17:38:57 +0200

---- On Wed, 25 Oct 2017 16:46:14 +0200 Natanael Copa <[email-masked]> wrote ---- 
> I wonder what you think about overriding the -Os compile flag for musl, 

I would be very careful with such changes.

There is misconception that the higher optimization level, the faster
code is generated.  That is the general -Olevel idea, but not what is
seen in practice.  Gains (or losses) from higher optimization levels
vary between archs and obviously depend on the code that is being
optimized.

Smaller code, beside being smaller, is also more cache-friendly, so -Os
can be faster than -O2 and often is.  OTOH higher optimization levels
for x86-64 usually tend to give better results than on other archs.

There is no rule.  It all depends on:
- source code,
- compiler,
- platform.

> that we want trade better performance at the cost of slightly bigger 

This makes sense if we really get better performance with -O2 on all
platforms AL supports.  And to be able to confirm that, it has te be
measured.

> We already do this with zlib. 

zlib is a different beast, because it's computational software.  It's
much more natural to see gains from higher -Olevel in that kind of apps.

> What do you think? 

There were similar changes in aports for various applications over
recent months, but I haven't seen even one proof behind them.

Performance improvements are imporant, and they may come from simply
bumping optimization level, but it should be verified, not blindly
assumed.

Regards,
Przemek

---

---

From: Jakub Jirutka <[email-masked]>
Date: Wed, 25 Oct 2017 21:49:43 +0200

Uh, I’ve accidentally sent my response to just specific people and not to ML. :( So once again and sorry for mess.

>>       LDFLAGS="$LDFLAGS -Wl,-soname,libc.musl-${CARCH}.so.1" \ 

This is IMO not correct, CFLAGS does not define just -Os, but more flags (-fomit-frame-pointer). Here you replace all the default flags with -O2.
I suggest to use `CFLAGS="${CFLAGS/-Os/-O2}"` instead, that’s what I already used in some abuilds.

> OTOH higher optimization levels for x86-64 usually tend to give better results than on other archs.

x86_64 is the most used platform, especially when you need performance (and can’t afford IBM’s proprietary architectures). I’d bet that it applies even to Alpine, but don’t have any numbers.

> There were similar changes in aports for various applications over recent months, but I haven’t seen even one proof behind them.
> 
> Performance improvements are imporant, and they may come from simply bumping optimization level, but it should be verified, not blindly assumed.

Technically you’re right, it’d be indeed nice to have some real proof. Unfortunately it’s quite hard to make a good benchmark and probably no one of us have time to do that. :(

However, there’s another reason why to prefer -O2. It’s the default optimization level that use almost all upstream projects and even downstream (other distros). So it’s the most tested variant and if the project care about performance, they typically optimize for -O2 (or -O3). But I must mention that I’m really not expert in this field, I can only say what I see as the most used, but not whether it really makes in technical aspects.

I’m the one who has changed -Os to -O2 in _some_ specific abuilds where performance is important, e.g. x264, opus, qemu, postgresql… I actually tried to find some proof for PostgreSQL, but the only performance good comparison I found [1] compares just -O2, -O3, -O4, -march=native and -flto, not -Os.

Just a note: we already compile many aports with -O2 and most of you don’t even know about it and/or didn’t care. These are almost all aports built by CMake. CMake by default doesn’t log what it is really executing, unless you set `-DCMAKE_VERBOSE_MAKEFILE=ON` (that’s why I usually enable it). If you’d enable verbose mode, you would see that -Os is passed to gcc, but it’s followed by -O2 added by CMake. That’s what `-DCMAKE_BUILD_TYPE=Release` do (among others); it’s a profile that predefines various flags including -O and it has higher priority than CFLAGS from environment. If you want -Os, there’s a built-in profile MinSizeRel. You can look which aports use this flag… ;)

However, as I have discovered over time, MinSizeRel is not always usable. Many projects have very bad CMakeLists, they fully or partially ignore CMAKE_BUILD_TYPE or foolishly assumes that release profile is just and only Release, so build sometimes even fails with MinSizeRel.

So, to sum it up, -O2 is the default and most used optimization level in most upstream projects and even other distributions, even we already use it for many aports, whatever you’re aware of it or not. So I’d support changing the default to -O2, at least for x86_64, and change it to -Os in specific abuilds where it makes sense (especially small static binaries).

As a (partial) academic person, I must agree with Przemysław about benchmarking vs. believing, but it’s unfortunately not realistic, at least not for all aports. It’d be really great and useful if someone can measure difference at least for very core components like musl libc.

I’d like to ask Skarnet and Shiz about their opinion and expertise in this topic.

Jakub

[1]: https://blog.pgaddict.com/posts/compiler-optimization-vs-postgresql

> On 25. Oct 2017, at 17:38, Przemysław Pawełczyk <[email-masked]> wrote:
> 

---

---

From: Natanael Copa <[email-masked]>
Date: Wed, 25 Oct 2017 21:46:03 +0200

On Wed, 25 Oct 2017 17:38:57 +0200

> There is no rule.  It all depends on:

As I understand -Os and -O2 are basically the same thing, with the
difference that -O2 enables some more alignments.

> > I think this makes sense since the functions in libc are so often
> > used that we want trade better performance at the cost of slightly
> > bigger binary.   
> 

You are right, of course.

I did a quick test with the code that made me think of -O2 in the first place. It is from here:
https://superuser.com/questions/1219609/why-is-the-alpine-docker-image-over-50-slower-than-the-ubuntu-image

BENCHMARK="import timeit; print(timeit.timeit('import json; json.dumps(list(range(10000)))', number=5000))"

# with -O2

$ for i in $(seq 0 6); do python3 -c "$BENCHMARK"; done
8.284425613994244
8.354899992002174
8.359624709992204
8.392496702988865
8.3223694319895
8.285188248992199
8.294311116012977

# with -Os

$ for i in $(seq 0 6); do python3 -c "$BENCHMARK"; done
8.267152725020424
8.260578833986074
8.232940819987562
8.224149581015809
8.27815192801063
8.31035227200482
8.29849099801504

So it looks that for this specific workload, -Os is actually slightly
faster.

It just highlights the point that you need to actually measure before
trying to optimize.

> much more natural to see gains from higher -Olevel in that kind of

I'm glad to hear that I'm not totally stupid :)

> bumping optimization level, but it should be verified, not blindly

Yeah, lets keep -Os and only change if its tested to have any impact.

---