Subject: performance drop with alpine linux upgrade

From: Riccardo Mottola <[email-masked]>
Date: Mon, 12 Sep 2022 10:47:05 +0200

Hello,

I upgraded Alpine Linux to 3.16 on my old Celeron-based Laptop.

now, the same task of compiling the Arctic Fox browser takes instead of
about 4 hours, almost 10 days!! I extra left it running until I get the
same compile error as before.
It is a bit extreme performance drop!
I noticed that while compiling, the laptop fan does not ran constantly
at high speed, but it remains quited for a period, then spins up, then
quiet again, even if CPU is 100%.

I can think of two causes, from my past experience with other systems:
- disabled caches
- CPU not runnig at full speed

In the BIOS I checked and CPU and L2 caches are enabled (small in this
crippled CPU anyway).
Could some APM/ACPI lower the cpu speed until I rise it withsome governor?

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Mobile Intel(R) Celeron(R) CPU 2.20GHz
stepping        : 9
microcode       : 0x10
cpu MHz         : 2194.366
cache size      : 256 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fdiv_bug        : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts
cpuid cid xtpr
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
l1tf mds swapgs itlb_multihit
bogomips        : 4390.59
clflush size    : 64
cache_alignment : 128
address sizes   : 36 bits physical, 32 bits virtual
power management:

Could some of the bug mitigation have such a strong impact? How can I
disable them in alpine? I don't think they would be a safety concern for
the kind of usage I have.

Riccardo

---

From: "alice" <[email-masked]>
Date: Mon, 12 Sep 2022 12:12:20 +0200

> quiet again, even if CPU is 100%.
you should monitor the frequency during that- it probably goes down/up
as the fans do
> I can think of two causes, from my past experience with other systems:
i don't think a kernel update would change this, either
> Could some APM/ACPI lower the cpu speed until I rise it withsome governor?
you can check the governors with:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

but i don't think even being stuck on "powersave" or something would do
that.. strange

you can change them to performance, which is usually the highest. the
sane default is "schedutil" when the cpufreq driver supports it to
balance power saving with performance, on some intel drivers the
performance one is basically that. i don't think anything changed in the
kernel defaults to make it powersave, so i'm not sure why it would be
that bad.

this sounds like some weird kernel regression in 5.15, if anything

> the kind of usage I have.
add mitigations=off to the kernel cmdline, via whatever booting setup
you have. usual warnings/caveats apply, but there you go. i doubt they
would be that impactful though..

---

From: Riccardo Mottola <[email-masked]>
Date: Mon, 12 Sep 2022 14:59:20 +0200

Hello Alice,

> you should monitor the frequency during that- it probably goes down/up

cpuinfo gives always the same frequency
> cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

my cpu9 does not have cpufreq, either it is not supported at all on this
old CPU model or something is not detected?
That would make me guess however that it runs at the speed shown in
cpuinfo, 2.2GHz.

also:

cervino:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to [email-masked], please.

> but i don't think even being stuck on "powersave" or something would do

No, I would, at the opposite, suppose it would be stuck to the frequency
reported in cpuinfo, that is, the maximum

> this sounds like some weird kernel regression in 5.15, if anything

strange indeed

5.15.64-0-lts #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000 i686 Linux

in dmesg I see:
[    1.605776] intel_pstate: CPU model not supported

but I don't know if this is relevant, this CPU is based on netburst
architecture. I think that back then, few cpu scaling methods where
available. Original speedstep and perhaps p4_clockmod - where the clock
doesn't really change?

I cannot identify this CPU here:
https://en.wikipedia.org/wiki/List_of_Intel_Celeron_processors#Netburst_based_Celerons

since 2.2Ghz appears to have 128K cache. It could be this one:
https://ark.intel.com/content/www/us/en/ark/products/27318/mobile-intel-celeron-processor-2-20-ghz-256k-cache-400-mhz-fsb.html

still i don't think too much relevant to have the exact detail.

> would be that impactful though..

I doubt too, on more modern CPUs I have found an impact of 1% to 10%
usually... but maybe on this old cache-limited CPU?

I have syslinux, I don't know hot do add that option to
|/etc/update-extlinux.conf|, but I will add it manually for a boot and
test the difference.

Riccardo

---

From: "alice" <[email-masked]>
Date: Mon, 12 Sep 2022 15:07:35 +0200

> That would make me guess however that it runs at the speed shown in
aha, i thought it was just a tad newer :) apologies. yeah, things from
that era don't usually change clocks that much.
> available. Original speedstep and perhaps p4_clockmod - where the clock
maybe there's a chance that if intel_pstate loads first it blocks the
others? (just guessing). both of the others are also available as
modules, maybe something funny happens with pstate. perhaps
modprobe.blacklist=intel_pstate does something (or similar things in
that vein)
> |/etc/update-extlinux.conf|, but I will add it manually for a boot and
the default_kernel_opts line can have it

---

From: Riccardo Mottola <[email-masked]>
Date: Tue, 20 Sep 2022 13:16:34 +0200

Hello Alice,

I discovered something unexpected. The impact of mitgations is extreme!

> modprobe.blacklist=intel_pstate does something (or similar things in

I tried playing with that, no relevant changes. The module wasn't even
loaded (I suppose the error is about that).

> the default_kernel_opts line can have it

That was it. mitigations=off makes the system almost as fast as before,
it takes now about 680minutes instead of... 7000!! (still slower than
the previous 500 minutes, but also gcc was upgraded, so other things
could take in account the slowdown).

It means one or more of the disabled mitgations has an extreme impact
(roughtly 1/10th performance) in the specific task of heavy compilation,
python and make scripts.
Being a celeron, this is a non-hyper-threaded CPU, but it was one of the
first CPUs with a very deep pipeline and high clock speed (2.2GHz).
Maybe some mitigations aren't even needed, however it would be
interesting to know which one(s) makes this huge impact, could be useful
for others. But to do that I need somehow a smaller testcase, I can't
wait a night vs. a fortnight everytime :)

Riccardo