Hello,
I upgraded Alpine Linux to 3.16 on my old Celeron-based Laptop.
now, the same task of compiling the Arctic Fox browser takes instead of
about 4 hours, almost 10 days!! I extra left it running until I get the
same compile error as before.
It is a bit extreme performance drop!
I noticed that while compiling, the laptop fan does not ran constantly
at high speed, but it remains quited for a period, then spins up, then
quiet again, even if CPU is 100%.
I can think of two causes, from my past experience with other systems:
- disabled caches
- CPU not runnig at full speed
In the BIOS I checked and CPU and L2 caches are enabled (small in this
crippled CPU anyway).
Could some APM/ACPI lower the cpu speed until I rise it withsome governor?
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Mobile Intel(R) Celeron(R) CPU 2.20GHz
stepping : 9
microcode : 0x10
cpu MHz : 2194.366
cache size : 256 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fdiv_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts
cpuid cid xtpr
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass
l1tf mds swapgs itlb_multihit
bogomips : 4390.59
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 32 bits virtual
power management:
Could some of the bug mitigation have such a strong impact? How can I
disable them in alpine? I don't think they would be a safety concern for
the kind of usage I have.
Riccardo
On Mon Sep 12, 2022 at 10:47 AM CEST, Riccardo Mottola wrote:
> Hello,>> I upgraded Alpine Linux to 3.16 on my old Celeron-based Laptop.>> now, the same task of compiling the Arctic Fox browser takes instead of> about 4 hours, almost 10 days!! I extra left it running until I get the> same compile error as before.> It is a bit extreme performance drop!> I noticed that while compiling, the laptop fan does not ran constantly> at high speed, but it remains quited for a period, then spins up, then> quiet again, even if CPU is 100%.
you should monitor the frequency during that- it probably goes down/up
as the fans do
>> I can think of two causes, from my past experience with other systems:> - disabled caches
i don't think a kernel update would change this, either
> - CPU not runnig at full speed>> In the BIOS I checked and CPU and L2 caches are enabled (small in this> crippled CPU anyway).> Could some APM/ACPI lower the cpu speed until I rise it withsome governor?
you can check the governors with:
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
but i don't think even being stuck on "powersave" or something would do
that.. strange
you can change them to performance, which is usually the highest. the
sane default is "schedutil" when the cpufreq driver supports it to
balance power saving with performance, on some intel drivers the
performance one is basically that. i don't think anything changed in the
kernel defaults to make it powersave, so i'm not sure why it would be
that bad.
this sounds like some weird kernel regression in 5.15, if anything
>> processor : 0> vendor_id : GenuineIntel> cpu family : 15> model : 2> model name : Mobile Intel(R) Celeron(R) CPU 2.20GHz> stepping : 9> microcode : 0x10> cpu MHz : 2194.366> cache size : 256 KB> physical id : 0> siblings : 1> core id : 0> cpu cores : 1> apicid : 0> initial apicid : 0> fdiv_bug : no> f00f_bug : no> coma_bug : no> fpu : yes> fpu_exception : yes> cpuid level : 2> wp : yes> flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe pebs bts> cpuid cid xtpr> bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass> l1tf mds swapgs itlb_multihit> bogomips : 4390.59> clflush size : 64> cache_alignment : 128> address sizes : 36 bits physical, 32 bits virtual> power management:>> Could some of the bug mitigation have such a strong impact? How can I> disable them in alpine? I don't think they would be a safety concern for> the kind of usage I have.
add mitigations=off to the kernel cmdline, via whatever booting setup
you have. usual warnings/caveats apply, but there you go. i doubt they
would be that impactful though..
>> Riccardo
Hello Alice,
alice wrote:
> On Mon Sep 12, 2022 at 10:47 AM CEST, Riccardo Mottola wrote:> you should monitor the frequency during that- it probably goes down/up> as the fans do
cpuinfo gives always the same frequency
>> I can think of two causes, from my past experience with other systems:>> - disabled caches> i don't think a kernel update would change this, either>> - CPU not runnig at full speed>>>> In the BIOS I checked and CPU and L2 caches are enabled (small in this>> crippled CPU anyway).>> Could some APM/ACPI lower the cpu speed until I rise it withsome governor?> you can check the governors with:> cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
my cpu9 does not have cpufreq, either it is not supported at all on this
old CPU model or something is not detected?
That would make me guess however that it runs at the speed shown in
cpuinfo, 2.2GHz.
also:
cervino:~# cpufreq-info
cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
>> but i don't think even being stuck on "powersave" or something would do> that.. strange
No, I would, at the opposite, suppose it would be stuck to the frequency
reported in cpuinfo, that is, the maximum
>> this sounds like some weird kernel regression in 5.15, if anything
strange indeed
5.15.64-0-lts #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000 i686 Linux
in dmesg I see:
[ 1.605776] intel_pstate: CPU model not supported
but I don't know if this is relevant, this CPU is based on netburst
architecture. I think that back then, few cpu scaling methods where
available. Original speedstep and perhaps p4_clockmod - where the clock
doesn't really change?
I cannot identify this CPU here:
https://en.wikipedia.org/wiki/List_of_Intel_Celeron_processors#Netburst_based_Celerons
since 2.2Ghz appears to have 128K cache. It could be this one:
https://ark.intel.com/content/www/us/en/ark/products/27318/mobile-intel-celeron-processor-2-20-ghz-256k-cache-400-mhz-fsb.html
still i don't think too much relevant to have the exact detail.
> add mitigations=off to the kernel cmdline, via whatever booting setup> you have. usual warnings/caveats apply, but there you go. i doubt they> would be that impactful though..
I doubt too, on more modern CPUs I have found an impact of 1% to 10%
usually... but maybe on this old cache-limited CPU?
I have syslinux, I don't know hot do add that option to
|/etc/update-extlinux.conf|, but I will add it manually for a boot and
test the difference.
Riccardo
On Mon Sep 12, 2022 at 2:59 PM CEST, Riccardo Mottola wrote:
> Hello Alice,>>> alice wrote:> > On Mon Sep 12, 2022 at 10:47 AM CEST, Riccardo Mottola wrote:> > you should monitor the frequency during that- it probably goes down/up> > as the fans do>> cpuinfo gives always the same frequency> >> I can think of two causes, from my past experience with other systems:> >> - disabled caches> > i don't think a kernel update would change this, either> >> - CPU not runnig at full speed> >>> >> In the BIOS I checked and CPU and L2 caches are enabled (small in this> >> crippled CPU anyway).> >> Could some APM/ACPI lower the cpu speed until I rise it withsome governor?> > you can check the governors with:> > cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor>> my cpu9 does not have cpufreq, either it is not supported at all on this> old CPU model or something is not detected?> That would make me guess however that it runs at the speed shown in> cpuinfo, 2.2GHz.
aha, i thought it was just a tad newer :) apologies. yeah, things from
that era don't usually change clocks that much.
>> also:>> cervino:~# cpufreq-info> cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009> Report errors and bugs to cpufreq@vger.kernel.org, please.> analyzing CPU 0:>> >> > but i don't think even being stuck on "powersave" or something would do> > that.. strange>> No, I would, at the opposite, suppose it would be stuck to the frequency> reported in cpuinfo, that is, the maximum>> >> > this sounds like some weird kernel regression in 5.15, if anything>> strange indeed>> 5.15.64-0-lts #1-Alpine SMP Mon, 05 Sep 2022 08:02:49 +0000 i686 Linux>>> in dmesg I see:> [ 1.605776] intel_pstate: CPU model not supported>>> but I don't know if this is relevant, this CPU is based on netburst> architecture. I think that back then, few cpu scaling methods where> available. Original speedstep and perhaps p4_clockmod - where the clock> doesn't really change?
maybe there's a chance that if intel_pstate loads first it blocks the
others? (just guessing). both of the others are also available as
modules, maybe something funny happens with pstate. perhaps
modprobe.blacklist=intel_pstate does something (or similar things in
that vein)
>> I cannot identify this CPU here:> https://en.wikipedia.org/wiki/List_of_Intel_Celeron_processors#Netburst_based_Celerons>> since 2.2Ghz appears to have 128K cache. It could be this one:> https://ark.intel.com/content/www/us/en/ark/products/27318/mobile-intel-celeron-processor-2-20-ghz-256k-cache-400-mhz-fsb.html>> still i don't think too much relevant to have the exact detail.>>> > add mitigations=off to the kernel cmdline, via whatever booting setup> > you have. usual warnings/caveats apply, but there you go. i doubt they> > would be that impactful though..>> I doubt too, on more modern CPUs I have found an impact of 1% to 10%> usually... but maybe on this old cache-limited CPU?>> I have syslinux, I don't know hot do add that option to> |/etc/update-extlinux.conf|, but I will add it manually for a boot and> test the difference.
the default_kernel_opts line can have it
>> Riccardo
Hello Alice,
I discovered something unexpected. The impact of mitgations is extreme!
alice wrote:
> maybe there's a chance that if intel_pstate loads first it blocks the> others? (just guessing). both of the others are also available as> modules, maybe something funny happens with pstate. perhaps> modprobe.blacklist=intel_pstate does something (or similar things in> that vein)
I tried playing with that, no relevant changes. The module wasn't even
loaded (I suppose the error is about that).
>> I cannot identify this CPU here:>> https://en.wikipedia.org/wiki/List_of_Intel_Celeron_processors#Netburst_based_Celerons>>>> since 2.2Ghz appears to have 128K cache. It could be this one:>> https://ark.intel.com/content/www/us/en/ark/products/27318/mobile-intel-celeron-processor-2-20-ghz-256k-cache-400-mhz-fsb.html>>>> still i don't think too much relevant to have the exact detail.>>>>>>> add mitigations=off to the kernel cmdline, via whatever booting setup>>> you have. usual warnings/caveats apply, but there you go. i doubt they>>> would be that impactful though..>> I doubt too, on more modern CPUs I have found an impact of 1% to 10%>> usually... but maybe on this old cache-limited CPU?>>>> I have syslinux, I don't know hot do add that option to>> |/etc/update-extlinux.conf|, but I will add it manually for a boot and>> test the difference.> the default_kernel_opts line can have it
That was it. mitigations=off makes the system almost as fast as before,
it takes now about 680minutes instead of... 7000!! (still slower than
the previous 500 minutes, but also gcc was upgraded, so other things
could take in account the slowdown).
It means one or more of the disabled mitgations has an extreme impact
(roughtly 1/10th performance) in the specific task of heavy compilation,
python and make scripts.
Being a celeron, this is a non-hyper-threaded CPU, but it was one of the
first CPUs with a very deep pipeline and high clock speed (2.2GHz).
Maybe some mitigations aren't even needed, however it would be
interesting to know which one(s) makes this huge impact, could be useful
for others. But to do that I need somehow a smaller testcase, I can't
wait a night vs. a fortnight everytime :)
Riccardo