Subject: [alpine-devel] Splitting up linux-firmware?

From: Oliver Smith <[email-masked]>
Date: Sat, 06 Jan 2018 22:13:00 +0000

Hey alpine-devel,

there's the >250 MB (install size) linux-firmware package, of which usually only few files are needed for one machine.
For other packages, Alpine already splits off -dev and even -doc to save as much space as possible, so it would make sense to me if we started splitting up linux-firmware as well.
Our use case in Alpine Linux based postmarketOS is, that we could only flash the firmware subpackges to the mobile device, which we actually need (thus making the total image size a lot smaller).

The package provides files in /lib/firmware and is already categorized in subfolders. So we could make one subpackage for each subfolder. Examples:

- linux-firmware-liquido (23.7 MB)
- linux-firmware-netrome (22.0 MB)
- linux-firmware-brcm (15.1 MB)
- linux-firmware-amdgpu (13.9 MB)
- linux-firmware-intel (12.8 MB)
- linux-firmware-ti-connectivity (8.6 MB)
- linux-firmware-bnx2x (8.6 MB)
- linux-firmware-mrvl (7.7 MB)
- linux-firmware-radeon (7.4 MB)

Files directly in /lib/firmware could go into "linux-firmware-other", and the "linux-firmware" package could depend on all its subpackages for compatibility.

What do you think, would it make sense to submit a PR for that?

Thanks,
Oliver Smith

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: "A. Wilcox" <[email-masked]>
Date: Sat, 6 Jan 2018 17:09:53 -0600

On 01/06/18 16:13, Oliver Smith wrote:
> Hey alpine-devel,
> 
> there's the >250 MB (install size) linux-firmware package, of which
> usually only few files are needed for one machine. For other
> packages, Alpine already splits off -dev and even -doc to save as
> much space as possible, so it would make sense to me if we started
> splitting up linux-firmware as well. Our use case in Alpine Linux
> based postmarketOS is, that we could only flash the firmware
> subpackges to the mobile device, which we actually need (thus making
> the total image size a lot smaller).
> 
> The package provides files in /lib/firmware and is already
> categorized in subfolders. So we could make one subpackage for each
> subfolder.
> 
> Files directly in /lib/firmware could go into "linux-firmware-other",
> and the "linux-firmware" package could depend on all its subpackages
> for compatibility.
> 

All my +1.

The size is so large and the download time is so long that I've taken to
just scp'ing around the /lib/firmware files I need on different machines
from a 'master' machine that has the full package installed.

I've thought about suggesting this before but I wasn't exactly sure the
best naming scheme (should it be amdgpu-firmware or etc), but your
proposal is actually very clean.  And it keeps compatibility.

Best,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org

---

From: Drew DeVault <[email-masked]>
Date: Sat, 6 Jan 2018 17:18:13 -0500

It should be possible to detect which firmwares the kernel attempts to
load and automatically download only the ones you need. IMO this would
be a better approach. Couple of issues:

- How would this interact with udev, if present
- Needs to be possible to download some firmware offline
- Should include all firmware on the ISO
- Should detect and pre-install relevant firmware during alpine-setup

--
Drew DeVault

---

---

From: "A. Wilcox" <[email-masked]>
Date: Sat, 6 Jan 2018 17:16:53 -0600

On 01/06/18 16:18, Drew DeVault wrote:
> load and automatically download only the ones you need.

There is a userspace firmware loader helper option in the kernel I think
(or maybe that was for modules?), but I don't think it'd be good to hang
up the boot process to download relevant firmware files.  Also, you have
to think about people who do not want firmware.  For example, I have one
system with a Broadcom WLAN chip but no other hardware requiring
firmware, so I just use it wired and run it fully libre.

> IMO this would be a better approach. Couple of issues:
> 

"download" "offline".

I mean, you can have a cache of firmware packages installed, but that's
basically the same as just having the large linux-firmware package
installed.

> - Should include all firmware on the ISO

I don't know the legalities of redistributing linux-firmware (I believe
it is "closed source, but freely distributable").  But beyond that, it
would mean that the Alpine ISO is no longer free/libre software.  I
don't agree that this is a good idea.  Perhaps there could *be* a spin
with firmware on the CD for people who know they need firmware for the
network or whatever, but it shouldn't be the default IMO.

> - Should detect and pre-install relevant firmware during alpine-setup

This is a decent idea for such a 'firmware CD spin'.  I don't know if
there is a 'lsfw' utility or such that can see what firmware is loaded,
but if so, that'd be all that would really be necessary.

Best,
--arw

---

From: Drew DeVault <[email-masked]>
Date: Sat, 6 Jan 2018 18:20:36 -0500

> firmware, so I just use it wired and run it fully libre.

In theory this wouldn't slow down the boot process if you had the ones
you need cached (and you should 1000% have the ones you need cached). It
might not make sense to download them on the fly, maybe, just write down
the failures and add a command to sort it out.

> basically the same as just having the large linux-firmware package

Not a full cache, just of the firmware you need.

> network or whatever, but it shouldn't be the default IMO.

This is a good point.

> but if so, that'd be all that would really be necessary.

I don't think there is, but hooking into the userspace loader should
make it pretty trivial to do this.

---

---

From: William Pitcock <[email-masked]>
Date: Sun, 7 Jan 2018 13:15:21 -0600

Hello,

> make it pretty trivial to do this.

NAK.  Hooking into the userspace loader with something that can block
is a really bad idea.

William

---

---

From: Drew DeVault <[email-masked]>
Date: Sun, 7 Jan 2018 14:32:20 -0500

> NAK.  Hooking into the userspace loader with something that can block

It needn't necessarily block.

---

---

From: William Pitcock <[email-masked]>
Date: Sun, 7 Jan 2018 13:20:35 -0600

Hi,

> firmware, so I just use it wired and run it fully libre.

Using the userspace firmware loader in this way could expose race
condition issues in poorly written drivers.

> network or whatever, but it shouldn't be the default IMO.

We already include all firmware on the ISO.  This is not likely to
change any time soon.

> but if so, that'd be all that would really be necessary.

I do like this idea.

William

---

---

From: William Pitcock <[email-masked]>
Date: Sun, 7 Jan 2018 13:21:14 -0600

Hi,

> What do you think, would it make sense to submit a PR for that?

Yes, please do it.

William

---

---

From: Oliver Smith <[email-masked]>
Date: Mon, 08 Jan 2018 16:51:00 +0000

>> What do you think, would it make sense to submit a PR for that?

Done: https://github.com/alpinelinux/aports/pull/3037

---

---

From: Timo Teras <[email-masked]>
Date: Tue, 9 Jan 2018 10:24:54 +0200

On Mon, 08 Jan 2018 16:51:00 +0000
Oliver Smith <[email-masked]> wrote:

> >> there's the >250 MB (install size) linux-firmware package, of
> >> which usually only few files are needed for one machine. For other
> >> subpackges to the mobile device, which we actually need (thus
> >> making the total image size a lot smaller).
> >>
> >> categorized in subfolders. So we could make one subpackage for
> >> each subfolder. Examples:
> >>
> >> Files directly in /lib/firmware could go into
> >> "linux-firmware-other", and the "linux-firmware" package could
> >> depend on all its subpackages for compatibility.
> >>

Thanks for working on this!

Few additional comments:

Should we now also change linux-grsec and linux-vanilla to not depend
on linux-firmware? Because they still pull in the meta-package pulling
in everything. Should the modules be split accordingly? Or should we
pull in some core firmwares only automatically?

Btw. mkinitfs properly picks up the firmwares for initramfs disks based
on kernel modules on initramfs. See:
https://git.alpinelinux.org/cgit/mkinitfs/tree/mkinitfs.in#n127

I earlier had the idea to do similar pruning for linux-firmware package
as it contains overlapping versions of firmwares, and firmwares for
modules we don't even build. However, this would require us to track the
firmwares requires for all of the kernel builds we do (all arches; all
configs). So it was not so trivial to implement.

And finally a minor nitpick: now have linux-firmware-RTL8192E;
generally we don't use uppercase in package names.

But thanks again for working on this. I have had on my todo list for a
while and good to see you're getting it done :)

Timo

---

---

From: Oliver Smith <[email-masked]>
Date: Tue, 09 Jan 2018 16:04:00 +0000

> pull in some core firmwares only automatically?

I think it makes sense for the linux-* packages not to depend on the
whole linux-firmware anymore, so people can really install only what they
need. As Jirutka wrote on GitHub, it's probably a good idea to show a
warning in post-upgrade, that most users need to install the subpackages
(or simply the whole linux-firmware package) then.

Regarding the modules I don't know if it's feasible to split them as well.

> generally we don't use uppercase in package names.

Thanks for noticing, I've made a follow-up PR that fixes this:
https://github.com/alpinelinux/aports/pull/3046

> while and good to see you're getting it done :)

You're welcome :)

---

---

From: William Pitcock <[email-masked]>
Date: Tue, 9 Jan 2018 16:16:31 -0600

Hi,

> (or simply the whole linux-firmware package) then.

I prefer to wait on removing the linux-firmware dependency until after
3.8 if possible.

> Regarding the modules I don't know if it's feasible to split them as well.

It should be possible to split by which module depends on what, but it
will be messy and require a significant rework of the package.
We could use modinfo to dump the firmware data for each module,
perhaps on kernel package build, and save that as a file somewhere
useful, perhaps in /lib/modules/$VERSION/firmwares.yaml or similar.

William

---