Hey alpine-devel,
there's the >250 MB (install size) linux-firmware package, of which usually only few files are needed for one machine.
For other packages, Alpine already splits off -dev and even -doc to save as much space as possible, so it would make sense to me if we started splitting up linux-firmware as well.
Our use case in Alpine Linux based postmarketOS is, that we could only flash the firmware subpackges to the mobile device, which we actually need (thus making the total image size a lot smaller).
The package provides files in /lib/firmware and is already categorized in subfolders. So we could make one subpackage for each subfolder. Examples:
- linux-firmware-liquido (23.7 MB)
- linux-firmware-netrome (22.0 MB)
- linux-firmware-brcm (15.1 MB)
- linux-firmware-amdgpu (13.9 MB)
- linux-firmware-intel (12.8 MB)
- linux-firmware-ti-connectivity (8.6 MB)
- linux-firmware-bnx2x (8.6 MB)
- linux-firmware-mrvl (7.7 MB)
- linux-firmware-radeon (7.4 MB)
Files directly in /lib/firmware could go into "linux-firmware-other", and the "linux-firmware" package could depend on all its subpackages for compatibility.
What do you think, would it make sense to submit a PR for that?
Thanks,
Oliver Smith
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On 01/06/18 16:13, Oliver Smith wrote:
> Hey alpine-devel,> > there's the >250 MB (install size) linux-firmware package, of which> usually only few files are needed for one machine. For other> packages, Alpine already splits off -dev and even -doc to save as> much space as possible, so it would make sense to me if we started> splitting up linux-firmware as well. Our use case in Alpine Linux> based postmarketOS is, that we could only flash the firmware> subpackges to the mobile device, which we actually need (thus making> the total image size a lot smaller).> > The package provides files in /lib/firmware and is already> categorized in subfolders. So we could make one subpackage for each> subfolder.> > Files directly in /lib/firmware could go into "linux-firmware-other",> and the "linux-firmware" package could depend on all its subpackages> for compatibility.> > What do you think, would it make sense to submit a PR for that?
All my +1.
The size is so large and the download time is so long that I've taken to
just scp'ing around the /lib/firmware files I need on different machines
from a 'master' machine that has the full package installed.
I've thought about suggesting this before but I wasn't exactly sure the
best naming scheme (should it be amdgpu-firmware or etc), but your
proposal is actually very clean. And it keeps compatibility.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
On 01/06/18 16:18, Drew DeVault wrote:
> It should be possible to detect which firmwares the kernel attempts to> load and automatically download only the ones you need.
There is a userspace firmware loader helper option in the kernel I think
(or maybe that was for modules?), but I don't think it'd be good to hang
up the boot process to download relevant firmware files. Also, you have
to think about people who do not want firmware. For example, I have one
system with a Broadcom WLAN chip but no other hardware requiring
firmware, so I just use it wired and run it fully libre.
> IMO this would be a better approach. Couple of issues:> > - How would this interact with udev, if present> - Needs to be possible to download some firmware offline
"download" "offline".
I mean, you can have a cache of firmware packages installed, but that's
basically the same as just having the large linux-firmware package
installed.
> - Should include all firmware on the ISO
I don't know the legalities of redistributing linux-firmware (I believe
it is "closed source, but freely distributable"). But beyond that, it
would mean that the Alpine ISO is no longer free/libre software. I
don't agree that this is a good idea. Perhaps there could *be* a spin
with firmware on the CD for people who know they need firmware for the
network or whatever, but it shouldn't be the default IMO.
> - Should detect and pre-install relevant firmware during alpine-setup
This is a decent idea for such a 'firmware CD spin'. I don't know if
there is a 'lsfw' utility or such that can see what firmware is loaded,
but if so, that'd be all that would really be necessary.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org
It should be possible to detect which firmwares the kernel attempts to
load and automatically download only the ones you need. IMO this would
be a better approach. Couple of issues:
- How would this interact with udev, if present
- Needs to be possible to download some firmware offline
- Should include all firmware on the ISO
- Should detect and pre-install relevant firmware during alpine-setup
--
Drew DeVault
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On 2018-01-06 5:16 PM, A. Wilcox wrote:
> There is a userspace firmware loader helper option in the kernel I think> (or maybe that was for modules?), but I don't think it'd be good to hang> up the boot process to download relevant firmware files. Also, you have> to think about people who do not want firmware. For example, I have one> system with a Broadcom WLAN chip but no other hardware requiring> firmware, so I just use it wired and run it fully libre.
In theory this wouldn't slow down the boot process if you had the ones
you need cached (and you should 1000% have the ones you need cached). It
might not make sense to download them on the fly, maybe, just write down
the failures and add a command to sort it out.
> "download" "offline".> > I mean, you can have a cache of firmware packages installed, but that's> basically the same as just having the large linux-firmware package> installed.
Not a full cache, just of the firmware you need.
> I don't know the legalities of redistributing linux-firmware (I believe> it is "closed source, but freely distributable"). But beyond that, it> would mean that the Alpine ISO is no longer free/libre software. I> don't agree that this is a good idea. Perhaps there could *be* a spin> with firmware on the CD for people who know they need firmware for the> network or whatever, but it shouldn't be the default IMO.
This is a good point.
> This is a decent idea for such a 'firmware CD spin'. I don't know if> there is a 'lsfw' utility or such that can see what firmware is loaded,> but if so, that'd be all that would really be necessary.
I don't think there is, but hooking into the userspace loader should
make it pretty trivial to do this.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hello,
On Sat, Jan 6, 2018 at 5:20 PM, Drew DeVault <sir@cmpwn.com> wrote:
> On 2018-01-06 5:16 PM, A. Wilcox wrote:>> There is a userspace firmware loader helper option in the kernel I think>> (or maybe that was for modules?), but I don't think it'd be good to hang>> up the boot process to download relevant firmware files. Also, you have>> to think about people who do not want firmware. For example, I have one>> system with a Broadcom WLAN chip but no other hardware requiring>> firmware, so I just use it wired and run it fully libre.>> In theory this wouldn't slow down the boot process if you had the ones> you need cached (and you should 1000% have the ones you need cached). It> might not make sense to download them on the fly, maybe, just write down> the failures and add a command to sort it out.>>> "download" "offline".>>>> I mean, you can have a cache of firmware packages installed, but that's>> basically the same as just having the large linux-firmware package>> installed.>> Not a full cache, just of the firmware you need.>>> I don't know the legalities of redistributing linux-firmware (I believe>> it is "closed source, but freely distributable"). But beyond that, it>> would mean that the Alpine ISO is no longer free/libre software. I>> don't agree that this is a good idea. Perhaps there could *be* a spin>> with firmware on the CD for people who know they need firmware for the>> network or whatever, but it shouldn't be the default IMO.>> This is a good point.>>> This is a decent idea for such a 'firmware CD spin'. I don't know if>> there is a 'lsfw' utility or such that can see what firmware is loaded,>> but if so, that'd be all that would really be necessary.>> I don't think there is, but hooking into the userspace loader should> make it pretty trivial to do this.
NAK. Hooking into the userspace loader with something that can block
is a really bad idea.
William
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hi,
On Sat, Jan 6, 2018 at 5:16 PM, A. Wilcox <awilfox@adelielinux.org> wrote:
> On 01/06/18 16:18, Drew DeVault wrote:>> It should be possible to detect which firmwares the kernel attempts to>> load and automatically download only the ones you need.>>> There is a userspace firmware loader helper option in the kernel I think> (or maybe that was for modules?), but I don't think it'd be good to hang> up the boot process to download relevant firmware files. Also, you have> to think about people who do not want firmware. For example, I have one> system with a Broadcom WLAN chip but no other hardware requiring> firmware, so I just use it wired and run it fully libre.
Using the userspace firmware loader in this way could expose race
condition issues in poorly written drivers.
>> IMO this would be a better approach. Couple of issues:>>>> - How would this interact with udev, if present>> - Needs to be possible to download some firmware offline>>> "download" "offline".>> I mean, you can have a cache of firmware packages installed, but that's> basically the same as just having the large linux-firmware package> installed.>>>> - Should include all firmware on the ISO>>> I don't know the legalities of redistributing linux-firmware (I believe> it is "closed source, but freely distributable"). But beyond that, it> would mean that the Alpine ISO is no longer free/libre software. I> don't agree that this is a good idea. Perhaps there could *be* a spin> with firmware on the CD for people who know they need firmware for the> network or whatever, but it shouldn't be the default IMO.
We already include all firmware on the ISO. This is not likely to
change any time soon.
>> - Should detect and pre-install relevant firmware during alpine-setup>> This is a decent idea for such a 'firmware CD spin'. I don't know if> there is a 'lsfw' utility or such that can see what firmware is loaded,> but if so, that'd be all that would really be necessary.
I do like this idea.
William
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hi,
On Sat, Jan 6, 2018 at 4:13 PM, Oliver Smith
<ollieparanoid@bitmessage.ch> wrote:
> Hey alpine-devel,>> there's the >250 MB (install size) linux-firmware package, of which usually only few files are needed for one machine.> For other packages, Alpine already splits off -dev and even -doc to save as much space as possible, so it would make sense to me if we started splitting up linux-firmware as well.> Our use case in Alpine Linux based postmarketOS is, that we could only flash the firmware subpackges to the mobile device, which we actually need (thus making the total image size a lot smaller).>> The package provides files in /lib/firmware and is already categorized in subfolders. So we could make one subpackage for each subfolder. Examples:>> - linux-firmware-liquido (23.7 MB)> - linux-firmware-netrome (22.0 MB)> - linux-firmware-brcm (15.1 MB)> - linux-firmware-amdgpu (13.9 MB)> - linux-firmware-intel (12.8 MB)> - linux-firmware-ti-connectivity (8.6 MB)> - linux-firmware-bnx2x (8.6 MB)> - linux-firmware-mrvl (7.7 MB)> - linux-firmware-radeon (7.4 MB)>> Files directly in /lib/firmware could go into "linux-firmware-other", and the "linux-firmware" package could depend on all its subpackages for compatibility.>> What do you think, would it make sense to submit a PR for that?
Yes, please do it.
William
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
William Pitcock:
> Hi,> > On Sat, Jan 6, 2018 at 4:13 PM, Oliver Smith> <ollieparanoid@bitmessage.ch> wrote:>> Hey alpine-devel,>>>> there's the >250 MB (install size) linux-firmware package, of which usually only few files are needed for one machine.>> For other packages, Alpine already splits off -dev and even -doc to save as much space as possible, so it would make sense to me if we started splitting up linux-firmware as well.>> Our use case in Alpine Linux based postmarketOS is, that we could only flash the firmware subpackges to the mobile device, which we actually need (thus making the total image size a lot smaller).>>>> The package provides files in /lib/firmware and is already categorized in subfolders. So we could make one subpackage for each subfolder. Examples:>>>> - linux-firmware-liquido (23.7 MB)>> - linux-firmware-netrome (22.0 MB)>> - linux-firmware-brcm (15.1 MB)>> - linux-firmware-amdgpu (13.9 MB)>> - linux-firmware-intel (12.8 MB)>> - linux-firmware-ti-connectivity (8.6 MB)>> - linux-firmware-bnx2x (8.6 MB)>> - linux-firmware-mrvl (7.7 MB)>> - linux-firmware-radeon (7.4 MB)>>>> Files directly in /lib/firmware could go into "linux-firmware-other", and the "linux-firmware" package could depend on all its subpackages for compatibility.>>>> What do you think, would it make sense to submit a PR for that?> > Yes, please do it.> > William>
Done: https://github.com/alpinelinux/aports/pull/3037> > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> ---> >
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Mon, 08 Jan 2018 16:51:00 +0000
Oliver Smith <ollieparanoid@bitmessage.ch> wrote:
> William Pitcock:> > On Sat, Jan 6, 2018 at 4:13 PM, Oliver Smith> > <ollieparanoid@bitmessage.ch> wrote: > >> Hey alpine-devel,> >>> >> there's the >250 MB (install size) linux-firmware package, of> >> which usually only few files are needed for one machine. For other> >> packages, Alpine already splits off -dev and even -doc to save as> >> much space as possible, so it would make sense to me if we started> >> splitting up linux-firmware as well. Our use case in Alpine Linux> >> based postmarketOS is, that we could only flash the firmware> >> subpackges to the mobile device, which we actually need (thus> >> making the total image size a lot smaller).> >>> >> The package provides files in /lib/firmware and is already> >> categorized in subfolders. So we could make one subpackage for> >> each subfolder. Examples:> >>> >> - linux-firmware-liquido (23.7 MB)> >> - linux-firmware-netrome (22.0 MB)> >> - linux-firmware-brcm (15.1 MB)> >> - linux-firmware-amdgpu (13.9 MB)> >> - linux-firmware-intel (12.8 MB)> >> - linux-firmware-ti-connectivity (8.6 MB)> >> - linux-firmware-bnx2x (8.6 MB)> >> - linux-firmware-mrvl (7.7 MB)> >> - linux-firmware-radeon (7.4 MB)> >>> >> Files directly in /lib/firmware could go into> >> "linux-firmware-other", and the "linux-firmware" package could> >> depend on all its subpackages for compatibility.> >>> >> What do you think, would it make sense to submit a PR for that? > > > > Yes, please do it.> > Done: https://github.com/alpinelinux/aports/pull/3037
Thanks for working on this!
Few additional comments:
Should we now also change linux-grsec and linux-vanilla to not depend
on linux-firmware? Because they still pull in the meta-package pulling
in everything. Should the modules be split accordingly? Or should we
pull in some core firmwares only automatically?
Btw. mkinitfs properly picks up the firmwares for initramfs disks based
on kernel modules on initramfs. See:
https://git.alpinelinux.org/cgit/mkinitfs/tree/mkinitfs.in#n127
I earlier had the idea to do similar pruning for linux-firmware package
as it contains overlapping versions of firmwares, and firmwares for
modules we don't even build. However, this would require us to track the
firmwares requires for all of the kernel builds we do (all arches; all
configs). So it was not so trivial to implement.
And finally a minor nitpick: now have linux-firmware-RTL8192E;
generally we don't use uppercase in package names.
But thanks again for working on this. I have had on my todo list for a
while and good to see you're getting it done :)
Timo
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Timo Teras:
> On Mon, 08 Jan 2018 16:51:00 +0000> Oliver Smith <ollieparanoid@bitmessage.ch> wrote:> >> William Pitcock:>>> On Sat, Jan 6, 2018 at 4:13 PM, Oliver Smith>>> <ollieparanoid@bitmessage.ch> wrote: >>>> Hey alpine-devel,>>>>>>>> there's the >250 MB (install size) linux-firmware package, of>>>> which usually only few files are needed for one machine. For other>>>> packages, Alpine already splits off -dev and even -doc to save as>>>> much space as possible, so it would make sense to me if we started>>>> splitting up linux-firmware as well. Our use case in Alpine Linux>>>> based postmarketOS is, that we could only flash the firmware>>>> subpackges to the mobile device, which we actually need (thus>>>> making the total image size a lot smaller).>>>>>>>> The package provides files in /lib/firmware and is already>>>> categorized in subfolders. So we could make one subpackage for>>>> each subfolder. Examples:>>>>>>>> - linux-firmware-liquido (23.7 MB)>>>> - linux-firmware-netrome (22.0 MB)>>>> - linux-firmware-brcm (15.1 MB)>>>> - linux-firmware-amdgpu (13.9 MB)>>>> - linux-firmware-intel (12.8 MB)>>>> - linux-firmware-ti-connectivity (8.6 MB)>>>> - linux-firmware-bnx2x (8.6 MB)>>>> - linux-firmware-mrvl (7.7 MB)>>>> - linux-firmware-radeon (7.4 MB)>>>>>>>> Files directly in /lib/firmware could go into>>>> "linux-firmware-other", and the "linux-firmware" package could>>>> depend on all its subpackages for compatibility.>>>>>>>> What do you think, would it make sense to submit a PR for that? >>>>>> Yes, please do it.>>>> Done: https://github.com/alpinelinux/aports/pull/3037> > Thanks for working on this!> > Few additional comments:> > Should we now also change linux-grsec and linux-vanilla to not depend> on linux-firmware? Because they still pull in the meta-package pulling> in everything. Should the modules be split accordingly? Or should we> pull in some core firmwares only automatically?
I think it makes sense for the linux-* packages not to depend on the
whole linux-firmware anymore, so people can really install only what they
need. As Jirutka wrote on GitHub, it's probably a good idea to show a
warning in post-upgrade, that most users need to install the subpackages
(or simply the whole linux-firmware package) then.
Regarding the modules I don't know if it's feasible to split them as well.
> > Btw. mkinitfs properly picks up the firmwares for initramfs disks based> on kernel modules on initramfs. See:> https://git.alpinelinux.org/cgit/mkinitfs/tree/mkinitfs.in#n127> > I earlier had the idea to do similar pruning for linux-firmware package> as it contains overlapping versions of firmwares, and firmwares for> modules we don't even build. However, this would require us to track the> firmwares requires for all of the kernel builds we do (all arches; all> configs). So it was not so trivial to implement.> > And finally a minor nitpick: now have linux-firmware-RTL8192E;> generally we don't use uppercase in package names.
Thanks for noticing, I've made a follow-up PR that fixes this:
https://github.com/alpinelinux/aports/pull/3046> > But thanks again for working on this. I have had on my todo list for a> while and good to see you're getting it done :)>
You're welcome :)
> Timo> > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> ---> >
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hi,
On Tue, Jan 9, 2018 at 10:04 AM, Oliver Smith
<ollieparanoid@bitmessage.ch> wrote:
> I think it makes sense for the linux-* packages not to depend on the> whole linux-firmware anymore, so people can really install only what they> need. As Jirutka wrote on GitHub, it's probably a good idea to show a> warning in post-upgrade, that most users need to install the subpackages> (or simply the whole linux-firmware package) then.
I prefer to wait on removing the linux-firmware dependency until after
3.8 if possible.
> Regarding the modules I don't know if it's feasible to split them as well.
It should be possible to split by which module depends on what, but it
will be messy and require a significant rework of the package.
We could use modinfo to dump the firmware data for each module,
perhaps on kernel package build, and save that as a file somewhere
useful, perhaps in /lib/modules/$VERSION/firmwares.yaml or similar.
William
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---