Subject: [alpine-aports] [RFC] Splitting linux-firmware

From: Isaac Dunham <[email-masked]>
Date: Sat, 2 Apr 2016 22:21:18 -0700

Hello,
I recently checked the size of installed packages on my system, and found
that linux-firmware was the largest (~120 megabytes).
I need the Realtek firmware, and perhaps could use the i915 firmware.
So I figured it would be a significant cutdown in installed size,
as well as a major reduction in downloads, if it were possible to just
select firmware from a couple of manufacturers.

While I was in the process of doing this, I noticed a few smaller issues:
-the non-firmware files that the APKBUILD tries to delete are actually
located in /lib/firmware, not /usr/lib/firmware
-the source for carl9170 firmware is included in the final package

Here's a tentative patch that fixes those issues and tries to split
things up.

Does this look like a good idea?
Should any further splitting be done? 
For example:
-splitting iwlwifi firmware off because it's the bulk of the intel
firmware, but not the bulk of the users;
-splitting qlogic, 3com, and chelsio firmware from -misc

Thanks,
Isaac Dunham
>From 0059b36e8677ca5a0567d68a2c7790566a099843 Mon Sep 17 00:00:00 2001
From: Isaac Dunham <[email-masked]>
Date: Sat, 2 Apr 2016 21:51:48 -0700
Subject: [PATCH] main/linux-firmware: split package

linux-firmware is about 120 megabytes, close to twice the size
of the kernel; most users only need about 1-5 files (gpu,
wireless, ethernet, bluetooth, cpu).
By splitting this package into smaller parts, it's possible to
cut the footprint down significantly; I would end up needing
linux-firmware-realtek and perhaps linux-firmware-intel,
for ~40 megabytes.
There is a case to be made for splitting -iwl from -intel:
iwl is roughly 23 megabytes, the bulk of -intel.

Smaller manfacturers didn't get split out, and form -misc.
---
 main/linux-firmware/APKBUILD | 87 ++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 83 insertions(+), 4 deletions(-)

diff --git a/main/linux-firmware/APKBUILD b/main/linux-firmware/APKBUILD
index 0a9bb49..a9b68fb 100644
--- a/main/linux-firmware/APKBUILD
+++ b/main/linux-firmware/APKBUILD
@@ -2,17 +2,19 @@
 # Maintainer: Natanael Copa <[email-masked]>
 pkgname=linux-firmware
 pkgver=20151021
-pkgrel=0
+pkgrel=1
 pkgdesc="firmware files for linux"
 #url="http://git.kernel.org/?p=linux/kernel/git/dwmw2/linux-firmware.git;a=summary"
 url="http://git.kernel.org/?p=linux/kernel/git/firmware/linux-firmware.git;a=summary"
 
 arch="noarch"
 license="GPL"
-depends=
+subpackages="${pkgname}-intel ${pkgname}-amd ${pkgname}-atheros
+	${pkgname}-broadcom ${pkgname}-nvidia ${pkgname}-marvell
+	${pkgname}-realtek ${pkgname}-ti ${pkgname}-misc"
+depends="${subpackages}"
 makedepends=
 install=""
-subpackages=
 replaces="linux-grsec linux-vserver"
 
 source="http://dev.alpinelinux.org/archive/$pkgname/$pkgname-${pkgver}.tar.gz"
@@ -49,7 +51,84 @@ build() {
 package() {
 	cd "${_builddir}"
 	make DESTDIR="${pkgdir}" FIRMWAREDIR="/lib/firmware" install || return 1
-	rm -f "${pkgdir}/usr/lib/firmware/{Makefile,README,configure,GPL-3}"
+	rm -f "${pkgdir}/lib/firmware/{Makefile,README,configure,GPL-3}" || return 1
+	# carl9170 firmware is included as prebuilt and source
+	# this is the source code.
+	rm -rf "${pkgdir}/lib/firmware/carl9170fw/" || return 1
+}
+
+intel() {
+	pkgdesc="firmware for Intel hardware"
+	mkdir -p "$subpkgdir/lib/firmware" || return 1
+	mv "$pkgdir"/lib/firmware/iwlwifi* "$subpkgdir/lib/firmware/" || return 1
+	for S in e100 intel i915; do
+		mv "$pkgdir"/lib/firmware/"$S" "$subpkgdir/lib/firmware/" || return 1
+	done
+}
+
+amd() {
+	pkgdesc="firmware for AMD/ATI hardware"
+	mkdir -p "$subpkgdir/lib/firmware" || return 1
+	for S in amd-ucode amdgpu radeon r128; do
+		mv "$pkgdir"/lib/firmware/"$S" "$subpkgdir"/lib/firmware || return 1
+	done
+}
+
+atheros() {
+	pkgdesc="firmware for Atheros and Qualcomm/Atheros hardware"
+	mkdir -p "$subpkgdir/lib/firmware" || return 1
+	for S in ar3k ath10k ath9k_htc ath6k qca; do
+		mv "$pkgdir"/lib/firmware/"$S" "$subpkgdir"/lib/firmware || return 1
+	done
+	for F in "$pkgdir"/lib/firmware/ath*.* \
+		"$pkgdir"/lib/firmware/ar*.* \
+		"$pkgdir"/lib/firmware/carl9170*.* 
+			do mv "$F" "$subpkgdir"/lib/firmware/ || return 1
+	done
+}
+
+broadcom() {
+	pkgdesc="firmware for Broadcom hardware"
+	mkdir -p "$subpkgdir/lib/firmware" || return 1
+	cd "$pkgdir"/lib/firmware
+	for S in brcm bnx2* tigon; do
+		mv $S "$subpkgdir"/lib/firmware/ || return 1
+	done
+}
+
+nvidia() {
+	pkgdesc="firmware for NVIDIA hardware"
+	mkdir -p "$subpkgdir"/lib/firmware || return 1
+	mv "$pkgdir"/lib/firmware/nvidia "$subpkgdir"/lib/firmware || return 1
+}
+
+marvell() {
+	pkgdesc="firmware for Marvell hardware (including Libertas wireless cards)"
+	mkdir -p "$subpkgdir"/lib/firmware || return 1
+	cd "$pkgdir"/lib/firmware || return 1
+	mv mwl8k mwlwifi mrvl libertas "$subpkgdir"/lib/firmware/ || return 1
+}
+
+realtek() {
+	pkgdesc="firmware for Realtek hardware (networking and bluetooth)"
+	mkdir -p "$subpkgdir/lib/firmware" || return 1
+	cd "$pkgdir"/lib/firmware || return 1
+	for S in RTL8192E rtl_bt rtl_nic rtlwifi; do
+		mv $S "$subpkgdir"/lib/firmware/ || return 1
+	done
+}
+
+ti() {
+	pkgdesc="firmware for TI hardware (usb-serial adapters, wireless, SoCs)"
+	mkdir -p "$subpkgdir"/lib/firmware || return 1
+	cd "$pkgdir"/lib/firmware || return 1
+	mv ti-* ti_*.fw "$subpkgdir"/lib/firmware || return 1
+}
+
+misc() {
+	pkgdesc="firmware for hardware from assorted manufacturers"
+	mkdir -p "$subpkgdir"/lib || return 1
+	mv "$pkgdir"/lib/firmware "$subpkgdir"/lib/
 }
 
 md5sums="2960b7241de622362f1ef93592ab16d4  linux-firmware-20151021.tar.gz"
-- 
2.7.4

---

From: Timo Teras <[email-masked]>
Date: Sun, 3 Apr 2016 16:25:18 +0300

On Sat, 2 Apr 2016 22:21:18 -0700
Isaac Dunham <[email-masked]> wrote:

> I recently checked the size of installed packages on my system, and
> found that linux-firmware was the largest (~120 megabytes).
> While I was in the process of doing this, I noticed a few smaller
> issues: -the non-firmware files that the APKBUILD tries to delete are
> actually located in /lib/firmware, not /usr/lib/firmware

Before doing this, a first step would be to delete the unneeded
firmwares.

We already do this for modloop. But to do it for the package, would
need a list of files needed to be maintained. I believe about half of
the package can be just deleted based on our module configuration and
specific kernel versions.

See http://git.alpinelinux.org/cgit/alpine-iso/tree/Makefile#n92 for
how to get info on what firmware is required by the kernel modules.

If the package is still too large after this, doing splitting might be
an idea.

Thanks,
Timo

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Isaac Dunham <[email-masked]>
Date: Sun, 3 Apr 2016 13:55:02 -0700

> how to get info on what firmware is required by the kernel modules.

Before doing that, there's a question of whether we're going to support
'noarch' "properly" (ie, with a single build/package upload for all
arches).
If so, it would be necessary to compile a single list of files for 
all arches.

Checking it against linux-firmware I get results along the lines of
- one third of the firmware is 'missing' (ie, not in linux-firmware)
A large chunk of this is in alsa-firmware and other packages.

- the remainder runs just over 40 megabytes

- there's lots of firmware that seems to be 'unneeded' even though the
relevant drivers are enabled (eg, only 2 files from brcm/ of 24 are
needed), which makes me wonder if something's wrong with modinfo...

> If the package is still too large after this, doing splitting might be

*If* the results are correct, it wouldn't be *needed*.

Thanks,
Isaac Dunham

---

---

From: Timo Teras <[email-masked]>
Date: Mon, 4 Apr 2016 08:43:05 +0300

On Sun, 3 Apr 2016 13:55:02 -0700
Isaac Dunham <[email-masked]> wrote:

> > > I recently checked the size of installed packages on my system,
> > > and found that linux-firmware was the largest (~120 megabytes).
> > > I need the Realtek firmware, and perhaps could use the i915
> > > firmware. So I figured it would be a significant cutdown in
> > > installed size, as well as a major reduction in downloads, if it
> > > were possible to just select firmware from a couple of
> > > manufacturers.
> > > 
> > > issues: -the non-firmware files that the APKBUILD tries to delete
> > > are actually located in /lib/firmware, not /usr/lib/firmware
> > > Here's a tentative patch that fixes those issues and tries to
> > > split things up.
> > > 
> > need a list of files needed to be maintained. I believe about half
> > of the package can be just deleted based on our module
> > configuration and specific kernel versions.
> > 
> Before doing that, there's a question of whether we're going to
> support 'noarch' "properly" (ie, with a single build/package upload
> for all arches).

We could make linux-firmware 'all' package and keep only the modules
relevant for that architectures kernel. Quite a bit of the hardware
is architecture specific. Well technically it would work, but nobody
really uses specific combinations and they are turned off in the
relevant configs.

> needed), which makes me wonder if something's wrong with modinfo...

I believe it's just a result of having firmware for all supported
kernel versions in the linux-firmware package. E.g. many of the intel
wifi drivers duplicates. The firmware version used really depends on the
kernel version you run. Often the module version and the
required firmware version go hand in hand.

Each module can (and do) list several firmware file names -- all
potential names they can request. So the list of modules names should
be accurate.

> > If the package is still too large after this, doing splitting might
> > be an idea.  
> 

There was request also for some other firmwares not in linux-firmware.
Such as the rpi3 wifi firmware. So we might need to split the firmware
package in any case.

Thanks,
Timo

---