Received: from mx.msys.ch (hq.msys.ch [46.175.8.8]) by nld3-dev1.alpinelinux.org (Postfix) with ESMTPS id 2B2E9782E15 for <~alpine/apk-tools@lists.alpinelinux.org>; Sun, 8 May 2022 07:22:26 +0000 (UTC) Received: from authenticated-user (PRIMARY_HOSTNAME [PUBLIC_IP]) by mx.msys.ch (Postfix) with ESMTPA id 01DF3D004A000 for <~alpine/apk-tools@lists.alpinelinux.org>; Sun, 8 May 2022 07:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=msys.ch; s=dkim; t=1651994545; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=jFlG8xihyJorwFyDuwM5Jhix5NylealpTlbnwcNCetM=; b=UxjTspGM/UvwUbXOG23DWGJiWO8pv81SBrTjhQk9P+JJSvUoun4QqshYds9pQ6dU9fiMdA 9AsIPIANogj0PHmTtd791R0VaGCfWV+N3J0O4gbvYDHyMIyAwxLxybVwUOV55lxGdSTPe2 UAlC0CHNttf2vZtpX6ivKKt5Bd9r4a0= From: Marc Balmer Content-Type: multipart/alternative; boundary="Apple-Mail=_7FFF0C3C-20B1-4572-A541-4C79FD820974" Mime-Version: 1.0 Subject: Creating an .apk package from local files, without downloading and building sources Message-Id: Date: Sun, 8 May 2022 09:22:23 +0200 To: ~alpine/apk-tools@lists.alpinelinux.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=msys.ch; s=dkim; t=1651994546; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=jFlG8xihyJorwFyDuwM5Jhix5NylealpTlbnwcNCetM=; b=iGL097QD/kn8UCkYZ1cLvCwX1a5xQRw7FT4nBi4Bn89DHjI2hNctyHRnao8FdSSYKS++yJ jAn7LWhgRFT/5uSlqJWyGWOiqK9BJQ5wytD59hgWkZaEcg+5BiJiUjSonbtmVhoh25Z2iS xF69i6Uwy9aRlzU9KghPGnnaPkOvKEg= ARC-Seal: i=1; s=dkim; d=msys.ch; t=1651994546; a=rsa-sha256; cv=none; b=i2hTsG0onJFnXJnUi5dzR3kw9SplSHuZ4n1iDJu2K5LffJttUpP5m2q0Fc9XpJciHxUS4i i1FK+s12vTw1GhrVrCSLR5ipusc4dfl9XIRzfiAKS3s50rFD7tyOt06+PslA7m+nS6qnjo fEpXvSW8KlExysieTGuJ3Ak2Q4n+DCQ= ARC-Authentication-Results: i=1; mx.msys.ch; auth=pass smtp.auth=marc@msys.ch smtp.mailfrom=marc@msys.ch Authentication-Results: mx.msys.ch; auth=pass smtp.auth=marc@msys.ch smtp.mailfrom=marc@msys.ch X-Spamd-Bar: / --Apple-Mail=_7FFF0C3C-20B1-4572-A541-4C79FD820974 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii G'day! TL;DR: We are looking for a way to build an .apk package from locally installed = files, without going through the download and build stages that abuild = normally does. I.e without an APKBUILD file. A caveat: My first post to this list... I write this E-Mail using HTML = for better readability of the code. If that is a no-go here, please = tell me and I will resend as plain text. Our setting We have a large git repo with the source code for many libraries and = binaries, much of them open source. We build and package them for = various linux distributions and architectures. We use docker containers = to build for the various systems. We think that our binaries, packaged = for AlpineLinux, would be a good fit for container use. We thus did = setup a build container for Alpine Linux, which builds all of our stuff = just fine. We are now trying to create .apk packages using the already compiled = libraries and binaries. Our approach is to create a directory for each = package we want to build and to describe the package using a Makefile. = The packages Makefile then includes a Makefile fragment that drives the = actual package building. PACKAGE=3D lua PKGVER=3D 5.4.4-r1 DESCRIPTION=3D "Lua interpreter and compiler" PACKAGER=3D micro systems > URL=3D https://lua.msys.ch MAINTAINER=3D micro systems LICENSE=3D MIT FILES=3D usr/bin/lua \ usr/bin/luac MKDIR?=3D ../../../mk/ include $(MKDIR)alpine.package.mk The actual binaries are installed in the docker build container under = /tmp/alpine. We copy the files from there to a local staging directory = "staging" and build the package from there. The Makefile Fragment = alpine.package.mk looks like this: LICENSE?=3D custom PACKAGER?=3D unknown DESCRIPTION?=3D unknwon PACKAGER?=3D unknown URL?=3D unknown MAINTAINER?=3D unknown INSTSCRIPTS!=3D find . -name .p\*-\*install -exec basename {} \; DATE_EPOCH!=3D date -u "+%s" .DEFAULT_GOAL=3D package package: # Create the data part @-mkdir -p staging @tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x find staging -exec touch -h -d @${DATE_EPOCH} {} + @(cd staging; tar --xattrs -f - -c *) | abuild-tar --hash | \ gzip > data.tar.gz (cd staging; find . -print0) | LC_ALL=3DC sort -z | tar -C = staging --xattrs \ --xattrs-exclude=3Dsecurity.selinux \ --format=3Dposix \ = --pax-option=3Dexthdr.name=3D%d/PaxHeaders/%f,atime:=3D0,ctime:=3D0 \ --mtime=3D"@${DATE_EPOCH}" \ --no-recursion --null -T - \ -f - -c | abuild-tar --hash | gzip -n -9 = >data.tar.gz # Create .PKGINFO @echo "# Generated by alpine.package.mk" > .PKGINFO @echo -n "# " >> .PKGINFO @date >> .PKGINFO @echo "package =3D ${PACKAGE}" >> .PKGINFO @echo "pkgver =3D ${PKGVER}" >> .PKGINFO @echo "pkgdesc =3D ${DESCRIPTION}" >> .PKGINFO @echo "url =3D ${URL}" >> .PKGINFO @echo "builddate =3D ${DATE_EPOCH}" >> .PKGINFO @echo "commit =3D 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> = .PKGINFO @echo "packager =3D ${PACKAGER}" >> .PKGINFO @echo -n "size =3D " >> .PKGINFO @(cd staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> = .PKGINFO @echo "arch =3D x86_64" >> .PKGINFO @echo "origin =3D ${PACKAGE}" >> .PKGINFO @echo "maintainer =3D ${MAINTAINER}" >> .PKGINFO @echo "license =3D ${LICENSE}" >> .PKGINFO @for dep in ${DEPENDS}; do echo "depend =3D $${dep}" >> = .PKGINFO; done @echo "# automatically detected:" >> .PKGINFO @for f in ${FILES}; \ do \ find /tmp/alpine -name $${f} -type f -perm +111; \ echo -n "provides =3D cmd:" >> .PKGINFO; \ basename $${f} | tr -d '\n' >> .PKGINFO; \ echo "=3D${PKGVER}" >> .PKGINFO; \ done @echo -n "datahash =3D " >>.PKGINFO @sha256sum data.tar.gz | cut -d " " -f 1 >> .PKGINFO # Create the control part (index) tar --format=3Dposix \ = --pax-option=3Dexthdr.name=3D%d/PaxHeaders/%f,atime:=3D0,ctime:=3D0 \ --mtime=3D"@${DATE_EPOCH}" \ -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \ | gzip -n -9 > control.tar.gz #@tar --owner=3D0 --group=3D0 --numeric-owner -c .PKGINFO = ${INSTSCRIPTS} | \ # abuild-tar --cut | gzip > control.tar.gz @abuild-sign -q control.tar.gz # Produce the final apk @cat control.tar.gz data.tar.gz > ${PACKAGE}-${PKGVER}.apk @cat control.tar.gz data.tar.gz > = /var/apk/${PACKAGE}-${PKGVER}.apk -(cd /var/apk; apk index ${PACKAGE}-${PKGVER}.apk -o = APKINDEX.tar.gz) abuild-sign -q /var/apk/APKINDEX.tar.gz # Show info and clean up @cat .PKGINFO @rm -rf data.tar.gz control.tar.gz .PKGINFO staging Essentially, we try to run the packaging steps in the same way as abuild = does it (we used the abuild source code as a reference). Of course our approach is not as elegant as = abuild. Moreover, it does not work, the apk package being produced can not be installed. While the resulting .apk package looks like any other .apk package, and = the content looks correct, apk exits with exit code 99 when we try to add or index the package. apk add = prints "ERROR: Not supported." (Not supported seems to be the result from musl libc for an unknow error = code). Do you have any ideas what we are doing wrong? Thanks, Marc --Apple-Mail=_7FFF0C3C-20B1-4572-A541-4C79FD820974 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii G'day!

TL;DR:

We are looking for a way to build an .apk package from = locally installed files, without going through the download and build = stages that abuild normally does. I.e without an APKBUILD = file.

A = caveat:  My first post to this list... I write this E-Mail using = HTML for better readability of the code.  If that is a no-go here, = please tell me and I will resend as plain text.


Our = setting

We = have a large git repo with the source code for many libraries and = binaries, much of them open source.  We build and package them for = various linux distributions and architectures.  We use docker = containers to build for the various systems.   We think that our = binaries, packaged for AlpineLinux, would be a good fit for container = use.  We thus did setup a build container for Alpine Linux, which = builds all of our stuff just fine.

We are now trying to create .apk = packages using the already compiled libraries and binaries.  Our = approach is to create a directory for each package we want to build and = to describe the package using a Makefile.  The packages Makefile = then includes a Makefile fragment that drives the actual package = building.

PACKAGE=3D = lua
PKGVER=3D = 5.4.4-r1
DESCRIPTION=3D "Lua interpreter and = compiler"
PACKAGER=3D micro = systems <info@msys.ch>
MAINTAINER=3D= micro systems <info@msys.ch>
LICENSE=3D = MIT

FILES=3D usr/bin/lua \
usr/bin/luac

MKDIR?=3D = ../../../mk/

include $(MKDIR)alpine.package.mk

The actual binaries are = installed in the docker build container under /tmp/alpine.  We copy = the files from there to a local staging directory "staging" and build = the package from there. The Makefile Fragment = alpine.package.mk looks like this:

LICENSE?=3D = custom
PACKAGER?=3D unknown
DESCRIPTION?=3D unknwon
PACKAGER?=3D = unknown
URL?=3D = unknown
MAINTAINER?=3D unknown

INSTSCRIPTS!=3D= find . -name .p\*-\*install -exec basename {} \;
DATE_EPOCH!=3D date -u "+%s"

.DEFAULT_GOAL=3D package

package:
# Create the data = part
@-mkdir -p staging
@tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x

find = staging -exec touch -h -d @${DATE_EPOCH} {} +

@(cd staging; tar --xattrs = -f - -c *) | abuild-tar --hash | \
= gzip > data.tar.gz

(cd = staging; find . -print0) | LC_ALL=3DC sort -z | tar -C staging --xattrs = \
= --xattrs-exclude=3Dsecurity.selinux \
= --format=3Dposix \
= = --pax-option=3Dexthdr.name=3D%d/PaxHeaders/%f,atime:=3D0,ctime:=3D0 = \
= --mtime=3D"@${DATE_EPOCH}" \
--no-recursion --null -T - \
-f - -c | = abuild-tar --hash | gzip -n -9 >data.tar.gz

# Create = .PKGINFO
@echo "# Generated by = alpine.package.mk" > .PKGINFO
@echo= -n "# " >> .PKGINFO
@date= >> .PKGINFO
= @echo "package =3D = ${PACKAGE}" >> .PKGINFO
@echo= "pkgver =3D ${PKGVER}" >> .PKGINFO
@echo "pkgdesc =3D ${DESCRIPTION}" >> .PKGINFO
@echo= "url =3D ${URL}" >> .PKGINFO
@echo "builddate =3D ${DATE_EPOCH}" >> .PKGINFO
@echo= "commit =3D 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> = .PKGINFO
@echo "packager =3D ${PACKAGER}" >> .PKGINFO
@echo= -n "size =3D " >> .PKGINFO
@(cd = staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> .PKGINFO
@echo "arch =3D x86_64" >> = .PKGINFO
@echo "origin =3D ${PACKAGE}" >> .PKGINFO
@echo= "maintainer =3D ${MAINTAINER}" >> = .PKGINFO
@echo "license =3D ${LICENSE}" >> .PKGINFO
@for = dep in ${DEPENDS}; do echo "depend =3D $${dep}" >> .PKGINFO; done
@echo "# automatically detected:" >> = .PKGINFO
@for f in ${FILES}; \
= do \
find /tmp/alpine -name = $${f} -type f -perm = +111; \
echo -n "provides =3D = cmd:" >> .PKGINFO; \
= basename $${f} | = tr -d '\n' >> .PKGINFO; \
= echo "=3D${PKGVER}" >> .PKGINFO; \
done
= @echo -n "datahash =3D = " >>.PKGINFO
= @sha256sum data.tar.gz = | cut -d " " -f 1 >> .PKGINFO

# Create = the control part (index)
tar --format=3Dposix \
= --pax-option=3Dexthdr.name=3D%d/PaxHeaders/%f,atime:=3D0,ctime:=3D0 = \
--mtime=3D"@${DATE_EPOCH}" \
= -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \
= | gzip -n -9 > control.tar.gz


#@tar --owner=3D0 --group=3D0 --numeric-owner -c = .PKGINFO ${INSTSCRIPTS} | \
# abuild-tar --cut | gzip > = control.tar.gz
@abuild-sign -q = control.tar.gz

# Produce the final = apk
@cat control.tar.gz = data.tar.gz > ${PACKAGE}-${PKGVER}.apk
= @cat control.tar.gz = data.tar.gz > /var/apk/${PACKAGE}-${PKGVER}.apk
= -(cd /var/apk; apk = index ${PACKAGE}-${PKGVER}.apk -o APKINDEX.tar.gz)
abuild-sign -q /var/apk/APKINDEX.tar.gz

# Show info and clean up
@cat = .PKGINFO
@rm -rf data.tar.gz = control.tar.gz .PKGINFO staging

Essentially, we try to run the packaging = steps in the same way as abuild does it (we used the abuild = source
code as a = reference). Of course our approach is not as elegant as abuild. = Moreover, it does not work, the
apk package being produced can not be = installed.

While the = resulting .apk package looks like any other .apk package, and the = content looks correct, apk exits
with exit code 99 when we try to add or index the package. = apk add prints "ERROR: <pkg name> Not supported."

(Not supported seems to be the = result from musl libc for an unknow error code).

Do you have any ideas what we = are doing wrong?

Thanks,
Marc


= --Apple-Mail=_7FFF0C3C-20B1-4572-A541-4C79FD820974--