Subject: Creating an .apk package from local files, without downloading and building sources

From: Marc Balmer <[email-masked]>
Date: Sun, 8 May 2022 09:22:23 +0200

G'day!

TL;DR:

We are looking for a way to build an .apk package from locally installed files, without going through the download and build stages that abuild normally does. I.e without an APKBUILD file.

A caveat:  My first post to this list... I write this E-Mail using HTML for better readability of the code.  If that is a no-go here, please tell me and I will resend as plain text.

Our setting

We have a large git repo with the source code for many libraries and binaries, much of them open source.  We build and package them for various linux distributions and architectures.  We use docker containers to build for the various systems.   We think that our binaries, packaged for AlpineLinux, would be a good fit for container use.  We thus did setup a build container for Alpine Linux, which builds all of our stuff just fine.

We are now trying to create .apk packages using the already compiled libraries and binaries.  Our approach is to create a directory for each package we want to build and to describe the package using a Makefile.  The packages Makefile then includes a Makefile fragment that drives the actual package building.

PACKAGE=        lua
PKGVER=         5.4.4-r1
DESCRIPTION=    "Lua interpreter and compiler"
PACKAGER=       micro systems <[email-masked] <mailto:[email-masked]>>
URL=            https://lua.msys.ch
MAINTAINER=     micro systems <[email-masked]>
LICENSE=        MIT

FILES=          usr/bin/lua \
                usr/bin/luac

MKDIR?=         ../../../mk/

include $(MKDIR)alpine.package.mk

The actual binaries are installed in the docker build container under /tmp/alpine.  We copy the files from there to a local staging directory "staging" and build the package from there. The Makefile Fragment alpine.package.mk looks like this:

LICENSE?=       custom
PACKAGER?=      unknown
DESCRIPTION?=   unknwon
PACKAGER?=      unknown
URL?=           unknown
MAINTAINER?=    unknown

INSTSCRIPTS!=   find . -name .p\*-\*install -exec basename {} \;
DATE_EPOCH!=    date -u "+%s"

.DEFAULT_GOAL=  package

package:
        # Create the data part
        @-mkdir -p staging
        @tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x

        find staging -exec touch -h -d @${DATE_EPOCH} {} +

        @(cd staging; tar --xattrs -f - -c *) | abuild-tar --hash | \
                gzip > data.tar.gz

        (cd staging; find . -print0) | LC_ALL=C sort -z | tar -C staging --xattrs \
                        --xattrs-exclude=security.selinux \
                        --format=posix \
                        --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
                        --mtime="@${DATE_EPOCH}" \
                        --no-recursion --null -T - \
                        -f - -c | abuild-tar --hash | gzip -n -9 >data.tar.gz

        # Create .PKGINFO
        @echo "# Generated by alpine.package.mk" > .PKGINFO
        @echo -n "# " >> .PKGINFO
        @date >> .PKGINFO
        @echo "package = ${PACKAGE}" >> .PKGINFO
        @echo "pkgver = ${PKGVER}" >> .PKGINFO
        @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO
        @echo "url = ${URL}" >> .PKGINFO
        @echo "builddate = ${DATE_EPOCH}" >> .PKGINFO
        @echo "commit = 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> .PKGINFO
        @echo "packager = ${PACKAGER}" >> .PKGINFO
        @echo -n "size = " >> .PKGINFO
        @(cd staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> .PKGINFO
        @echo "arch = x86_64" >> .PKGINFO
        @echo "origin = ${PACKAGE}" >> .PKGINFO
        @echo "maintainer = ${MAINTAINER}" >> .PKGINFO
        @echo "license = ${LICENSE}" >> .PKGINFO
        @for dep in ${DEPENDS}; do echo "depend = $${dep}" >> .PKGINFO; done
        @echo "# automatically detected:" >> .PKGINFO
        @for f in ${FILES}; \
        do \
                find /tmp/alpine -name $${f} -type f -perm +111; \
                echo -n "provides = cmd:" >> .PKGINFO; \
                basename $${f} | tr -d '\n' >> .PKGINFO; \
                echo "=${PKGVER}" >> .PKGINFO; \
        done
        @echo -n "datahash = " >>.PKGINFO
        @sha256sum data.tar.gz | cut -d " " -f 1 >> .PKGINFO

        # Create the control part (index)
        tar --format=posix \
                --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
                --mtime="@${DATE_EPOCH}" \
                -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \
                | gzip -n -9 > control.tar.gz

        #@tar --owner=0 --group=0 --numeric-owner -c .PKGINFO ${INSTSCRIPTS} | \
        #       abuild-tar --cut | gzip > control.tar.gz
        @abuild-sign -q control.tar.gz

        # Produce the final apk
        @cat control.tar.gz data.tar.gz > ${PACKAGE}-${PKGVER}.apk
        @cat control.tar.gz data.tar.gz > /var/apk/${PACKAGE}-${PKGVER}.apk
        -(cd /var/apk; apk index ${PACKAGE}-${PKGVER}.apk -o APKINDEX.tar.gz)
        abuild-sign -q /var/apk/APKINDEX.tar.gz

        # Show info and clean up
        @cat .PKGINFO
        @rm -rf data.tar.gz control.tar.gz .PKGINFO staging

Essentially, we try to run the packaging steps in the same way as abuild does it (we used the abuild source
code as a reference).  Of course our approach is not as elegant as abuild.  Moreover, it does not work, the
apk package being produced can not be installed.

While the resulting .apk package looks like any other .apk package, and the content looks correct, apk exits
with exit code 99 when we try to add or index the package.  apk add prints "ERROR: <pkg name> Not supported."

(Not supported seems to be the result from musl libc for an unknow error code).

Do you have any ideas what we are doing wrong?

Thanks,
Marc

---

From: Carl Chave <[email-masked]>
Date: Sun, 8 May 2022 07:43:42 -0400

Marc,

Not entirely the same use case as you, but a couple years ago I wanted
to resign some already built packages with a different key without
rebuilding from scratch and came up with the below (the main snippet).
It's not elegant and there's probably a better way etc. but caveats
aside, it does work. Maybe there's something in there that jumps out
at you:

for apk in ${apks}
do
  tmp=$(mktemp -dp .)
  chmod 0755 ${tmp}
  printf "===> Processing package ${apk}\n"
  tar -xzpf ${apk} -C ${tmp}
  rm ${apk}
  rm ${tmp}/.SIGN*.pub
  mv ${tmp}/.PKGINFO .
  cd ${tmp}
  tar --xattrs -f - -c $(ls -A1) | abuild-tar --hash | gzip -9 > ../data.tar.gz
  cd ..
  sha256=$(sha256sum data.tar.gz | cut -d' ' -f1)
  sed -i '$d' .PKGINFO
  echo "datahash = ${sha256}" >> .PKGINFO
  tar -f - -c .PKGINFO | abuild-tar --cut | gzip -9 > control.tar.gz
  abuild-sign -k ${priv_key} control.tar.gz
  cat control.tar.gz data.tar.gz > ${apk}
  rm -rf control.tar.gz data.tar.gz .PKGINFO ${tmp}
  i=$(( ${i} +1 ))
done

On Sun, May 8, 2022 at 3:22 AM Marc Balmer <[email-masked]> wrote:
> PACKAGER= micro systems <[email-masked]>
> URL= https://lua.msys.ch

---

From: Timo Teras <[email-masked]>
Date: Mon, 9 May 2022 11:57:54 +0300

Hi,

On Sun, 8 May 2022 09:22:23 +0200

>         @echo "package = ${PACKAGE}" >> .PKGINFO

Should be 'pkgname'.

>         @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO

Do note that the current 'master' of apk-tools has a new package
format, and tooling in apk-tools itself for building those packages. We
are looking to change all formats in future releases.

Timo

---

From: Marc Balmer <[email-masked]>
Date: Mon, 9 May 2022 12:56:05 +0200

Hi Timo

>>        @echo "package = ${PACKAGE}" >> .PKGINFO

Oh what a stupid mistake....  Now it works, thank you!

> are looking to change all formats in future releases.

Is the new format being documented somewhere?

- Marc

---

From: Konstantin Kulikov <[email-masked]>
Date: Sun, 8 May 2022 13:46:46 +0300

Maybe you can use abuild directly? Redefine fetch() like below and you
should be good.

fetch() {
default_fetch
cp -r $path_to_your_sources $builddir
}