~alpine/apk-tools

4 4

Creating an .apk package from local files, without downloading and building sources

Marc Balmer <marc@msys.ch>
Details
Message ID
<D194B900-6A60-4A2C-B520-513D8F1263D6@msys.ch>
DKIM signature
missing
Download raw message
2 years ago 
G'day!

TL;DR:

We are looking for a way to build an .apk package from locally installed files, without going through the download and build stages that abuild normally does. I.e without an APKBUILD file.

A caveat:  My first post to this list... I write this E-Mail using HTML for better readability of the code.  If that is a no-go here, please tell me and I will resend as plain text.


Our setting

We have a large git repo with the source code for many libraries and binaries, much of them open source.  We build and package them for various linux distributions and architectures.  We use docker containers to build for the various systems.   We think that our binaries, packaged for AlpineLinux, would be a good fit for container use.  We thus did setup a build container for Alpine Linux, which builds all of our stuff just fine.

We are now trying to create .apk packages using the already compiled libraries and binaries.  Our approach is to create a directory for each package we want to build and to describe the package using a Makefile.  The packages Makefile then includes a Makefile fragment that drives the actual package building.

PACKAGE=        lua
PKGVER=         5.4.4-r1
DESCRIPTION=    "Lua interpreter and compiler"
PACKAGER=       micro systems <info@msys.ch <info@msys.ch>>
URL=            https://lua.msys.ch
MAINTAINER=     micro systems <info@msys.ch>
LICENSE=        MIT

FILES=          usr/bin/lua \
                usr/bin/luac

MKDIR?=         ../../../mk/

include $(MKDIR)alpine.package.mk

The actual binaries are installed in the docker build container under /tmp/alpine.  We copy the files from there to a local staging directory "staging" and build the package from there. The Makefile Fragment alpine.package.mk looks like this:

LICENSE?=       custom
PACKAGER?=      unknown
DESCRIPTION?=   unknwon
PACKAGER?=      unknown
URL?=           unknown
MAINTAINER?=    unknown

INSTSCRIPTS!=   find . -name .p\*-\*install -exec basename {} \;
DATE_EPOCH!=    date -u "+%s"

.DEFAULT_GOAL=  package

package:
        # Create the data part
        @-mkdir -p staging
        @tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x

        find staging -exec touch -h -d @${DATE_EPOCH} {} +

        @(cd staging; tar --xattrs -f - -c *) | abuild-tar --hash | \
                gzip > data.tar.gz

        (cd staging; find . -print0) | LC_ALL=C sort -z | tar -C staging --xattrs \
                        --xattrs-exclude=security.selinux \
                        --format=posix \
                        --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
                        --mtime="@${DATE_EPOCH}" \
                        --no-recursion --null -T - \
                        -f - -c | abuild-tar --hash | gzip -n -9 >data.tar.gz

        # Create .PKGINFO
        @echo "# Generated by alpine.package.mk" > .PKGINFO
        @echo -n "# " >> .PKGINFO
        @date >> .PKGINFO
        @echo "package = ${PACKAGE}" >> .PKGINFO
        @echo "pkgver = ${PKGVER}" >> .PKGINFO
        @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO
        @echo "url = ${URL}" >> .PKGINFO
        @echo "builddate = ${DATE_EPOCH}" >> .PKGINFO
        @echo "commit = 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> .PKGINFO
        @echo "packager = ${PACKAGER}" >> .PKGINFO
        @echo -n "size = " >> .PKGINFO
        @(cd staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> .PKGINFO
        @echo "arch = x86_64" >> .PKGINFO
        @echo "origin = ${PACKAGE}" >> .PKGINFO
        @echo "maintainer = ${MAINTAINER}" >> .PKGINFO
        @echo "license = ${LICENSE}" >> .PKGINFO
        @for dep in ${DEPENDS}; do echo "depend = $${dep}" >> .PKGINFO; done
        @echo "# automatically detected:" >> .PKGINFO
        @for f in ${FILES}; \
        do \
                find /tmp/alpine -name $${f} -type f -perm +111; \
                echo -n "provides = cmd:" >> .PKGINFO; \
                basename $${f} | tr -d '\n' >> .PKGINFO; \
                echo "=${PKGVER}" >> .PKGINFO; \
        done
        @echo -n "datahash = " >>.PKGINFO
        @sha256sum data.tar.gz | cut -d " " -f 1 >> .PKGINFO

        # Create the control part (index)
        tar --format=posix \
                --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
                --mtime="@${DATE_EPOCH}" \
                -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \
                | gzip -n -9 > control.tar.gz


        #@tar --owner=0 --group=0 --numeric-owner -c .PKGINFO ${INSTSCRIPTS} | \
        #       abuild-tar --cut | gzip > control.tar.gz
        @abuild-sign -q control.tar.gz

        # Produce the final apk
        @cat control.tar.gz data.tar.gz > ${PACKAGE}-${PKGVER}.apk
        @cat control.tar.gz data.tar.gz > /var/apk/${PACKAGE}-${PKGVER}.apk
        -(cd /var/apk; apk index ${PACKAGE}-${PKGVER}.apk -o APKINDEX.tar.gz)
        abuild-sign -q /var/apk/APKINDEX.tar.gz

        # Show info and clean up
        @cat .PKGINFO
        @rm -rf data.tar.gz control.tar.gz .PKGINFO staging

Essentially, we try to run the packaging steps in the same way as abuild does it (we used the abuild source
code as a reference).  Of course our approach is not as elegant as abuild.  Moreover, it does not work, the
apk package being produced can not be installed.

While the resulting .apk package looks like any other .apk package, and the content looks correct, apk exits
with exit code 99 when we try to add or index the package.  apk add prints "ERROR: <pkg name> Not supported."

(Not supported seems to be the result from musl libc for an unknow error code).

Do you have any ideas what we are doing wrong?

Thanks,
Marc
Konstantin Kulikov <k.kulikov2@gmail.com>
Details
Message ID
<CAD+eXGRpRMHyS8LTaiXuG2OAMe4SQ7MGSLYPbUauf3su-oPaiA@mail.gmail.com>
In-Reply-To
<D194B900-6A60-4A2C-B520-513D8F1263D6@msys.ch> (view parent)
DKIM signature
missing
Download raw message
2 years ago 
Maybe you can use abuild directly? Redefine fetch() like below and you
should be good.

fetch() {
default_fetch
cp -r $path_to_your_sources $builddir
}

On Sun, May 8, 2022 at 10:22 AM Marc Balmer <marc@msys.ch> wrote:
>
> G'day!
>
> TL;DR:
>
> We are looking for a way to build an .apk package from locally installed files, without going through the download and build stages that abuild normally does. I.e without an APKBUILD file.
>
> A caveat:  My first post to this list... I write this E-Mail using HTML for better readability of the code.  If that is a no-go here, please tell me and I will resend as plain text.
>
>
> Our setting
>
> We have a large git repo with the source code for many libraries and binaries, much of them open source.  We build and package them for various linux distributions and architectures.  We use docker containers to build for the various systems.   We think that our binaries, packaged for AlpineLinux, would be a good fit for container use.  We thus did setup a build container for Alpine Linux, which builds all of our stuff just fine.
>
> We are now trying to create .apk packages using the already compiled libraries and binaries.  Our approach is to create a directory for each package we want to build and to describe the package using a Makefile.  The packages Makefile then includes a Makefile fragment that drives the actual package building.
>
> PACKAGE= lua
> PKGVER= 5.4.4-r1
> DESCRIPTION= "Lua interpreter and compiler"
> PACKAGER= micro systems <info@msys.ch>
> URL= https://lua.msys.ch
> MAINTAINER= micro systems <info@msys.ch>
> LICENSE= MIT
>
> FILES= usr/bin/lua \
> usr/bin/luac
>
> MKDIR?= ../../../mk/
>
> include $(MKDIR)alpine.package.mk
>
> The actual binaries are installed in the docker build container under /tmp/alpine.  We copy the files from there to a local staging directory "staging" and build the package from there. The Makefile Fragment alpine.package.mk looks like this:
>
> LICENSE?= custom
> PACKAGER?= unknown
> DESCRIPTION?= unknwon
> PACKAGER?= unknown
> URL?= unknown
> MAINTAINER?= unknown
>
> INSTSCRIPTS!= find . -name .p\*-\*install -exec basename {} \;
> DATE_EPOCH!= date -u "+%s"
>
> .DEFAULT_GOAL= package
>
> package:
> # Create the data part
> @-mkdir -p staging
> @tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x
>
> find staging -exec touch -h -d @${DATE_EPOCH} {} +
>
> @(cd staging; tar --xattrs -f - -c *) | abuild-tar --hash | \
> gzip > data.tar.gz
>
> (cd staging; find . -print0) | LC_ALL=C sort -z | tar -C staging --xattrs \
> --xattrs-exclude=security.selinux \
> --format=posix \
> --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
> --mtime="@${DATE_EPOCH}" \
> --no-recursion --null -T - \
> -f - -c | abuild-tar --hash | gzip -n -9 >data.tar.gz
>
> # Create .PKGINFO
> @echo "# Generated by alpine.package.mk" > .PKGINFO
> @echo -n "# " >> .PKGINFO
> @date >> .PKGINFO
> @echo "package = ${PACKAGE}" >> .PKGINFO
> @echo "pkgver = ${PKGVER}" >> .PKGINFO
> @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO
> @echo "url = ${URL}" >> .PKGINFO
> @echo "builddate = ${DATE_EPOCH}" >> .PKGINFO
> @echo "commit = 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> .PKGINFO
> @echo "packager = ${PACKAGER}" >> .PKGINFO
> @echo -n "size = " >> .PKGINFO
> @(cd staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> .PKGINFO
> @echo "arch = x86_64" >> .PKGINFO
> @echo "origin = ${PACKAGE}" >> .PKGINFO
> @echo "maintainer = ${MAINTAINER}" >> .PKGINFO
> @echo "license = ${LICENSE}" >> .PKGINFO
> @for dep in ${DEPENDS}; do echo "depend = $${dep}" >> .PKGINFO; done
> @echo "# automatically detected:" >> .PKGINFO
> @for f in ${FILES}; \
> do \
> find /tmp/alpine -name $${f} -type f -perm +111; \
> echo -n "provides = cmd:" >> .PKGINFO; \
> basename $${f} | tr -d '\n' >> .PKGINFO; \
> echo "=${PKGVER}" >> .PKGINFO; \
> done
> @echo -n "datahash = " >>.PKGINFO
> @sha256sum data.tar.gz | cut -d " " -f 1 >> .PKGINFO
>
> # Create the control part (index)
> tar --format=posix \
> --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
> --mtime="@${DATE_EPOCH}" \
> -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \
> | gzip -n -9 > control.tar.gz
>
>
> #@tar --owner=0 --group=0 --numeric-owner -c .PKGINFO ${INSTSCRIPTS} | \
> # abuild-tar --cut | gzip > control.tar.gz
> @abuild-sign -q control.tar.gz
>
> # Produce the final apk
> @cat control.tar.gz data.tar.gz > ${PACKAGE}-${PKGVER}.apk
> @cat control.tar.gz data.tar.gz > /var/apk/${PACKAGE}-${PKGVER}.apk
> -(cd /var/apk; apk index ${PACKAGE}-${PKGVER}.apk -o APKINDEX.tar.gz)
> abuild-sign -q /var/apk/APKINDEX.tar.gz
>
> # Show info and clean up
> @cat .PKGINFO
> @rm -rf data.tar.gz control.tar.gz .PKGINFO staging
>
> Essentially, we try to run the packaging steps in the same way as abuild does it (we used the abuild source
> code as a reference). Of course our approach is not as elegant as abuild. Moreover, it does not work, the
> apk package being produced can not be installed.
>
> While the resulting .apk package looks like any other .apk package, and the content looks correct, apk exits
> with exit code 99 when we try to add or index the package. apk add prints "ERROR: <pkg name> Not supported."
>
> (Not supported seems to be the result from musl libc for an unknow error code).
>
> Do you have any ideas what we are doing wrong?
>
> Thanks,
> Marc
>
>
Carl Chave <online@chave.us>
Details
Message ID
<CAGP1gyPfhGBi2BjAq7D1fCfG0RyLo4zyyw_=Uhx69pfECmB9XQ@mail.gmail.com>
In-Reply-To
<D194B900-6A60-4A2C-B520-513D8F1263D6@msys.ch> (view parent)
DKIM signature
missing
Download raw message
2 years ago 
Marc,

Not entirely the same use case as you, but a couple years ago I wanted
to resign some already built packages with a different key without
rebuilding from scratch and came up with the below (the main snippet).
It's not elegant and there's probably a better way etc. but caveats
aside, it does work. Maybe there's something in there that jumps out
at you:

for apk in ${apks}
do
  tmp=$(mktemp -dp .)
  chmod 0755 ${tmp}
  printf "===> Processing package ${apk}\n"
  tar -xzpf ${apk} -C ${tmp}
  rm ${apk}
  rm ${tmp}/.SIGN*.pub
  mv ${tmp}/.PKGINFO .
  cd ${tmp}
  tar --xattrs -f - -c $(ls -A1) | abuild-tar --hash | gzip -9 > ../data.tar.gz
  cd ..
  sha256=$(sha256sum data.tar.gz | cut -d' ' -f1)
  sed -i '$d' .PKGINFO
  echo "datahash = ${sha256}" >> .PKGINFO
  tar -f - -c .PKGINFO | abuild-tar --cut | gzip -9 > control.tar.gz
  abuild-sign -k ${priv_key} control.tar.gz
  cat control.tar.gz data.tar.gz > ${apk}
  rm -rf control.tar.gz data.tar.gz .PKGINFO ${tmp}
  i=$(( ${i} +1 ))
done

On Sun, May 8, 2022 at 3:22 AM Marc Balmer <marc@msys.ch> wrote:
>
> G'day!
>
> TL;DR:
>
> We are looking for a way to build an .apk package from locally installed files, without going through the download and build stages that abuild normally does. I.e without an APKBUILD file.
>
> A caveat:  My first post to this list... I write this E-Mail using HTML for better readability of the code.  If that is a no-go here, please tell me and I will resend as plain text.
>
>
> Our setting
>
> We have a large git repo with the source code for many libraries and binaries, much of them open source.  We build and package them for various linux distributions and architectures.  We use docker containers to build for the various systems.   We think that our binaries, packaged for AlpineLinux, would be a good fit for container use.  We thus did setup a build container for Alpine Linux, which builds all of our stuff just fine.
>
> We are now trying to create .apk packages using the already compiled libraries and binaries.  Our approach is to create a directory for each package we want to build and to describe the package using a Makefile.  The packages Makefile then includes a Makefile fragment that drives the actual package building.
>
> PACKAGE= lua
> PKGVER= 5.4.4-r1
> DESCRIPTION= "Lua interpreter and compiler"
> PACKAGER= micro systems <info@msys.ch>
> URL= https://lua.msys.ch
> MAINTAINER= micro systems <info@msys.ch>
> LICENSE= MIT
>
> FILES= usr/bin/lua \
> usr/bin/luac
>
> MKDIR?= ../../../mk/
>
> include $(MKDIR)alpine.package.mk
>
> The actual binaries are installed in the docker build container under /tmp/alpine.  We copy the files from there to a local staging directory "staging" and build the package from there. The Makefile Fragment alpine.package.mk looks like this:
>
> LICENSE?= custom
> PACKAGER?= unknown
> DESCRIPTION?= unknwon
> PACKAGER?= unknown
> URL?= unknown
> MAINTAINER?= unknown
>
> INSTSCRIPTS!= find . -name .p\*-\*install -exec basename {} \;
> DATE_EPOCH!= date -u "+%s"
>
> .DEFAULT_GOAL= package
>
> package:
> # Create the data part
> @-mkdir -p staging
> @tar -C /tmp/alpine -f - -c ${FILES} | tar -C staging -x
>
> find staging -exec touch -h -d @${DATE_EPOCH} {} +
>
> @(cd staging; tar --xattrs -f - -c *) | abuild-tar --hash | \
> gzip > data.tar.gz
>
> (cd staging; find . -print0) | LC_ALL=C sort -z | tar -C staging --xattrs \
> --xattrs-exclude=security.selinux \
> --format=posix \
> --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
> --mtime="@${DATE_EPOCH}" \
> --no-recursion --null -T - \
> -f - -c | abuild-tar --hash | gzip -n -9 >data.tar.gz
>
> # Create .PKGINFO
> @echo "# Generated by alpine.package.mk" > .PKGINFO
> @echo -n "# " >> .PKGINFO
> @date >> .PKGINFO
> @echo "package = ${PACKAGE}" >> .PKGINFO
> @echo "pkgver = ${PKGVER}" >> .PKGINFO
> @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO
> @echo "url = ${URL}" >> .PKGINFO
> @echo "builddate = ${DATE_EPOCH}" >> .PKGINFO
> @echo "commit = 5ff2c19564c162690aa5abbfb46fd6d3960c1c24" >> .PKGINFO
> @echo "packager = ${PACKAGER}" >> .PKGINFO
> @echo -n "size = " >> .PKGINFO
> @(cd staging; du -b -c ${FILES}) | tail -n 1 | cut -f 1 >> .PKGINFO
> @echo "arch = x86_64" >> .PKGINFO
> @echo "origin = ${PACKAGE}" >> .PKGINFO
> @echo "maintainer = ${MAINTAINER}" >> .PKGINFO
> @echo "license = ${LICENSE}" >> .PKGINFO
> @for dep in ${DEPENDS}; do echo "depend = $${dep}" >> .PKGINFO; done
> @echo "# automatically detected:" >> .PKGINFO
> @for f in ${FILES}; \
> do \
> find /tmp/alpine -name $${f} -type f -perm +111; \
> echo -n "provides = cmd:" >> .PKGINFO; \
> basename $${f} | tr -d '\n' >> .PKGINFO; \
> echo "=${PKGVER}" >> .PKGINFO; \
> done
> @echo -n "datahash = " >>.PKGINFO
> @sha256sum data.tar.gz | cut -d " " -f 1 >> .PKGINFO
>
> # Create the control part (index)
> tar --format=posix \
> --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 \
> --mtime="@${DATE_EPOCH}" \
> -f - -c .PKGINFO ${INSTSCRIPTS} | abuild-tar --cut \
> | gzip -n -9 > control.tar.gz
>
>
> #@tar --owner=0 --group=0 --numeric-owner -c .PKGINFO ${INSTSCRIPTS} | \
> # abuild-tar --cut | gzip > control.tar.gz
> @abuild-sign -q control.tar.gz
>
> # Produce the final apk
> @cat control.tar.gz data.tar.gz > ${PACKAGE}-${PKGVER}.apk
> @cat control.tar.gz data.tar.gz > /var/apk/${PACKAGE}-${PKGVER}.apk
> -(cd /var/apk; apk index ${PACKAGE}-${PKGVER}.apk -o APKINDEX.tar.gz)
> abuild-sign -q /var/apk/APKINDEX.tar.gz
>
> # Show info and clean up
> @cat .PKGINFO
> @rm -rf data.tar.gz control.tar.gz .PKGINFO staging
>
> Essentially, we try to run the packaging steps in the same way as abuild does it (we used the abuild source
> code as a reference). Of course our approach is not as elegant as abuild. Moreover, it does not work, the
> apk package being produced can not be installed.
>
> While the resulting .apk package looks like any other .apk package, and the content looks correct, apk exits
> with exit code 99 when we try to add or index the package. apk add prints "ERROR: <pkg name> Not supported."
>
> (Not supported seems to be the result from musl libc for an unknow error code).
>
> Do you have any ideas what we are doing wrong?
>
> Thanks,
> Marc
>
>
Timo Teras <timo.teras@iki.fi>
Details
Message ID
<20220509115754.5cb9fd0b@vostro>
In-Reply-To
<D194B900-6A60-4A2C-B520-513D8F1263D6@msys.ch> (view parent)
DKIM signature
missing
Download raw message
2 years ago 
Hi,

On Sun, 8 May 2022 09:22:23 +0200
Marc Balmer <marc@msys.ch> wrote:

>         # Create .PKGINFO
>         @echo "# Generated by alpine.package.mk" > .PKGINFO
>         @echo -n "# " >> .PKGINFO
>         @date >> .PKGINFO
>         @echo "package = ${PACKAGE}" >> .PKGINFO

Should be 'pkgname'.

>         @echo "pkgver = ${PKGVER}" >> .PKGINFO
>         @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO


Do note that the current 'master' of apk-tools has a new package
format, and tooling in apk-tools itself for building those packages. We
are looking to change all formats in future releases.

Timo
Marc Balmer <marc@msys.ch>
Details
Message ID
<214BF52B-EB48-4970-B1A8-6576B888A3F4@msys.ch>
In-Reply-To
<20220509115754.5cb9fd0b@vostro> (view parent)
DKIM signature
missing
Download raw message
2 years ago 
Hi Timo


>>        # Create .PKGINFO
>>        @echo "# Generated by alpine.package.mk" > .PKGINFO
>>        @echo -n "# " >> .PKGINFO
>>        @date >> .PKGINFO
>>        @echo "package = ${PACKAGE}" >> .PKGINFO
> 
> Should be 'pkgname'.

Oh what a stupid mistake....  Now it works, thank you!

> 
>>        @echo "pkgver = ${PKGVER}" >> .PKGINFO
>>        @echo "pkgdesc = ${DESCRIPTION}" >> .PKGINFO
> 
> 
> Do note that the current 'master' of apk-tools has a new package
> format, and tooling in apk-tools itself for building those packages. We
> are looking to change all formats in future releases.

Is the new format being documented somewhere?

- Marc
Reply to thread Export thread (mbox)