Subject: [alpine-devel] Extend APKBUILD format with a variable to check framework versions

From: Oliver Smith <[email-masked]>
Date: Tue, 08 Jan 2019 08:44:00 +0000

Hey folks,

various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE
Framework have a lot of components that get released together and
therefore must have the same pkgver. In postmarketOS we have a CI check
in place, that makes sure, that the version matches for all these
components. Right now we are categorizing the frameworks by their URL,
but that is of course not the purpose of the URL variable.

It was suggested to add a new variable instead, and since Alpine and
Adélie would probably benefit from such a check and variable as well, I
wonder if it makes sense to make this variable official and add it to
the APKBUILD reference in Alpine's wiki. I could add a simple shell
script that checks the package versions based on that variable in
Alpine's aports.git to be used by Travis.

I would like to keep "framework" out of the name, as this may also be
the case for non-frameworks, such as the KDE Applications, which are
also released together.

One idea for the name would be "relgroup", as in "release group":

  relgroup="qt5"
  relgroup="kde-applications"

What do you think?

Regards,
Oliver

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: "A. Wilcox" <[email-masked]>
Date: Tue, 8 Jan 2019 07:21:38 -0600

>   relgroup="kde-applications"

This would also allow removal of those God-awful checks in the poppler
packages that make me sick to my stomach just reading them.

And the gcj/gcc thingies.

Do note, though, that a significant number of KDE Applications have
their own versioning (and Okteta just *changed* from YY.MM.X to 0.25).

Further still, Qt5 WebKit isn't on the same release cycle that the rest
is.  Version 5.9.1 for the 5.9 series, 5.202.1 for post-5.9 (though I
believe 5.202.1 can build against 5.9).

I guess these just wouldn't be in the relgroup.

My main concern with this is that abuild would then have to do the
equivalent of `grep relgroup ../*/APKBUILD` and check if any don't
match.  "Analysing dependencies" already takes multiple seconds on arm64
(where I/O is slow) and ppc32 (where fork(2) is slow).  The last thing
we need is for another extremely slow process.  Perhaps it could only
even bother checking if the target APKBUILD has relgroup set.

Additionally, this check should be late, and overrideable (maybe with -f
for force).  Our Plasma bump scripts look like:

for i in $(grep -l 'pkgver=5.12.6' */APKBUILD | cut -f1 -d'/'); do cd
$i; sed -i 's/5.12.6/5.12.7/g' APKBUILD; abuild checksum; cd ..; done

so if it checked early, we wouldn't be able to bump at all.
Additionally, sometimes we are forced to go down for a package - I
believe during the 18.08.0 cycle, one of the KDE games failed to build
and we shipped 18.04.3 until 18.08 was fixed.

Anyway, these are just our thoughts over at Adélie.  You do you.

Best,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org

---

From: Chloe Kudryavtsev <[email-masked]>
Date: Tue, 8 Jan 2019 20:17:26 -0500

> but that is of course not the purpose of the URL variable.

I wonder if this could be handled using meta-packages, or something 
along those lines.
Nothing deep/further here, but it seemed worth mentioning (since meta 
packages are effectively a way to group other packages, and would be a 
more "whitelist" approach).

> Alpine's aports.git to be used by Travis.

This sounds interesting.
APKBUILD is basically treated as a shell script (see: abuild.in:2560), 
so, theoretically, adding a variable (even into the spec) doesn't have 
any real downsides (besides an extra couple of optional bytes around).

In the meantime, you could always use _variables (_ being the non-spec 
prefix for variables) to do the same thing (with the same script).
This way if it gets officially supported, upstreaming would be very easy 
(just remove the _s).

> One idea for the name would be "relgroup", as in "release group":

I like that name - it seems fairly representative of the problem being 
solved.

---

---

From: Natanael Copa <[email-masked]>
Date: Wed, 9 Jan 2019 09:53:32 +0100

On Tue, 08 Jan 2019 08:44:00 +0000

> Hey folks,

Hi!
 
> the case for non-frameworks, such as the KDE Applications, which are

There are various other packages that would befit from this, for example:

- llvm/clang/compiler-rt
- kernel and 3rd party modules
- poppler/poppler-qt4
- python/python-tkinter
- wpa_supplicant/wpa_gui
- shorewall/shorewall-core/shorewall6
- audacious/audacious-plugins

Some of those are like qt5 a bigger project splitted up in minor
pieces, and some are built from same source but needs to be built from
different APKBUILDs to avoid circular dependencies.

>   relgroup="kde-applications"

relgroup sounds good.

I am not sure I would like to implement this in abuild though. It would
be an external tool (which may be called from abuild if configured to do
so). I think it would be good to perform the check and error out on
mismatches from a git push hook.

I'd like to continue discuss the implementation details after v3.9
release, which is top prio right now.

Thanks!

-nc

---

---

From: Oliver Smith <[email-masked]>
Date: Tue, 12 Mar 2019 08:50:00 +0000

Hello Natanael and Alpine ML,

3.9 is released, so I would like to follow up with this discussion.

> mismatches from a git push hook.

I would also prefer not to run the check every time in abuild. Besides
the performance issues, that A. Wilcox pointed out, it wouldn't work
well for the postmarketOS/pmbootstrap use case: we only copy one aport
into the Alpine chroot at a time and build it. So we would have to
re-implement it.

Having an external tool sounds good, besides having it in a git push
hook, I would make sure that it also runs in Alpine's CI scripts. We
could use the same script in postmarketOS CI conveniently.

So it seems everybody (who replied) is on board with the idea. I'll
create a simple shell script that does the relgroup check, and submit a
pull request for it.

Would it make more sense to place that script in abuild.git, or aports.git?

Cheers,
Oliver

---

---

From: Oliver Smith <[email-masked]>
Date: Wed, 13 Mar 2019 08:47:00 +0000

Hello again,

here is the script. I placed it in its own git repository, to add tests
and a CI config in the same place, without cluttering Alpine's existing
repositories. The script itself is less than 100 lines (with comments)
and should basically work everywhere out of the box without installing
anything.

https://github.com/ollieparanoid/aports-relgroup-check/blob/master/relgroup-check.sh

See the "tests" dir of the same repository for example output.

My idea is, that this gets forked and maintained over at Alpine's github
organization. Then aports.git (and postmarketOS' pmaports.git, and
Adélie's repo) can download and run the script in their CI code.

What do you think folks?

Best regards,
Oliver

---

---

From: Natanael Copa <[email-masked]>
Date: Thu, 14 Mar 2019 10:31:23 +0100

On Wed, 13 Mar 2019 08:47:00 +0000

Hi!

Thank you for working on this.
 
> Adélie's repo) can download and run the script in their CI code.

I looked at the script and tested it a bit. A few general comments:

- Would be nice to be able to specify APORTS dir as an arg, and if its
  omitted  (and $APORTS is empty) then use current dir.

- I like that you use `find` to find the APKBUILDs

- I think we may need parse/execute the entire APKBUILD to get the
  relgroup, so that things like relgroup=qt${pkgver%%.*} works.

- the script is a bit slow. Probably because there are forks for each
  APKBUILD.

Sounds like a good idea to keep it in the alpinelinux' org. I don't
know if we want keep it on git.alpinelinux.org? What does infra team
think?

-nc

---

---

From: Oliver Smith <[email-masked]>
Date: Thu, 14 Mar 2019 23:14:00 +0000

Hello Natanael,

> I looked at the script and tested it a bit. A few general comments:>

done

>   relgroup, so that things like relgroup=qt${pkgver%%.*} works.

So right now the script limits how the relgroup lines may look like with
a regex. It checks for any invalid relgroup lines and errors out as
necessary before it actually reads all relgroups from the APKBUILDs.
Thanks to the simple format, both steps can be done with grep and should
therefore be a lot faster than actually parsing each APKBUILD. (At least
that's my assumption. I could do some benchmarks if anybody doubts this.)

That means we would need to write 'relgroup="qt5"' in the given example.

What do you think, is this a good trade-off between speed and writing
anything you want in relgroup?

> - the script is a bit slow. Probably because there are forks for each

Just to clarify: it only sources the APKBUILDs where it found out with
grep, that it has a relgroup. In order to parse the pkgver properly.

On my laptop (with SSD) and a few relgroups added to Alpine's
aports.git, it takes 3.9 seconds after a fresh boot. (All subsequent
calls take about 0.4 seconds due to caching.)

I'm not sure if we could speed it up more, suggestions welcome.

> know if we want keep it on git.alpinelinux.org? What does infra team

Not that you have asked me, but I think it is good for consistency.

Thanks for your review and testing!

Cheers,
Oliver

---