Hey folks,
various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE
Framework have a lot of components that get released together and
therefore must have the same pkgver. In postmarketOS we have a CI check
in place, that makes sure, that the version matches for all these
components. Right now we are categorizing the frameworks by their URL,
but that is of course not the purpose of the URL variable.
It was suggested to add a new variable instead, and since Alpine and
Adélie would probably benefit from such a check and variable as well, I
wonder if it makes sense to make this variable official and add it to
the APKBUILD reference in Alpine's wiki. I could add a simple shell
script that checks the package versions based on that variable in
Alpine's aports.git to be used by Travis.
I would like to keep "framework" out of the name, as this may also be
the case for non-frameworks, such as the KDE Applications, which are
also released together.
One idea for the name would be "relgroup", as in "release group":
relgroup="qt5"
relgroup="kde-applications"
What do you think?
Regards,
Oliver
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On 01/08/19 02:44, Oliver Smith wrote:
> Hey folks,> > various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE> Framework have a lot of components that get released together and> therefore must have the same pkgver. In postmarketOS we have a CI check> in place, that makes sure, that the version matches for all these> components. Right now we are categorizing the frameworks by their URL,> but that is of course not the purpose of the URL variable.> > It was suggested to add a new variable instead, and since Alpine and> Adélie would probably benefit from such a check and variable as well, I> wonder if it makes sense to make this variable official and add it to> the APKBUILD reference in Alpine's wiki. I could add a simple shell> script that checks the package versions based on that variable in> Alpine's aports.git to be used by Travis.> > I would like to keep "framework" out of the name, as this may also be> the case for non-frameworks, such as the KDE Applications, which are> also released together.> > One idea for the name would be "relgroup", as in "release group":> > relgroup="qt5"> relgroup="kde-applications"> > What do you think?> > Regards,> Oliver
This would also allow removal of those God-awful checks in the poppler
packages that make me sick to my stomach just reading them.
And the gcj/gcc thingies.
Do note, though, that a significant number of KDE Applications have
their own versioning (and Okteta just *changed* from YY.MM.X to 0.25).
Further still, Qt5 WebKit isn't on the same release cycle that the rest
is. Version 5.9.1 for the 5.9 series, 5.202.1 for post-5.9 (though I
believe 5.202.1 can build against 5.9).
I guess these just wouldn't be in the relgroup.
My main concern with this is that abuild would then have to do the
equivalent of `grep relgroup ../*/APKBUILD` and check if any don't
match. "Analysing dependencies" already takes multiple seconds on arm64
(where I/O is slow) and ppc32 (where fork(2) is slow). The last thing
we need is for another extremely slow process. Perhaps it could only
even bother checking if the target APKBUILD has relgroup set.
Additionally, this check should be late, and overrideable (maybe with -f
for force). Our Plasma bump scripts look like:
for i in $(grep -l 'pkgver=5.12.6' */APKBUILD | cut -f1 -d'/'); do cd
$i; sed -i 's/5.12.6/5.12.7/g' APKBUILD; abuild checksum; cd ..; done
so if it checked early, we wouldn't be able to bump at all.
Additionally, sometimes we are forced to go down for a package - I
believe during the 18.08.0 cycle, one of the KDE games failed to build
and we shipped 18.04.3 until 18.08 was fixed.
Anyway, these are just our thoughts over at Adélie. You do you.
Best,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
On 1/8/2019 3:44 AM, Oliver Smith wrote:
> various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE> Framework have a lot of components that get released together and> therefore must have the same pkgver. In postmarketOS we have a CI check> in place, that makes sure, that the version matches for all these> components. Right now we are categorizing the frameworks by their URL,> but that is of course not the purpose of the URL variable.
I wonder if this could be handled using meta-packages, or something
along those lines.
Nothing deep/further here, but it seemed worth mentioning (since meta
packages are effectively a way to group other packages, and would be a
more "whitelist" approach).
> It was suggested to add a new variable instead, and since Alpine and> Adélie would probably benefit from such a check and variable as well, I> wonder if it makes sense to make this variable official and add it to> the APKBUILD reference in Alpine's wiki. I could add a simple shell> script that checks the package versions based on that variable in> Alpine's aports.git to be used by Travis.
This sounds interesting.
APKBUILD is basically treated as a shell script (see: abuild.in:2560),
so, theoretically, adding a variable (even into the spec) doesn't have
any real downsides (besides an extra couple of optional bytes around).
In the meantime, you could always use _variables (_ being the non-spec
prefix for variables) to do the same thing (with the same script).
This way if it gets officially supported, upstreaming would be very easy
(just remove the _s).
> I would like to keep "framework" out of the name, as this may also be> the case for non-frameworks, such as the KDE Applications, which are> also released together.> One idea for the name would be "relgroup", as in "release group":
I like that name - it seems fairly representative of the problem being
solved.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, 08 Jan 2019 08:44:00 +0000
Oliver Smith <ollieparanoid@bitmessage.ch> wrote:
> Hey folks,
Hi!
> various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE> Framework have a lot of components that get released together and> therefore must have the same pkgver. In postmarketOS we have a CI check> in place, that makes sure, that the version matches for all these> components. Right now we are categorizing the frameworks by their URL,> but that is of course not the purpose of the URL variable.> > It was suggested to add a new variable instead, and since Alpine and> Adélie would probably benefit from such a check and variable as well, I> wonder if it makes sense to make this variable official and add it to> the APKBUILD reference in Alpine's wiki. I could add a simple shell> script that checks the package versions based on that variable in> Alpine's aports.git to be used by Travis.> > I would like to keep "framework" out of the name, as this may also be> the case for non-frameworks, such as the KDE Applications, which are> also released together.
There are various other packages that would befit from this, for example:
- llvm/clang/compiler-rt
- kernel and 3rd party modules
- poppler/poppler-qt4
- python/python-tkinter
- wpa_supplicant/wpa_gui
- shorewall/shorewall-core/shorewall6
- audacious/audacious-plugins
Some of those are like qt5 a bigger project splitted up in minor
pieces, and some are built from same source but needs to be built from
different APKBUILDs to avoid circular dependencies.
> One idea for the name would be "relgroup", as in "release group":> > relgroup="qt5"> relgroup="kde-applications"> > What do you think?
relgroup sounds good.
I am not sure I would like to implement this in abuild though. It would
be an external tool (which may be called from abuild if configured to do
so). I think it would be good to perform the check and error out on
mismatches from a git push hook.
I'd like to continue discuss the implementation details after v3.9
release, which is top prio right now.
Thanks!
-nc
> > Regards,> Oliver> > > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hello Natanael and Alpine ML,
3.9 is released, so I would like to follow up with this discussion.
> I am not sure I would like to implement this in abuild though. It would> be an external tool (which may be called from abuild if configured to do> so). I think it would be good to perform the check and error out on> mismatches from a git push hook.
I would also prefer not to run the check every time in abuild. Besides
the performance issues, that A. Wilcox pointed out, it wouldn't work
well for the postmarketOS/pmbootstrap use case: we only copy one aport
into the Alpine chroot at a time and build it. So we would have to
re-implement it.
Having an external tool sounds good, besides having it in a git push
hook, I would make sure that it also runs in Alpine's CI scripts. We
could use the same script in postmarketOS CI conveniently.
So it seems everybody (who replied) is on board with the idea. I'll
create a simple shell script that does the relgroup check, and submit a
pull request for it.
Would it make more sense to place that script in abuild.git, or aports.git?
Cheers,
Oliver
Natanael Copa:
> On Tue, 08 Jan 2019 08:44:00 +0000> Oliver Smith <ollieparanoid@bitmessage.ch> wrote:> >> Hey folks,> > Hi!> >> various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE>> Framework have a lot of components that get released together and>> therefore must have the same pkgver. In postmarketOS we have a CI check>> in place, that makes sure, that the version matches for all these>> components. Right now we are categorizing the frameworks by their URL,>> but that is of course not the purpose of the URL variable.>>>> It was suggested to add a new variable instead, and since Alpine and>> Adélie would probably benefit from such a check and variable as well, I>> wonder if it makes sense to make this variable official and add it to>> the APKBUILD reference in Alpine's wiki. I could add a simple shell>> script that checks the package versions based on that variable in>> Alpine's aports.git to be used by Travis.>>>> I would like to keep "framework" out of the name, as this may also be>> the case for non-frameworks, such as the KDE Applications, which are>> also released together.> > There are various other packages that would befit from this, for example:> > - llvm/clang/compiler-rt> - kernel and 3rd party modules> - poppler/poppler-qt4> - python/python-tkinter> - wpa_supplicant/wpa_gui> - shorewall/shorewall-core/shorewall6> - audacious/audacious-plugins> > Some of those are like qt5 a bigger project splitted up in minor> pieces, and some are built from same source but needs to be built from> different APKBUILDs to avoid circular dependencies.> >> One idea for the name would be "relgroup", as in "release group":>>>> relgroup="qt5">> relgroup="kde-applications">>>> What do you think?> > relgroup sounds good.> > I am not sure I would like to implement this in abuild though. It would> be an external tool (which may be called from abuild if configured to do> so). I think it would be good to perform the check and error out on> mismatches from a git push hook.> > I'd like to continue discuss the implementation details after v3.9> release, which is top prio right now.> > Thanks!> > -nc> >>>> Regards,>> Oliver>>>>>>>> --->> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>> Help: alpine-devel+help@lists.alpinelinux.org>> --->>> > > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> ---> >
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hello again,
here is the script. I placed it in its own git repository, to add tests
and a CI config in the same place, without cluttering Alpine's existing
repositories. The script itself is less than 100 lines (with comments)
and should basically work everywhere out of the box without installing
anything.
https://github.com/ollieparanoid/aports-relgroup-check/blob/master/relgroup-check.sh
See the "tests" dir of the same repository for example output.
My idea is, that this gets forked and maintained over at Alpine's github
organization. Then aports.git (and postmarketOS' pmaports.git, and
Adélie's repo) can download and run the script in their CI code.
What do you think folks?
Best regards,
Oliver
Oliver Smith:
> Hello Natanael and Alpine ML,> > 3.9 is released, so I would like to follow up with this discussion.> >> I am not sure I would like to implement this in abuild though. It would>> be an external tool (which may be called from abuild if configured to do>> so). I think it would be good to perform the check and error out on>> mismatches from a git push hook.> > I would also prefer not to run the check every time in abuild. Besides> the performance issues, that A. Wilcox pointed out, it wouldn't work> well for the postmarketOS/pmbootstrap use case: we only copy one aport> into the Alpine chroot at a time and build it. So we would have to> re-implement it.> > Having an external tool sounds good, besides having it in a git push> hook, I would make sure that it also runs in Alpine's CI scripts. We> could use the same script in postmarketOS CI conveniently.> > So it seems everybody (who replied) is on board with the idea. I'll> create a simple shell script that does the relgroup check, and submit a> pull request for it.> > Would it make more sense to place that script in abuild.git, or aports.git?> > Cheers,> Oliver> > Natanael Copa:>> On Tue, 08 Jan 2019 08:44:00 +0000>> Oliver Smith <ollieparanoid@bitmessage.ch> wrote:>>>>> Hey folks,>>>> Hi!>> >>> various frameworks, such as Qt (available in Alpine), KDE Plasma, KDE>>> Framework have a lot of components that get released together and>>> therefore must have the same pkgver. In postmarketOS we have a CI check>>> in place, that makes sure, that the version matches for all these>>> components. Right now we are categorizing the frameworks by their URL,>>> but that is of course not the purpose of the URL variable.>>>>>> It was suggested to add a new variable instead, and since Alpine and>>> Adélie would probably benefit from such a check and variable as well, I>>> wonder if it makes sense to make this variable official and add it to>>> the APKBUILD reference in Alpine's wiki. I could add a simple shell>>> script that checks the package versions based on that variable in>>> Alpine's aports.git to be used by Travis.>>>>>> I would like to keep "framework" out of the name, as this may also be>>> the case for non-frameworks, such as the KDE Applications, which are>>> also released together.>>>> There are various other packages that would befit from this, for example:>>>> - llvm/clang/compiler-rt>> - kernel and 3rd party modules>> - poppler/poppler-qt4>> - python/python-tkinter>> - wpa_supplicant/wpa_gui>> - shorewall/shorewall-core/shorewall6>> - audacious/audacious-plugins>>>> Some of those are like qt5 a bigger project splitted up in minor>> pieces, and some are built from same source but needs to be built from>> different APKBUILDs to avoid circular dependencies.>>>>> One idea for the name would be "relgroup", as in "release group":>>>>>> relgroup="qt5">>> relgroup="kde-applications">>>>>> What do you think?>>>> relgroup sounds good.>>>> I am not sure I would like to implement this in abuild though. It would>> be an external tool (which may be called from abuild if configured to do>> so). I think it would be good to perform the check and error out on>> mismatches from a git push hook.>>>> I'd like to continue discuss the implementation details after v3.9>> release, which is top prio right now.>>>> Thanks!>>>> -nc>>>>>>>> Regards,>>> Oliver>>>>>>>>>>>> --->>> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>>> Help: alpine-devel+help@lists.alpinelinux.org>>> --->>>>>>>>>>> --->> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>> Help: alpine-devel+help@lists.alpinelinux.org>> --->>>>> > > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> ---> >
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Wed, 13 Mar 2019 08:47:00 +0000
Oliver Smith <ollieparanoid@bitmessage.ch> wrote:
> Hello again,
Hi!
Thank you for working on this.
> here is the script. I placed it in its own git repository, to add tests> and a CI config in the same place, without cluttering Alpine's existing> repositories. The script itself is less than 100 lines (with comments)> and should basically work everywhere out of the box without installing> anything.> > https://github.com/ollieparanoid/aports-relgroup-check/blob/master/relgroup-check.sh> > See the "tests" dir of the same repository for example output.> > My idea is, that this gets forked and maintained over at Alpine's github> organization. Then aports.git (and postmarketOS' pmaports.git, and> Adélie's repo) can download and run the script in their CI code.> > What do you think folks?
I looked at the script and tested it a bit. A few general comments:
- Would be nice to be able to specify APORTS dir as an arg, and if its
omitted (and $APORTS is empty) then use current dir.
- I like that you use `find` to find the APKBUILDs
- I think we may need parse/execute the entire APKBUILD to get the
relgroup, so that things like relgroup=qt${pkgver%%.*} works.
- the script is a bit slow. Probably because there are forks for each
APKBUILD.
Sounds like a good idea to keep it in the alpinelinux' org. I don't
know if we want keep it on git.alpinelinux.org? What does infra team
think?
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Hello Natanael,
> I looked at the script and tested it a bit. A few general comments:>> - Would be nice to be able to specify APORTS dir as an arg, and if its> omitted (and $APORTS is empty) then use current dir.
done
> - I like that you use `find` to find the APKBUILDs> > - I think we may need parse/execute the entire APKBUILD to get the> relgroup, so that things like relgroup=qt${pkgver%%.*} works.
So right now the script limits how the relgroup lines may look like with
a regex. It checks for any invalid relgroup lines and errors out as
necessary before it actually reads all relgroups from the APKBUILDs.
Thanks to the simple format, both steps can be done with grep and should
therefore be a lot faster than actually parsing each APKBUILD. (At least
that's my assumption. I could do some benchmarks if anybody doubts this.)
That means we would need to write 'relgroup="qt5"' in the given example.
What do you think, is this a good trade-off between speed and writing
anything you want in relgroup?
> - the script is a bit slow. Probably because there are forks for each> APKBUILD.
Just to clarify: it only sources the APKBUILDs where it found out with
grep, that it has a relgroup. In order to parse the pkgver properly.
On my laptop (with SSD) and a few relgroups added to Alpine's
aports.git, it takes 3.9 seconds after a fresh boot. (All subsequent
calls take about 0.4 seconds due to caching.)
I'm not sure if we could speed it up more, suggestions welcome.
> Sounds like a good idea to keep it in the alpinelinux' org. I don't> know if we want keep it on git.alpinelinux.org? What does infra team> think?
Not that you have asked me, but I think it is good for consistency.
Thanks for your review and testing!
Cheers,
Oliver
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---