I'd like to start working on normalizing the Python packages in aports.
The quality of Python packages is pretty hit-or-miss: many packages have
only Python 2 support, split packages are addressed inconsistently, and
checks() are missing on most.
I've put together a proposal on the wiki for Python packaging standards.
If we like this proposal, the next step would be applying these
standards to the existing packages in aports. I'll compile a list of
affected packages and some simple scripts to help track progress/todo
and we can get started in a separate branch.
Here's my proposal for packaging standards:
https://wiki.alpinelinux.org/wiki/Python_package_policies
Another issue we'll have to face soon enough is Python 2 end-of-life. Do
we want to address this at the same time? I think the best balance of
conservatism and speedy removal of python 2 would be:
- Stop adding new python 2 packages to aports
- As we normalize existing packages, drop python 2 if it's easy, or make
a note if not
- Make case-by-case judgement calls on the more difficult packages,
based on the upstream's progress/amicability towards a python 3 port.
Upstreams which are unwilling to port to python 3 should probably just
be dropped.
If we bikeshed too hard on the Python 2 deprecation let's drop it and
focus just on normalization for now - it'll be easier to deal with
Python 2 from a sane baseline anyway.
Thoughts?
--
Drew DeVault
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Both hands up.
6 березня 2019 р. 23:25:02 GMT+02:00, Drew DeVault <sir@cmpwn.com> написав:
>I'd like to start working on normalizing the Python packages in aports.>The quality of Python packages is pretty hit-or-miss: many packages>have>only Python 2 support, split packages are addressed inconsistently, and>checks() are missing on most.>>I've put together a proposal on the wiki for Python packaging>standards.>If we like this proposal, the next step would be applying these>standards to the existing packages in aports. I'll compile a list of>affected packages and some simple scripts to help track progress/todo>and we can get started in a separate branch.>>Here's my proposal for packaging standards:>>https://wiki.alpinelinux.org/wiki/Python_package_policies>>Another issue we'll have to face soon enough is Python 2 end-of-life.>Do>we want to address this at the same time? I think the best balance of>conservatism and speedy removal of python 2 would be:>>- Stop adding new python 2 packages to aports>- As we normalize existing packages, drop python 2 if it's easy, or>make> a note if not>- Make case-by-case judgement calls on the more difficult packages,> based on the upstream's progress/amicability towards a python 3 port.> Upstreams which are unwilling to port to python 3 should probably just> be dropped.>>If we bikeshed too hard on the Python 2 deprecation let's drop it and>focus just on normalization for now - it'll be easier to deal with>Python 2 from a sane baseline anyway.>>Thoughts?>>-->Drew DeVault>>>--->Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org>Help: alpine-devel+help@lists.alpinelinux.org>---
--
Відправлено з мого Android пристрою з K-9 Mail. Вибачте за стислість.
On 03/06/19 15:25, Drew DeVault wrote:
> Another issue we'll have to face soon enough is Python 2 end-of-life. Do> we want to address this at the same time? I think the best balance of> conservatism and speedy removal of python 2 would be:> > - Stop adding new python 2 packages to aports> - As we normalize existing packages, drop python 2 if it's easy, or make> a note if not> - Make case-by-case judgement calls on the more difficult packages,> based on the upstream's progress/amicability towards a python 3 port.> Upstreams which are unwilling to port to python 3 should probably just> be dropped.> > If we bikeshed too hard on the Python 2 deprecation let's drop it and> focus just on normalization for now - it'll be easier to deal with> Python 2 from a sane baseline anyway.>
Hi Drew,
We (Adélie) are already shipping only Python 3. (Actually, We have been
since 2016.)
At one point there was interest in merging these changes in to Alpine
but as we continued to diverge and there was some tension with Alpine
core devs (I'd rather not rehash) we never did. Our Wiki still has the
notes I wrote about Python 2->3. They may help you somewhat. Note that
this is current as of June; I don't know how much has changed in aports
in the intervening months.
https://wiki.adelielinux.org/wiki/Project:Harmony/Current_merge_list#Python_2_removal
I hope this helps your efforts in some way. I would like to see Python
2 removed from all distros.
Best regards,
--arw
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
On Mar 11, 2019, at 10:29 AM, Drew DeVault <sir@cmpwn.com> wrote:
> > [disclaimer: I work on a mailman competitor]> > I'm not terribly fond of mailman v2 either, but also not of v1, and I> don't think that we should hold up Python 2 for its sake. You should> probably convince the upstream to migrate to Python 3 rather than try to> convince downstream to hold onto deprecated software.
Mailman 3 is the one that uses Python 3. We use it at Adélie.
Mailman 2 is deprecated and no longer maintained. Upstream is not going to port it; they did that already, giving Mailman 3.
Best,
—arw
--
A. Wilcox (Sent from my iPhone - not signed)
Project Lead, Adélie Linux
https://adelielinux.org
[disclaimer: I work on a mailman competitor]
I'm not terribly fond of mailman v2 either, but also not of v1, and I
don't think that we should hold up Python 2 for its sake. You should
probably convince the upstream to migrate to Python 3 rather than try to
convince downstream to hold onto deprecated software.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Drew DeVault wrote in <20190310161822.GC1674@homura.localdomain>:
|On 2019-03-09 7:48 PM, A. Wilcox wrote:
|> https://wiki.adelielinux.org/wiki/Project:Harmony/Current_merge_list#Pyt\
|> hon_2_removal
|>
|> I hope this helps your efforts in some way. I would like to see Python
|> 2 removed from all distros.
|
|Cheers, this should make it easier.
Python 2 is needed for the mailman ML manager.
Only the plain python2 though, dnspython must be .. ah it is in
ports now. Never tried this via package until now, however, it
was not there when i set things up. Will try out soon.
I have no opinion on any other python package, but it would be
tremendous if at least the mailman related binaries were kept --
am i really the only one who uses the elder but still supported
mailman that just works (which should be interpreted as a personal
impression not as a statement on newer mailman or so) on Alpine?
All the -- to me, at least -- important MLs seem to run with the
old mailman that uses Python2. (In fact i cannot recall at the
moment that i ever have subscribed to one which uses the new!)
Thanks,
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Drew DeVault wrote in <20190311152940.GC11876@homura.localdomain>:
|[disclaimer: I work on a mailman competitor]
|
|I'm not terribly fond of mailman v2 either, but also not of v1, and I
|don't think that we should hold up Python 2 for its sake. You should
|probably convince the upstream to migrate to Python 3 rather than try to
|convince downstream to hold onto deprecated software.
Is a killer.
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
A. Wilcox wrote in <83ECE5F4-ED37-4DA8-AD42-894EC106FDE9@adelielinux.org>:
|On Mar 11, 2019, at 10:29 AM, Drew DeVault <[1]sir@cmpwn.com[/1]> wrote:
|
| [1] sir@cmpwn.com
|
||[disclaimer: I work on a mailman competitor]
|
||I'm not terribly fond of mailman v2 either, but also not of v1, and I
||don't think that we should hold up Python 2 for its sake. You should
||probably convince the upstream to migrate to Python 3 rather than try to
||convince downstream to hold onto deprecated software.
|
|Mailman 3 is the one that uses Python 3. We use it at Adélie.
|
|Mailman 2 is deprecated and no longer maintained. Upstream is not going \
|to port it; they did that already, giving Mailman 3.
In how far is Mailman 2 deprecated? As far as i know it gets
security fixes.
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Steffen Nurpmeso wrote in <20190311160447.h55sm%steffen@sdaoden.eu>:
|A. Wilcox wrote in <83ECE5F4-ED37-4DA8-AD42-894EC106FDE9@adelielinux.org>:
||On Mar 11, 2019, at 10:29 AM, Drew DeVault <[1]sir@cmpwn.com[/1]> wrote:
||
|| [1] sir@cmpwn.com
||
|||[disclaimer: I work on a mailman competitor]
||
|||I'm not terribly fond of mailman v2 either, but also not of v1, and I
|||don't think that we should hold up Python 2 for its sake. You should
|||probably convince the upstream to migrate to Python 3 rather than try to
|||convince downstream to hold onto deprecated software.
||
||Mailman 3 is the one that uses Python 3. We use it at Adélie.
||
||Mailman 2 is deprecated and no longer maintained. Upstream is not going \
||to port it; they did that already, giving Mailman 3.
|
|In how far is Mailman 2 deprecated? As far as i know it gets
|security fixes.
It is also still listed on their web page of current releases.
--steffen
|
|Der Kragenbaer, The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Wed, 6 Mar 2019 14:25:02 -0700
Drew DeVault <sir@cmpwn.com> wrote:
> I'd like to start working on normalizing the Python packages in aports.
Very good!
> The quality of Python packages is pretty hit-or-miss: many packages have> only Python 2 support, split packages are addressed inconsistently, and> checks() are missing on most.> > I've put together a proposal on the wiki for Python packaging standards.> If we like this proposal, the next step would be applying these> standards to the existing packages in aports. I'll compile a list of> affected packages and some simple scripts to help track progress/todo> and we can get started in a separate branch.> > Here's my proposal for packaging standards:> > https://wiki.alpinelinux.org/wiki/Python_package_policies
I'll comment the proposal here.
> # Maintainer: Joe Bloe <joe@example.org>> pkgname=py-alpine-name
What do we for python2 only modules? pkgname=py2-alpine-name?
What do we when python4 arrives? should we start plan for that already
now? Does it make sense to use pkgname=py3-alpine-name?
> _pkgname=pypi-name
I dont really like the variable name "_pkgname" as it does not really
say anything. I think something like "_pypiname", "_pyname" or
"_upstream_name" is better.
> pkgver=1.2.3> pkgrel=0> pkgdesc="Example Python package"> url=http://example.org/
we should use https:// in our example.
> arch=noarch> license=MIT> depends="python3"> makedepends="py-setuptools"
I think this is somewhat tricky, because the real dependency here is
py3-setuptools which gets pulled in indirectly. I would prefer that we
use the direct dependency instead of rely that the indirect
dependencies will be done correctly.
Using indirect dependencies has caused problems before. The build time
dependency resolver (to detect build order) may not be aware of the
indirect dependency, and a future rename of {py,py2,py3}-setuptools may
lead to unexpected consequences.
Using direct dependencies avoids lots of those.
> source="https://files.pythonhosted.org/packages/source/${_pkgname:0:1}/$_pkgname/$_pkgname-$pkgver.tar.gz"
the ${_pkgname:0:1} expansion is a bash extension which happens to be
supported by busybox ash. We do use some of those, but it would be
better to avoid if possible.
_firstchar=${_pkgname%${_pkgname#?}}
> builddir=$srcdir/$_pkgname-$pkgver>> build() {> cd "$builddir"> python3 setup.py build> }> > check() {> cd "$builddir"> python3 setup.py test> }> > package() {> python3 setup.py install --prefix=/usr --root="$pkgdir"> }
Some comments on the Python 2 & 3:
> # Maintainer: Joe Bloe <joe@example.org>> pkgname=py-alpine-name> _pkgname=pypi-name
maybe use _pyname instead of _pkgname?
> pkgver=1.2.3> pkgrel=0> pkgdesc="Example Python package"> url=http://example.org/> arch=noarch> license=MIT> subpackages="py2-${pkgname#py-}:_py2 py3-${pkgname#py-}:_py3"> makedepends="py-setuptools"> source="https://files.pythonhosted.org/packages/source/${_pkgname:0:1}/$_pkgname/$_pkgname-$pkgver.tar.gz"> builddir=$srcdir/$_pkgname-$pkgver>> prepare() {> cp -r "$builddir" "$builddir"-py3
setuptools does not support out-of-tree builds?
> }> > build() {> cd "$builddir"> python2 setup.py build> cd "$builddir"-py3> python3 setup.py build> }>> check() {> cd "$builddir"> python2 setup.py test> cd "$builddir"-py3> python3 setup.py test> }>> package() {> mkdir -p "$pkgdir"> }
I would prefer do something like:
package() {
mkdir -p "$pkgdir"
python2 setup.py install --prefix=/usr --root="$pkgdir"
python3 setup.py install --prefix=/usr --root="$pkgdir"
}
and then move the /usr/lib/python2.* and /usr/lib/python3.* to its
subpkgdir.
>> _py2() {> cd "$builddir"> _py python2> }>> _py3() {> cd "$builddir"-py3> _py python3> }>> _py() {> python="$1"
we need 'local' here.
> pkgdesc="$pkgdesc (for $python)"> depends="$depends $python"> install_if="$pkgname=$pkgver-r$pkgrel $python"> $python setup.py install --prefix=/usr --root="$subpkgdir"> }
There is an optional way to do it, without the _py2 and _py3 functions:
subpackages="py2-${pkgname#py-}:_py py3-${pkgname#py-}:_py"
makedepends="py-setuptools $_py2_deps $_py3_deps" # make sure depends are built first
...
_py() {
local _py=${subpkgname##-*}
local _python="python${_py#py}"
pkgdesc="$pkgdesc (for $_python)"
install_if="$pkgname=$pkgver-r$pkgrel $_python"
case $_py in
py2) depends="$_py2_deps";;
py3) depends="$_py3_deps";;
esac
mkdir -p "$subpkgdir"/usr/lib
mv "$pkgdir"/usr/lib/${_python}* "$subpkgdir"/usr/lib/
}
But I suppose using _py2 and _py3 functions makes things more readable.
We should make sure all the direct runtime dependencies are specified
in the makedepends, or the build server may not be able to detect that
the dependencies needs to be built first.
> Another issue we'll have to face soon enough is Python 2 end-of-life. Do> we want to address this at the same time? I think the best balance of> conservatism and speedy removal of python 2 would be:> > - Stop adding new python 2 packages to aports> - As we normalize existing packages, drop python 2 if it's easy, or make> a note if not> - Make case-by-case judgement calls on the more difficult packages,> based on the upstream's progress/amicability towards a python 3 port.> Upstreams which are unwilling to port to python 3 should probably just> be dropped.
I think those are good suggestions.
> If we bikeshed too hard on the Python 2 deprecation let's drop it and> focus just on normalization for now - it'll be easier to deal with> Python 2 from a sane baseline anyway.
I think removing python2 completely may be tricky and I support doing
it in small steps. We should work in that direction.
> Thoughts?
I think we should already now start think how to deal with python 4,
whenever that comes.
> > --> Drew DeVault> > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
Thanks for the feedback!
On 2019-03-14 11:24 AM, Natanael Copa wrote:
> > # Maintainer: Joe Bloe <joe@example.org>> > pkgname=py-alpine-name> > What do we for python2 only modules?
Delete them.
> What do we when python4 arrives? should we start plan for that already> now? Does it make sense to use pkgname=py3-alpine-name?> -%<-> I think we should already now start think how to deal with python 4,> whenever that comes.
Nack. I will eat my hat (and volunteer for another overhaul effort) if
another backwards-incompatible Python shitstorm happens in the next 25
years.
> I dont really like the variable name "_pkgname" as it does not really> say anything. I think something like "_pypiname", "_pyname" or> "_upstream_name" is better.
+1 for _pyname.
> we should use https:// in our example.> -%<-> the ${_pkgname:0:1} expansion is a bash extension which happens to be> supported by busybox ash. We do use some of those, but it would be> better to avoid if possible.> -%<-> We should make sure all the direct runtime dependencies are specified> in the makedepends, or the build server may not be able to detect that> the dependencies needs to be built first.
Ack
> > makedepends="py-setuptools"> > I think this is somewhat tricky, because the real dependency here is> py3-setuptools which gets pulled in indirectly. I would prefer that we> use the direct dependency instead of rely that the indirect> dependencies will be done correctly.> -%<-> I would prefer that we use the direct dependency instead of rely that> the indirect dependencies will be done correctly.
Using `makedepends="py3-setuptools"` presents a problem as well, for
Python 2 deprecation/removal. If we rename py3-* to py-*, we will have
to go through and fix all of these later. If we use meta-packages, this
problem with fix itself when Python 2 is removed, and I'd err on the
side of spot-fixing weirdness until then.
> > prepare() {> > cp -r "$builddir" "$builddir"-py3> > setuptools does not support out-of-tree builds?
Not without hacks, I'm afraid. I'm not too torn up about it since we'll
want to be leaving Python 2 behind anyway.
> > _py() {> > python="$1"> > we need 'local' here.
local is also a bash extension. It's undefined behavior in POSIX shell.
> There is an optional way to do it, without the _py2 and _py3 functions:> -%<-> > But I suppose using _py2 and _py3 functions makes things more readable.
Is doing it without the _py2 and _py3 functions desirable if it makes
things less readable?
> > Another issue we'll have to face soon enough is Python 2 end-of-life. Do> > we want to address this at the same time? I think the best balance of> > conservatism and speedy removal of python 2 would be:> > > > - Stop adding new python 2 packages to aports> > - As we normalize existing packages, drop python 2 if it's easy, or make> > a note if not> > - Make case-by-case judgement calls on the more difficult packages,> > based on the upstream's progress/amicability towards a python 3 port.> > Upstreams which are unwilling to port to python 3 should probably just> > be dropped.> > I think those are good suggestions.> > I think removing python2 completely may be tricky and I support doing> it in small steps. We should work in that direction.
I've written this up into the wiki page, along with incorporating the
rest of your feedback.
https://wiki.alpinelinux.org/w/index.php?title=Python_package_policies&type=revision&diff=15799&oldid=15755
Another question: how should we structure this work? I would appreciate
volunteers (I think there already were some on IRC) to participate and
help with code reviews. We probably ought to work out-of-tree and
periodically request-pull our work into aports.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On 03/14/19 15:37, Drew DeVault wrote:
> On 2019-03-14 3:37 PM, A. Wilcox wrote:>> Unspecified, not undefined.>>>> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_01_01> > Is there a meaningful difference?
Yes.
Quoting the Rationale for Base Definitions[1]:
-8<-
The definitions for "unspecified" and "undefined" appear nearly
identical at first examination, but are not. The term "unspecified"
means that a conforming application may deal with the unspecified
behavior, and it should not care what the outcome is. The term
"undefined" says that a conforming application should not do it because
no definition is provided for what it does (and implicitly it would care
what the outcome was if it tried it). It is important to remember,
however, that if the syntax permits the statement at all, it must have
some outcome in a real implementation.
Thus, the terms "undefined" and "unspecified" apply to the way the
application should think about the feature. In terms of the
implementation, it is always "defined''-there is always some result,
even if it is an error. The implementation is free to choose the
behavior it prefers.
-8<-
It is therefore conformant for a POSIX shell script to use any of the
unspecified keywords, as long as they can "deal with" the unspecified
behaviour.
This can be trivially accomplished using `type` and defining a small
`local` function for shells that don't have it[2], *if* abuild were ever
to *use* such a shell. Note that I doubt that would be possible anyway,
as abuild require multiple changes that were rejected when we proposed
strict POSIX compatibility. (This was so we could use zsh or dash as
/bin/sh and let abuild use a /bin/sh hashbang instead of requiring a
hard dependency on BusyBox. We gave up and gave it a hashbang of
/bin/bash in our soft-fork.) In addition, all build scripts inside of
packages which use /bin/sh as their hashbang would *also* have to be
compatible with this theoretical shell.
Best,
--arw
[1]:
https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xbd_chap01.html#tag_21_01_15
[2]: example:
type local 2>/dev/null || local() { [ ... ] }
--
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org
Thanks for explaining. FWIW I'm in favor of strict POSIX compatibility.
I use subshells to good effect to work around the lack of local:
_python() (
python="$1" # does not affect caller's environment
...
)
In this case, because it also sets the pkgdesc et al (e.g. *wants* to
affect the caller's environment), some additional finesee is required.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Thu, Mar 14, 2019 at 05:30:00PM -0500, A. Wilcox wrote:
> Thus, the terms "undefined" and "unspecified"
[skipped]
> It is therefore conformant for a POSIX shell script to use any of the> unspecified keywords, as long as they can "deal with" the unspecified> behaviour.
A shell script does not "deal" with the behaviour, unless it actually
does.
> This can be trivially accomplished using `type` and defining a small> `local` function for shells that don't have it[2], *if* abuild were ever
Then this has to be done, before being able to say that the script "deals"
with a statement possibly unknown to the shell interpreter.
> to *use* such a shell. Note that I doubt that would be possible anyway,> as abuild require multiple changes that were rejected when we proposed> strict POSIX compatibility. (This was so we could use zsh or dash as> /bin/sh and let abuild use a /bin/sh hashbang instead of requiring a> hard dependency on BusyBox. We gave up and gave it a hashbang of> /bin/bash in our soft-fork.) In addition, all build scripts inside of> packages which use /bin/sh as their hashbang would *also* have to be> compatible with this theoretical shell.
This is a separate argument, saying that because other parties are
not disciplined enough, you choose not to bother. Of course, this
may be practically sound, but how big is your gain by choosing
to write non-portable code?
Other parties' questionable behaviour does not preclude one from writing
portable scripts. This actually does not cost too much overhead, but
makes the reuse of the code and of the trained coding skills (including
people who for any reason read your code) more reliable and portable,
I would say more efficient.
Regards,
Rune
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Thu, 14 Mar 2019 15:12:41 -0400
Drew DeVault <sir@cmpwn.com> wrote:
> Thanks for the feedback!> > On 2019-03-14 11:24 AM, Natanael Copa wrote:> > > # Maintainer: Joe Bloe <joe@example.org>> > > pkgname=py-alpine-name > > > > What do we for python2 only modules? > > Delete them.
Ok if I redirect affected people who needs support to you, and let you
respond on opened issues?
I agree that we should try move away from python2, but I don't we we
can remove it yet. And even if we decided to do so, it is a relatively
big project. Should we sacrifice llvm 6 upgrade? should we sacrifice
openjdk upgrade? Should we delay or skip alpine 3.10 release?
> > What do we when python4 arrives? should we start plan for that> > already now? Does it make sense to use pkgname=py3-alpine-name?> > -%<-> > I think we should already now start think how to deal with python 4,> > whenever that comes. > > Nack. I will eat my hat (and volunteer for another overhaul effort) if> another backwards-incompatible Python shitstorm happens in the next 25> years.
Well. Sorry if I am skeptic here, but it is not first time a
contributor show with strong opinions on how we should do things,
claiming that the consequences does not matter. Years later, when the
contributor moved on to something else, I am still the one who will do
the extra work maintaining it.
Python 4 is already mentioned:
https://www.python.org/dev/peps/pep-0563/#implementation
and i highly doubt that /usr/bin/python3 symlink will point
to /usr/bin/python4.0. So I kind of expect it to be "incompatible".
So I think we already now should think about what will happen when
python4 shows up (probably after python3.9), and I don't know if
you will still be around when that happens.
> > > makedepends="py-setuptools" > > > > I think this is somewhat tricky, because the real dependency here is> > py3-setuptools which gets pulled in indirectly. I would prefer that we> > use the direct dependency instead of rely that the indirect> > dependencies will be done correctly.> > -%<-> > I would prefer that we use the direct dependency instead of rely that> > the indirect dependencies will be done correctly. > > Using `makedepends="py3-setuptools"` presents a problem as well, for> Python 2 deprecation/removal. If we rename py3-* to py-*, we will have> to go through and fix all of these later. If we use meta-packages, this> problem with fix itself when Python 2 is removed, and I'd err on the> side of spot-fixing weirdness until then.
But it also require that build time dependency resolver is aware of it,
which it is not. I can tell you that it is far easier to keep the build
time dependency resolver stupid and simple and manually set direct
dependencies in APKBUILDs than fix the build time dependency resolver.
Also errors caused by direct dependencies (eg missed rename of py3- to
py-) are easy to spot and trivial to fix, while errors caused by broken
indirect dependencies are often tricky to understand why it happens. In
practice what happens is people tell me "builder is broken, i don't know
why, please fix asap!"
So I would prefer to have direct dependencies and go over and replace
py3-* with py-* in the future.
> > > prepare() {> > > cp -r "$builddir" "$builddir"-py3 > > > > setuptools does not support out-of-tree builds? > > Not without hacks, I'm afraid. I'm not too torn up about it since we'll> want to be leaving Python 2 behind anyway.
ok.
> > > _py() {> > > python="$1" > > > > we need 'local' here. > > local is also a bash extension. It's undefined behavior in POSIX shell.
At a minimum it should be prefixed with _ (eg _python) to make sure
that it does not conflict with any internal variable in abuild itself.
We still have accepted use of 'local' in both abuild and the APKBUILDs.
There are a few extension we have accepted:
local
${var//search/replace}
${var:0:1}
If we should accept those or go for strict POSIX compliance is another
discussion which I don't think we should mix with the python packaging
policy.
> > There is an optional way to do it, without the _py2 and _py3 functions:> > -%<-> > > > But I suppose using _py2 and _py3 functions makes things more readable. > > Is doing it without the _py2 and _py3 functions desirable if it makes> things less readable?
I think not. I
> > > Another issue we'll have to face soon enough is Python 2 end-of-life. Do> > > we want to address this at the same time? I think the best balance of> > > conservatism and speedy removal of python 2 would be:> > > > > > - Stop adding new python 2 packages to aports> > > - As we normalize existing packages, drop python 2 if it's easy, or make> > > a note if not> > > - Make case-by-case judgement calls on the more difficult packages,> > > based on the upstream's progress/amicability towards a python 3 port.> > > Upstreams which are unwilling to port to python 3 should probably just> > > be dropped. > > > > I think those are good suggestions.> > > > I think removing python2 completely may be tricky and I support doing> > it in small steps. We should work in that direction. > > I've written this up into the wiki page, along with incorporating the> rest of your feedback.> > https://wiki.alpinelinux.org/w/index.php?title=Python_package_policies&type=revision&diff=15799&oldid=15755> > Another question: how should we structure this work? I would appreciate> volunteers (I think there already were some on IRC) to participate and> help with code reviews. We probably ought to work out-of-tree and> periodically request-pull our work into aports.
Once we decided on how we want it, we should probably start with the
current PRs so new additions is done correctly and contributors are
trained to the new standard. Then I think we should take things in
chunks. For example we could do py-django-* in one go, py-flask-* on
another.
Earler, before python3 was supported, all the py-* packages was
python2. We should probably also look over all py-* packages and make
sure that all python2 packages are renamed to py2-*.
Thanks!
-nc
> > > ---> Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org> Help: alpine-devel+help@lists.alpinelinux.org> --->
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, Mar 19, 2019 at 10:31:53AM +0100, Natanael Copa wrote:
> Earler, before python3 was supported, all the py-* packages was> python2. We should probably also look over all py-* packages and make> sure that all python2 packages are renamed to py2-*.
2c from a lurker.
A large part of this thread looks dedicated to surviving a mess created
by the reuse of the same name "python" (also "py-" as a prefix or similar)
for different and incompatible softwares.
The problems would hardly exist if python2 and python3 worlds
were separated from the beginning.
That's why I would like to emphasize the need to always use strictly
specified references for the purposes of packaging.
In other words:
Do not use "default" names which change their meaning with time and
then their usage must be traced and adjusted everywhere (!)
>From this point of view, it is important to (also) name all python3
packages as py3-*.
This is not the same as handling the preferences of the _users_. They
can get a convenience link of "python" to something they like - but then
per-user, not in the common part of $PATH. This is not hard.
(It _is_ hard to prevent the misuse of "default" "python" if it is present
at hand, with its unpredictable semantics. No "python" should be there
in a common location like /usr/{,local/}bin unless a local administrator
adds a link there to some pythonN or whatnot and takes himself all the
consequences. "python2" and/or "python3" should be there today, possibly
"python3" and "python4" tomorrow, but not some "python"-of-the-day.)
Please do not perpetuate the others' badly chosen practices - they can
not be made consistent other than only temporary and only with a large
effort each time, if at all.
Rune
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, 19 Mar 2019 12:32:30 +0100
u-3qpt@aetey.se wrote:
> On Tue, Mar 19, 2019 at 10:31:53AM +0100, Natanael Copa wrote:> > Earler, before python3 was supported, all the py-* packages was> > python2. We should probably also look over all py-* packages and make> > sure that all python2 packages are renamed to py2-*. > > 2c from a lurker.> > A large part of this thread looks dedicated to surviving a mess created> by the reuse of the same name "python" (also "py-" as a prefix or similar)> for different and incompatible softwares.> > The problems would hardly exist if python2 and python3 worlds> were separated from the beginning.> > That's why I would like to emphasize the need to always use strictly> specified references for the purposes of packaging.> > In other words:> > Do not use "default" names which change their meaning with time and> then their usage must be traced and adjusted everywhere (!)> > From this point of view, it is important to (also) name all python3> packages as py3-*.
always name all python3 packages py3-* would also make it possible to
install py4-* and py3-* in parallel if for some reason that would be
needed in future.
python3.6 and python3.7 will not be possible to install in parallel.
> This is not the same as handling the preferences of the _users_. They> can get a convenience link of "python" to something they like - but then> per-user, not in the common part of $PATH. This is not hard.> > (It _is_ hard to prevent the misuse of "default" "python" if it is present> at hand, with its unpredictable semantics. No "python" should be there> in a common location like /usr/{,local/}bin unless a local administrator> adds a link there to some pythonN or whatnot and takes himself all the> consequences. "python2" and/or "python3" should be there today, possibly> "python3" and "python4" tomorrow, but not some "python"-of-the-day.)> > Please do not perpetuate the others' badly chosen practices - they can> not be made consistent other than only temporary and only with a large> effort each time, if at all.
what about `python-dev`? should you get latest python or failure?
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Wed, 6 Mar 2019 14:25:02 -0700
Drew DeVault <sir@cmpwn.com> wrote:
> I'd like to start working on normalizing the Python packages in aports.> The quality of Python packages is pretty hit-or-miss: many packages have> only Python 2 support, split packages are addressed inconsistently, and> checks() are missing on most.> > I've put together a proposal on the wiki for Python packaging standards.> If we like this proposal, the next step would be applying these> standards to the existing packages in aports. I'll compile a list of> affected packages and some simple scripts to help track progress/todo> and we can get started in a separate branch.> > Here's my proposal for packaging standards:> > https://wiki.alpinelinux.org/wiki/Python_package_policies> > Another issue we'll have to face soon enough is Python 2 end-of-life. Do> we want to address this at the same time? I think the best balance of> conservatism and speedy removal of python 2 would be:> > - Stop adding new python 2 packages to aports> - As we normalize existing packages, drop python 2 if it's easy, or make> a note if not> - Make case-by-case judgement calls on the more difficult packages,> based on the upstream's progress/amicability towards a python 3 port.> Upstreams which are unwilling to port to python 3 should probably just> be dropped.> > If we bikeshed too hard on the Python 2 deprecation let's drop it and> focus just on normalization for now - it'll be easier to deal with> Python 2 from a sane baseline anyway.> > Thoughts?
I would like to hear what Fabian Affolter thinks of your proposal.
Fabian maintains alot of python packages and has done so for a while.
-nc
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, Mar 19, 2019 at 12:57:38PM +0100, Natanael Copa wrote:
> On Tue, 19 Mar 2019 12:32:30 +0100> u-3qpt@aetey.se wrote:> > > On Tue, Mar 19, 2019 at 10:31:53AM +0100, Natanael Copa wrote:> > > Earler, before python3 was supported, all the py-* packages was> > > python2. We should probably also look over all py-* packages and make> > > sure that all python2 packages are renamed to py2-*. > > > > 2c from a lurker.> > > > A large part of this thread looks dedicated to surviving a mess created> > by the reuse of the same name "python" (also "py-" as a prefix or similar)> > for different and incompatible softwares.> > > > The problems would hardly exist if python2 and python3 worlds> > were separated from the beginning.> > > > That's why I would like to emphasize the need to always use strictly> > specified references for the purposes of packaging.> > > > In other words:> > > > Do not use "default" names which change their meaning with time and> > then their usage must be traced and adjusted everywhere (!)> > > > From this point of view, it is important to (also) name all python3> > packages as py3-*.> > always name all python3 packages py3-* would also make it possible to> install py4-* and py3-* in parallel if for some reason that would be> needed in future.> > python3.6 and python3.7 will not be possible to install in parallel.
I think we should make a difference between software and libraries. For
software, it should not matter what python version it is, they are
written against a specific version and just need the apropriate
dependencies.
Libraries ofcourse it does make sense to differentiate, because the
python interpreter is looking for those packages in it's library
directories.
In the end, it does make sense to phase out python2, because at some
point (at the moment January 2020), python 2.7 will be no longer
maintained.
Before that time, I don't think we should remove python 2 only packages,
but try to migrate as much as possible to python3. After that time, we
would need to make the choice to keep packages in the repo for a python
version that is no longer maintained.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, Mar 19, 2019 at 12:57:38PM +0100, Natanael Copa wrote:
> On Tue, 19 Mar 2019 12:32:30 +0100> u-3qpt@aetey.se wrote:> > From this point of view, it is important to (also) name all python3> > packages as py3-*.> > always name all python3 packages py3-* would also make it possible to> install py4-* and py3-* in parallel if for some reason that would be> needed in future.
It is always useful to be able to do parallell installations,
this avoids artificial collisions when different packages need different
packages which want to occupy the same naming slot.
> python3.6 and python3.7 will not be possible to install in parallel.
Why not? As long as "python3" is the only one which is named "python3"
and, say, depends on a certain one of 3.6 or 3.7.
Then the semantics of "python3" and the corresponding contract
against other packages is clear: "only the features which are
present in 3.x for all x<=N". Then the consumer can either accept this
contract or explicitly refer to, say "python3.7" if it needs
a feature not present in the "pyhton3" contract.
Of course there will be still possibilities to choose wrong,
but at least the semantics can be specified unambiguously
and there will be no collisions.
Redundancy may appear if different consumers pull in both 3.6 and
3.7. But as well this can mean that really both 3.6 and 3.7 are needed
on the computer in question. No breakage in either case.
> > Please do not perpetuate the others' badly chosen practices - they can> > not be made consistent other than only temporary and only with a large> > effort each time, if at all.> > what about `python-dev`? should you get latest python or failure?
If you ask me - this is clearly underspecified and _must_ fail.
How do I as a packager know which of the "python-family" languages the
local sysadmin will want his users to develop in?
"Latest" has no stable mapping, how can a consumer (another package or
a local administrator) rely on it without checking what "latest" means
in the context of "python-dev"? Somewhat later this may no longer be
the same? No, thanks.
Of course nothing prevents having a package "pythonLatest-dev" with a
description "this will correspond to yet unknown versions of "Python"
depending on the pace of the development of the language and on how fast
we will update the corresponding packages, updates will change what you
get via this package, possibly to something incompatible".
This might be exactly what someone needs for a day, but probably not what
someone else needs to specify dependencies in his python-related package.
Regards,
Rune
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On Tue, Mar 19, 2019 at 02:08:24PM +0100, u-3qpt@aetey.se wrote:
> How do I as a packager know which of the "python-family" languages the> local sysadmin will want his users to develop in?
This looks ambiguous. It was of course about the packager role in general,
not to pretend being an Alpine Linux packager.
Rune
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---
On 2019-03-19 10:31 AM, Natanael Copa wrote:
> > > What do we for python2 only modules? > > > > Delete them.> > Ok if I redirect affected people who needs support to you, and let you> respond on opened issues?> > I agree that we should try move away from python2, but I don't we we> can remove it yet. And even if we decided to do so, it is a relatively> big project. Should we sacrifice llvm 6 upgrade? should we sacrifice> openjdk upgrade? Should we delay or skip alpine 3.10 release?
Sorry, in retrospect this was a snarky and unrealistic answer. Instead
I'd like to make case-by-case judgements on python2 packages, and I
don't think that establishing a standard is necessary for packages which
will be short lived by nature - since I think we very much ought to
ditch them when Python 2 is EoL.
> Well. Sorry if I am skeptic here, but it is not first time a> contributor show with strong opinions on how we should do things,> claiming that the consequences does not matter. Years later, when the> contributor moved on to something else, I am still the one who will do> the extra work maintaining it.> > Python 4 is already mentioned:> https://www.python.org/dev/peps/pep-0563/#implementation> > and i highly doubt that /usr/bin/python3 symlink will point> to /usr/bin/python4.0. So I kind of expect it to be "incompatible".> > So I think we already now should think about what will happen when> python4 shows up (probably after python3.9), and I don't know if> you will still be around when that happens.
The Python devs have already committed to making Python 4, if it
happens at all, as compatible with Python 3 as possible, with only
conservative changes announced well in advance. They aren't going to
change how strings work again - those kinds of sweeping changes that
break large swaths of the ecosystem are extremely unlikely. I would, in
fact, be surprised if symlinking /usr/bin/python3 to /usr/bin/python4
wouldn't work fine for all but a few programs. I'm certain the
disruption to Alpine will be smaller than that of, for example, the
OpenSSL vs. LibreSSL issues.
> > Using `makedepends="py3-setuptools"` presents a problem as well, for> > Python 2 deprecation/removal. If we rename py3-* to py-*, we will have> > to go through and fix all of these later. If we use meta-packages, this> > problem with fix itself when Python 2 is removed, and I'd err on the> > side of spot-fixing weirdness until then.> > But it also require that build time dependency resolver is aware of it,> which it is not. I can tell you that it is far easier to keep the build> time dependency resolver stupid and simple and manually set direct> dependencies in APKBUILDs than fix the build time dependency resolver.> > Also errors caused by direct dependencies (eg missed rename of py3- to> py-) are easy to spot and trivial to fix, while errors caused by broken> indirect dependencies are often tricky to understand why it happens. In> practice what happens is people tell me "builder is broken, i don't know> why, please fix asap!"> > So I would prefer to have direct dependencies and go over and replace> py3-* with py-* in the future.
Alright, +1. Let's go with moving to py3-* for all Python packages.
> > local is also a bash extension. It's undefined behavior in POSIX> > shell.> > At a minimum it should be prefixed with _ (eg _python) to make sure> that it does not conflict with any internal variable in abuild itself.> > We still have accepted use of 'local' in both abuild and the APKBUILDs.> There are a few extension we have accepted:> > local> ${var//search/replace}> ${var:0:1}> > If we should accept those or go for strict POSIX compliance is another> discussion which I don't think we should mix with the python packaging> policy.
I don't like this mixed-leniency policy, it seems kind of arbitrary.
However, I agree that this isn't the venue for this discussion. Let's
call it _python - without the local keyword - for the time being. I do
have some thoughts around POSIX compliance that I'll mention when the
time comes.
> > I've written this up into the wiki page, along with incorporating the> > rest of your feedback.> > > > https://wiki.alpinelinux.org/w/index.php?title=Python_package_policies&type=revision&diff=15799&oldid=15755> > > > Another question: how should we structure this work? I would appreciate> > volunteers (I think there already were some on IRC) to participate and> > help with code reviews. We probably ought to work out-of-tree and> > periodically request-pull our work into aports.> > Once we decided on how we want it, we should probably start with the> current PRs so new additions is done correctly and contributors are> trained to the new standard. Then I think we should take things in> chunks. For example we could do py-django-* in one go, py-flask-* on> another.
Starting with the existing PRs is a good idea, as well as requiring new
patches to follow these conventions. As far as planning the broader
normalization project goes, my plan is to collect a list of all Python
packages, assign them a status (something like [unreviewed, reviewed,
normalized]), and do some light scripting to help us keep up with this
work. This should make it easy for someone to pick up work, and give us
a place to record any issues that come up for addressing later (like
Python 2-only upstreams). From there, I think doing it one chunk at a
time seems wise indeed.
Updated the wiki page to incorporate feedback since my last reply.
---
Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org
Help: alpine-devel+help@lists.alpinelinux.org
---