Date: Thu, 14 Mar 2019 11:24:13 +0100
From: Natanael Copa <ncopa@alpinelinux.org>
To: Drew DeVault <sir@cmpwn.com>
Cc: alpine-devel@lists.alpinelinux.org
Subject: Re: [alpine-devel] Normalizing Python packages in aports
Message-ID: <20190314112413.29dd9816@ncopa-desktop.copa.dup.pw>
In-Reply-To: <20190306212501.GD2800@cirno.localdomain>
References: <20190306212501.GD2800@cirno.localdomain>
Precedence: list
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

On Wed, 6 Mar 2019 14:25:02 -0700
Drew DeVault <sir@cmpwn.com> wrote:

> I'd like to start working on normalizing the Python packages in aports.

Very good!

> The quality of Python packages is pretty hit-or-miss: many packages have
> only Python 2 support, split packages are addressed inconsistently, and
> checks() are missing on most.
> 
> I've put together a proposal on the wiki for Python packaging standards.
> If we like this proposal, the next step would be applying these
> standards to the existing packages in aports. I'll compile a list of
> affected packages and some simple scripts to help track progress/todo
> and we can get started in a separate branch.
> 
> Here's my proposal for packaging standards:
> 
> https://wiki.alpinelinux.org/wiki/Python_package_policies

I'll comment the proposal here.

> # Maintainer: Joe Bloe <joe@example.org>
> pkgname=py-alpine-name

What do we for python2 only modules? pkgname=py2-alpine-name?

What do we when python4 arrives? should we start plan for that already
now? Does it make sense to use pkgname=py3-alpine-name?

> _pkgname=pypi-name

I dont really like the variable name "_pkgname" as it does not really
say anything. I think something like "_pypiname", "_pyname" or
"_upstream_name" is better.

> pkgver=1.2.3
> pkgrel=0
> pkgdesc="Example Python package"
> url=http://example.org/

we should use https:// in our example.

> arch=noarch
> license=MIT
> depends="python3"
> makedepends="py-setuptools"

I think this is somewhat tricky, because the real dependency here is
py3-setuptools which gets pulled in indirectly. I would prefer that we
use the direct dependency instead of rely that the indirect
dependencies will be done correctly.

Using indirect dependencies has caused problems before. The build time
dependency resolver (to detect build order) may not be aware of the
indirect dependency, and a future rename of {py,py2,py3}-setuptools may
lead to unexpected consequences.

Using direct dependencies avoids lots of those.

> source="https://files.pythonhosted.org/packages/source/${_pkgname:0:1}/$_pkgname/$_pkgname-$pkgver.tar.gz"

the ${_pkgname:0:1} expansion is a bash extension which happens to be
supported by busybox ash. We do use some of those, but it would be
better to avoid if possible.

_firstchar=${_pkgname%${_pkgname#?}}

> builddir=$srcdir/$_pkgname-$pkgver
>
> build() {
> 	cd "$builddir"
> 	python3 setup.py build
> }
> 
> check() {
> 	cd "$builddir"
> 	python3 setup.py test
> }
> 
> package() {
> 	python3 setup.py install --prefix=/usr --root="$pkgdir"
> }


Some comments on the Python 2 & 3:

> # Maintainer: Joe Bloe <joe@example.org>
> pkgname=py-alpine-name
> _pkgname=pypi-name

maybe use _pyname instead of _pkgname?

> pkgver=1.2.3
> pkgrel=0
> pkgdesc="Example Python package"
> url=http://example.org/
> arch=noarch
> license=MIT
> subpackages="py2-${pkgname#py-}:_py2 py3-${pkgname#py-}:_py3"
> makedepends="py-setuptools"
> source="https://files.pythonhosted.org/packages/source/${_pkgname:0:1}/$_pkgname/$_pkgname-$pkgver.tar.gz"
> builddir=$srcdir/$_pkgname-$pkgver
>
> prepare() {
> 	cp -r "$builddir" "$builddir"-py3

setuptools does not support out-of-tree builds?

> }
> 
> build() {
> 	cd "$builddir"
>	python2 setup.py build
>	cd "$builddir"-py3
>	python3 setup.py build
> }
>
> check() {
>	cd "$builddir"
>	python2 setup.py test
>	cd "$builddir"-py3
>	python3 setup.py test
> }
>
> package() {
>	mkdir -p "$pkgdir"
> }

I would prefer do something like:

package() {
	mkdir -p "$pkgdir"
	python2 setup.py install --prefix=/usr --root="$pkgdir"
	python3 setup.py install --prefix=/usr --root="$pkgdir"
}

and then move the /usr/lib/python2.* and /usr/lib/python3.* to its
subpkgdir.

>
> _py2() {
>	cd "$builddir"
>	_py python2
> }
>
> _py3() {
>	cd "$builddir"-py3
>	_py python3
> }
>
> _py() {
>	python="$1"

we need 'local' here.

>	pkgdesc="$pkgdesc (for $python)"
>	depends="$depends $python"
>	install_if="$pkgname=$pkgver-r$pkgrel $python"
>	$python setup.py install --prefix=/usr --root="$subpkgdir"
> }

There is an optional way to do it, without the _py2 and _py3 functions:

subpackages="py2-${pkgname#py-}:_py py3-${pkgname#py-}:_py"
makedepends="py-setuptools $_py2_deps $_py3_deps" # make sure depends are built first

...

_py() {
	local _py=${subpkgname##-*}
	local _python="python${_py#py}"

	pkgdesc="$pkgdesc (for $_python)"
	install_if="$pkgname=$pkgver-r$pkgrel $_python"

	case $_py in
	py2) depends="$_py2_deps";;
	py3) depends="$_py3_deps";;
	esac

	mkdir -p "$subpkgdir"/usr/lib
	mv "$pkgdir"/usr/lib/${_python}* "$subpkgdir"/usr/lib/
}

But I suppose using _py2 and _py3 functions makes things more readable.

We should make sure all the direct runtime dependencies are specified
in the makedepends, or the build server may not be able to detect that
the dependencies needs to be built first.

> Another issue we'll have to face soon enough is Python 2 end-of-life. Do
> we want to address this at the same time? I think the best balance of
> conservatism and speedy removal of python 2 would be:
> 
> - Stop adding new python 2 packages to aports
> - As we normalize existing packages, drop python 2 if it's easy, or make
>   a note if not
> - Make case-by-case judgement calls on the more difficult packages,
>   based on the upstream's progress/amicability towards a python 3 port.
>   Upstreams which are unwilling to port to python 3 should probably just
>   be dropped.

I think those are good suggestions.
 
> If we bikeshed too hard on the Python 2 deprecation let's drop it and
> focus just on normalization for now - it'll be easier to deal with
> Python 2 from a sane baseline anyway.

I think removing python2 completely may be tricky and I support doing
it in small steps. We should work in that direction.

> Thoughts?

I think we should already now start think how to deal with python 4,
whenever that comes.

> 
> --
> Drew DeVault
> 
> 
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
> 



---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---