Subject: [alpine-devel] Python 3 in Alpine 3.5

From: Bartłomiej Piotrowski <[email-masked]>
Date: Thu, 02 Jun 2016 12:29:17 +0200

Hi all,

we already discussed Python 3 support at least 3 times. As we recently
released new stable series, edge is again open for all happy breakage
so let's make use of it.

Before I start though, I ran a simple grep on aports and it turns out
there are 575 packages providing various Python libraries. I think this
is much beyond our resources to keep all of them up to date, including
possible security fixes.  Following recent Ruby example, I would love to
limit this set to only very popular libraries (Flask, Requests, etc) and
these that require to be patched to successfully build (numpy). I can 
also
see an exception for all compiled libraries. The question is how to 
measure
popularity; if that becomes a concern, I would rather drop all pure 
Python
modules instead.

I have no idea what to do with tools. We probably should leave them with
dependencies, preferably using Python 3.

With all of the above done, it would ease Python 3 transition a lot. I 
am
thinking of renaming all py-* packages to py2-* and then add py3-*
subpackages where applicable. Then we can open a champagne bottle and 
celebrate.

I want to gather your opinions before I start pushing anything to 
master.

Bartłomiej

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 3 Jun 2016 12:12:28 +0200

On Thu, 02 Jun 2016 12:29:17 +0200

> popularity; if that becomes a concern, I would rather drop all pure 

The difference between python and ruby module packages is that python
are much easier to package an maintain. Upgrading python packages is
fast too, so I actually don't mind keeping then.

Ruby packages on the other hand was almost impossible to maintain,
which is why they got removed.
 
> dependencies, preferably using Python 3.

I suppose the biggest job is to figure out what to keep and what to
remove. I suspect there will be complains as soon as we start remove
python packages. There were very few complains when we removed ruby, I
suspect that nobody used them (because they were likely broken).

> I want to gather your opinions before I start pushing anything to 

Sounds good. so to repeat:

1. clean up. (figure out what we can remove and remove it)
2. rename py-* to py2-*
3. add py3-*

---

---

From: Jiri Horner <[email-masked]>
Date: Fri, 3 Jun 2016 16:14:43 +0200

>which is why they got removed.

Have you considered integrating pip with apk? Like `apk add py-foo` would 
invoke pip to install `foo` for you and manage upgrades too. I was using simple 
script that did something similar, so I don't need to switch between apk and 
pip and it was quite comfortable.

This way it'll be only needed to maintain packages that need patches and still
python packages would remain fist-class citizens and they would be upgraded with 
system packages. Also packaged applications could declare dependencies on things 
in PyPI easily.

Cheers,
Jiri

---

---

From: Sander Maijers <[email-masked]>
Date: Fri, 3 Jun 2016 17:20:53 +0200

Hi all,

I use a lot of Python including Aports, and would not mind if they
were all removed. It would be good though if documentation is provided
how to get up and running installing python packages. E.g.
ca-certificates must be installed first for pip to work properly with
PyPI.

You can measure popularity by checking the number of downloads of
Python Aports. This will give a more direct measure than general
popularity, namely the likelihood that Alpine Linux users will be
dissatisfied once those Aports are no longer available. Please check
who is currently maintaining py3 Aports and query them, as they may be
more offended if Aports are removed without a lot of consultation than
users in general.

I also think Jiri's idea is interesting. For this apk extension, do
consider defaulting to the pip options:
--disable-pip-version-check
--isolated
--no-cache-dir

Best,
Sander

On Fri, Jun 3, 2016 at 4:14 PM, Jiri Horner <[email-masked]> wrote:
> Have you considered integrating pip with apk? Like `apk add py-foo` would invoke pip to install `foo` for you and manage upgrades too. I was using simple script that did something similar, so I don't need to switch between apk and pip and it was quite comfortable.
>
> python packages would remain fist-class citizens and they would be upgraded with system packages. Also packaged applications could declare dependencies on things in PyPI easily.
>
> Cheers,
> Jiri
>
>
>
> ---

---

---

From: Bartłomiej Piotrowski <[email-masked]>
Date: Mon, 6 Jun 2016 19:21:40 +0200

> ca-certificates must be installed first for pip to work properly with

That would be certainly resolved on dependencies level, so py-pip
automatically pulls ca-certificates.

> more offended if Aports are removed without a lot of consultation than

Good idea. I'll see if we can get such stats from somewhere.

Bartłomiej

---

From: Bartłomiej Piotrowski <[email-masked]>
Date: Tue, 7 Jun 2016 16:38:59 +0200

> Good idea. I'll see if we can get such stats from somewhere.

ScrumpyJack has shared his (low traffic) mirror logs from past 12 weeks
and I poured it through some pipes. Barely 102 modules were ever
downloaded, where top 15 is:

    235 py-gdbm
    221 py-libxml2
    215 py-libxslt
    203 py-curl
    194 py-libmount
    161 py-setuptools
     78 py-pip
     36 py-requests
     20 py-pillow
     19 py-crypto
     18 py-oauth2
     18 py-httplib2
     17 py-talloc
     16 py-markupsafe
     16 py-jinja2

This pretty much confirms my theory of unused modules and what should be
kept in repositories.

Bartłomiej

---

From: Kaarle Ritvanen <[email-masked]>
Date: Wed, 8 Jun 2016 01:07:23 +0300 (EEST)

> This pretty much confirms my theory of unused modules and what should be

I think this just proves that some modules are installed more often than 
others. The same is probably true for non-Python packages, but we 
generally do not remove packages because too few people use them.

If the renaming from py-* to py2-* is absolutely necessary, surely that 
can easily be done using find, xargs, sed etc. regardless of the number of 
existing packages.

BR,
Kaarle

---

---

From: Sander Maijers <[email-masked]>
Date: Thu, 9 Jun 2016 11:22:32 +0200

These numbers do help to find out what Aports you could remove without
much objection. I've personally installed Python package Aports just
because they existed(!), favoring them over installing using pip and
making sure any cache files are cleaned. This does not mean I think it
is wise to use them in general. The numbers are just informative,
since the reason for this proposal is not mainly lack of popularity,
but the maintenance overhead of keeping too many Python package
Aports. Pip must first be installed for Python 2, BTW, another reason
for using Aports instead. My only independent motivation for
installing Python package OS packages is if they depend on native
libraries that could well be installed OS-wide separately and that you
may want to switch between, such as Numpy and a BLAS library. I do
that on Arch Linux. I think the latter should be the criterion for
keeping Python packages as Aports: only if they have significant
dependencies that are covered by Aports, that are ultimately not
Python packages themselves.

---

---

From: Kaarle Ritvanen <[email-masked]>
Date: Thu, 9 Jun 2016 15:12:33 +0300 (EEST)

On Thu, 9 Jun 2016, Sander Maijers wrote:

> The numbers are just informative, since the reason for this proposal is 
> not mainly lack of popularity, but the maintenance overhead of keeping 
> too many Python package Aports.

Where does the overhead come from? As explained in an earlier post in this 
thread, the cost of maintaining Python packages is low in comparison to 
Ruby.

I fail to see how the mere existence of a package that has no native 
dependencies would carry a significant maintenance overhead. Usually they 
work just fine. And if no one has time to regularly apply fixes on a 
particular package, py-* or anything else, the package should be moved to 
the community repository.

BR,
Kaarle

---

---

From: Christian Kampka <[email-masked]>
Date: Fri, 03 Jun 2016 16:10:48 +0000

Hi,

> system packages. Also packaged applications could declare dependencies on

I am very much against this.
apk should only offer packages for which we can
a) verified content, pypi packages are not even signed, installing them
does warrent special security considerations since installing python
packages from pypi usually means running setup.py, in a system-wide
installation probably as root.

b) dependency management will be difficult, as several python packages have
non-python dependencies. Often these need to invoke a compiler to build
correctly.

c) apk should only install things for which we can guarantee (to a degree)
that they are functional on alpine, which is pretty much impossible for
pypi packages, especially if b) applies.

Natanael Copa <[email-masked]> schrieb am Fr., 3. Juni 2016 um

> 1. clean up. (figure out what we can remove and remove it)

While we are at it, we should probably check which of the remaining
packages need to remain in main and which ones can be moved to community.

Cheers,
Christian

---

From: Bartłomiej Piotrowski <[email-masked]>
Date: Mon, 6 Jun 2016 19:17:38 +0200

On 2016-06-03 16:14, Jiri Horner wrote:
> Have you considered integrating pip with apk? Like `apk add py-foo`
> would invoke pip to install `foo` for you and manage upgrades too. I was
> using simple script that did something similar, so I don't need to
> switch between apk and pip and it was quite comfortable.

That would be excellent idea, but apk-tools being written in C is rather
unhappy place for such task. As far as I know, one of the design goals
of packaging system Alpine is simplicity as in KISS, and apk<->pip
translation definitely counts as magic.

Bartłomiej

---

From: Bartłomiej Piotrowski <[email-masked]>
Date: Mon, 6 Jun 2016 19:15:12 +0200

> fast too, so I actually don't mind keeping then.

It's easier because Python modules rarely enforce dependencies versions
on runtime. It doesn't mean we don't have possible issues now, but I
generally agree that it is not so bad.

> suspect that nobody used them (because they were likely broken).

There are some packages that are surely very popular in the community or
as a dependency. Setuptools, packaging, requests, Flask, Django,
urllib3, pip, six, jinja, yaml, cffi, unit tests libraries, Pillow,
Sphinx, to name a few… It sounds better to me to start from "scratch"
basing on requests from users than carefully check each of 576 py-*
packages we have now.

Bartłomiej