~alpine/devel

11 6

[alpine-devel] Python 3 in Alpine 3.5

Bartłomiej Piotrowski
Details
Message ID
<a58597d021c9b6da1284fa0ce62235f5@server-speed.net>
Sender timestamp
1464863357
DKIM signature
missing
Download raw message
Hi all,

we already discussed Python 3 support at least 3 times. As we recently
released new stable series, edge is again open for all happy breakage
so let's make use of it.

Before I start though, I ran a simple grep on aports and it turns out
there are 575 packages providing various Python libraries. I think this
is much beyond our resources to keep all of them up to date, including
possible security fixes.  Following recent Ruby example, I would love to
limit this set to only very popular libraries (Flask, Requests, etc) and
these that require to be patched to successfully build (numpy). I can 
also
see an exception for all compiled libraries. The question is how to 
measure
popularity; if that becomes a concern, I would rather drop all pure 
Python
modules instead.

I have no idea what to do with tools. We probably should leave them with
dependencies, preferably using Python 3.

With all of the above done, it would ease Python 3 transition a lot. I 
am
thinking of renaming all py-* packages to py2-* and then add py3-*
subpackages where applicable. Then we can open a champagne bottle and 
celebrate.

I want to gather your opinions before I start pushing anything to 
master.

Bartłomiej


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Natanael Copa
Details
Message ID
<20160603121228.5d26f586@ncopa-desktop.alpinelinux.org>
In-Reply-To
<a58597d021c9b6da1284fa0ce62235f5@server-speed.net> (view parent)
Sender timestamp
1464948748
DKIM signature
missing
Download raw message
On Thu, 02 Jun 2016 12:29:17 +0200
Bart*omiej Piotrowski <b@bpiotrowski.pl> wrote:

> Hi all,
> 
> we already discussed Python 3 support at least 3 times. As we recently
> released new stable series, edge is again open for all happy breakage
> so let's make use of it.
> 
> Before I start though, I ran a simple grep on aports and it turns out
> there are 575 packages providing various Python libraries. I think this
> is much beyond our resources to keep all of them up to date, including
> possible security fixes.  Following recent Ruby example, I would love to
> limit this set to only very popular libraries (Flask, Requests, etc) and
> these that require to be patched to successfully build (numpy). I can 
> also
> see an exception for all compiled libraries. The question is how to 
> measure
> popularity; if that becomes a concern, I would rather drop all pure 
> Python
> modules instead.

The difference between python and ruby module packages is that python
are much easier to package an maintain. Upgrading python packages is
fast too, so I actually don't mind keeping then.

Ruby packages on the other hand was almost impossible to maintain,
which is why they got removed.
 
> I have no idea what to do with tools. We probably should leave them with
> dependencies, preferably using Python 3.

I suppose the biggest job is to figure out what to keep and what to
remove. I suspect there will be complains as soon as we start remove
python packages. There were very few complains when we removed ruby, I
suspect that nobody used them (because they were likely broken).

> 
> With all of the above done, it would ease Python 3 transition a lot. I 
> am
> thinking of renaming all py-* packages to py2-* and then add py3-*
> subpackages where applicable. Then we can open a champagne bottle and 
> celebrate.
> 
> I want to gather your opinions before I start pushing anything to 
> master.

Sounds good. so to repeat:

1. clean up. (figure out what we can remove and remove it)
2. rename py-* to py2-*
3. add py3-*




---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Christian Kampka
Details
Message ID
<CADq4isRzkZTMjBZuVs3s_wp+5aXF+xyf3uQvqr98+ZEh58pS9g@mail.gmail.com>
In-Reply-To
<20160603141443.GB4441@eucalyptus> (view parent)
Sender timestamp
1464970248
DKIM signature
missing
Download raw message
Hi,

Jiri Horner <laeqten@gmail.com> schrieb am Fr., 3. Juni 2016 um 16:14 Uhr:

> Have you considered integrating pip with apk? Like `apk add py-foo` would
> invoke pip to install `foo` for you and manage upgrades too. I was using
> simple
> script that did something similar, so I don't need to switch between apk
> and
> pip and it was quite comfortable.
>
> This way it'll be only needed to maintain packages that need patches and
> still
> python packages would remain fist-class citizens and they would be
> upgraded with
> system packages. Also packaged applications could declare dependencies on
> things
> in PyPI easily.
>

I am very much against this.
apk should only offer packages for which we can
a) verified content, pypi packages are not even signed, installing them
does warrent special security considerations since installing python
packages from pypi usually means running setup.py, in a system-wide
installation probably as root.

b) dependency management will be difficult, as several python packages have
non-python dependencies. Often these need to invoke a compiler to build
correctly.

c) apk should only install things for which we can guarantee (to a degree)
that they are functional on alpine, which is pretty much impossible for
pypi packages, especially if b) applies.

Natanael Copa <ncopa@alpinelinux.org> schrieb am Fr., 3. Juni 2016 um
12:12 Uhr:

> 1. clean up. (figure out what we can remove and remove it)
> 2. rename py-* to py2-*
> 3. add py3-*
>

While we are at it, we should probably check which of the remaining
packages need to remain in main and which ones can be moved to community.

Cheers,
Christian
Jiri Horner
Details
Message ID
<20160603141443.GB4441@eucalyptus>
In-Reply-To
<20160603121228.5d26f586@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1464963283
DKIM signature
missing
Download raw message
On Fri, Jun 03, 2016 at 12:12:28PM +0200, Natanael Copa wrote:
>On Thu, 02 Jun 2016 12:29:17 +0200
>Bart*omiej Piotrowski <b@bpiotrowski.pl> wrote:
>
>> Hi all,
>>
>> we already discussed Python 3 support at least 3 times. As we recently
>> released new stable series, edge is again open for all happy breakage
>> so let's make use of it.
>>
>> Before I start though, I ran a simple grep on aports and it turns out
>> there are 575 packages providing various Python libraries. I think this
>> is much beyond our resources to keep all of them up to date, including
>> possible security fixes.  Following recent Ruby example, I would love to
>> limit this set to only very popular libraries (Flask, Requests, etc) and
>> these that require to be patched to successfully build (numpy). I can
>> also
>> see an exception for all compiled libraries. The question is how to
>> measure
>> popularity; if that becomes a concern, I would rather drop all pure
>> Python
>> modules instead.
>
>The difference between python and ruby module packages is that python
>are much easier to package an maintain. Upgrading python packages is
>fast too, so I actually don't mind keeping then.
>
>Ruby packages on the other hand was almost impossible to maintain,
>which is why they got removed.
>

Have you considered integrating pip with apk? Like `apk add py-foo` would 
invoke pip to install `foo` for you and manage upgrades too. I was using simple 
script that did something similar, so I don't need to switch between apk and 
pip and it was quite comfortable.

This way it'll be only needed to maintain packages that need patches and still
python packages would remain fist-class citizens and they would be upgraded with 
system packages. Also packaged applications could declare dependencies on things 
in PyPI easily.

Cheers,
Jiri


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Sander Maijers
Details
Message ID
<CAGRPH5VJyh+41ZfMOJA5pFcNWp5r4dy3uH6SbG4L5jh2CSVVuA@mail.gmail.com>
In-Reply-To
<20160603141443.GB4441@eucalyptus> (view parent)
Sender timestamp
1464967253
DKIM signature
missing
Download raw message
Hi all,

I use a lot of Python including Aports, and would not mind if they
were all removed. It would be good though if documentation is provided
how to get up and running installing python packages. E.g.
ca-certificates must be installed first for pip to work properly with
PyPI.

You can measure popularity by checking the number of downloads of
Python Aports. This will give a more direct measure than general
popularity, namely the likelihood that Alpine Linux users will be
dissatisfied once those Aports are no longer available. Please check
who is currently maintaining py3 Aports and query them, as they may be
more offended if Aports are removed without a lot of consultation than
users in general.

I also think Jiri's idea is interesting. For this apk extension, do
consider defaulting to the pip options:
--disable-pip-version-check
--isolated
--no-cache-dir

Best,
Sander

On Fri, Jun 3, 2016 at 4:14 PM, Jiri Horner <laeqten@gmail.com> wrote:
>
> On Fri, Jun 03, 2016 at 12:12:28PM +0200, Natanael Copa wrote:
>>
>> On Thu, 02 Jun 2016 12:29:17 +0200
>> Bart*omiej Piotrowski <b@bpiotrowski.pl> wrote:
>>
>>> Hi all,
>>>
>>> we already discussed Python 3 support at least 3 times. As we recently
>>> released new stable series, edge is again open for all happy breakage
>>> so let's make use of it.
>>>
>>> Before I start though, I ran a simple grep on aports and it turns out
>>> there are 575 packages providing various Python libraries. I think this
>>> is much beyond our resources to keep all of them up to date, including
>>> possible security fixes.  Following recent Ruby example, I would love to
>>> limit this set to only very popular libraries (Flask, Requests, etc) and
>>> these that require to be patched to successfully build (numpy). I can
>>> also
>>> see an exception for all compiled libraries. The question is how to
>>> measure
>>> popularity; if that becomes a concern, I would rather drop all pure
>>> Python
>>> modules instead.
>>
>>
>> The difference between python and ruby module packages is that python
>> are much easier to package an maintain. Upgrading python packages is
>> fast too, so I actually don't mind keeping then.
>>
>> Ruby packages on the other hand was almost impossible to maintain,
>> which is why they got removed.
>>
>
> Have you considered integrating pip with apk? Like `apk add py-foo` would invoke pip to install `foo` for you and manage upgrades too. I was using simple script that did something similar, so I don't need to switch between apk and pip and it was quite comfortable.
>
> This way it'll be only needed to maintain packages that need patches and still
> python packages would remain fist-class citizens and they would be upgraded with system packages. Also packaged applications could declare dependencies on things in PyPI easily.
>
> Cheers,
> Jiri
>
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
>


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Bartłomiej Piotrowski
Details
Message ID
<5d25e0be-609f-7fe6-39b8-a8da17a23d1e@bpiotrowski.pl>
In-Reply-To
<20160603121228.5d26f586@ncopa-desktop.alpinelinux.org> (view parent)
Sender timestamp
1465233312
DKIM signature
missing
Download raw message
On 2016-06-03 12:12, Natanael Copa wrote:
> The difference between python and ruby module packages is that python
> are much easier to package an maintain. Upgrading python packages is
> fast too, so I actually don't mind keeping then.

It's easier because Python modules rarely enforce dependencies versions
on runtime. It doesn't mean we don't have possible issues now, but I
generally agree that it is not so bad.

> I suppose the biggest job is to figure out what to keep and what to
> remove. I suspect there will be complains as soon as we start remove
> python packages. There were very few complains when we removed ruby, I
> suspect that nobody used them (because they were likely broken).

There are some packages that are surely very popular in the community or
as a dependency. Setuptools, packaging, requests, Flask, Django,
urllib3, pip, six, jinja, yaml, cffi, unit tests libraries, Pillow,
Sphinx, to name a few… It sounds better to me to start from "scratch"
basing on requests from users than carefully check each of 576 py-*
packages we have now.

Bartłomiej
Bartłomiej Piotrowski
Details
Message ID
<2ca3860c-c5ad-0c0d-4bf2-d2724b913f24@bpiotrowski.pl>
In-Reply-To
<20160603141443.GB4441@eucalyptus> (view parent)
Sender timestamp
1465233458
DKIM signature
missing
Download raw message
On 2016-06-03 16:14, Jiri Horner wrote:
> Have you considered integrating pip with apk? Like `apk add py-foo`
> would invoke pip to install `foo` for you and manage upgrades too. I was
> using simple script that did something similar, so I don't need to
> switch between apk and pip and it was quite comfortable.

That would be excellent idea, but apk-tools being written in C is rather
unhappy place for such task. As far as I know, one of the design goals
of packaging system Alpine is simplicity as in KISS, and apk<->pip
translation definitely counts as magic.

Bartłomiej
Bartłomiej Piotrowski
Details
Message ID
<252f68bc-f838-19ac-5cf3-650a929cd7aa@bpiotrowski.pl>
In-Reply-To
<CAGRPH5VJyh+41ZfMOJA5pFcNWp5r4dy3uH6SbG4L5jh2CSVVuA@mail.gmail.com> (view parent)
Sender timestamp
1465233700
DKIM signature
missing
Download raw message
On 2016-06-03 17:20, Sander Maijers wrote:
> Hi all,
>
> I use a lot of Python including Aports, and would not mind if they
> were all removed. It would be good though if documentation is provided
> how to get up and running installing python packages. E.g.
> ca-certificates must be installed first for pip to work properly with
> PyPI.

That would be certainly resolved on dependencies level, so py-pip
automatically pulls ca-certificates.

> You can measure popularity by checking the number of downloads of
> Python Aports. This will give a more direct measure than general
> popularity, namely the likelihood that Alpine Linux users will be
> dissatisfied once those Aports are no longer available. Please check
> who is currently maintaining py3 Aports and query them, as they may be
> more offended if Aports are removed without a lot of consultation than
> users in general.

Good idea. I'll see if we can get such stats from somewhere.

Bartłomiej
Bartłomiej Piotrowski
Details
Message ID
<33f33d97-e079-b726-58e4-433e5ca18b37@bpiotrowski.pl>
In-Reply-To
<252f68bc-f838-19ac-5cf3-650a929cd7aa@bpiotrowski.pl> (view parent)
Sender timestamp
1465310339
DKIM signature
missing
Download raw message
On 2016-06-06 19:21, Bartłomiej Piotrowski wrote:
> Good idea. I'll see if we can get such stats from somewhere.

ScrumpyJack has shared his (low traffic) mirror logs from past 12 weeks
and I poured it through some pipes. Barely 102 modules were ever
downloaded, where top 15 is:

    235 py-gdbm
    221 py-libxml2
    215 py-libxslt
    203 py-curl
    194 py-libmount
    161 py-setuptools
     78 py-pip
     36 py-requests
     20 py-pillow
     19 py-crypto
     18 py-oauth2
     18 py-httplib2
     17 py-talloc
     16 py-markupsafe
     16 py-jinja2

This pretty much confirms my theory of unused modules and what should be
kept in repositories.

Bartłomiej
Kaarle Ritvanen
Details
Message ID
<alpine.LNX.2.20.1606080044570.19057@dell.my.domain>
In-Reply-To
<33f33d97-e079-b726-58e4-433e5ca18b37@bpiotrowski.pl> (view parent)
Sender timestamp
1465337243
DKIM signature
missing
Download raw message
On Tue, 7 Jun 2016, Bartłomiej Piotrowski wrote:

> ScrumpyJack has shared his (low traffic) mirror logs from past 12 weeks
> and I poured it through some pipes. Barely 102 modules were ever
> downloaded, where top 15 is:
> 
> This pretty much confirms my theory of unused modules and what should be
> kept in repositories.

I think this just proves that some modules are installed more often than 
others. The same is probably true for non-Python packages, but we 
generally do not remove packages because too few people use them.

If the renaming from py-* to py2-* is absolutely necessary, surely that 
can easily be done using find, xargs, sed etc. regardless of the number of 
existing packages.

BR,
Kaarle


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Sander Maijers
Details
Message ID
<CAGRPH5UKCgq1v3cHq-f45CXE8O_K1ExnW6upJWMtmcppv7WjMA@mail.gmail.com>
In-Reply-To
<alpine.LNX.2.20.1606080044570.19057@dell.my.domain> (view parent)
Sender timestamp
1465464152
DKIM signature
missing
Download raw message
These numbers do help to find out what Aports you could remove without
much objection. I've personally installed Python package Aports just
because they existed(!), favoring them over installing using pip and
making sure any cache files are cleaned. This does not mean I think it
is wise to use them in general. The numbers are just informative,
since the reason for this proposal is not mainly lack of popularity,
but the maintenance overhead of keeping too many Python package
Aports. Pip must first be installed for Python 2, BTW, another reason
for using Aports instead. My only independent motivation for
installing Python package OS packages is if they depend on native
libraries that could well be installed OS-wide separately and that you
may want to switch between, such as Numpy and a BLAS library. I do
that on Arch Linux. I think the latter should be the criterion for
keeping Python packages as Aports: only if they have significant
dependencies that are covered by Aports, that are ultimately not
Python packages themselves.

On Wed, Jun 8, 2016 at 12:07 AM, Kaarle Ritvanen
<kaarle.ritvanen@datakunkku.fi> wrote:
> On Tue, 7 Jun 2016, Bartłomiej Piotrowski wrote:
>
>> ScrumpyJack has shared his (low traffic) mirror logs from past 12 weeks
>> and I poured it through some pipes. Barely 102 modules were ever
>> downloaded, where top 15 is:
>>
>> This pretty much confirms my theory of unused modules and what should be
>> kept in repositories.
>
> I think this just proves that some modules are installed more often than
> others. The same is probably true for non-Python packages, but we
> generally do not remove packages because too few people use them.
>
> If the renaming from py-* to py2-* is absolutely necessary, surely that
> can easily be done using find, xargs, sed etc. regardless of the number of
> existing packages.
>
> BR,
> Kaarle
>
>
> ---
> Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
> Help:         alpine-devel+help@lists.alpinelinux.org
> ---
>


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---
Kaarle Ritvanen
Details
Message ID
<alpine.LFD.2.20.1606091508230.31504@kanala.kunkku.net>
In-Reply-To
<CAGRPH5UKCgq1v3cHq-f45CXE8O_K1ExnW6upJWMtmcppv7WjMA@mail.gmail.com> (view parent)
Sender timestamp
1465474353
DKIM signature
missing
Download raw message
On Thu, 9 Jun 2016, Sander Maijers wrote:

> The numbers are just informative, since the reason for this proposal is 
> not mainly lack of popularity, but the maintenance overhead of keeping 
> too many Python package Aports.

Where does the overhead come from? As explained in an earlier post in this 
thread, the cost of maintaining Python packages is low in comparison to 
Ruby.

I fail to see how the mere existence of a package that has no native 
dependencies would carry a significant maintenance overhead. Usually they 
work just fine. And if no one has time to regularly apply fixes on a 
particular package, py-* or anything else, the package should be moved to 
the community repository.

BR,
Kaarle


---
Unsubscribe:  alpine-devel+unsubscribe@lists.alpinelinux.org
Help:         alpine-devel+help@lists.alpinelinux.org
---