Subject: [alpine-devel] Avoiding bloat for Lua 5.1/5.2 modules

From: Natanael Copa <[email-masked]>
Date: Tue, 31 Dec 2013 12:07:58 +0100

Hi,

I am a bit in doubt how to handle the Lua modules and 5.1 and 5.2 compat.

I originally wanted to have lua5.1-<module> and lua5.2-<module>
packages because that would allow us to avoid bloat. You could have a
pure lua5.2 system.

I am not so sure anymore.

What I wanted:
* separate arch specific modules (compiled C code) for 5.1 and 5.2 so
  we don't get both installed unless wanted. Example: lua-filesystem.
* join arch agnostic lua modules (pure lua modules) and use symlinks
  for 5.1/5.2 compat. Since they normally support both 5.1 and 5.2 the
  code itself is identical. Rather than install 2 copies of same lua
  code, we want use symlinks. Example: lua-penlight.
* the lua-<module> (without lua version) should pull in both
  lua5.1-module and lua5.2-module. (or pull in the "current" lua
  version of the module)

Now, this becomes a bit compilcated when dealing with dependencies.
Lets take the example with lua-penlight and lua-filesystem.

Since lua-penlight is a pure lua module and supports both 5.1 and 5.2
we want use symlinks. So lua5.1-penlight is just a symlink to
lua5.2-penlight. However, if lua5.1-penlight depends on lua5.2-penlight
it will also pull in the lua5.2-penlight dependency lua5.2-filesystem -
which we don't want for lua5.1-penlight.

I think we have the following options:

1) Continue try avoid bloat and have a common package without dependency:

   * lua5.1-filesystem:   has lua 5.1 module

   * lua5.2-filesystem:   had lua 5.2 module

   * lua-filesystem:      depends on lua5.1-filesystem and
                          lua5.2-filesystem.

   * lua-penlight-common: has the .lua files but does not depend on
                          lua-filesystem. in it self not useful. (maybe
                          install the .lua files to /usr/share/lua/common)

   * lua5.1-penlight:     depends on lua-penlight-common and
                          lua5.1-filesystem. It has symlinks to the
                          files provided by lua-penlight-common.

   * lua5.2-penlight:     depends on lua-penlight-common and
                          lua5.2-filesystem. (might have symlinks too)

   * lua-penlight:        depends on lua5.1-penlight and lua5.2-penlight.

   This means 7(!) packages for 2 Lua modules.

2) We embrace the bloat and always ship both lua 5.1 and 5.2 in same
   package (like Debian does):

   * lua-filesystem:  ships both 5.1 and 5.2 modules.

   * lua-penlight:    ships both 5.1 and 5.2 modues and depends only
                      lua-filesystem.

   This is way simpler, but will mean that you will always get the
   moduels for both lua versions, regardless if you use both or not (I
   assume you normally only will want run one version of Lua). This
   means bloat.

What do you think? We keep it 'simple' and accept the bloat, or we make
it 'complicated' to avoid the bloat?

Some things to keep in mind: we have many more C modules (arch
specific) modules than noarch (pure Lua) modules.

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Ted Trask <[email-masked]>
Date: Thu, 2 Jan 2014 11:27:27 -0800 (PST)

Is there anything besides ACF that is preventing the wholesale move to Lua 5.2? My goal is to get ACF working with Lua 5.2 before the next release.

Ted

________________________________
 From: Natanael Copa <[email-masked]>
To: [email-masked] 
Sent: Tuesday, December 31, 2013 6:07 AM
Subject: [alpine-devel] Avoiding bloat for Lua 5.1/5.2 modules

Hi,

What I wanted:

-nc

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 3 Jan 2014 08:23:24 +0100

On Thu, 2 Jan 2014 11:27:27 -0800 (PST)

> Is there anything besides ACF that is preventing the wholesale move to Lua 5.2? My goal is to get ACF working with Lua 5.2 before the next release.

not really.

The stopper atm is lua-process i think, which seems abandoned from
upstream[1].

I had a short look at it 2 months ago but there was use of environments
etc so I think it will need some refactoring to support 5.2.

If we could get rid of Lua 5.1 within close future, then we could
likely ship both 5.1 and 5.2 in the same lua-<module> package. (and
then we'll get lua 5.3...)

-nc

[1] https://github.com/xlq/lua-subprocess/issues/2

---

---

From: Ted Trask <[email-masked]>
Date: Fri, 3 Jan 2014 09:07:04 -0500

lua-subprocess was used to replace io.popen calls. Barthalion started to fork the project ( https://github.com/Barthalion/lua-subprocess ) for 5.2 support, but did not finish. Right now, lua-subprocess is only referenced from 6 files because the vast majority of calls are done through a helper library. So, if we cannot get lua-subprocess to work with Lua 5.2, we could try replacing it with another package. I have never had the time to investigate writing lua libraries in C, so finishing the fork would take some time for me. Any other suggestions?

Ted

On Jan 3, 2014, at 2:23 AM, Natanael Copa <[email-masked]> wrote:

> l

---

From: Ted Trask <[email-masked]>
Date: Mon, 19 Jan 2015 16:57:20 +0000 (UTC)

Now that Lua 5.3 is officially released, I just finished upgrading ACF to work with Lua 5.2. :/
Mostly I updated a bunch of packages to create and depend on subpackages for Lua 5.1 and Lua 5.2 support. This included some minor code updates. The most difficult change was forking lua-subprocess to properly support Lua 5.1 and Lua 5.2. The most significant change for Alpine Linux was to modify the haserl package. Two subpackages were created to install separate haserl binaries with Lua 5.1 and 5.2 support. The /usr/bin/haserl file is now a symlink to haserl-lua5.2. Since ACF is based upon haserl, installs based upon edge will now use Lua 5.2 for ACF. The change to haserl could present backward compatibility issues with other haserl scripts and should be noted in the release notes for Alpine Linux 3.2.
Please test and let me know if you find any problems.
Ted
PS. Wow, just realized while writing this email that the thread is over a year old. :(
PPS. ACF is no longer blocking the official switch from Lua 5.1 to Lua 5.2. So, we can possibly move toward making Lua 5.2 the default on Alpine Linux 3.2.
PPPS. If you're wondering about Lua 5.3 support, the current blocker is haserl, which doesn't build with Lua 5.3.
      From: Ted Trask <[email-masked]>
 To: Natanael Copa <[email-masked]> 
Cc: "[email-masked]" <[email-masked]> 
 Sent: Friday, January 3, 2014 9:07 AM
 Subject: Re: [alpine-devel] Avoiding bloat for Lua 5.1/5.2 modules

Ted

not really.

l

-nc

---

---

From: "V.Krishn" <[email-masked]>
Date: Fri, 3 Jan 2014 01:19:26 +0530

> which we don't want for lua5.1-penlight.

I am little confused if,
lua5.1-penlight ===============.
   |                                                             ||
   |  depends on                                        ||
   |                                                             ||
   V                    depends on                     V
 lua5.2-penlight --------------> lua5.2-filesystem

does this not mean lua5.2-filesystem is required by 
lua5.1-penlight to run ?

1. Is it possible to estimate the % usage of users still wanting v5.1 ?

What I was initially thinking if AL v2.7 was marked as transitional version
then 2.8, 2.9 to 3.0 could be made to be a clean-up phase.
(my initial thouht was of lots of other pkgs newer additions)

In all, any higher and better version should have least overhead/bloat and 
clean installation , meaning for users in AL v2.8 installing lua 5.1 would 
come with some minor-over head, and for AL v3.0 lua 5.1 may/maynot be 
available. If available the over-head may be even larger and likely having 
both versions may not work.
In reason with available developer resources, I think such scenario may be 
acceptable.

2. Is it not easy to just add older repo and install lua 5.1 ?

-- 
Regards,
V.Krishn

---

---

From: Natanael Copa <[email-masked]>
Date: Fri, 3 Jan 2014 08:45:50 +0100

On Fri, 3 Jan 2014 01:19:26 +0530

> does this not mean lua5.2-filesystem is required by 

You are right. This is how it is right now. lua5.2-filesystem is pulled
in as a dependency for lua5.1-penlight, but it is lua5.1-filesystem
that is required for the penlight runtime to actually run, not
lua5.2-filessytem (which is "unused", bloat). And since
lua5.1-filesystem is currently not pulled in, it is actually broken.

That's why I raised this question. Its currently broken, how do we fix
it properly?
 
> 1. Is it possible to estimate the % usage of users still wanting v5.1 ?

I don't think anybody actually wants 5.1, but there might be some that
depends on modules that is not yet ported. Oh, the lighttpd developers
don't want add support for lua 5.2 for the current stable branch.

> (my initial thouht was of lots of other pkgs newer additions)

Well, the idea is more or less, alpine edge is a rolling release. Every
6 months we take what we have and ship a vX.Y release that we support
for 2 years.

> In reason with available developer resources, I think such scenario may be 

So if you pick Lua 5.2 (or Lua 'current') you'll don't get the compat
bloat. If you use lua5.1 (or Lua 'previous') you might get some unused
compat packages.

I suppose that would be a better option than do the full separation.

Ok...

Then the options are:

1) lua5.2-<module> -> no bloat
   lua5.1-<module> -> might give some bloat for compatibility (might or
   might not pull in some unused lua <latest> packages)

2) don't have version number in lua-<module> packages. The module will
   always include the lua <previous> variant til its no longer needed.

Option 1 will make it more visible what packages that are available for
each lua version. (apk search lua5.2-*)

I think I am slightly in favor for option #2 though.

> 2. Is it not easy to just add older repo and install lua 5.1 ?

Problem are if some package that we need/want (acf, prosody, awesome)
depends on lua 5.1. We cannot depend on packages in other repositories.

-nc

---

---

From: "V.Krishn" <[email-masked]>
Date: Fri, 3 Jan 2014 16:31:36 +0530

On Friday, January 03, 2014 01:15:50 PM Natanael Copa wrote:
> > What I was initially thinking if AL v2.7 was marked as transitional
> > version then 2.8, 2.9 to 3.0 could be made to be a clean-up phase.
> > In all, any higher and better version should have least overhead/bloat
> > and clean installation , meaning for users in AL v2.8 installing lua 5.1
> > would come with some minor-over head, and for AL v3.0 lua 5.1 may/maynot
> > be available. If available the over-head may be even larger and likely
> > having both versions may not work.
> > In reason with available developer resources, I think such scenario may
> > be acceptable.
> 

Full separation has more benefits:
1. no efforts require to test/check common pkgs compat.
2. more visible what packages that are available for each lua version.
3. overall less efforts to pkgs both version.
4. would be easier to pkgs even newer v5.3 or above.
5. easer to set backward compat with other application.
6. ensures both work properly (important).
7. cleaner install/uninstall.

If need for 5.1 is less, why bother put efforts in creating common pkgs.
Anyone wanting older version may understand and take the bloat of installing 
both or just the older one.

But than, the best way to weigh this would be against efforts and resources 
we have / required to achive this.

> depends on lua 5.1. We cannot depend on packages in other repositories.

We need to ensure 5,6,7 above. 

-- 
Regards.
V.Krishn

---