Subject: [alpine-devel] reorganize the repositories

From: Natanael Copa <[email-masked]>
Date: Fri, 17 Jul 2015 16:59:16 +0200

Hi,

The 'main' repository has grown big.

I feel that we have more packages than we can maintain for 2 years. I
don't mind that we have tons of packages, but there are many packages
that I only want give best-effort support for security fixes instead of
supporting it for 2 years.

Too many files/dirs in a directory is not good for performance also.

So the idea is to split main, or reorganize the repositories as:

- main
  All core packages and all the package that we can provide 2 year
  support for.

- community
  All the extra packages that community maintains. This repository will
  be available in stable releases but we only provide best-effort
  support for bug and security fixes. None of the packages in 'main'
  repository can depend on anything in 'community'.

- staging
  This is basically current 'testing', but we rename it to make it
  clear that packages are not supposed to stay here forever. We might
  want autopurge packages that has not been moved to 'main' or
  'community' repo. Once a package is tested and confirmed to work, it
  can be moved to either community or main.

We would have a gitolite group, 'maindevs' and another 'communitydevs'.
People in community group may git push to 'community' repo but not
'main' repo. This is so that maintainers can push packages themselves,
without needing to send patches, like they currently do. The recent
move to gitolite makes this possible.

We could also give maintainers git push access to only the packages
they maintain, but I think that would complicate things more than
needed.

Other alternatives is that we introduce categories ala gentoo, and the
bsd ports, but I would prefer avoid this. We'd need to decide what
package belongs in what category. They can often fit into many.

What do you think?

Other ideas?

-nc

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Alan Messias <[email-masked]>
Date: Fri, 17 Jul 2015 15:57:40 -0300

Natanael,

I lilke the idea of *staging and then move to 'main' or to 'community' repo*.
But just to make sure it is clear, let me quote you:

"
> *People in community group may git push to 'community' repo but not*
> *'main' repo. This is so that maintainers can push packages themselves,**without
> needing to send patches, like they currently do.*"

Does it mean that *everybody *that asks for write access in 'community'
will be accepted to git push to it? In case of an 'yes', this is the piece
that I don't like - I'd vote to everybody continue sending patches to
'community' except some approved ones.

Another quote ref. staging repo:

"
> *We might want autopurge packages that has not been moved to 'main'
> or'community' repo*"

It might be great! But i think we'll need to keep track of packages purged
just in case someone requests a package that were in staging for a period
and was purged - Or any other way to avoid that we fall in a loop adding
package and purging it over and over again (nowadays there are packages in
testing being used for users and, it may happen that, once we purge it, the
user will miss the package and request *new [old]* package).

"
> *We could also give maintainers git push access to only the packages*
> *they maintain, but I think that would complicate things more than*
> *needed.*"

I agree with you - I wouldn't vote to use this.

About the 'staging' repo: I like this name but I also like NetBSD's idea of
*wip* (stands for *w*ork *i*n *p*rogress). Maybe staging could be called
*wip* repository.

:wq!
alacerda

-- 
Atenciosamente,
Alan Messias Cordeiro

*The nice thing about standards is that there are so many to choose from.
And if you really don't like all the standards you just have to wait
another year until the one arises you are looking for. -- A. Tanenbaum,
"Introduction to Computer Networks"*

---

From: Bartłomiej Piotrowski <[email-masked]>
Date: Sat, 18 Jul 2015 11:20:59 +0200

On Fri, 17 Jul 2015 15:57:40 -0300
Alan Messias <[email-masked]> wrote:
> Does it mean that *everybody *that asks for write access in
> 'community' will be accepted to git push to it? In case of an 'yes',
> this is the piece that I don't like - I'd vote to everybody continue
> sending patches to 'community' except some approved ones.

Of course not – only regular contributors could apply for
additional permissions. There are many people like that, who maintain
only particular packages. They don't need permissions to modify
packages like kernel, just theirs are of expertise.

> It might be great! But i think we'll need to keep track of packages
> purged just in case someone requests a package that were in staging
> for a period and was purged

Everything resides in git history. No need to worry about it.

> About the 'staging' repo: I like this name but I also like NetBSD's
> idea of *wip* (stands for *w*ork *i*n *p*rogress). Maybe staging
> could be called *wip* repository.

It is not exactly "wip". I just felt that testing is wrong name,
given how many packages rot there for releases. Staging gives clear
indication that we want to know if package works good enough to move it
to "stable" repositories.

BP

---

From: Nathan Angelacos <[email-masked]>
Date: Sun, 19 Jul 2015 20:46:08 -0400

> Everything resides in git history. No need to worry about it.

Yeah, but someone asks for ZorkmidMiner2000, checking git history 
wouldn't be the first place I'd look to see if we have an old APKBUILD.  
See below.
I'm not disagreeing, just saying that its not the first thing I would 
think of.
> to "stable" repositories.

Agreed on the rot thing.  I actually prefer the "unmaintained" 
repository we had for a while.

As Natanael mentions, "Main" is BIG - it doesn't exactly scream "small 
and simple" and that calls into question secure.  But hey - if all of 
those packages are actually the latest upstream.  Cool.

I also like staging - because it means "its about to go somewhere else"  
I'd see staging for edge only - stuff that is up for acceptance.

Community to me says "this is stuff that main developers don't maintain, 
so we don't sign off on it, but someone cares enough to keep it updated, 
at least every other release or so."

I wonder it we create a 4th repo "abandoned" or "unowned"  - That would 
give everyone a clear indication of packages that need an owner.  
Someone new comes along and says "how can I help?"  we point them to 
"unowned" and say "take ownership" - they do? it goes to "community." 
They prove it works, it goes to "staging."   They lose interest in the 
project? Back to "unowned"

Probably we should have some criteria for pushing things from community 
to unowned.  For example: "Upstream source is updated more than 6 months 
ago... move from community to unowned"

So as a user, setting my repositories would be:

main - if it doesn't work, I can file a bug report and it will get fixed
staging - same, but there is a stronger possibility it won't work
community - if it doesn't work, I'll almost definitely have to 1) send 
the patch, or 2) become the maintainer
unowned - here be dragons.  I will almost definitely step up and become 
the maintainer.

----

Of course, I could have read Natanael's proposal all wrong, and he's 
thinking of moving stuff like 
abiword/asterisk/claws-mail/firefox/kamailio/xfce out of "main" and into 
"community." Then Alpine goes back to its "roots" of being a headless 
network appliance.

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 20 Jul 2015 11:26:42 +0200

On Sun, 19 Jul 2015 20:46:08 -0400
Nathan Angelacos <[email-masked]> wrote:

> On 07/18/2015 05:20 AM, Bart*omiej Piotrowski wrote:
> >

I don't have strong opinions regarding "unmaintained" repo. One
argument for removing it would be that we keep the tree cleaner by
remving non-functional stuff.

> project? Back to "unowned"

I kinda like this idea. "unmaintained" or "unowned" could serve as a
final stage before things gets purged. Then we purge stuff things that
has been there for a certain amount of time.

> main - if it doesn't work, I can file a bug report and it will get fixed

And you can expect getting security fixes for those packages for 2 years.

> staging - same, but there is a stronger possibility it won't work

And there will be no packages in "staging" repo in the stable branches,
simply because we don't ship things to stable that is not ready.

> the patch, or 2) become the maintainer

I think you should be able to file a bug and expect it be fixed, even
if it is in community. However, you might not be able to do so for
anything else but edge and latest stable.

The community repo will be available from stable branches too, however,
you may or may not get security fixes.

> unowned - here be dragons.  I will almost definitely step up and become 

And the build servers will not even try build those, so you will have
to build it yourself.

The unmaintained has been a place we push things that gets in the way
for us. Things that does not build, and nobody take the time to fix.
 
> abiword/asterisk/claws-mail/firefox/kamailio/xfce out of "main" and
> into "community." Then Alpine goes back to its "roots" of being a
> headless network appliance.

No, I think you got it right. The examples you mention:

abiword, asterisk, claws-mail kamailio and xfce are good candidates for
"main". They are slow moving projects. They provide fixes for older
stable and they don't give us headache to fix security bugs in older
maintained stable branches.

Firefox on the other hand is an interesting case. They do provide long
term support with the ESR variant, thus, firefox ESR may be a candidate
for "main".

However the normal firefox releases often pull in new libs as
dependencies, or often require newer non-ABI compatible versions of
dependency libs. Maintaining those for long time is a problem, thus, it
belongs to "community".

Other example for typical "community" packages is qemu. They often
don't provide backports of fixes for their older qemu versions and
sometimes I have not been able to provide security fixes for qemu for
older stable branches.

All those packages where upstream say "just use latest git" instead of
providing proper releases are not suitable for "main" and belongs in
"community".

-nc

---

---

From: Isaac Dunham <[email-masked]>
Date: Mon, 20 Jul 2015 11:31:47 -0700

> has been there for a certain amount of time.

Overall, the "four branches" proposal does sound good, but there are some
points where I'm not sure it makes sense or would like clarification.

* Currently, the aports tree has four subdirectories:
 main
 non-free
 testing
 unmaintained
I assume that the proposal is not deleting non-free, but merely ignoring
it because it's not in the main flow of packages.
(By the way...I've been thinking that I'd like to package "xephem", but
it needs to go in non-free, and I'm not sure where it should start.)

* If I'm reading the latest proposal right, the package flow would be:
new or fresh from unmaintained -> community
main developers adopt: community -> staging
becomes stable: staging -> main

Further, the proposal is to have "community" and "main" as the only ones
that are part of a release.
I don't think this makes sense, given the flow:
 - A newly fixed or repaired package should probably spend *at least* one
   revision in a repository where it's available for testing but not part
   of the release, regardless whether the core maintainers adopted it.
   In other words, it's good to have something before community.
 - Why would a core maintainer adopting a "community" package result in it
   moving out of a released repository before it moves to main?

In short, I'm thinking that switching the order of community and staging
seems better:
-Any package that someone makes build goes in "staging", regardless who
the packager is; so do proposed major updates.
-When someone can confirm that the package as built works, it goes into
community. (We might have a maintainer requirement.)
-When a core developer is happy with the packaging and how it works, and
is willing to handle support, it can go into main.
-If a package outside main ceases to build, it drops to "unmaintained";
if a package in main ceases to build, the maintainer may move it to
unmaintained if they don't wish to deal with fixing it; if a maintainer
announces intent to orphan a package, it will be moved to unmaintained
if it isn't adopted before the next release.

For what it's worth, "contrib" would be more traditional than
"community".

> > ago... move from community to unowned"

That's a bit naive.
I use a window manager that last saw a release a little under 5 years ago,
but it's still quite functional.

I'd say it's necessary to add some qualification such as 
"homepage or download gone" or "no longer builds".

(By the way...the sourceforge-hosted packages may be unbuildable for a while.)

Thanks,
Isaac Dunham

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 20 Jul 2015 22:44:05 +0200

On Mon, 20 Jul 2015 11:31:47 -0700

> it needs to go in non-free, and I'm not sure where it should start.)

Yes non-free would stay as is. Here we put stuff that we are not
allowed to provide binaries for, but still want make it easy for people
to build a package themselves if they want.

> becomes stable: staging -> main

sorry for being unclear. What I propose is that the flow is:

new package -> staging.

Depending on who ends up maintaining it and for how long, it goes from
staging to either "community" or "main".

That is, if there is a package that i do use and don't mind maintain
(for example qemu/libvirt/virt-manager), but I don't can (or want)
provide security fixes for 2 years, then I would add myself as
maintainer and place it in "community".

> that are part of a release.

Currently we have only "main" as a part of a release. The proposal is
that we do both "main" and "community"

>    In other words, it's good to have something before community.

What I had in mind was that "staging" could be entry point for all new
packages. That way we could bring in a new package into the system, and
test it a bit before deciding what kind of support we want give it.

>    moving out of a released repository before it moves to main?

If a core maintainer is prepared to support it for 2 years then it is
qualified for "main". A core developer might want maintain the package
but as more a best-effort, then it would still be in "community". You
cannot expect security fixes in more than latest stable branch.

> the packager is; so do proposed major updates.

I don't mind pushing major updates directly to edge/main if the plan is
to include this in next major release.

The problem with always pushing new major releases of a package to
"staging" first is that an ABI breakage might require rebuild of *many*
packages. I would prefer not copy all the packages that needs rebuild
to staging, build them there and then copy them back to main (or
"community").

I do see the value of it, but I think it costs more work. What I am
trying to do is reduce amount of work.

We have the recent example with udev -> eudev. I did push it to testing
first (proposed to be renamed to "staging") but I did not copy all the
50+ packages that needed rebuild. Doing so would have prevented
breakages for edge users (unless you had edge/testing in your repo
ofcourse). They could been there for a a week or two while sorting out
the issues before moving it to main.

But that would also meant that none of the updates, minor or major of
any of those 50 packages would go to main til all eudev issues was
resolved or I would have to update same package twice. That is what I
wanted avoid.

> is willing to handle support, it can go into main.

when someone confirm it works it could got to either main or community,
depending on who wants be maintainer and depending on what kind of
support the maintainer is prepared to provide for it.

I don't think there is any point that a package with good support needs
to go via community before going to main.

> if it isn't adopted before the next release.

The idea was, if a maintainer is no longer interested in maintaining a
package it would go to "unmaintained" as an orphaned package.

I do like the idea of purging those when preparing next release. We
should probably clean the "unmaintained" directory before starting the
builders for a new release because when building a release is a time
when you typically discover that a package no longer builds - and a
time when maintainers are already super busy.

> For what it's worth, "contrib" would be more traditional than

Might be. But I think core developers might want be interested in
maintaining some of those packages. At least I am.

What i think is:
main = longer term support, 2 years.
community (or contrib) = only latest stable is supported. older than
that are best-effort only.

> "homepage or download gone" or "no longer builds".

I agree. I would not mind maintaining a package "main" that has not got
any new release in 5+ years. Specially if upstream still provide
security fixes in a git branch, even if they don't do new releases.

---

---

From: Isaac Dunham <[email-masked]>
Date: Mon, 20 Jul 2015 16:24:02 -0700

> to build a package themselves if they want.

Ah, I'd somewhat misunderstood the purpose, but that's probably still
the right place for it.
I assumed that non-free was more like Debian: "here's a bunch of
packages we can distribute, but you'd better check the license to make
sure you can use them."
The exact terms and conditions of the software state:
 1. Permission to build and run XEphem from this source code is granted only
 for personal use or for bona fide educational situations or research funded
 by public funds. Permission is expressly prohibited in commercial or military
 situations without prior agreement.
 ...
 4. Redistribution rights: You may redistribute XEphem source code only in its
 entirety and without modification. You may redistribute binaries only if they
 were built from the original source code, or from source code with very minor
 changes made for the purpose of porting and not for the purposes of changing
 functionality.

> maintainer and place it in "community".

I'd gotten that impression from reading your comments, but Nathan's comments
seemed to go the other way.
I don't see any major discrepancies between your proposal and what I
had been thinking of, though I'm not clear on exactly when or whether
packages get deleted from unmaintained.

I'd think some packages should get more leeway than others, depending
on things like whether dependencies build, how much patching they already
have, how far behind they are, and how complicated the build recipe is.
If it's a trivial recipe and it no longer builds, a short stay in 
unmaintained would be good (so those of us who occasionally glance
through there and fix up a package or two get a chance to notice it).
If it's harder to package, a longer stay would be good so that the work
already put in doesn't have to get duplicated...until the package falls
behind. Then it should probably get purged, since forwards-porting a
bunch of workarounds is usually hard.

Thanks,
Isaac Dunham

---

---

From: "V.Krishn" <[email-masked]>
Date: Tue, 28 Jul 2015 18:31:59 +0530

> has been there for a certain amount of time.

I also notice some pkgs from main/ getting purged directly.
If such process is adopted it would be better if it got pushed to "abandoned" 
or "unowned" which will mark it as something that will/maybe be dropped off 
aports.

-- 
Regards.
V.Krishn

---

---

From: Timo Teras <[email-masked]>
Date: Sun, 19 Jul 2015 21:18:58 +0300

On Fri, 17 Jul 2015 16:59:16 +0200
Natanael Copa <[email-masked]> wrote:

> that I only want give best-effort support for security fixes instead
> of supporting it for 2 years.
> 
>   All the extra packages that community maintains. This repository
> will be available in stable releases but we only provide best-effort
> We would have a gitolite group, 'maindevs' and another
> 'communitydevs'. People in community group may git push to
> 'community' repo but not 'main' repo. This is so that maintainers can
> push packages themselves, without needing to send patches, like they
> currently do. The recent move to gitolite makes this possible.
> 

Sounds Good.

Might be an idea to improve apk-tools at same time a bit. I'd prefer to
not need to edit repositories when doing upgrade. Would be nice if we
shipped all the mirrors in etc/apk/repositories.d/alpine.list or
similar. And then have config to select the correct branch and
preferred mirror, or similar. This would allow to great dist-upgrade or
similar that would know the latest stable branch and upgrade config to
that.

Though, not sure if this simplification or complication.

---