Subject: mailing lists, processes, modernization

From: "Ariadne Conill" <[email-masked]>
Date: Sat, 18 Apr 2020 03:48:25 +0000

Hello,

When Alpine was first started, we had a single mailing list, alpine-devel,
which was running on ezmlm.  This served us well for many years, but a few
years ago, we started to realize that we needed to modernize the software
being used in order to mitigate various security and spam issues with the
ezmlm setup.

This lead to Drew DeVault proposing that Alpine use his Sourcehut platform
for mailing lists, which we have used for the past 8 months.  Initially,
this was seen as a good solution for the original problems we were having
with the ezmlm-based setup, but some have discontinued using the mailing
lists, in some cases leaving the project entirely due to issues the
Sourcehut software has processing HTML e-mail, which caused incompatibility
with popular e-mail clients such as the Android GMail client and the iOS
EMail client, as well as differences in behavior verses the earlier mailing
list system involving the Sourcehut solution not mailing a copy of the
message back to its author.

The reduced participation on the mailing lists, and thus reduced adherence
to procedure for coordinating system changes have reduced its usefulness as
single source of truth for tracking system changes.

Unfortunately, this brings us to a crossroads: mailing lists that are
boycotted by participants are not a useful communications tool for a
project, because of the source of truth issue.  In addition, the infra
team has expressed a desire to migrate all applications they manage to
run under Docker, a desire not shared by Drew for the Sourcehut
infrastructure.

Since adoption of the Sourcehut mailing list platform, we have also
implemented GitLab as a Git forge.  GitLab features high quality issue
tracking and integrates with e-mail.  This leads us to my proposal to
replace development mailing lists with GitLab issues and merge requests.

Initially, we discussed the aports mailing list.  This has been deprecated
and replaced with the new mkmr tool[1].  Many users are now using the mkmr
tool and that effort has been considered largely successful.

So, now we talk about alpine-devel.  In most cases, it is believed that
change proposals could simply be tracked and commented on by filing issues
against the aports project in GitLab.  For larger meta-issues, it is proposed
that we create a project for meta-issues.  This replaces the working
side of alpine-devel with a more efficient workflow and restores the single
source of truth for system changes.

For the support inquiries that get sent to alpine-devel and alpine-user,
I propose that we use Discourse forums instead.  Discourse is more accessible
to the average user, featuring mobile apps and tons of features.  Discourse
also supports e-mail for people who still desire to interact with the system
using e-mail, but the support is limited to replying and creating new threads.
This is believed to be adequate for die-hard e-mail users, however.

That leads us to decommissioning of the Sourcehut platform.  We need some
mechanism to maintain the list archives for historical reasons, as the
mailing lists served as a single source of truth for a long time.  As the
Sourcehut software is a mostly typical Flask application, it seems plausible
that we could write a script to dump the mailing list archives as static
HTML pages.  The Sourcehut software could then be decommissioned, and
maintaining the mailing list archive would be of minimal impact to the
infra team.

Thoughts?

Ariadne

---

From: David Demelier <[email-masked]>
Date: Sun, 19 Apr 2020 11:04:13 +0200

> message back to its author.

I personally use mlmmj [0], it's pretty minimalist and simple to use.
Could be nice to use to replace this one at least for other lists. And
at least it does one basic thing for a mailing list, send your own email
back.
 
> run under Docker, a desire not shared by Drew for the Sourcehut

I think it's useful to discuss new problematics but not for sending
patches. For example, proposals, help for development and such.

> This is believed to be adequate for die-hard e-mail users, however.

Here I strongly suggest to not use discourse. It's very painful to use
and its interface is not intuitive. It has a mailing-list mode but it
still hard to use and not very handy.

Regards,

[0]: http://mlmmj.org

-- 
David

---

From: "Ariadne Conill" <[email-masked]>
Date: Sun, 19 Apr 2020 09:22:30 +0000

Hello,

> at least it does one basic thing for a mailing list, send your own email

mlmmj is a non-starter for us.  The whole reason why we wound up on
Sourcehut's implementation to begin with is because the infra team no
longer wished to maintain mlmmj due to security concerns.

> patches. For example, proposals, help for development and such.

Any mechanism, including GitLab issues, can be used to track proposals,
likely with more efficiency than e-mail.  The proposal does not involve
using Discourse for development discussions, that would be replaced by
GitLab issues.  Discourse would be primarily used for user-facing
support.

Many other distributions and projects have begun to use issue tracking
in this way.  It is certainly not novel to this proposal.

> still hard to use and not very handy.

The alternative would be to use a traditional forum package.  Users
are not using the Alpine lists for support, instead using third-party
solutions like StackExchange, because users do not wish to interact
with mailing lists to have their question solved.  Discourse is
particularly attractive here *because* it can support the Q&A format
that users have become accustomed to for support.  That it also
supports a mailing list mode is a nice feature, but ultimately not
the primary reason for the suggestion.

Ariadne

---

From: Wolf <[email-masked]>
Date: Mon, 20 Apr 2020 21:17:59 +0200

Hello,

On 2020-04-19 09:22:30 +0000, Ariadne Conill wrote:
> [..]
> The alternative would be to use a traditional forum package.

I personally would definitely prefer that alternative. Well, I still
prefer the mailing list, but if that is off the table, traditional
bulletin board would be my next choice.

> [..]

> Discourse is particularly attractive here *because* it can support the
> Q&A format that users have become accustomed to for support.

Are we talking about self-hosting Discourse? Or about using the hosted
solution (where I would assume Discourse's Terms of Service would
apply)?

W.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

---

From: Ariadne Conill <[email-masked]>
Date: Mon, 20 Apr 2020 13:23:01 -0600

Hello,

> bulletin board would be my next choice.

Well, the problem is that we do not own our own support
pipeline right now.  Yes, there is the mailing lists, but almost
nobody is using them.  Instead, they go to third-party websites
like StackExchange.  We would like to own our support
pipeline, so that we can ensure it continues to exist regardless
of what happens to some otherwise unrelated company.

> solution (where I would assume Discourse's Terms of Service would

This would be a self-hosted solution, managed, presumably,
by the infra team, in lieu of the mailing lists.

Ariadne

---

From: j3s <[email-masked]>
Date: Mon, 20 Apr 2020 15:38:23 -0500

Please tell me if I'm overstepping, but as a (very happy) Alpine user, I 
feel obligated to speak up here.

> of what happens to some otherwise unrelated company.

My opinion is that this isn't going to happen regardless of the software 
that the Alpine project sets up. People like StackExchange, and if those 
users were interested in hearing from the Alpine folks, they'd probably 
be using the mailing list today.

Currently, it takes 3 clicks and two distinct webpages to get to the 
support mailing list. I bet that having a "support" header on your home 
page that provides a mailto: link would drive as many people to the 
mailing list as standing up Discourse would.

I guess my point is twofold; I don't think you can "own" the support 
pipeline, and I don't think that lack of a shiny forum is the solution 
to what I perceive as a UX problem.

j3s

---

From: Consus <[email-masked]>
Date: Tue, 21 Apr 2020 00:16:47 +0300

On Mon, Apr 20, 2020 at 03:38:23PM -0500, j3s wrote:
> users were interested in hearing from the Alpine folks, they'd probably be
> using the mailing list today.
> 
> Currently, it takes 3 clicks and two distinct webpages to get to the support
> mailing list. I bet that having a "support" header on your home page that
> provides a mailto: link would drive as many people to the mailing list as
> standing up Discourse would.
> 
> pipeline, and I don't think that lack of a shiny forum is the solution to
> what I perceive as a UX problem.

This whole thing boils down to UX. Writing an email to the mailing list
is much harder than writing a post to reddit, mostly due to UX issues.

E.g. it's impossible to write a plaintext email in my mobile mail client
at all.

More to that, mobile clients mostly suck at displaying mailing lists due
to the hard newlines and 80 characters. You need a really BIG screen in
order to display an email properly. Yes, this looks ugly and makes
things harder to read.

What makes the situation even more complicated is that by default all
mail is coming to your inbox. And there is no standard protocol across
mail clients, you'll probably have to login to the web UI and setup a
filter there.  But first you need an example email to learn filter
conditions.

So joining a mailing list becomes kind of a big deal. You have to setup
a mailing list account, you have to configure your inbox filters, you
have to make sure that your client can even produce plaintext emails...

At this point most people just go to reddit and ask questions there.

---

From: Ariadne Conill <[email-masked]>
Date: Tue, 21 Apr 2020 00:22:56 -0600

Hello,

> > using the mailing list today.

There are many people in the Alpine community that post on
StackExchange and similar websites.  People *do* want to hear
from "the Alpine folks," just not in the form of e-mail.

> > mailing list as standing up Discourse would.

I really doubt it.  Interacting with mailing lists is inconvenient,
and the Sourcehut software has exacerbated that inconvenience
by being incompatible with the most commonly used e-mail
clients in 2020 (namely the ones built into your phone), due to
Drew's decision to fight political battles and use our mailing
lists (as well as anyone else who uses Sourcehut's mailing lists)
as a weapon in that fight.

If the mailing lists were unpopular in the ezmlm/mlmmj days, they
are basically terminally ill at this point.

> > what I perceive as a UX problem.

We can certainly do a better job of it than what we are doing now.
This argument essentially boils down to "I like e-mail, so we shouldn't
try to do something else because it may not work."
 
> At this point most people just go to reddit and ask questions there.

Right and this is really my point.  People were unhappy with the mailing
list solution we offer before migration to Sourcehut, the additional
restrictions added by the Sourcehut software have exacerbated that
unhappiness.

We need to focus on fixing our communications tools.  They were
unpopular before the Sourcehut migration, and they are less popular
now.  Asking the infra team to maintain resources that month over
month are seeing a continued decline in utilization because people
are using alternative tools (critical development discussion in IRC,
support on StackExchange, etc) does not make sense.

Ariadne

---

From: Rasmus Thomsen <[email-masked]>
Date: Wed, 22 Apr 2020 09:37:02 +0000

>are basically terminally ill at this point.

Yes, the reply rate on our MLs is rather low for most mails (well, basically all but the ones where we discuss shutting the ML down...). I personally don't see a future for the MLs either.

Regards,

Rasmus

---

From: David Demelier <[email-masked]>
Date: Tue, 21 Apr 2020 08:51:37 +0200

On Tue, Apr 21, 2020 at 12:16:47AM +0300, Consus wrote:
> mail clients

Yes, there is one it's called Sieve [0]. I personally use sieve-connect
to edit my rules from command line.

[0]: https://en.wikipedia.org/wiki/Sieve_(mail_filtering_language)

-- 
David

---

From: Consus <[email-masked]>
Date: Tue, 21 Apr 2020 10:04:48 +0300

> to edit my rules from command line.

By "standard" I don't mean that there is an RFC, I mean that there is a
well-known and accepted by most implementations way to do things. And
Sieve is no such thing. GMail does not support it, my mobile email
client does not support it. Stock Google Mail client does not support
it. Even mutt does not support it :)

---

From: Nathan Angelacos <[email-masked]>
Date: Sat, 18 Apr 2020 09:16:52 -0400

Hi,

On Sat, 2020-04-18 at 03:48 +0000, Ariadne Conill wrote:
> Hello,
> 
> When Alpine was first started we had a single mailing list, alpine-
> devel,which was running on ezmlm.

<snip>

> Initially, we discussed the aports mailing list.  This has been
> deprecated and replaced with the new mkmr tool[1].  Many users are
> now using the mkmr tool and that effort has been considered largely
> successful.

<snip>

> For the support inquiries that get sent to alpine-devel and alpine-
> user, I propose that we use Discourse forums instead.  

<snip>

> it seems plausible that we could write a script to dump the mailing
> list archives as static HTML pages.  
> 
> Thoughts?

As the person who set up the original ezmlm lists, and someone who is a
"die-hard email user",  I don't see a problem with this proposal.

You addressed one of my pain points with mentioning the mkmr tool,
which, to be honest, I was not aware of.  I certainly don't mind
learning new tools, or doing things differently, as long as there's
some coaching for us old "grey-beards" :)

I'm not a huge fan of Discourse, but I agree that it would probably 
ease the friction for new users to get assistance.  And that's the goal
at this point, right?

And yes, we want to keep the old mailing lists for archival.

Thumbs up from here.

---

From: Leo <[email-masked]>
Date: Sat, 18 Apr 2020 11:55:08 -0300

On Sat, 18 Apr 2020 03:48:25 +0000
"Ariadne Conill" <[email-masked]> wrote:

> Hello,
> 
> When Alpine was first started, we had a single mailing list,
> alpine-devel, which was running on ezmlm.  This served us well for
> many years, but a few years ago, we started to realize that we needed
> to modernize the software being used in order to mitigate various
> security and spam issues with the ezmlm setup.
> 
> This lead to Drew DeVault proposing that Alpine use his Sourcehut
> platform for mailing lists, which we have used for the past 8 months.
>  Initially, this was seen as a good solution for the original
> problems we were having with the ezmlm-based setup, but some have
> discontinued using the mailing lists, in some cases leaving the
> project entirely due to issues the Sourcehut software has processing
> HTML e-mail, which caused incompatibility with popular e-mail clients
> such as the Android GMail client and the iOS EMail client, as well as
> differences in behavior verses the earlier mailing list system
> involving the Sourcehut solution not mailing a copy of the message
> back to its author.
> 
> The reduced participation on the mailing lists, and thus reduced
> adherence to procedure for coordinating system changes have reduced
> its usefulness as single source of truth for tracking system changes.
> 

Correct me if I'm wrong (hopefully I am wrong), but e-mail integration
is only for Enterprise Edition.

> now using the mkmr tool and that effort has been considered largely

I'm the author of the mentioned tool and i can say it has basic
functionality mostly working (with a few quirks), you can create (mkmr),
merge (mgmr), and edit (programmatically and interactively, edmr, vimr)
a Merge Request's attributes.

There are a few things missing that are super useful for people used to
SourceHut and E-Mail workflow in general:

- List Merge Requests open
- Comment on Merge Requests and its threads (also resolve them)
- List changes of a merge request before merging
- Checkout a merge request in a local branch

There is also no tooling to deal with issues (comment on an issue,
open/close an issue)

When the above are complete people will be able to have a
decent to good experience contributing to Alpine Linux with only a CLI

> So, now we talk about alpine-devel.  In most cases, it is believed
> that change proposals could simply be tracked and commented on by
> filing issues against the aports project in GitLab.  For larger
> meta-issues, it is proposed that we create a project for meta-issues.
>  This replaces the working side of alpine-devel with a more efficient
> workflow and restores the single source of truth for system changes.
> 
> For the support inquiries that get sent to alpine-devel and
> alpine-user, I propose that we use Discourse forums instead.
> Discourse is more accessible to the average user, featuring mobile
> apps and tons of features.  Discourse also supports e-mail for people
> who still desire to interact with the system using e-mail, but the
> support is limited to replying and creating new threads. This is
> believed to be adequate for die-hard e-mail users, however.
> 

About the email support, i don't see that as a big problem since that
is basically what e-mail can do at all

> That leads us to decommissioning of the Sourcehut platform.  We need
> some mechanism to maintain the list archives for historical reasons,
> as the mailing lists served as a single source of truth for a long
> time.  As the Sourcehut software is a mostly typical Flask
> application, it seems plausible that we could write a script to dump
> the mailing list archives as static HTML pages.  The Sourcehut
> software could then be decommissioned, and maintaining the mailing
> list archive would be of minimal impact to the infra team.
> 
> Thoughts?
> 
> Ariadne

---

From: "Ariadne Conill" <[email-masked]>
Date: Sat, 18 Apr 2020 17:57:03 +0000

Hello,

> is only for Enterprise Edition.

The FOSS version of GitLab supports e-mail integration for replies, but
not e-mail integration for submitting patches.

See also: https://docs.gitlab.com/ce/administration/reply_by_email.html

Ariadne

---

From: Ariadne Conill <[email-masked]>
Date: Wed, 22 Apr 2020 08:51:08 -0600

Hello,

I would like to thank everyone for contributing their thoughts and general
discussion to this latest round of discussions about the current situation
with our mailing lists.

It seems that we have mostly gained consensus for using GitLab to track
planning-related development discussions.  Now, we will need to figure
out how best to accomplish this.  I appreciate everyone's patience while
we work through ideas and come to a consensus on how the GitLab-based
development planning will work.  There are a few possibilities that have
been discussed in IRC and elsewhere already, that I will outline in another
thread.

As you are likely aware, the initial position of this round was to propose
using GitLab for planning (which allows all sorts of advantages, such as
assigning deadlines and milestones to system change proposals), and to
move all other discussion to Discourse.

Well, feedback on using Discourse instead of a traditional mailing list
solution was quite mixed, but most contributors that contributed to
this discussion appreciated the goal of having a system that allowed
users to post via either e-mail or the web.

Which leads us to an alternative proposal for the mailing lists: adopting
mailman3 and hyperkitty.  Hyperkitty is a frontend to Mailman3 which
provides many of the same features we were looking for in Discourse.
If you're interested about the web experience, the fedora mailing lists
system uses Hyperkitty for user-facing components, which you can see
at https://lists.fedoraproject.org/.

In this case, we will still need to write a dumper for Sourcehut to convert
our current archives to mbox format so they can be imported into
Hyperkitty.  We will also need to migrate the subscriptions, obviously,
but both of these tasks should not prove that difficult.

Please let us know your thoughts on Discourse verses Hyperkitty as
a path forward for the remaining discussion that would not be well
served by using GitLab.

Thanks everyone,
Ariadne

---

From: Rasmus Thomsen <[email-masked]>
Date: Wed, 22 Apr 2020 17:42:19 +0200

Hello,

> at https://lists.fedoraproject.org/.

Sounds good to me and Hyperkitty seems to be battle-tested by the
Fedora project already so that's nice. Are there other users of
Hyperkitty?

Thanks,

Rasmus

---

From: Max Rees <[email-masked]>
Date: Wed, 22 Apr 2020 17:44:08 -0500

> Fedora project already so that's nice. Are there other users of

Adélie Linux has also been using Mailman 3 / Postorius / Hyperkitty for
some time now. However, since our team is quite small the mailing lists
get much less use than IRC in terms of development, and for support we
primarily see activity on our support channel and Reddit. But for what
it's worth we've had only a few problems with the whole setup and I'd
say it's worked quite well.

https://lists.adelielinux.org

Max

---

From: j3s <[email-masked]>
Date: Wed, 22 Apr 2020 11:06:47 -0500

> a path forward for the remaining discussion that would not be well

One benefit of Discourse is that posts could be made from either the web 
UI or email, improving accessibility & UX. With Hyperkitty, posting from 
the web is removed - and email would once again become the de-facto way 
of posting.

If we are concerned about meeting users where they are - which in many 
cases, is the web - I think it would be ill-advised to remove that option.

---

From: Wolf <[email-masked]>
Date: Wed, 22 Apr 2020 18:21:04 +0200

Hello,

On 2020-04-22 11:06:47 -0500, j3s wrote:
> One benefit of Discourse is that posts could be made from either the web UI
> or email, improving accessibility & UX. With Hyperkitty, posting from the
> web is removed

This does not seem to be true. At least on the lists.fedoraproject.org
it *is* possible to post via the web UI.

W.

---

From: Ariadne Conill <[email-masked]>
Date: Wed, 22 Apr 2020 10:27:21 -0600

Hello,

On Wednesday, April 22, 2020 10:21:04 AM MDT Wolf wrote:
> > UI

As Wolf mentioned, modern Hyperkitty versions can post from the web UI, which 
is why it was proposed as an alternative to Discourse that allows us to meet 
the same goal of allowing web and e-mail users to make use of the same 
communications channel.

Ariadne

---

From: Wolf <[email-masked]>
Date: Wed, 22 Apr 2020 19:06:08 +0200

> the same goal of allowing web and e-mail users to make use of the same 

IMO it seems like a reasonable compromise.

At the same time, I do not know how hard it would be to add the same
option to sourcehut. Or if ddevault would be willing to. But if yes, it
would achieve the same goal without having to migrate anything.

W.

---

From: Ariadne Conill <[email-masked]>
Date: Wed, 22 Apr 2020 14:07:43 -0600

Hello,

On Wednesday, April 22, 2020 11:06:08 AM MDT Wolf wrote:
> > As Wolf mentioned, modern Hyperkitty versions can post from the web UI,
> > which is why it was proposed as an alternative to Discourse that allows
> > us to meet the same goal of allowing web and e-mail users to make use of
> > the same communications channel.
> 

Unfortunately, as a reminder, due to the way Sourcehut processes mail from 
several mail clients, including the default iOS and Android clients (rejecting 
such mail), we are going to have to switch anyway.  This has caused several 
contributors to stop contributing to the project already because they find 
being scolded by our mailing list server when they reply to an inquiry from 
their phone to be unacceptable.

While it is true that the Sourcehut mailing list software now has the 
capability to accept HTML e-mail, it requires that there be a text/plain part 
as well.  This unfortunately breaks both the iOS and Android e-mail clients, 
resulting in the mailing list advising you to read a website that simply 
suggests using an alternative client.

Asking people to make significant changes to their workflow simply to 
participate on our lists is obviously absurd, and that's where this process 
started six months ago.  Had we known last year that the Sourcehut software 
would be incompatible with commonly used e-mail software, such as the e-mail 
clients built in to iOS and Android, we would not have moved forward with the 
switch.

Ariadne

---

From: j3s <[email-masked]>
Date: Wed, 22 Apr 2020 16:19:14 -0500

> On Apr 22, 2020, at 3:08 PM, Ariadne Conill
> such mail)

For what it’s worth, I’m sending this email from the default iOS mail client.

---

From: Ariadne Conill <[email-masked]>
Date: Wed, 22 Apr 2020 15:23:09 -0600

Hello,

On Wednesday, April 22, 2020 3:19:14 PM MDT j3s wrote:
> > several mail clients, including the default iOS and Android clients
> > (rejecting such mail)
> 

iOS I guess sends text/plain and text/html parts.  Android default client 
(GMail) only sends text/html.  Originally, both were issues.

Ariadne

---

From: Consus <[email-masked]>
Date: Thu, 23 Apr 2020 09:58:50 +0300

> at https://lists.fedoraproject.org/.

Discourse is kinda everywhere now, so users will be more familiar with
it. If email communication is going to slowly fade away, it would make
more sense to use tool that will last longer.