Subject: [alpine-devel] Improving Alpine Linux documentation

From: Consus <[email-masked]>
Date: Tue, 12 Dec 2017 23:13:05 +0300

Hello,

I'm writing this email to start an archived discussion about the future
of the Alpine Linux documentation.

Earlier this day we (clandmeter, _ikke_, ncopa, TBK and your humble
author) were discussing on IRC (check out the new #alpine-docs channel)
how the current state of the project documentation can be improved.

For now the primary source of knowledge is the MediaWiki running on
https://wiki.alpinelinux.org. It does contain a lot of useful info,
however it lacks proper structure, the spam level is very high and
newcomers often get banned during the infamous grace period after the
account creation.

The other problem with the publicly edited Wiki (mentioned by Natanael)
is that one cannot fully trust the content. So malicious edits in Wiki
sections describing policies, BTC wallets for donations or secure
network configurations can harm the project. The question if existing
security methods (user groups with different permissions, ability to
lock a Wiki page) are enough to address this issue is open for
discussion.

We currently have (at least) three proposals:

1. The MediaWiki stays as it is (we just need to add the proper
   structure and harden it against the Evil Bot Horde).

2. We move the core project documentation somewhere else
   (https://alpinelinux.org or https://docs.alpinelinux.org) but keep
   Wiki in place for the community. This way the core project
   documentation is stored somewhere safe (probably in Git) and GitHub
   pull-requests or patches via ML are used to update it.

3. We completely move to some sort of https://docs.alpinelinux.org
   (no matter in it's current form or not) and all user contributions.
   Hardcore variant of #2.

Of course, every proposal have have it's own Pros and Cons.

The main concern with Git-based documentation solutions is that the
standalone patch review system should be rolled out (patchworks,
ReviewBoard, you name it) because as (AFAIK) clandmeter stated we cannot
fully rely on GitHub. However, it gives the developers the full control
over the content

It worth mentioning that the problems with Wiki can be addressed in a
number of ways.

First of all, currently Alpine Wiki does not require a email address to
create an account. Requiring a validated email address should reduce the
spam level considerably.

Sensitive information (like PGP keys, donation options, etc) stored in
the Wiki can be restricted for developer accounts only. If user finds an
error in the document he can raise an issue in the Discussion tab.

Please, share your opinions on this topic so we can improve the state of
things.

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Drew DeVault <[email-masked]>
Date: Tue, 12 Dec 2017 15:23:02 -0500

I don't really have any advice for what to do with the wiki, but
regarding Alpine's core documentation I have some thoughts. Something
that bothers me about Alpine (and that I'd like to contribute to
improving) is the lack of man pages for Alpine core stuff. I think a
substantial portion of the core docs can and should be done this way,
rather than on the wiki - though it would be nice to render the man
pages to HTML for display on the web, too.

No one likes writing roff so I've been working on a tool recently to
help with this:

https://git.sr.ht/~sircmpwn/scdoc

If there's no opposition to this I was hoping to start writing man pages
for apk shortly.

---

---

From: Consus <[email-masked]>
Date: Tue, 12 Dec 2017 23:26:26 +0300

> If there's no opposition to this I was hoping to start writing man pages

Try http://mdocml.bsd.lv. It is used by the OpenBSD project. I was writing man pages
using it myself, it's pretty okay.

---

---

From: Drew DeVault <[email-masked]>
Date: Tue, 12 Dec 2017 15:31:47 -0500

> using it myself, it's pretty okay.

mdocml is basically roff with lipstick on, I don't like it much more
than I liked roff. There are other man page generators, too, but they're
almost all written in high level languages and require some kind of
runtime (god forbid, XSLT). The new tool I'm working on is a small POSIX
C99 tool with no dependencies. Its syntax is much easier to write, read,
and maintain than mdoc.

---

---

From: Consus <[email-masked]>
Date: Tue, 12 Dec 2017 23:34:56 +0300

> C99 tool with no dependencies. Its syntax is much easier to write, read,

True. But this is more about the high-level documentation, not the
tool's synopsis.

---

---

From: "A. Wilcox" <[email-masked]>
Date: Tue, 12 Dec 2017 15:40:50 -0600

On 12/12/17 14:34, Consus wrote:
>>> Try http://mdocml.bsd.lv. It is used by the OpenBSD project. I 
>>> was writing man pages using it myself, it's pretty okay.
>> 
>> mdocml is basically roff with lipstick on, I don't like it much 
>> more than I liked roff. There are other man page generators,
>> too, but they're almost all written in high level languages and
>> require some kind of runtime (god forbid, XSLT). The new tool I'm
>> working on is a small POSIX C99 tool with no dependencies. Its
>> syntax is much easier to write, read, and maintain than mdoc.
> 

(Haha, I didn't even read this part of the thread before sending my
last mail.)

- -- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
http://adelielinux.org

---

---

From: Drew DeVault <[email-masked]>
Date: Tue, 12 Dec 2017 16:53:15 -0500

> True. But this is more about the high-level documentation, not the

Man pages are not limited to the synopsis of tools. Large portions of
information currently on the wiki is on-topic for man pages:

https://wiki.alpinelinux.org/wiki/Alpine_Linux_package_management

apk(8)

https://wiki.alpinelinux.org/wiki/Apk_spec

apk-database(7)

https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package
https://wiki.alpinelinux.org/wiki/APKBUILD_Reference

abuild(1)

APKBUILD(7)

and so on. If an overhaul of the core docs on the wiki is due, then I
think lots of it should be moved into man pages as the single source of
truth.

---

---

From: William Pitcock <[email-masked]>
Date: Tue, 12 Dec 2017 16:08:44 -0600

Hi,

> think lots of it should be moved into man pages as the single source of

I agree, with the exception that the apk-related manpages should be
written in mdoc (as I said elsewhere).  I think it makes sense also to
be consistent with that policy in apk for abuild, too.

William

---

---

From: William Pitcock <[email-masked]>
Date: Tue, 12 Dec 2017 16:05:24 -0600

Hi,

> No one likes writing roff so I've been working on a tool recently to

I don't mind writing roff with the mdoc macro package.  It seems fine to me.

> If there's no opposition to this I was hoping to start writing man pages

We prefer to have apk's documentation in mdoc, since it does not
require any preprocessing, but maybe your tool can be useful for other
Alpine projects.

William

---

---

From: Drew DeVault <[email-masked]>
Date: Tue, 12 Dec 2017 17:30:49 -0500

> require any preprocessing, but maybe your tool can be useful for other

Summarizing the discussion from IRC on the ML:

mdoc requires either preprocessing or an extra runtime dependency.
Saying mdoc doesn't require preprocessing is like saying JVM bytecode
doesn't require compilation.

scdoc is a very light compile-time dependency and is extremely portable.
The syntax is much friendlier than mdoc (which is unavoidable as long as
mdoc is based on roff), which makes it easier for more people to
contribute. mdoc/roff syntax _has_ been a significant barrier for people
to contribute to projects that use them in the past.

It seems like it's been set in some of your minds that mdoc is the way
forward. I don't think it's a very rational choice.

--
Drew DeVault

---

---

From: Kiyoshi Aman <[email-masked]>
Date: Tue, 12 Dec 2017 22:59:43 +0000

Hi,

As an end user, I really couldn't care less how the documentation is
generated. It needs to be complete, thorough, readable for plebes such as
myself, and available. (It also needs to be compatible with enduser
toolchains like mandb and mdoc, since those are the man providers used by
Adélie and Alpine, respectively.)

Please stop squabbling over manpage backend formats and focus on the
relevant issues. Namely, what to do about the wiki, what documentation
needs to be provided and where, and so on.

Thanks,
Kiyoshi

> forward. I don't think it's a very rational choice.
-- Kiyoshi Aman

---

From: William Pitcock <[email-masked]>
Date: Tue, 12 Dec 2017 17:00:01 -0600

Hello,

> doesn't require compilation.

That is not an accurate summarization.  Alpine's man command includes
the mdoc macro package.

> to contribute to projects that use them in the past.

In the case of apk and probably also abuild, it makes most sense that
the documentation be written by the maintainers of that software.
Accordingly, it makes sense that they should use the tools they wish
to use, which in the case of apk, would be mdoc.

> forward. I don't think it's a very rational choice.

Finally, as somebody who is involved in maintenance of apk, I am
stating that I intend to write a manpage for apk at some point during
the 3.8 development cycle.  I just need to get pkgconf 1.4 out the
door first, which ironically, is also blocking on docs right now.

William

---

---

From: "A. Wilcox" <[email-masked]>
Date: Tue, 12 Dec 2017 15:36:07 -0600

On 12/12/17 14:13, Consus wrote:
> Earlier this day we (clandmeter, _ikke_, ncopa, TBK and your
> humble author) were discussing on IRC (check out the new
> #alpine-docs channel) how the current state of the project
> documentation can be improved.

Is this channel available on Matrix as well?

> documentation is stored somewhere safe (probably in Git) and
> GitHub pull-requests or patches via ML are used to update it.

This is how we do it in Adélie.  It seems to work well.

> The main concern with Git-based documentation solutions is that
> the standalone patch review system should be rolled out
> (patchworks, ReviewBoard, you name it) because as (AFAIK)
> clandmeter stated we cannot fully rely on GitHub. However, it gives
> the developers the full control over the content

I'm glad that Alpine core still realises that.  I have been worried
lately that there is too much buy-in to GitHub from the Alpine
developers.  Do we really want all of that valuable discussion about
PRs and such locked away in a corporation's databases?  How long will
GitHub stay afloat before turning to ads, pay-to-use, or even just
plain disappear from the Internet?

> First of all, currently Alpine Wiki does not require a email
> address to create an account. Requiring a validated email address
> should reduce the spam level considerably.

Reduce, but not eliminate.  Honestly, the only true way to eliminate
spam is to have closed registrations (you have to poke an Alpine
person to get an account on the wiki).  But that adds a lot of work,
so it probably isn't a good solution.  Requiring valid email will take
out 80% of it so I suppose that's a very good start.

> Sensitive information (like PGP keys, donation options, etc) stored
> in the Wiki can be restricted for developer accounts only. If user
> finds an error in the document he can raise an issue in the
> Discussion tab.

I don't think any of this should be stored on the wiki.  All of this
should be on the main site, and never on the Wiki.  It certainly won't
change very often, so there's no reason to need for the community to
be able to edit it.

Meanwhile, I am going to work on mdoc'ing up apk and friends and hope
to have this done by the weekend.

Best,
- --arw

- -- 
A. Wilcox (awilfox)

---

---

From: Consus <[email-masked]>
Date: Wed, 13 Dec 2017 01:59:33 +0300

> Is this channel available on Matrix as well?

Honestly, I don't know. It was not me who created it.
 
> This is how we do it in Adélie.  It seems to work well.

Do you have third-party (non-core) contributors? I believe that this
model adds additional complexity for the regular folks. It would be nice
to have a positive example.
 
> plain disappear from the Internet?

Well, it's Git. If GitHub goes dark, there is gitlab / bitbucket / etc.
The migration will be instant. But still, it's better to have the tool
that you can control yourself.
 
> out 80% of it so I suppose that's a very good start.

Actually, I want to have a chat with Archlinux wiki maintainers about
the spam-reducing. They solved this problem somehow.
 
> change very often, so there's no reason to need for the community to

Yeah, sure, why not. Archlinux an Gentoo just keep this stuff on the
site directly. This opens a question: what else may be considered
sensitive enough? If nothing more than that, than (in my opinion) Wiki
is the perfect match for now, if the spam problem will be solved.

---

---

From: Oliver Smith <[email-masked]>
Date: Thu, 14 Dec 2017 00:14:00 +0000

Hi there,

I like writing documentation in wikis (I'm less interested in man pages, though I
think it's good to have them).

> Honestly, I don't know. It was not me who created it.
clandmeter said he could "fix" this after the matrix server is migrated to
new hardware. I'd also like to join the chat once the matrix channel is up
and running.

> to have a positive example.
I see it the same way, I'd much rather edit a wiki by clicking on "edit",
changing something, looking at a preview, and then saving it. There are more
arguments, which in my opinion, make MediaWiki a superior choice for Alpine,
when compared with a git repository:
* It's used for a lot of good distro wikis already (e.g. Arch, Gentoo) and people
  coming from those distros will have it easier to contribute content.
* It's easier to edit when you don't need to make a pull request for everything
* You have actual previews
* You can search the wiki easily
* It is possible to use templates and categorie

> the spam-reducing. They solved this problem somehow.

I think the Arch wiki has a clever registration questions, which reduces
a lot of spam accounts from being created in the first place. Right now
they have (I don't know if/how often they rotate these, and pacman is
their package manager, so you'd need to run Arch Linux more or less to
answer this one):
> What is the output of "pacman -V|base32|head -1"

So we could do something similar with `apk` and/or busybox instead of pacman.

Source: <https://wiki.archlinux.org/index.php?title=Special:CreateAccount&returnto=Main+page>

Meanwhile, Alpine's questions are autogenerated and in the form of:

> Given the number three hundred forty-four million one hundred thirty thousand
> two, what is the seventh digit?

Source: <https://wiki.alpinelinux.org/w/index.php?title=Special:UserLogin&returnto=Main+Page&type=signup>

Not fun to answer, less effective because you can answer it without having Alpine
running. And it could be that bots already know how to answer these automatically
(unless Alpine wrote a specific plugin for this).

But if all that does not help with the spam problem, I have found an extension
that might be suitable. Citing from <https://bugs.alpinelinux.org/issues/6774>
with updated dates:
>> How does it work?
>> 1. Every edit (or image upload) by a new user is being sent to moderation.
>> 2. Until the moderator approves this edit, the page is unchanged. Pending 
>>    edit is neither in page history nor on RecentChanges.
>> 3. The user can see his/her edit and continue editing his/her own version of 
>>    the page.
> https://www.mediawiki.org/wiki/Extension:Moderation
> https://github.com/edwardspec/mediawiki-moderation
> From what I can tell they have a maintained code base (last git commit was on 
> 2017-11-06, last release on 2017-10-16), some test cases and also use it in 
> production (that's what they say on the MediaWiki page). Looks good to me, but 
> I have not tried out the extension myself or reviewed the code.

PS: I don't think requiring a valid e-mail address helps at all. It's just annoying
and very easy to work around.

> Help:         [email-masked]

Thanks,
Oliver Smith

---

---

From: Paul Zillmann <[email-masked]>
Date: Thu, 14 Dec 2017 12:26:15 +0100

>From a user's perspective:

The wiki registration isn't hard at all. Trouble I had: the proxyserver
gave me an error message[1]
Two days later I registered myself successfully. I wanted to edit a section
which doesn't contain any IP addresses [2], but I couldn't save.
The error message was that I'm not allowed to publish IP addresses in the
wiki.

After that I looked up any contact information about the doc team (Wiki
administrators, moderators or any mailing list etc.) but I couldn't find
any.
Obviosly the alpine-user list doesn't bother with documentation problems.

I heard about docs.alpinelinux.org in this thread for the first time.
Where is the difference to the wiki?

[1] https://lists.alpinelinux.org/alpine-user/0205.html
[2]
https://wiki.alpinelinux.org/wiki/Configure_Networking#Enabling_IPv6_.28Optional.29

Greetings,
Paul

---

From: "A. Wilcox" <[email-masked]>
Date: Thu, 14 Dec 2017 00:03:22 -0600

On 13/12/17 18:14, Oliver Smith wrote:
> clandmeter said he could "fix" this after the matrix server is 
> migrated to new hardware. I'd also like to join the chat once the 
> matrix channel is up and running.

Good to know, thanks!

>> Do you have third-party (non-core) contributors? I believe that 
>> this model adds additional complexity for the regular folks. It 
>> would be nice to have a positive example.
>> 
> I see it the same way, I'd much rather edit a wiki by clicking on 
> "edit", changing something, looking at a preview, and then saving
> it. There are more arguments, which in my opinion, make MediaWiki
> a superior choice for Alpine, when compared with a git repository

Yes, actually, we only have 3 people on core and 7 total developers
(everyone with commit).  Everyone else is a contributor with no
affiliation.  Our core docs are written in DocBook (an install guide,
which is mostly done, and a developer guide, which is still in draft),
and are written entirely by horst@ and I.  The wiki is written by
everyone.

>> Actually, I want to have a chat with Archlinux wiki maintainers 
>> about the spam-reducing. They solved this problem somehow.
>> 
> 
> I think the Arch wiki has a clever registration questions, which 
> reduces a lot of spam accounts from being created in the first
> place. Right now they have (I don't know if/how often they rotate
> these, and pacman is their package manager, so you'd need to run
> Arch Linux more or less to answer this one):

This might still be confusing for people who are not used to command
lines.  Let's remember that not everyone who uses Alpine wants to (or
/can/) use the command line like that.

I do agree that some Alpine-related questions might be useful.  There
could be more than one, even.

> PS: I don't think requiring a valid e-mail address helps at all.
> It's just annoying and very easy to work around.

Annoying?  I'm genuinely curious to know why having a valid email is
annoying, considering that virtually everyone with an internet
connection at least has an ISP email box.

It is somewhat easy to work around with a sophisticated enough script,
but it is just another way to verify reality.  Maybe not necessary if
we get Alpine-related questions in the wiki sign-up page, but it never
hurts.

Best,
- --arw

- -- 
A. Wilcox (awilfox)

---

---

From: Carlo Landmeter <[email-masked]>
Date: Thu, 14 Dec 2017 11:23:09 +0100

On 14 December 2017 at 07:03, A. Wilcox <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
synapse migration is finished and the room is available.

I also talked to nangel and he would make an email address mandatory (not
sure thats done already).

-carlo

---

From: Natanael Copa <[email-masked]>
Date: Fri, 15 Dec 2017 09:12:36 +0100

On Tue, 12 Dec 2017 23:13:05 +0300

> of the Alpine Linux documentation.

Hi!

Thank you for bringing this up. I agree that we need to improve
documentation and I have been thinking about it for a while.

...

>    pull-requests or patches via ML are used to update it.

This is the solution I would like to go for, that is, some of the
official documentation is moved out from mediawiki to something else,
like docs.alpinelinux.org and manpages, while much of the docs stays on
wiki.

We need to keep in mind that there are different kinds of
documentation. For example:

install doc: how to do the install
administrator doc: the overview doc on how to do basic things in alpine
manpages: technical docs and specifications for individual tools
contributor doc: how to get involved in the project
developer doc: how to get started with development
infra doc: documentation of our infra structure
community docs: how to set up some specific hard/software

If you are first time user of alpine, and wonder how to get started,
then man pages may not be that helpful. You need an install doc that
can guide you throught the install. What is the difference between
sys/data/diskless install? When do you use what?

After the install is done, the user may need some guidance to do the
basic admin stuff, like (re)configure network, add users, change
password, find and install software, start and stop services.

Those docs can link to the manpages, which are more in-depth docs for
each tool. I don't have strong opinion on the format of the man pages.
Personally I would prefer write things in markdown or asciidoc, but I
don't mind writing directly in mdoc. (would be nice to have a filter
for cgit to parse mdoc)

The contributor and developer docs includes the Alpine linux project
processes and policies. How do you report bugs? How do you submit
patches? How do we include new developers? How do you become an aports
maintainer? How/when do new developers get git push access? What do we
do when someone leaves? How do a developer tag a new Alpine release?
Who are on the infra team? Who are the core developers? How/when are
releases made?

The community docs are things like, how do I get this USB wifi dongle
to work under alpine? How to install and set up apache, nginx, postfix
etc.

The question is now, how much of the above do we keep in wiki and how
much do we move out from wiki?

I think that manpages should be kept in the project it belongs to (eg
apk man pages should be stored in the apk-tools git repo, etc).

I think we should move things that are policies out of wiki too, things
like how to become a developer, how to contribute monetary, how the
release process works
(https://wiki.alpinelinux.org/wiki/Alpine_Linux:Releases)

I also don't think we can expect to move everything out from wiki and
moving things will take time. So we need to keep the wiki in good shape.

What I think we should do with the wiki:

- fix the registration process for new users
  (https://lists.alpinelinux.org/alpine-user/0205.html)

- Remove duplicates:
  https://wiki.alpinelinux.org/wiki/Alpine_Linux:Mailing_lists
  https://wiki.alpinelinux.org/wiki/Alpine_Linux:IRC

- Remove obsolete/outdated docs

- Write how to create good content on the wiki (something like
  https://wiki.archlinux.org/index.php/Help:Editing)

- Update the style/css?

I think that would be a good start at least.

What documentation is currently missing?

Other thing I think we should do is to make sure that every git repo
has a README, which includes:
- what is the purpose of the software project
- how do you get started (how to build)
- where to find info on how to report bugs and get involved (links)

Thanks!

-nc

---

---

From: Stuart Cardall <[email-masked]>
Date: Fri, 15 Dec 2017 20:23:39 +0000

I started creating README.alpine docs with notes to get things working:

musl64 [~/aports]$ find . -type f -name README.alpine
./community/elasticsearch/README.alpine
./community/runit/README.alpine
./community/lxcfs/README.alpine
./community/autossh/README.alpine
./community/mini-sendmail/README.alpine
./testing/pass/README.alpine

Stuart.

---

From: "A. Wilcox" <[email-masked]>
Date: Fri, 15 Dec 2017 17:29:06 -0600

On 15/12/17 02:12, Natanael Copa wrote:
> install doc: how to do the install administrator doc: the overview
> doc on how to do basic things in alpine manpages: technical docs
> and specifications for individual tools contributor doc: how to get
> involved in the project developer doc: how to get started with
> development infra doc: documentation of our infra structure 

I agree this is a good split of the docs.

> Those docs can link to the manpages, which are more in-depth docs
> for each tool. I don't have strong opinion on the format of the man
> pages. Personally I would prefer write things in markdown or
> asciidoc, but I don't mind writing directly in mdoc. (would be nice
> to have a filter for cgit to parse mdoc)

This may already be true.  It depends on what syntax highlighter
Alpine cgit is using:

https://wiki.archlinux.org/index.php/Cgit#Syntax_highlighting

Unfortunately `highlight` does not support man (or asciidoc!), but
`pygments` does support both:

http://pygments.org/docs/lexers/#lexers-for-non-html-markup-languages

> I think that manpages should be kept in the project it belongs to
> (eg apk man pages should be stored in the apk-tools git repo,
> etc).

Yes, definitely.

> Other thing I think we should do is to make sure that every git
> repo has a README, which includes: - what is the purpose of the
> software project - how do you get started (how to build) - where to
> find info on how to report bugs and get involved (links)

Would rST format be okay for this?  I could use the templates we have
in Adélie for README / CONTRIBUTING files and write some for apk-tools
and abuild if you like.  Here is an example:

https://code.foxkit.us/adelie/shimmy/raw/master/README.rst
https://code.foxkit.us/adelie/shimmy/raw/master/CONTRIBUTING.rst

Best to you and yours,
- --arw

- -- 
A. Wilcox (awilfox)

---