From nobody Fri Mar 29 13:49:56 2024 X-Original-To: alpine-devel@lists.alpinelinux.org Received: from mail-ua0-f179.google.com (mail-ua0-f179.google.com [209.85.217.179]) by lists.alpinelinux.org (Postfix) with ESMTP id 8C1AA5C4D2A for ; Tue, 17 Oct 2017 05:01:06 +0000 (GMT) Received: by mail-ua0-f179.google.com with SMTP id n38so313398uai.11 for ; Mon, 16 Oct 2017 22:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=Bxhay/XC/dmFrJiudRJRh0dBknVIfnaLIvYg9zHMcEQ=; b=fI1aibjJftWEZNkXF6wpllFtuYU5HxpM3UdnUcYSj4gJg0nvLyE7b6QGxUsRrMMFB4 8kLjqvOdnM6NCIzNRb8+vKwX2fYUW27mzayNynkRrQXzg8SExFvI5hjLfJ/k3T0PGy0/ 7hYcpsMGPRWxndsMWqA2tEp2UxuJHYyFvPVzj2IIHLx9tueu1Oi76KkU9jUuXRPQhQcH 8zKNBsQX0v2q/ssHcpPffbJwX8ILVCh4JZMRO1W3ZfAH0ApEhUl5cWFoHmaDEuIKs1MM jwydRlIKkAdwYxBdTh2kskB7Etfmd/odr6IGY5aH4HVYite+hnf5ECuxNjpX7BRI2+xO n0jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=Bxhay/XC/dmFrJiudRJRh0dBknVIfnaLIvYg9zHMcEQ=; b=tj6qTmbanzaWISMQhaPp+XiyKcSoi4QFqapZUpLVCdHetQFxr2z2NSwGeo/YG6xL5t erpuqNfZpvdYXJmypk76y7RRCXzYUz5bfe4FO/XxRHcNQAln6wt3O4J9GgDKXyrtWPvu 9P3mW+m+CJAh2my0jWv8p5BMfu/IGsaw6+PfWR9z7RqUqOzR3BHV6rJIT2mddgylmv6b 2Xg19M9Gxoh1TwSB/huRMFSvdDPFHg1BbRjDVy/Uj1lAJ9iTgU9pJn1uSw5/0F73WiEe uMYD7ISHtkttxwJgMMvXDQZsHpTR3jgL9B5Xy3aNnMExZw03PJo6hD1WNWiAo4gbe1GQ AhSQ== X-Gm-Message-State: AMCzsaVsmEN2vQsCAvWKrrmJfGye+F8d27qQBuchRZz96Br5WxQw+6Jj 9LxdAY8jVKfAByhD0wdxq7/alDSPjdRCvPTjw38dOw== X-Google-Smtp-Source: AOwi7QCVi/Hv/uGJCtfTPdaCccEIdHtV+ki2zqFt/qiw/a79q7vCXJ97v7iC45BWwu3NqypOi5l57il20tc6oIywCqQ= X-Received: by 10.159.56.9 with SMTP id p9mr9623720uad.50.1508216465616; Mon, 16 Oct 2017 22:01:05 -0700 (PDT) X-Mailinglist: alpine-devel Precedence: list List-Id: Alpine Development List-Unsubscribe: List-Post: List-Help: List-Subscribe: MIME-Version: 1.0 Received: by 10.103.96.68 with HTTP; Mon, 16 Oct 2017 22:01:05 -0700 (PDT) From: Andy Chu Date: Mon, 16 Oct 2017 22:01:05 -0700 Message-ID: Subject: [alpine-devel] Parsing all APKBUILD files To: alpine-devel@lists.alpinelinux.org Content-Type: multipart/alternative; boundary="001a11495908ad7399055bb702ce" --001a11495908ad7399055bb702ce Content-Type: text/plain; charset="UTF-8" Hi there, I just sent some minor bug fixes here: https://github.com/alpinelinux/aports/pull/2506 and wanted to give some context. I've been playing with Alpine for a few weeks and I like it! After being deep in the bowels of Debian a few years ago, the code is a breath of fresh air! I'm writing a new shell, and as a torture test for the parser, I parsed ~700K lines of shell scripts I have on my hard drive. This includes the entire aports repository, which is ~5000 APKBUILD files in ~250K lines. Results here: http://www.oilshell.org/git-branch/master/21552818/wild.wwz/distro/alpine-aports/index.html Only two scripts failed, because of the unicode whitespace issue I noted in the pull request. (I only noticed this because of a bug in my shell. However no shells I tested, including busybox ash, accept this code, so they are real bugs.) ----- The latest post on my blog has a little background, including why I'm interested in Alpine: http://www.oilshell.org/blog/2017/10/06.html >From my perspective it seems as functional as Debian in many ways, but the build scripts are more sane. One big caveat is that my shell is very slow right now, due to a couple of implementation shortcuts, which I plan to remove, as well as implementing it in Python. However the algorithm is principled, as well as fast in the algorithmic sense [1]. I did the hard part of making it correct; now I have to make it fast in practice :) So what I'll probably be doing in the near future is speeding up the parser, so I can build packages with abuild running under OSH. (It takes a few seconds to even parse abuild before it runs right now, especially in a VM.) [1] http://www.oilshell.org/blog/tags.html?tag=parsing-shell#parsing-shell ----- I also have some longer term goals for improving shell. As mentioned in the blog post, I think Linux distros are the projects that really push shell to its limits. They're almost all built on shell in some way that is not very satisfactory. For example, Debian is a big mess of auto-generated Makefiles and shell scripts. Red Hat embeds shell snippets in a text file with metadata. Alpine has some of the cleaner shell code I've seen, but I have been lurking and noticed a couple threads where shell is brought up: 2017/06/30 - Script to parse APKBUILDS and output table of arch support http://lists.alpinelinux.org/alpine-devel/5741.html "I agree. The way to parse the APKBUILDs is to execute them. " I looked at the read_apkbuild code here, which pipes a program to bash that sources an APKBUILD to bash and echos variables: https://git.alpinelinux.org/cgit/user/ncopa/aports-cache/ I'm not sure if there is any benefit to this right now, but this can be done in my shell by writing a Python program that pokes into the interpreter state to eval and fetch variables. I would be interested in use cases for querying APKBUILD metadata. ---- 2017/06/12 - A few questions about Alpine http://lists.alpinelinux.org/alpine-devel/5689.html "The major issue with APKBUILD is that it is not possible to specify the metadata for subpackages (pkgdesc, depends, arch etc) in global scope, and regardless of how we define it in shell, it will be clumsy at best. " I didn't quite understand this, but I plan to add structured metadata to Oil, which seems to be a major thing missing from shell. In other words, you should be able to parse scripts and extract data without executing. There are tons of tools that need to mix declarative metadata with shell snippets, like essentially every package manager, Docker files, Chef/Puppet-type configuration, systemd config files, Upstart config files, every Makefile ever written, etc. ---- Also on IRC a few weeks ago, foxkit proposed a lint tool for APKBUILD: "repoman uses python to static read the ebuild files in gentoo tree and find common mistakes. rpmlint uses perl to do same with redhat rpms" OSH has a very detailed AST so it may be possible to write lint tools on top of it. Here is an example: http://www.oilshell.org/git-branch/master/21552818/wild.wwz/distro/alpine-aports/main/bash/APKBUILD__ast.html However as mentioned I have to speed things up a lot first. ----- None of these ideas are particularly urgent, but I thought I'd mention them. If there is something that you want from a shell, I'm interested. I suspect Alpine is not going to switch shells any time soon, but having a second shell implementation to validate/analyze the large amount of code seems like it could be useful. Let me know what you think! thanks, Andy --001a11495908ad7399055bb702ce Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi there,

I just sent some mino= r bug fixes here:

https://github.com/alpinelinux/aports/pull/2506

= and wanted to give some context.

I've been playing with Al= pine for a few weeks and I like it!=C2=A0 After being deep in the bowels of= Debian a few years ago, the code is a breath of fresh air!

I&= #39;m writing a new shell, and as a torture test for the parser, I parsed ~= 700K lines of shell scripts I have on my hard drive.=C2=A0 This includes th= e entire aports repository, which is ~5000 APKBUILD files in ~250K lines.
Results here:

Only two scripts = failed, because of the unicode whitespace issue I noted in the pull request= .=C2=A0 (I only noticed this because of a bug in my shell.=C2=A0 However no= shells I tested, including busybox ash, accept this code, so they are real= bugs.)

-----

The latest = post on my blog has a little background, including why I'm interested i= n Alpine:

=

From my perspective it seems as functional as Debian in= many ways, but the build scripts are more sane.

O= ne big caveat is that my shell is very slow right now, due to a couple of i= mplementation shortcuts, which I plan to remove, as well as implementing it= in Python.=C2=A0 However the algorithm is principled, as well as fast in t= he algorithmic sense [1].=C2=A0 I did the hard part of making it correct; n= ow I have to make it fast in practice :)

So what I= 'll probably be doing in the near future is speeding up the parser, so = I can build packages with abuild running under OSH.=C2=A0 (It takes a few s= econds to even parse abuild before it runs right now, especially in a VM.)<= /div>


-----=

I also have some longer term goals for improving = shell.=C2=A0 As mentioned in the blog post, I think Linux distros are the p= rojects that really push shell to its limits.=C2=A0 They're almost all = built on shell in some way that is not very satisfactory.=C2=A0 For example= , Debian is a big mess of auto-generated Makefiles and shell scripts.=C2=A0= =C2=A0 Red Hat embeds shell snippets in a text file with metadata.

Alpine has some of the cleaner shell code I've see= n, but I have been lurking and noticed a couple threads where shell is brou= ght up:



I'm not sure if there = is any benefit to this right now, but this can be done in my shell by writi= ng a Python program that pokes into the interpreter state to eval and fetch= variables.

I would be interested in use cases= for querying APKBUILD metadata.

----

2= 017/06/12 - A few questions about Alpine

http://lists.alpinelinux.org/alpine-de= vel/5689.html

"The major issue with APKBUILD is that it is = not possible to specify the
metadata for subpackages (pkgdesc, depends, = arch etc) in global scope,
and regardless of how we define it in shell, = it will be clumsy at best. "

I didn't qui= te understand this, but I plan to add structured metadata to Oil, which see= ms to be a major thing missing from shell.=C2=A0 In other words, you should= be able to parse scripts and extract data without executing.

There are tons of tools that need to mix declarative metada= ta with shell snippets, like essentially every package manager, Docker file= s, Chef/Puppet-type configuration, systemd config files, Upstart config fil= es, every Makefile ever written, etc.

----

Also on IRC a few weeks ago, foxkit proposed a lint t= ool for APKBUILD:

"repoman uses python to static r= ead the ebuild files in gentoo tree and find common mistakes. rpmlint uses = perl to do same with redhat rpms"

OSH has a v= ery detailed AST so it may be possible to write lint tools on top of it.=C2= =A0 Here is an example:


<= /div>
However as mentioned I have to speed things up a lot first.
=

-----

None of these ideas = are particularly urgent, but I thought I'd mention them.=C2=A0 If there= is something that you want from a shell, I'm interested.=C2=A0 I suspe= ct Alpine is not going to switch shells any time soon, but having a second = shell implementation to validate/analyze the large amount of code seems lik= e it could be useful.

Let me know what you thi= nk!

thanks,
Andy

--001a11495908ad7399055bb702ce-- --- Unsubscribe: alpine-devel+unsubscribe@lists.alpinelinux.org Help: alpine-devel+help@lists.alpinelinux.org ---