Subject: Re: [alpine-aports] [PATCH] main/ca-certificates: change update-ca-certificates lua script to a shell script

From: Ramon Soares <[email-masked]>
Date: Mon, 4 May 2015 08:48:26 -0300

Is really necessary to have performance in this script? 

To a script that should have little use, I believe the dependency of other packages to be unnecessary.

--
Ramon Soares
[email-masked]

CodeCraft
http://www.codecraft63.com.br/

+55 (71) 9162-1704
+55 (71) 3387-3814

> On May 3, 2015, at 11:02 AM, [email-masked] wrote:
> 
> Nak.
> 
> We had shell script earlier but it was slow. I wrote lua script to fix performance.
> 
> Either add features you need to the lua version. Or if you dont like lua dependency (which really is small) the alternative is C rewrite.
> 
> See git log and issue tracker for details.
> 
> On Sun May 3 14:21:36 2015 GMT+0300, Ramon Soares wrote:
>> ---
>> main/ca-certificates/APKBUILD               |  10 +-
>> main/ca-certificates/update-ca-certificates | 250 ++++++++++++++++++----------
>> 2 files changed, 168 insertions(+), 92 deletions(-)
>> 
>> diff --git a/main/ca-certificates/APKBUILD b/main/ca-certificates/APKBUILD
>> index 2db2af0..8f344f1 100644
>> --- a/main/ca-certificates/APKBUILD
>> +++ b/main/ca-certificates/APKBUILD
>> @@ -7,12 +7,12 @@ _nmu="+nmu${pkgver#*_p}"
>> [ "$_nmu" = "+nmu${pkgver}" ] && _nmu=""
>> _ver=${pkgver}
>> 
>> -pkgrel=1
>> +pkgrel=2
>> pkgdesc="Common CA certificates PEM files"
>> url="http://packages.debian.org/sid/ca-certificates"
>> arch="noarch"
>> license="MPL 2.0 GPL2+"
>> -depends="run-parts openssl lua5.2 lua5.2-posix"
>> +depends="run-parts openssl"
>> makedepends="python"
>> subpackages="$pkgname-doc"
>> options="!fhs"
>> @@ -62,8 +62,8 @@ EOF
>> }
>> 
>> md5sums="f619282081c8bfc65ea64c37fa5285ed  ca-certificates_20141019.tar.xz
>> -bb24a9e2caf6150053a981e617a209a9  update-ca-certificates"
>> +5d3218ff481e9236fc217080e325c2ca  update-ca-certificates"
>> sha256sums="684902d3f4e9ad27829f4af0d9d2d588afed03667997579b9c2be86fcd1eb73a  ca-certificates_20141019.tar.xz
>> -4f109a22d74ef36f691933716a83c015b8475ed4e47d21bd0d7f8baef4de046b  update-ca-certificates"
>> +aa7d262788b36ff786ff94055b3f5d23d588a35e59aecd767fb31f9c75985fa8  update-ca-certificates"
>> sha512sums="5b0e8fb917f5642a5a2b4fde46a706db0c652ff3fb31a5053d9123a5b670b50c6e3cf2496915cc01c613dcbe964d6432f393c12d8a697baedfad58f9d13e568b  ca-certificates_20141019.tar.xz
>> -dadf046999f226cef0b14bd9014e59f04fa05a984339ff84940a2beb0f33f70bc921233d22291a294f9366c67989f5d337febd1832574c2e28317c14de84ff00  update-ca-certificates"
>> +5104acd36b623a6cf65f13a1ce542d57fa9aeec351d46bc535c220007c876b1d1790a85d3c51da9b723778e8a3dac18f1d90833268a2e4bf2f29a831a64a903b  update-ca-certificates"
>> diff --git a/main/ca-certificates/update-ca-certificates b/main/ca-certificates/update-ca-certificates
>> index 53f45df..5375950 100755
>> --- a/main/ca-certificates/update-ca-certificates
>> +++ b/main/ca-certificates/update-ca-certificates
>> @@ -1,87 +1,163 @@
>> -#!/usr/bin/lua5.2
>> -
>> -local CERTSDIR='/usr/share/ca-certificates/'
>> -local LOCALCERTSDIR='/usr/local/share/ca-certificates/'
>> -local ETCCERTSDIR='/etc/ssl/certs/'
>> -local CERTBUNDLE='ca-certificates.crt'
>> -local CERTSCONF='/etc/ca-certificates.conf'
>> -
>> -local posix = require 'posix'
>> -function string.begins(str, prefix) return str:sub(1,#prefix)==prefix end
>> -
>> -local function add(fn, out, links)
>> -	-- Map fn to file in etc
>> -	local pem = "ca-cert-"..fn:gsub('.*/', ''):gsub('.crt$',''):gsub('[, ]','_'):gsub('[()]','=')..".pem"
>> -	links[pem] = fn
>> -	-- Read the certificate for the bundle
>> -	local f = io.open(fn, "rb")
>> -	if f ~= nil then
>> -		local content = f:read("*all")
>> -		f:close()
>> -		out:write(content)
>> -		if content:sub(-1) ~= '\n' then out:write('\n') end
>> -	end
>> -end
>> -
>> -local calinks = {}
>> -local cacerts = {}
>> -
>> -local fd, tmpfile = posix.mkstemp(ETCCERTSDIR..'bundleXXXXXX')
>> -if not fd then
>> -	print("Failed to open temporary file for ca bundle")
>> -	return 1
>> -end
>> -posix.close(fd)
>> -posix.chmod(tmpfile, "rw-r--r--")
>> -local bundle = io.open(tmpfile, "wb")
>> -
>> --- Handle global CA certs from config file
>> -for l in io.lines(CERTSCONF) do
>> -	local firstchar = l:sub(1,1)
>> -	if firstchar ~= "#" and firstchar ~= "!" then
>> -		add(CERTSDIR..l, bundle, calinks)
>> -	end
>> -end
>> -
>> --- Handle local CA certificates
>> -local certlist = posix.glob(LOCALCERTSDIR..'*.crt')
>> -if certlist ~= nil then
>> -	table.sort(certlist)
>> -	for f in pairs(certlist) do
>> -		local fn = LOCALCERTSDIR..f
>> -		if posix.stat(fn, 'type') == 'regular' then
>> -			add(fn, bundle, calinks)
>> -		end
>> -	end
>> -end
>> -
>> --- Update etc cert dir for additions and deletions
>> -local f, target
>> -for f in posix.files(ETCCERTSDIR) do
>> -	local fn = ETCCERTSDIR..f
>> -	if posix.stat(fn, 'type') == 'link' then
>> -		local curtgt = posix.readlink(fn)
>> -		local target = calinks[f]
>> -		if target == nil then
>> -			-- Symlink exists but is not wanted
>> -			-- Delete it if it points to 'our' directory
>> -			if curtgt:begins(CERTSDIR) or curtgt:begins(LOCALCERTSDIR) then
>> -				os.remove(fn)
>> -			end
>> -		elseif curtgt ~= target then
>> -			-- Symlink exists but points wrong
>> -			posix.link(target, ETCCERTSDIR..f, true)
>> -		else
>> -			-- Symlink exists and is ok
>> -			calinks[f] = nil
>> -		end
>> -	end
>> -end
>> -for f, target in pairs(calinks) do
>> -	posix.link(target, ETCCERTSDIR..f, true)
>> -end
>> -
>> --- Update hashes and the bundle
>> -bundle:close()
>> -os.rename(tmpfile, ETCCERTSDIR..CERTBUNDLE)
>> -os.execute("c_rehash "..ETCCERTSDIR.." > /dev/null")
>> +#!/bin/sh -e
>> +#
>> +# update-ca-certificates
>> +#
>> +# Copyright (c) 2003 Fumitoshi UKAI <[email-masked]>
>> +# Copyright (c) 2009 Philipp Kern <[email-masked]>
>> +# 
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write to the Free Software
>> +# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02111-1301,
>> +# USA.
>> +#
>> +
>> +verbose=0
>> +fresh=0
>> +while [ $# -gt 0 ];
>> +do
>> +  case $1 in
>> +  --verbose|-v)
>> +  	verbose=1;;
>> +  --fresh|-f)
>> +	fresh=1;;
>> +  --help|-h|*)
>> +	echo "$0: [--verbose] [--fresh]"
>> +	exit;;
>> +  esac
>> +  shift
>> +done
>> +
>> +CERTSCONF=/etc/ca-certificates.conf
>> +CERTSDIR=/usr/share/ca-certificates
>> +LOCALCERTSDIR=/usr/local/share/ca-certificates
>> +CERTBUNDLE=ca-certificates.crt
>> +ETCCERTSDIR=/etc/ssl/certs
>> +
>> +cleanup() {
>> +  rm -f "$TEMPBUNDLE"
>> +  rm -f "$ADDED"
>> +  rm -f "$REMOVED"
>> +}
>> +trap cleanup 0
>> +
>> +# Helper files.  (Some of them are not simple arrays because we spawn
>> +# subshells later on.)
>> +TEMPBUNDLE="$(mktemp -t "${CERTBUNDLE}.tmp.XXXXXX")"
>> +ADDED="$(mktemp -t "ca-certificates.tmp.XXXXXX")"
>> +REMOVED="$(mktemp -t "ca-certificates.tmp.XXXXXX")"
>> +
>> +# Adds a certificate to the list of trusted ones.  This includes a symlink
>> +# in /etc/ssl/certs to the certificate file and its inclusion into the
>> +# bundle.
>> +add() {
>> +  CERT="$1"
>> +  PEM="$ETCCERTSDIR/$(basename "$CERT" .crt | sed -e 's/ /_/g' \
>> +                                                  -e 's/[()]/=/g' \
>> +                                                  -e 's/,/_/g').pem"
>> +  if ! test -e "$PEM" || [ "$(readlink "$PEM")" != "$CERT" ]
>> +  then
>> +    ln -sf "$CERT" "$PEM"
>> +    echo +$PEM >> "$ADDED"
>> +  fi
>> +  cat "$CERT" >> "$TEMPBUNDLE"
>> +}
>> +
>> +remove() {
>> +  CERT="$1"
>> +  PEM="$ETCCERTSDIR/$(basename "$CERT" .crt).pem"
>> +  if test -L "$PEM"
>> +  then
>> +    rm -f "$PEM"
>> +    echo -$PEM >> "$REMOVED"
>> +  fi
>> +}
>> +
>> +cd $ETCCERTSDIR
>> +if [ "$fresh" = 1 ]; then
>> +  echo -n "Clearing symlinks in $ETCCERTSDIR..."
>> +  find . -type l -print | while read symlink
>> +  do
>> +     case $(readlink $symlink) in
>> +     $CERTSDIR*) rm -f $symlink;;
>> +     esac
>> +  done
>> +  find . -type l -print | while read symlink
>> +  do
>> +     test -f $symlink || rm -f $symlink
>> +  done
>> +  echo "done."
>> +fi
>> +
>> +echo -n "Updating certificates in $ETCCERTSDIR... "
>> +
>> +# Handle certificates that should be removed.  This is an explicit act
>> +# by prefixing lines in the configuration files with exclamation marks (!).
>> +sed -n -e '/^$/d' -e 's/^!//p' $CERTSCONF | while read crt
>> +do
>> +  remove "$CERTSDIR/$crt"
>> +done
>> +
>> +sed -e '/^$/d' -e '/^#/d' -e '/^!/d' $CERTSCONF | while read crt
>> +do
>> +  if ! test -f "$CERTSDIR/$crt"
>> +  then
>> +    echo "W: $CERTSDIR/$crt not found, but listed in $CERTSCONF." >&2
>> +    continue
>> +  fi
>> +  add "$CERTSDIR/$crt"
>> +done
>> +
>> +# Now process certificate authorities installed by the local system
>> +# administrator.
>> +if [ -d "$LOCALCERTSDIR" ]
>> +then
>> +  find -L "$LOCALCERTSDIR" -type f -name '*.crt' | while read crt
>> +  do
>> +    add "$crt"
>> +  done
>> +fi
>> +
>> +rm -f "$CERTBUNDLE"
>> +
>> +ADDED_CNT=$(wc -l < "$ADDED")
>> +REMOVED_CNT=$(wc -l < "$REMOVED")
>> +
>> +if [ "$ADDED_CNT" -gt 0 ] || [ "$REMOVED_CNT" -gt 0 ]
>> +then
>> +  # only run if set of files has changed
>> +  if [ "$verbose" = 0 ]
>> +  then
>> +    c_rehash . > /dev/null
>> +  else
>> +    c_rehash .
>> +  fi
>> +fi
>> +
>> +chmod 0644 "$TEMPBUNDLE"
>> +mv -f "$TEMPBUNDLE" "$CERTBUNDLE"
>> +
>> +echo "$ADDED_CNT added, $REMOVED_CNT removed; done."
>> +
>> +HOOKSDIR=/etc/ca-certificates/update.d
>> +echo -n "Running hooks in $HOOKSDIR...."
>> +VERBOSE_ARG=
>> +[ "$verbose" = 0 ] || VERBOSE_ARG=--verbose
>> +eval run-parts $VERBOSE_ARG --test -- $HOOKSDIR | while read hook
>> +do
>> +  ( cat $ADDED
>> +    cat $REMOVED ) | $hook || echo E: $hook exited with code $?.
>> +done
>> +echo "done."
>> +
>> +# vim:set et sw=2:
>> +
>> -- 
>> 2.3.7
>> 
>> 
>> 
>> ---
>> Unsubscribe:  [email-masked]
>> Help:         [email-masked]
>> ---
>>

---

From: Timo Teras <[email-masked]>
Date: Mon, 4 May 2015 15:23:43 +0300

On Mon, 4 May 2015 08:48:26 -0300

> other packages to be unnecessary.

It directly affects the boottime of tmpfs installs, as well as
upgrading packages.

See: http://bugs.alpinelinux.org/issues/2846

The shell script can run 3-20 seconds on ARM, where as the lua version
is 0.25 seconds.

Having to wait extra 20 seconds on boot, or "apk upgrade" is
unacceptable if it's fixed with lua dependency. Especially since lua is
the preferred scripting/interpreted language in alpine.

The total installed size for lua and the posix module is roughly 300
kB. It is still less than bash (700+ kB).

Also; the patch is also incorrect, since the orignal
update-ca-certificates is in the tarball, it should not be copied to
aports. See the original commit introducing the lua script:
http://git.alpinelinux.org/cgit/aports/commit/?id=af18a975d8494f923d0ff3754dd250ffc641b6ef

But that said, we are not going to go back to something that is slower;
just because someone else is doing it. If you find the lua dependency
unacceptable, please rewrite the code in C, or some other compiled
language.

Thanks,
Timo

---

---

From: Natanael Copa <[email-masked]>
Date: Mon, 4 May 2015 16:42:00 +0200

On Mon, 4 May 2015 15:23:43 +0300

> kB. It is still less than bash (700+ kB).

...

> unacceptable, please rewrite the code in C, or some other compiled

I think we want it rewritten in C. Does not look difficult.

-nc

---