Subject: [alpine-devel] apk-tools - In need of a minimal spec

From: Orion <[email-masked]>
Date: Mon, 2 Feb 2015 00:32:18 -0800

I have began reading through the apk-tools source code with the goal of
teaching myself more about how package managers work by porting
it into go-lang. I would appreciate help compiling some minimally
sufficient specification for this.

I ask for your help because I have had little luck finding any kind of
documentation, design thoughts, comments, or self describing
functions/variables. Let me know what the easiest means of discussion
would be.

# Stuff I Found Confusing
 * blob.c
	* memspn()
	* memcspn()
	* apk_blob_spn()
	* apk_blob_cspn()
	* digitdecode & dx()
		- is data taken, encoded with your own custom codec and
		  then put into a blow?
 * archive.c & gunzip.c
	* is the tar and gzip code used custom or pulled from existing
	  implementations?

Thank you for your time.

-- 
keybase.io/systmkor

---
Unsubscribe:  [email-masked]
Help:         [email-masked]
---

---

From: Timo Teras <[email-masked]>
Date: Mon, 2 Feb 2015 11:17:18 +0200

On Mon, 2 Feb 2015 00:32:18 -0800
Orion <[email-masked]> wrote:

> I have began reading through the apk-tools source code with the goal
> of teaching myself more about how package managers work by porting

Yes, we should add some overview documentation. And specs of how the
package format is handled.

> # Stuff I Found Confusing

It's basically similar to strspn() but with two differences:
 - it works with blobs, so the source string is not zero-terminated
 - the list of characters to accept or reject is a bitmap instead of
   zero-terminated string (see the header for apk_spn_match_def typedef)

memspn() would probably be the libc name, but there is no such
function. Use of that name is probably misleading.

I wrote this earlier when I had tendency to micro-optimize things. If
I'd write it now, I'd leave the memspn() stuff out, and keep only the
C-versions.

> 	* digitdecode & dx()
> 		- is data taken, encoded with your own custom codec

This is just a lookup-table based ASCII hexnumber to number conversion.
Basically atoi() and strtol() equivalent, but it's hand written again,
so we can work with "blobs" (length limited, not zero-terminated).

> and then put into a blow?
>  * archive.c & gunzip.c

gunzip.c ends up using libz. It's just glue for the internal
abstractionlayer.

archive.c contains full tar decoder and encoder. It's mostly custom
code. And currently it is required due to how the .apk file is
constructed.

/Timo

---

---

From: Orion <[email-masked]>
Date: Mon, 2 Feb 2015 19:15:47 -0800

Thanks. That starts to clear up a few things.

I've created a wiki page for the apk-tools specification. 
http://wiki.alpinelinux.org/wiki/Apk_spec

I think this place would make sense for us to start. Let me know if you
think there is a better way of going about this. Thank you for your
time.

-- 
keybase.io/systmkor

---

---

From: Orion <[email-masked]>
Date: Wed, 4 Feb 2015 01:42:31 -0800

I've began to try define the syntax/keywords used inside
the /lib/apk/db/installed . Let me know if I'm on the right track.
http://wiki.alpinelinux.org/wiki/Apk_spec

-- 
keybase.io/systmkor

---

---

From: Timo Teras <[email-masked]>
Date: Wed, 4 Feb 2015 12:01:12 +0200

On Wed, 4 Feb 2015 01:42:31 -0800

> http://wiki.alpinelinux.org/wiki/Apk_spec

Yes, mostly. I could probably fix / update some details too.

It might be an idea to explain that the same file format is used for
both: installed database, and the package index file. The index file
contains a subset, and installed database has some additional fields.

Please remove the 'blob' term there. Blobs are internal implementation
detail. apk_blob is basically a pointer + length structure. On the
index/database file it is just a specifically formatted text string.

Also, there's a convention that capital letter fields are mandatory.
Apk refuses to install the package if it recognize it. Lower case
letters are optional support, and apk will give warning but accept to
install it.

Maybe it'd be good as a table with columns:
 - Field letter
 - Description
 - Field data format
 - Allowed in installed-db / package-index

For APKBUILD we already have some other wiki pages, and it's not read
by apk-tools so that can be left out.

Over all, for the fileformats. It'd need also a section for index file
and it's structure, the installed db file structure, and the package
file structure.

I do have some notes on these. But it's not in any formatted way. I
should take time to write it to the wiki page.

Thanks for the effort so far :)

/Timo

---

---

From: Orion <[email-masked]>
Date: Fri, 6 Feb 2015 18:58:16 -0800

On Wed, 4 Feb 2015 12:01:12 +0200
Timo Teras <[email-masked]> wrote:

> Please remove the 'blob' term there.

'blob' terms has been removed.

> letters are optional support, and apk will give warning but accept to

One line noting this but leaving out the warning until I know more
about what causes it etc.

>  - Allowed in installed-db / package-index

Table with these fields have been created. That said I haven't filled
the entire table out yet. I think the table should probably be a quick
reference and then if needed we have sections for each one going into
depth about that field.

> by apk-tools so that can be left out.

Section removed.

> should take time to write it to the wiki page.

Maybe add some of your notes to the wiki page this weekend? :3 :P

> Thanks for the effort so far :)

Your welcome. Thanks for responding and not leaving me stranded to
reverse engineer everything. 

-- 
keybase.io/systmkor

---