Subject: Re: Packaging go projects
To: ~alpine/devel@lists.alpinelinux.org
References: <CAD+eXGRiTnrSnkE5iyS5cYB1bTR8JXOLzvYyW0-eEc0VtusZKA@mail.gmail.com>
 <CAD+eXGQmWT+2ASHj+423TyosrHFRF974ogY+SqSCxYP99JQrdA@mail.gmail.com>
From: Paul Zillmann <p.zillmann@h6g.de>
Message-ID: <aec684cf-93c9-ab1f-89d3-5a5ddd8a3a5a@h6g.de>
Date: Thu, 12 Sep 2019 10:46:16 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.0
MIME-Version: 1.0
In-Reply-To: <CAD+eXGQmWT+2ASHj+423TyosrHFRF974ogY+SqSCxYP99JQrdA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US

Hello Konstantin,

that definitely looks easy to do.
I have one concern though.

As far as I'm concerned abuild downloads any network sources when an URL 
is given in the apkbuild file.
Buildozer then takes these (additional) source files and archives them 
for later retrieval.

Now with that system in mind it is possible to lose sources of old packages.
Before go modules, people used to use go vendoring to get the sources of 
libs into their own source tree.

- Paul

Am 11.09.19 um 21:14 schrieb Konstantin Kulikov:
> Working with abuild a bit more I've came up with a simpler way.
> Instead of redefining fetch, making it depend on go and putting stuff
> in SRCDEST,
> we can instead define a few env vars in APKBUILD like this:
>
> export GOPROXY=${GOPROXY:-https://proxy.golang.org}  # Default in
> go1.13 and can be removed.
> export GOPATH=${GOPATH:-$srcdir/go}
> export GOCACHE=${GOCACHE:-$srcdir/go-build}
> export GOTMPDIR=${GOTMPDIR:-$srcdir}
>
> This way "go build" will download modules from GOPROXY and place all
> build files inside srcdir.
> Hoverwer if user has those vars in their environment (ie local dev
> machine) or in abuild.conf (ie ci checks or build server),
> they will be used instead. This way devs work as they usually do (no
> difference between running "go build" manually and from abuild)
> and CI can opt in into module and/or build caches if they want to
> remove calls to external network and speed up builds.
>
> Build cache has one case when it doesn't always rebuild packages when
> it should so enabling it on build servers now might be problematic.
> https://github.com/golang/go/issues/24355
>
> Module cache should work well.
>
> I've updated my example:
> https://github.com/alpinelinux/aports/commit/ac34f80cdadd78523fde160e1bbc032dd04bfbca
> Previous version is here:
> https://github.com/alpinelinux/aports/commit/7892cf94cd917c080a0c381caf5ac5e90aa84c97
>
> On Wed, Jul 31, 2019 at 7:59 PM Konstantin Kulikov <k.kulikov2@gmail.com> wrote:
>> Hey everyone.
>>
>> I'm trying to package a go project which uses modules. This is what I
>> came up with.
>>
>> https://github.com/alpinelinux/aports/pull/9800
>>
>> Notes below:
>> 1. fetch() depends on go command which means that abuild checksum will
>> fail if go is not installed.
>> 2. $SRCDEST will contain "go" directory with module archives using
>> layout defined by goproxy protocol (see
>> https://godoc.org/cmd/go#hdr-Module_proxy_protocol). The good thing is
>> that directory can be served over http and used as real GOPROXY,
>> avoiding contacting google or other proxies unnecessarily. Current go
>> projects in aports run "glide install" or "dep ensure" inside
>> prepare() or build() and do not cache sources at all.
>> 3. GOPROXY should be defined in abuild.conf and default to
>> https://proxy.golang.org or any other proxy to avoid depending on
>> git,svn,bzr and others and allow users to use their own proxies
>> (googles proxy is unavailable in China for example).
>> 4. sha512sums doesn't record hashes of module files, instead they are
>> recorded in go.sum file and go tool verifies them itself.
>> 5. chmod line in fetch() is unfortunate, without it cleanup step will
>> fail. Go devs previously refused to add option to make unpacked module
>> files writable.
>> 6. fetch() will be the same for every go project and can be defined in
>> abuild and used with options="gomodule" for example, same goes for
>> GOPATH var which should be set to the same value for all steps.
>> 7. To build a project we create dummy module which I called "apk" (but
>> can be any name). It allows to record exact version of every used
>> dependency, selectively upgrade or replace those dependencies and
>> record that info in binaries (it can be inspected using
>> rsc.io/goversion or, since go 1.13, "go version").
>> 8. Patching is undoable in a way it's done in alpine (using .patch
>> files) because checksums are recorded in binaries and can be verified
>> against google's sumdb. Instead maintainer will need to fork project,
>> apply patches, tag release and use their fork in APKBUILD.
>> 9. To create new package contibutor would need to create apk.go and
>> "blank import" all binaries in it, run "go mod init apk && go mod
>> tidy", then write APKBUILD as usual.
>> 10. To update a package `go get $(go list -f '{{ join .Imports " "
>> }}')` can be used. If specific version is wanted instead of @latest,
>> run manually "go get github.com/9seconds/mtg@v0.16"
>>
>> Does this look sane to you? Should we package it like this? Any other ideas?