Working with abuild a bit more I've came up with a simpler way.
Instead of redefining fetch, making it depend on go and putting stuff
in SRCDEST,
we can instead define a few env vars in APKBUILD like this:
export GOPROXY=${GOPROXY:-https://proxy.golang.org} # Default in
go1.13 and can be removed.
export GOPATH=${GOPATH:-$srcdir/go}
export GOCACHE=${GOCACHE:-$srcdir/go-build}
export GOTMPDIR=${GOTMPDIR:-$srcdir}
This way "go build" will download modules from GOPROXY and place all
build files inside srcdir.
Hoverwer if user has those vars in their environment (ie local dev
machine) or in abuild.conf (ie ci checks or build server),
they will be used instead. This way devs work as they usually do (no
difference between running "go build" manually and from abuild)
and CI can opt in into module and/or build caches if they want to
remove calls to external network and speed up builds.
Build cache has one case when it doesn't always rebuild packages when
it should so enabling it on build servers now might be problematic.
https://github.com/golang/go/issues/24355
Module cache should work well.
I've updated my example:
https://github.com/alpinelinux/aports/commit/ac34f80cdadd78523fde160e1bbc032dd04bfbca
Previous version is here:
https://github.com/alpinelinux/aports/commit/7892cf94cd917c080a0c381caf5ac5e90aa84c97
On Wed, Jul 31, 2019 at 7:59 PM Konstantin Kulikov <k.kulikov2@gmail.com> wrote:
>> Hey everyone.>> I'm trying to package a go project which uses modules. This is what I> came up with.>> https://github.com/alpinelinux/aports/pull/9800>> Notes below:> 1. fetch() depends on go command which means that abuild checksum will> fail if go is not installed.> 2. $SRCDEST will contain "go" directory with module archives using> layout defined by goproxy protocol (see> https://godoc.org/cmd/go#hdr-Module_proxy_protocol). The good thing is> that directory can be served over http and used as real GOPROXY,> avoiding contacting google or other proxies unnecessarily. Current go> projects in aports run "glide install" or "dep ensure" inside> prepare() or build() and do not cache sources at all.> 3. GOPROXY should be defined in abuild.conf and default to> https://proxy.golang.org or any other proxy to avoid depending on> git,svn,bzr and others and allow users to use their own proxies> (googles proxy is unavailable in China for example).> 4. sha512sums doesn't record hashes of module files, instead they are> recorded in go.sum file and go tool verifies them itself.> 5. chmod line in fetch() is unfortunate, without it cleanup step will> fail. Go devs previously refused to add option to make unpacked module> files writable.> 6. fetch() will be the same for every go project and can be defined in> abuild and used with options="gomodule" for example, same goes for> GOPATH var which should be set to the same value for all steps.> 7. To build a project we create dummy module which I called "apk" (but> can be any name). It allows to record exact version of every used> dependency, selectively upgrade or replace those dependencies and> record that info in binaries (it can be inspected using> rsc.io/goversion or, since go 1.13, "go version").> 8. Patching is undoable in a way it's done in alpine (using .patch> files) because checksums are recorded in binaries and can be verified> against google's sumdb. Instead maintainer will need to fork project,> apply patches, tag release and use their fork in APKBUILD.> 9. To create new package contibutor would need to create apk.go and> "blank import" all binaries in it, run "go mod init apk && go mod> tidy", then write APKBUILD as usual.> 10. To update a package `go get $(go list -f '{{ join .Imports " "> }}')` can be used. If specific version is wanted instead of @latest,> run manually "go get github.com/9seconds/mtg@v0.16">> Does this look sane to you? Should we package it like this? Any other ideas?
Hello Konstantin,
that definitely looks easy to do.
I have one concern though.
As far as I'm concerned abuild downloads any network sources when an URL
is given in the apkbuild file.
Buildozer then takes these (additional) source files and archives them
for later retrieval.
Now with that system in mind it is possible to lose sources of old packages.
Before go modules, people used to use go vendoring to get the sources of
libs into their own source tree.
- Paul
Am 11.09.19 um 21:14 schrieb Konstantin Kulikov:
> Working with abuild a bit more I've came up with a simpler way.> Instead of redefining fetch, making it depend on go and putting stuff> in SRCDEST,> we can instead define a few env vars in APKBUILD like this:>> export GOPROXY=${GOPROXY:-https://proxy.golang.org} # Default in> go1.13 and can be removed.> export GOPATH=${GOPATH:-$srcdir/go}> export GOCACHE=${GOCACHE:-$srcdir/go-build}> export GOTMPDIR=${GOTMPDIR:-$srcdir}>> This way "go build" will download modules from GOPROXY and place all> build files inside srcdir.> Hoverwer if user has those vars in their environment (ie local dev> machine) or in abuild.conf (ie ci checks or build server),> they will be used instead. This way devs work as they usually do (no> difference between running "go build" manually and from abuild)> and CI can opt in into module and/or build caches if they want to> remove calls to external network and speed up builds.>> Build cache has one case when it doesn't always rebuild packages when> it should so enabling it on build servers now might be problematic.> https://github.com/golang/go/issues/24355>> Module cache should work well.>> I've updated my example:> https://github.com/alpinelinux/aports/commit/ac34f80cdadd78523fde160e1bbc032dd04bfbca> Previous version is here:> https://github.com/alpinelinux/aports/commit/7892cf94cd917c080a0c381caf5ac5e90aa84c97>> On Wed, Jul 31, 2019 at 7:59 PM Konstantin Kulikov <k.kulikov2@gmail.com> wrote:>> Hey everyone.>>>> I'm trying to package a go project which uses modules. This is what I>> came up with.>>>> https://github.com/alpinelinux/aports/pull/9800>>>> Notes below:>> 1. fetch() depends on go command which means that abuild checksum will>> fail if go is not installed.>> 2. $SRCDEST will contain "go" directory with module archives using>> layout defined by goproxy protocol (see>> https://godoc.org/cmd/go#hdr-Module_proxy_protocol). The good thing is>> that directory can be served over http and used as real GOPROXY,>> avoiding contacting google or other proxies unnecessarily. Current go>> projects in aports run "glide install" or "dep ensure" inside>> prepare() or build() and do not cache sources at all.>> 3. GOPROXY should be defined in abuild.conf and default to>> https://proxy.golang.org or any other proxy to avoid depending on>> git,svn,bzr and others and allow users to use their own proxies>> (googles proxy is unavailable in China for example).>> 4. sha512sums doesn't record hashes of module files, instead they are>> recorded in go.sum file and go tool verifies them itself.>> 5. chmod line in fetch() is unfortunate, without it cleanup step will>> fail. Go devs previously refused to add option to make unpacked module>> files writable.>> 6. fetch() will be the same for every go project and can be defined in>> abuild and used with options="gomodule" for example, same goes for>> GOPATH var which should be set to the same value for all steps.>> 7. To build a project we create dummy module which I called "apk" (but>> can be any name). It allows to record exact version of every used>> dependency, selectively upgrade or replace those dependencies and>> record that info in binaries (it can be inspected using>> rsc.io/goversion or, since go 1.13, "go version").>> 8. Patching is undoable in a way it's done in alpine (using .patch>> files) because checksums are recorded in binaries and can be verified>> against google's sumdb. Instead maintainer will need to fork project,>> apply patches, tag release and use their fork in APKBUILD.>> 9. To create new package contibutor would need to create apk.go and>> "blank import" all binaries in it, run "go mod init apk && go mod>> tidy", then write APKBUILD as usual.>> 10. To update a package `go get $(go list -f '{{ join .Imports " ">> }}')` can be used. If specific version is wanted instead of @latest,>> run manually "go get github.com/9seconds/mtg@v0.16">>>> Does this look sane to you? Should we package it like this? Any other ideas?
Previously developers chose whether to vendor or not and when they didn't,
abuild would download all libs on every build (which takes a few
minutes for telegraf for example).
Some other languages, like rust and javascript suffer from the same problem.
Now with go, once module is published, it can never be changed and thus
can be cached by any goproxy forever. There are currently a few available
publicly (proxy.golang.org, goproxy.io, goproxy.cn). Availability of
modules no longer
depends on developer, but instead on any third-party you choose.
Furthermore alpine can set up their own goproxy (it's implemented by 4
GET requests
and can be served by static filesystem, see
https://golang.org/pkg/cmd/go/#hdr-Module_proxy_protocol
for details) and use that on build and ci servers.
On Thu, Sep 12, 2019 at 8:46 AM Paul Zillmann <p.zillmann@h6g.de> wrote:
>> Hello Konstantin,>> that definitely looks easy to do.> I have one concern though.>> As far as I'm concerned abuild downloads any network sources when an URL> is given in the apkbuild file.> Buildozer then takes these (additional) source files and archives them> for later retrieval.>> Now with that system in mind it is possible to lose sources of old packages.> Before go modules, people used to use go vendoring to get the sources of> libs into their own source tree.>> - Paul>> Am 11.09.19 um 21:14 schrieb Konstantin Kulikov:> > Working with abuild a bit more I've came up with a simpler way.> > Instead of redefining fetch, making it depend on go and putting stuff> > in SRCDEST,> > we can instead define a few env vars in APKBUILD like this:> >> > export GOPROXY=${GOPROXY:-https://proxy.golang.org} # Default in> > go1.13 and can be removed.> > export GOPATH=${GOPATH:-$srcdir/go}> > export GOCACHE=${GOCACHE:-$srcdir/go-build}> > export GOTMPDIR=${GOTMPDIR:-$srcdir}> >> > This way "go build" will download modules from GOPROXY and place all> > build files inside srcdir.> > Hoverwer if user has those vars in their environment (ie local dev> > machine) or in abuild.conf (ie ci checks or build server),> > they will be used instead. This way devs work as they usually do (no> > difference between running "go build" manually and from abuild)> > and CI can opt in into module and/or build caches if they want to> > remove calls to external network and speed up builds.> >> > Build cache has one case when it doesn't always rebuild packages when> > it should so enabling it on build servers now might be problematic.> > https://github.com/golang/go/issues/24355> >> > Module cache should work well.> >> > I've updated my example:> > https://github.com/alpinelinux/aports/commit/ac34f80cdadd78523fde160e1bbc032dd04bfbca> > Previous version is here:> > https://github.com/alpinelinux/aports/commit/7892cf94cd917c080a0c381caf5ac5e90aa84c97> >> > On Wed, Jul 31, 2019 at 7:59 PM Konstantin Kulikov <k.kulikov2@gmail.com> wrote:> >> Hey everyone.> >>> >> I'm trying to package a go project which uses modules. This is what I> >> came up with.> >>> >> https://github.com/alpinelinux/aports/pull/9800> >>> >> Notes below:> >> 1. fetch() depends on go command which means that abuild checksum will> >> fail if go is not installed.> >> 2. $SRCDEST will contain "go" directory with module archives using> >> layout defined by goproxy protocol (see> >> https://godoc.org/cmd/go#hdr-Module_proxy_protocol). The good thing is> >> that directory can be served over http and used as real GOPROXY,> >> avoiding contacting google or other proxies unnecessarily. Current go> >> projects in aports run "glide install" or "dep ensure" inside> >> prepare() or build() and do not cache sources at all.> >> 3. GOPROXY should be defined in abuild.conf and default to> >> https://proxy.golang.org or any other proxy to avoid depending on> >> git,svn,bzr and others and allow users to use their own proxies> >> (googles proxy is unavailable in China for example).> >> 4. sha512sums doesn't record hashes of module files, instead they are> >> recorded in go.sum file and go tool verifies them itself.> >> 5. chmod line in fetch() is unfortunate, without it cleanup step will> >> fail. Go devs previously refused to add option to make unpacked module> >> files writable.> >> 6. fetch() will be the same for every go project and can be defined in> >> abuild and used with options="gomodule" for example, same goes for> >> GOPATH var which should be set to the same value for all steps.> >> 7. To build a project we create dummy module which I called "apk" (but> >> can be any name). It allows to record exact version of every used> >> dependency, selectively upgrade or replace those dependencies and> >> record that info in binaries (it can be inspected using> >> rsc.io/goversion or, since go 1.13, "go version").> >> 8. Patching is undoable in a way it's done in alpine (using .patch> >> files) because checksums are recorded in binaries and can be verified> >> against google's sumdb. Instead maintainer will need to fork project,> >> apply patches, tag release and use their fork in APKBUILD.> >> 9. To create new package contibutor would need to create apk.go and> >> "blank import" all binaries in it, run "go mod init apk && go mod> >> tidy", then write APKBUILD as usual.> >> 10. To update a package `go get $(go list -f '{{ join .Imports " "> >> }}')` can be used. If specific version is wanted instead of @latest,> >> run manually "go get github.com/9seconds/mtg@v0.16"> >>> >> Does this look sane to you? Should we package it like this? Any other ideas?