gopls downloads things at build time

[ryandesign (Ryan Carsten Schmidt)]

When I build gopls I see:

go: downloading mvdan.cc/xurls/v2 v2.2.0
go: downloading honnef.co/go/tools v0.0.1-2020.1.4
go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
go: downloading github.com/sergi/go-diff v1.1.0
go: downloading golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
go: downloading golang.org/x/mod v0.2.0
go: downloading github.com/BurntSushi/toml v0.3.1

Ports shouldn't download files at build time. Download everything you need using distfiles in the fetch phase.

[ra1nb0w]

I need to investigate more deeply go.vendors because it doesn't work with external domains. Is there a good skeleton that I can use as reference?

[ryandesign (Ryan Carsten Schmidt)]

I have not familiarized myself with go... maybe check who maintains other go ports and ask them if they have a solution.

[ra1nb0w]

I can't understand how to manage mvdan.cc; can you suggest a developer to contact? thank you

[danieltrautmann (Daniel Trautmann)]

I'm not sure what the exact issue is you have, but I guess it's the v2 part of the module name. I'm quite new to MacPorts and tried to figure out how go.vendors is implemented, but failed to find the code. My guess is that the implementation doesn't support Go modules well.

However, I can try to help with the Go part of this issue.

[ra1nb0w]

mvdan.cc uses go-import/go-source to redirect to the github repository (we need to maintain this hierarchy) but the go portgroup doesn't support those and doesn't work with repository different from github/gitlab/bitbucket. Those redirect should be something related to GO111MODULE but I didn't understood the whole implications. Anyway, any help will be appreciate :) You can read the portgroup at ​https://github.com/macports/macports-ports/blob/master/_resources/port1.0/group/golang-1.0.tcl

[danieltrautmann (Daniel Trautmann)]

I had a look at the code of the portgroup. I don't know much about tcl, but it seems you are right that it can't handle this case. Even module names starting with gopkg.in or golang.org are handled by rewriting some variables.

Can you check if it is possible to use the GitHub URL github.com/mvdan/xurls in the go.vendors option? I guess lock should be v2.2.0.

Also there is a tool called go2port, which is available as port, which can generate the portfile based on the Go module. For now it seems that it doesn't support these package names either (there is an open issue), but someone already made a pull request for that. Maybe you can checkout is fork and see if it works. ​https://github.com/amake/go2port/pull/3

[ra1nb0w]

Replying to danieltrautmann:

Can you check if it is possible to use the GitHub URL github.com/mvdan/xurls in the go.vendors option? I guess lock should be v2.2.0.

we need to respect the hierarchy and this is done with the url therefore I can't use the true source without moving back to the name used on the dependency declaration.

Also there is a tool called go2port, which is available as port, which can generate the portfile based on the Go module. For now it seems that it doesn't support these package names either (there is an open issue), but someone already made a pull request for that. Maybe you can checkout is fork and see if it works. ​https://github.com/amake/go2port/pull/3

Tried and seems it doesn't work and unfortunately gopls is contained in another repository with many bins (see go-tools). The other option is to use a custom pre-build or pre-fetch but I didn't found a way that worked well.

[danieltrautmann (Daniel Trautmann)]

Okay, I will have a look at the pre-build / pre-fetch too.

Essentially what you are searching for is something like the go.package option for vendors. Am I right?

IMHO the best way to do that stuff is to let go get handle it, because then you don't have to reinvent the wheel, but I guess there are some good reasons why it is that way.

[ra1nb0w]

Probably there are other packages that run go during the fetch phase or something similar but I did not have time to check yet.

[danieltrautmann (Daniel Trautmann)]

I've went through all other Go based ports and found one hack which you can see here: ​https://github.com/macports/macports-ports/blob/master/devel/shfmt/Portfile#L130

The dependency is fetched via go.vendors and then moved inside the GOPATH.

Nearly no other port seems to care about downloading stuff in the build phase.

Besides that I reverse engineered the way how Bazel (A build tool I use) does fetch Go dependencies, which boils down to using plain go mod download. I also checked hat go mod download actually does, which is this:

$ go mod download -x mvdan.cc/xurls/v2
# get https://proxy.golang.org/honnef.co/go/tools/@v/v0.0.1-2020.1.4.mod
# get https://proxy.golang.org/golang.org/x/mod/@v/v0.3.0.mod
# get https://proxy.golang.org/github.com/yuin/goldmark/@v/v1.1.32.mod
# get https://proxy.golang.org/github.com/sergi/go-diff/@v/v1.1.0.mod
# get https://proxy.golang.org/golang.org/x/sync/@v/v0.0.0-20200625203802-6e8e738ad208.mod
# get https://proxy.golang.org/golang.org/x/net/@v/v0.0.0-20200625001655-4c5254603344.mod
# get https://proxy.golang.org/mvdan.cc/gofumpt/@v/v0.0.0-20200709182408-4fd085cb6d5f.mod
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.mod
# get https://proxy.golang.org/golang.org/x/sync/@v/v0.0.0-20200625203802-6e8e738ad208.mod: 200 OK (1.217s)
# get https://proxy.golang.org/github.com/yuin/goldmark/@v/v1.1.32.mod: 200 OK (1.217s)
# get https://proxy.golang.org/mvdan.cc/gofumpt/@v/v0.0.0-20200709182408-4fd085cb6d5f.mod: 200 OK (1.217s)
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.mod: 200 OK (1.242s)
# get https://proxy.golang.org/golang.org/x/mod/@v/v0.3.0.mod: 200 OK (1.242s)
# get https://proxy.golang.org/golang.org/x/net/@v/v0.0.0-20200625001655-4c5254603344.mod: 200 OK (1.242s)
# get https://proxy.golang.org/golang.org/x/sys/@v/v0.0.0-20200323222414-85ca7c5b95cd.mod
# get https://proxy.golang.org/gopkg.in/check.v1/@v/v1.0.0-20190902080502-41f04d3bba15.mod
# get https://proxy.golang.org/github.com/rogpeppe/go-internal/@v/v1.6.0.mod
# get https://proxy.golang.org/github.com/rogpeppe/go-internal/@v/v1.5.2.mod
# get https://proxy.golang.org/honnef.co/go/tools/@v/v0.0.1-2020.1.4.mod: 200 OK (1.446s)
# get https://proxy.golang.org/github.com/sergi/go-diff/@v/v1.1.0.mod: 200 OK (1.445s)
# get https://proxy.golang.org/golang.org/x/sys/@v/v0.0.0-20200323222414-85ca7c5b95cd.mod: 200 OK (0.131s)
# get https://proxy.golang.org/gopkg.in/check.v1/@v/v1.0.0-20190902080502-41f04d3bba15.mod: 200 OK (0.131s)
# get https://proxy.golang.org/github.com/rogpeppe/go-internal/@v/v1.5.2.mod: 200 OK (0.134s)
# get https://proxy.golang.org/github.com/rogpeppe/go-internal/@v/v1.6.0.mod: 200 OK (0.153s)
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.info
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.info: 200 OK (0.026s)
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.zip
# get https://proxy.golang.org/mvdan.cc/xurls/v2/@v/v2.2.0.zip: 200 OK (0.026s)

So in the end it downloads a Zip file which contains the files and fit the structure required for GOPATH.

[Davide Gerhard <ra1nb0w@…>]

In 5c28b37efeaf39c4d956277f7bdbcd650897e0d9/macports-ports (master):

gopls: update version to 0.4.4

Closes: #60802

[ra1nb0w]

thank you very much Daniel! Done but it is very cumbersome to implement and maintain!