cmd/go: repeated `run pkg@version` calls hit the proxy for no apparent reason

[mvdan]

Go version

go version go1.24.1 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/mvdan/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/mvdan/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build878653294=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/mvdan/go/pkg/mod'
GONOPROXY='github.com/cue-unity'
GONOSUMDB='github.com/cue-unity'
GOOS='linux'
GOPATH='/home/mvdan/go'
GOPRIVATE='github.com/cue-unity'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='on'
GOTELEMETRYDIR='/home/mvdan/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.1'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Ran go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 -version a number of times.

What did you see happen?

$ go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 -version
WORK=/tmp/go-build3841505877
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.043s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)
$ go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 -version
WORK=/tmp/go-build1725445701
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.044s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)
$ go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 -version
WORK=/tmp/go-build1793441905
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.049s)
/home/mvdan/.cache/go-build/ce/cefc08a55ed89b9e6ccbf96cc8155a15c7b548b63404c0988cc5b9998be24db2-d/staticcheck -version
staticcheck 2024.1.1 (0.5.1)

What did you expect to see?

I would expect the repeated runs to not hit GOPROXY at all; the module version is fully cached locally via GOMODCACHE.

Note that this is separate from go run caching binaries now. Even without caching of binaries, I would expect the local module cache to be sufficient so that the build can happen without hitting GOPROXY.

[gabyhelp]

Related Issues

• cmd/go: go get hits proxy to check retracted versions for transitive dependencies too much #42185 (closed)
• cmd/go 'go mod download <package name>' fetches all modules if module cache is empty/missing #40245 (closed)
• cmd/go: get of module already present in proxy can be slow #50158
• cmd/go: GOPROXY default can make 'get -u' lag upstream repository #32870
• go get #56314 (closed)
• proxy.golang.org: update to Go 1.13.3 to pick up the fix for #34477 #34991 (closed)
• proxy.golang.org: slow resolution of commit already pulled through the proxy #54977 (closed)
• proxy.golang.org: missing package #32542 (closed)
• cmd/go: tests fail when GOROOT is stale #48698 (closed)
• cmd/go: package resolution fails after selecting incorrect module when using multiple proxies #40672

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[rogpeppe]

Here's another interesting observation: how much the network is accessed depends a lot on whether GOPRIVATE is set or not (apparently regardless of the actual value of GOPRIVATE).

Here's a testscript reproducer:

env GOCACHE=$WORK/.cache
go env
# without GOPRIVATE
go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version

# without GOPRIVATE but with cache
go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version

# without GOPRIVATE but with cache (again, to be sure)
go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version

# with GOPRIVATE
env GOPRIVATE=something.unrelated
go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version

# with GOPRIVATE with cache
go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version

I see this output from testscript -v, with some lengthy output removed for brevity.

> env GOCACHE=$WORK/.cache
> go env
[stdout]
AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='$WORK/.cache'
GOCACHEPROG=''
GODEBUG=''
GOENV='/no-home/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=$WORK/.tmp/go-build1245423453=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='$WORK/.gopath/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='$WORK/.gopath'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/rogpeppe/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/no-home/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/rogpeppe/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.0'
GOWORK=''
PKG_CONFIG='pkg-config'

# without GOPRIVATE (8.745s)
> go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
go: downloading honnef.co/go/tools v0.5.1
WORK=$WORK/.tmp/go-build3168116774
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
....
[7618 lines elided]
....
cp $WORK/b001/exe/staticcheck $WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck # internal
$WORK/b001/exe/staticcheck --version

# without GOPRIVATE but with cache (0.239s)
> go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build3961093243
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info: 404 Not Found (0.034s)
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info: 404 Not Found (0.034s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info: 404 Not Found (0.035s)
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info: 404 Not Found (0.035s)
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.013s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# without GOPRIVATE but with cache (again, to be sure) (0.224s)
> go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build2190875445
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info
# get https://proxy.golang.org/honnef.co/@v/v0.5.1.info: 404 Not Found (0.027s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/@v/v0.5.1.info: 404 Not Found (0.028s)
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.013s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# with GOPRIVATE (0.185s)
> env GOPRIVATE=something.unrelated
> go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build2012329811
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.082s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

# with GOPRIVATE with cache (0.173s)
> go run -x honnef.co/go/tools/cmd/staticcheck@v0.5.1 --version
[stdout]
staticcheck 2024.1.1 (0.5.1)

[stderr]
WORK=$WORK/.tmp/go-build3915661201
# get https://proxy.golang.org/honnef.co/go/tools/@v/list
# get https://proxy.golang.org/honnef.co/go/tools/@v/list: 200 OK (0.074s)
$WORK/.cache/a5/a569e2ffa607a72fa041c20dea4798a04ee909df2be86bea26ed867e645ed1b1-d/staticcheck --version

PASS

I see the same behaviour under Go tip as of go1.25-24b395119b

[seankhliao]

maybe because of #59230

[rogpeppe]

@seankhliao that would explain the hit on @latest, but not difference with respect to GOPRIVATE AFAICS.

[matloob]

So the reason we're hitting the proxy is to get the deprecations.

The GOPRIVATE behavior is a bit harder to explain but I think it's still correct. So what's going on is that when GOPRIVATE is set GOPROXY implicitly has a first element that will lookup only the GOPRIVATE matching paths directly before trying other proxies with the GOPRIVATE matching paths disabled.

When we're trying to figure out which module honnef.co/go/tools/cmd/staticcheck@v0.5.1 belongs to, we try all the prefixes to see which ones exist. In the non-GOPRIVATE case, we directly send all of them to the module proxy. That's the normal behavior you see. But in the GOPRIVATE case, we first try to only resolve the GOPRIVATE matching paths directly. I think the thing that introduces a bit of confusion here is that we're always allowed to use the module cache regardless of the setting of GOPRIVATE. So essentially, with GOPRIVATE set to something that doesn't match any of the staticcheck prefixes, we try to look them all up using the module cache first. The honnef.co/go/tools lookup succeeds so the operation of finding the modules that exist succeeds and doesn't fail over to the first real element of GOPROXY.

[cherrymui]

Sounds like this works as intended? Is there any suggestion for improvement? Otherwise perhaps we can close this. Thanks.

[matloob]

Yes I think this is working as intended. In the discussion offline, @seankhliao suggested the proposal #43646 as a potential option to skip some of this work. That's probably the way we want to go.

[cherrymui]

Thanks. Sounds like we can close this, and use #43646 (or new proposals) for potential improvements.

[rogpeppe]

Sorry for the delayed response.

Yes I think this is working as intended. In the discussion offline, @seankhliao suggested the proposal #43646 as a potential option to skip some of this work. That's probably the way we want to go.

I don't think that GOPROXY=cache per se would really fit my use case. We have go run pkg@version statements in shell scripts and the like, and when the script is run, we don't know whether anything is installed yet. So to take advantage of #43646, we'd need to first run some command to pre-populate the cache, which seems like it would require knowing in advance (or scanning for) all the go run occurrences that might or might happen, because otherwise the command will fail when the content isn't available.

I guess you're going to say "just set GOPROXY=cache,https://proxy.golang.org,direct" and I guess that's a possibility, but I don't want to have to remember to set it for just this use case and I wouldn't want it to synthesise results for @latest.

So what's going on is that when GOPRIVATE is set GOPROXY implicitly has a first element that will lookup only the GOPRIVATE matching paths directly before trying other proxies with the GOPRIVATE matching paths disabled.

This feels to me like an implementation detail not a reason that it should work this way. AFAICS nothing in the module docs say anything about GOPRIVATE acting like an implicit first element. Instead, GOPRIVATE is always phrased as if it were orthogonal to the order of lookup (e.g. "GOPRIVATE — list of glob patterns of module path prefixes that should be considered private.").

In general, I at least do have a workaround for now, but explaining it to people will be odd: "set GOPRIVATE to something, anything, and go run will run faster".

In fact, given that the behaviour this behaviour is considered OK when GOPRIVATE has no relationship to any module that's being used, why not always just behave as if there's an implicit first element in GOPROXY that allows us to check the cache before doing anything else? Is there any down side to doing that?