cmd/go: buildmode=shared config should not be read back from $GOROOT/pkg

[rsc]

The $GOROOT/pkg (or $GOPATH/pkg) directory is meant to be a cache of precomputed results, all of which can be recomputed as needed. In particular the ideal is that you get the same build results both before and after doing rm -r pkg, and that you get the same build results regardless of what commands have run before.

This property is not true of buildmode=shared: the package groupings passed to go install -buildmode=shared commands are recorded only in the content of the pkg directory. That is, if you run

go install -buildmode=shared runtime sync/atomic
go install -linkshared myprog

then you get a myprog installed that links against pkg/libruntime,sync-atomic.so. It knows to do this because it read that string from both pkg/runtime.shlibname and pkg/sync/atomic.shlibname. If pkg had been removed, the resulting myprog would be different (it wouldn't depend on any shared libraries).

Also, if you run:

go install -buildmode=shared runtime sync/atomic
<modify files in runtime>
go install -linkshared myprog

then the install of myprog is actually expected to (and does) update pkg/libruntime,sync-atomic.so. Even if myprog made no mention of sync/atomic, cmd/go is expected to know what packages are built into pkg/libruntime,sync-atomic.so because it reads the list back out of the .so file.

Both the .shlibname files and the package list in the .so file are irreplaceable: if they are deleted, the go command cannot work out what they should contain. They make pkg not a proper cache but a source of truth for build configuration.

In the long term we need to move all configuration of this sort outside the pkg directory. I don't expect this to happen for Go 1.10, but I'd like to try to figure out a plan for Go 1.11.

It seems to me that the questions that must be answered are (1) what form does the configuration take, and (2) where is it kept? The form is not interesting here; what matters is where we keep the config.

One possibility is that the configuration file lives in a well-known location, like $GOROOT/src/shared.cfg and $GOPATH/src/shared.cfg.

One possibility is that the configuration file must be specified in any particular build:

go install -buildmode=shared -sharedcfg=x.cfg std
go install -linkshared -sharedcfg=x.cfg myprog

One possibility is that there's a well-known location but the file can be overridden, or perhaps augmented, by specifying a config.

I don't know enough about how this buildmode is used to say what the best approach is. Obviously if the people making decisions about the shared library structure are not the owners of $GOROOT, then a file in $GOROOT is not sufficient by itself. But maybe a default file in $GOROOT plus the ability to override makes more sense so that everyone doesn't reinvent the wheel. I don't know.

Thoughts?

/cc @mwhudson @ianlancetaylor @crawshaw

[mwhudson]

I agree that storing irreplaceable data in pkg/ is bad. I've never really been happy about this aspect of buildmode=shared.

I don't know how relevant it is, but I think the package list in the .so and the .shlibname files basically contain the same information -- the groupings of packages that are intended to go into a shared library together.

From the distro POV, any of your proposals would be OK I think. TBH, from a distro POV assuming each repo (in the sense of a thing you pass to go get) gets its own shared library would be fine and obviate your concerns. Where that falls down is testing, writing misc/cgo/testshared would be a lot harder in this model! And it would be much much slower if it had to build all of the standard library into a shared library.

So I don't know if that helps any, I'm afraid.

[seankhliao]

Obsoleted by #47788