os: os.Chtimes is corrupting very old timestamps

[ncw]

Go version

go version go1.25.0 linux/amd64

Output of go env in your module/workspace:

Details

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/ncw/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/ncw/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build512443455=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/dev/null'
GOMODCACHE='/home/ncw/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/ncw/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/go/go1.25'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/ncw/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/go/go1.25/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.25.0'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

Running this program demonstrates the problem. It needs to be run on a file system that supports a wide timerange. ext4 doesn't but ntfs-3g does.

package main

import (
	"fmt"
	"log"
	"os"
	"os/exec"
	"path/filepath"
	"time"
)

func statAndPrint(label, path string, expected time.Time) {
	fi, err := os.Stat(path)
	if err != nil {
		log.Fatalf("stat (%s): %v", label, err)
	}
	actual := fi.ModTime().UTC()
	fmt.Printf("%s\n", label)
	fmt.Printf("Expected mtime: %sZ (unix nano=%d)\n", expected.Format("2006-01-02T15:04:05"), expected.UnixNano())
	fmt.Printf("Actual   mtime: %s (unix nano=%d)\n", actual.Format(time.RFC3339), actual.UnixNano())
	if expected.Equal(actual) {
		fmt.Printf("ALL OK - times equal\n")
	} else {
		fmt.Printf("FAIL - times differ by %v\n", actual.Sub(expected))
	}
}

func main() {
	if len(os.Args) < 2 {
		log.Fatalf("usage: %s <directory>", os.Args[0])
	}
	dir := os.Args[1]
	fname := filepath.Join(dir, "chtimes_test.txt")

	const expectedStr = "1665-12-01T13:57:13" // interpreted as UTC

	// Create (or truncate) the file
	if err := os.WriteFile(fname, []byte{}, 0o644); err != nil {
		log.Fatalf("create: %v", err)
	}

	// Parse expected time in UTC and set both atime & mtime via os.Chtimes
	expected, err := time.ParseInLocation("2006-01-02T15:04:05", expectedStr, time.UTC)
	if err != nil {
		log.Fatalf("parse: %v", err)
	}
	if err := os.Chtimes(fname, expected, expected); err != nil {
		log.Fatalf("chtimes: %v", err)
	}
	statAndPrint("Set timestamp with os.Chtimes", fname, expected)

	fmt.Printf("------------------------------------------------------------\n")

	// Now set the timestamp via the system `touch` command.
	// Use TZ=UTC so the -t timestamp is not affected by local timezone.
	touchArg := expected.Format("200601021504.05") // CCYYMMDDhhmm.SS
	cmd := exec.Command("touch", "-t", touchArg, fname)
	cmd.Env = append(os.Environ(), "TZ=UTC")
	if out, err := cmd.CombinedOutput(); err != nil {
		log.Fatalf("touch: %v; output: %s", err, string(out))
	}
	statAndPrint("Set timestamp with touch (TZ=UTC)", fname, expected)
}

You can create a suitable file system to check this on with an NTFS-3g loopback on Linux like this

truncate -s 10M ntfs.img
sudo mkfs.ntfs -F -L NTFS_VOL ntfs.img
sudo mkdir -p /mnt/ntfsimg
sudo mount -t ntfs-3g -o loop,uid=$UID,gid=$(id -g) ntfs.img /mnt/ntfsimg

What did you see happen?

It attempts to set a 1665 date as a timestamp on a file. This is read back as a 2250 timestamp.

Set timestamp with os.Chtimes
Expected mtime: 1665-12-01T13:57:13Z (unix nano=8850864706709551616)
Actual   mtime: 2250-06-22T13:31:46Z (unix nano=8850864706709551600)
FAIL - times differ by 2562047h47m16.854775807s

It then sets the timestamp with touch which does work as expected (showing that this date is representable by the file system and by Go time.Time).

Set timestamp with touch (TZ=UTC)
Expected mtime: 1665-12-01T13:57:13Z (unix nano=8850864706709551616)
Actual   mtime: 1665-12-01T13:57:13Z (unix nano=8850864706709551616)
ALL OK - times equal

Note how all the unix nano values are the same (+/- 16nS)

What did you expect to see?

I expected the go standard library not to lose the extra precision in the Go timestamp when applying it to the file.

The problem appears to be here

go/src/os/file_posix.go

Lines 179 to 199 in 3cf1aaf

	func Chtimes(name string, atime time.Time, mtime time.Time) error {
	utimes := chtimesUtimes(atime, mtime)
	if e := syscall.UtimesNano(fixLongPath(name), utimes[0:]); e != nil {
	return &PathError{Op: "chtimes", Path: name, Err: e}
	}
	return nil
	}
	func chtimesUtimes(atime, mtime time.Time) [2]syscall.Timespec {
	var utimes [2]syscall.Timespec
	set := func(i int, t time.Time) {
	if t.IsZero() {
	utimes[i] = syscall.Timespec{Sec: _UTIME_OMIT, Nsec: _UTIME_OMIT}
	} else {
	utimes[i] = syscall.NsecToTimespec(t.UnixNano())
	}
	}
	set(0, atime)
	set(1, mtime)
	return utimes
	}

In particular the use of UnixNano() to convert the time as the 1665 date is not representable as an int64 nS since the epoch but it is representable as a syscall.Timespec

type Timespec struct {
	Sec  int64
	Nsec int64
}

This could probably fixed by constructing the Timespec something like this instead of utimes[i] = syscall.NsecToTimespec(t.UnixNano())

utimes[i] = syscall.Timespec{
    Sec: t.Unix(),
    Nsec: int64(t.Nanosecond()),
}

Though I am uncertain as to whether there is any rounding in t.Unix() which might affect things.

Note that this likely affects Windows too as the time is roundtripped through an int64 there too

go/src/os/root_windows.go

Lines 367 to 372 in 3cf1aaf

	if !atime.IsZero() {
	a = syscall.NsecToFiletime(atime.UnixNano())
	}
	if !mtime.IsZero() {
	w = syscall.NsecToFiletime(mtime.UnixNano())
	}

This problem was discovered by an rclone user in rclone/rclone#8834

[gabyhelp]

Related Issues

• os, cmd/gofmt: tests fail on ext4 with 128byte inodes due to insufficient time precision #75042
• os: Chtimes: support nanosecond resolution on Mac OS X #22528 (closed)
• syscall: UtimesNano does not use nanosecond system call on BSD, Solaris #16480 (closed)
• time: Now() faster than reported timestamps from filesystem #33510 (closed)
• x/sys/unix: TestUtimesNanoAt fails #26034 (closed)
• proposal: os: add Touch to set access/mod times to current time #31880 (closed)
• time: time.Before return an unexpected result #42056 (closed)

Related Code Changes

• os: make Chtimes accept empty time values to skip file time modification

Related Discussions

• Improving the accuracy of os.Chtimes
• os.Chtimes bounds on time (32 and 64 bit file time)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[Jorropo]

I don't see any rounding in t.Unix (only truncation). It looks safe to me, please send a patch.

We ideally want a test but it might or might not be pain to do on the runners.
Your new implementation looks obviously correct and it will still be tested in the existing time ranges so I would still merge it even if you don't add a test.

[ncw]

Thanks for the feedback @Jorropo

I'll work on a patch.

[ncw]

@Jorropo looking into this further I see that the idea that a datetime can fit into UnixNano int64 is throughout the syscall code and also in golang.org/x/sys/unix and golang.org/x/sys/windows whereas this isn't true for any 64 bit Linux and has never been true for Windows.

For example uses like this

src/os/root_windows.go:         a = syscall.NsecToFiletime(atime.UnixNano())
src/os/root_windows.go:         w = syscall.NsecToFiletime(mtime.UnixNano())

Or this

src/syscall/syscall_linux_arm64.go:             NsecToTimespec(TimevalToNsec(tv[0])),
src/syscall/syscall_linux_arm64.go:             NsecToTimespec(TimevalToNsec(tv[0])),
src/syscall/syscall_linux_loong64.go:           NsecToTimespec(TimevalToNsec(tv[0])),

Or this

src/syscall/syscall_bsd.go:             NsecToTimeval(TimespecToNsec(ts[0])),

All of which are needlessly truncating date/times into an int64.

This could do with some support from syscall, eg something like

 
 package syscall
 
+import "time"
+
 // TimespecToNsec returns the time stored in ts as nanoseconds.
 func TimespecToNsec(ts Timespec) int64 { return ts.Nano() }
 
@@ -20,9 +22,21 @@ func NsecToTimespec(nsec int64) Timespec {
 	return setTimespec(sec, nsec)
 }
 
+// TimeToTimespec converts a time.Time into a [Timespec].
+func TimeToTimespec(t time.Time) Timespec {
+	return setTimespec(t.Unix(), int64(t.Nanosecond()))
+}
+
 // TimevalToNsec returns the time stored in tv as nanoseconds.
 func TimevalToNsec(tv Timeval) int64 { return tv.Nano() }
 
+// TimevalToTimespec converts a [Timeval] into a [Timespec].
+func TimevalToTimespec(tv Timeval) Timespec {
+	sec, usec := tv.Unix()
+	nsec := usec * 1e3
+	setTimespec(tv, nsec)
+}
+
 // NsecToTimeval converts a number of nanoseconds into a [Timeval].
 func NsecToTimeval(nsec int64) Timeval {
 	nsec += 999 // round up to microsecond
@@ -34,3 +48,16 @@ func NsecToTimeval(nsec int64) Timeval {
 	}
 	return setTimeval(sec, usec)
 }
+
+// TimespecToTimeval converts a [Timespec] into a [Timeval].
+func TimespecToTimeval(ts Timespec) Timeval {
+	sec, nsec := ts.Unix()
+	nsec += 999 // round up to microsecond
+	usec := nsec % 1e9 / 1e3
+	sec := nsec / 1e9
+	if usec < 0 {
+		usec += 1e6
+		sec--
+	}
+	return setTimeval(sec, usec)
+}

The same changes would need to be made in golang.org/x/sys/unix too.

This is starting to sound like a big job!

Should I

1. Just do the small fix outlined above in os
2. Do the full fix in syscall and os (am I allowed to make new public functions in syscall - I thought it was frozen?)
3. Do the full fix in syscall and os and golang.org/x/sys/unix?
4. Something else?

Thanks

[Jorropo]

@ncw unless there is some code that depends on the truncation behavior (which would be surprising) it makes sense to fix absolutely all the cases where we find this pattern.

Making conversions functions between the different types would be fine but you need to be really careful with import order in the std since it's a import cycle hotspot,
there is a test checking the import order against a hardcoded list, you can change the hardcoded list but it forces to ask questions.
It's fine to change the list but it might require refactoring stuff unrelated to your CL and require more people to take a look, best to avoid to go smoothly.

I don't know if syscall is frozen or not but if you want to make a public function in syscall you need to submit a proposal anyway.
There isn't a problem creating a new package under src/internal or private syscall functions.
From memory golang.org/x/sys/unix is not covered by the proposal process, for this one probably just send what you think is the best solution and relevant people might take a look at the CL.

Tl;Dr: changing a public std package API requires a proposal, this will take longer and require more peoples to take a look.
Ideally you can fix std consumers without changing any public API.
Then you would have really good points to open a proposal adding time.Time ←→ syscall.Time* functions like you shown in your last message (doesn't have to be sequenced in this particular order).