"too many open files" since update to v1.43.0

[julienschmidt]

Welcome

• Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).
• Yes, I've tried with the standalone linter if available. (https://golangci-lint.run/usage/linters/)

Description of the problem

With v1.42.1 everything was fine, however since the update to v1.43.0 I often get some sort of "too many open file" errors when running it on a rather big (closed-source) project, either in combination with "failed to save|get data to low-level cache", linters not being able to run like gci in the example below, or errors like could not import <...> (<...>.go:1: too many open files) (typecheck).

The errors only occur when gocritic is enabled and stand-alone gocritic works just fine. The likelihood of such errors greatly increases when I run gociritic together with some other linters. My best guess is, that this is some resource leak connected with gocritic.

Version of golangci-lint

$ golangci-lint --version
golangci-lint has version 1.43.0 built from 861262b on 2021-11-02T20:54:42Z

Configuration file

run:
  skip-dirs-use-default: true
  modules-download-mode: vendor
  timeout: 5m

output:
  uniq-by-line: true
  sort-results: true

linters-settings:
  gocritic:
    disabled-checks:
      - sloppyReassign
      - whyNoLint

    enabled-tags:
      - diagnostic
      - style
      - performance

linters:
  enable:
    - asciicheck
    - bodyclose
    - deadcode
    - durationcheck
    - errcheck
    - errorlint
    - exhaustive
    - exportloopref
    - forcetypeassert
    - gci
    - gocritic
    - gofmt
    - goimports
    - gomoddirectives
    - goprintffuncname
    - gosec
    - gosimple
    - govet

Go environment

$ go version && go env
go version go1.17.2 darwin/amd64
GO111MODULE=""
GOARCH="amd64"
GOBIN="/usr/local/bin"
GOCACHE="/Users/julienschmidt/Library/Caches/go-build"
GOENV="/Users/julienschmidt/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/julienschmidt/go/pkg/mod"
GONOPROXY="<REDACTED>"
GONOSUMDB="<REDACTED>"
GOOS="darwin"
GOPATH="/Users/julienschmidt/go"
GOPRIVATE="<REDACTED>"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.17.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.17.2/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="<REDACTED>"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3v/sp9m614906g_7xcfhfhycyzm0000gn/T/go-build343025515=/tmp/go-build -gno-record-gcc-switches -fno-common"

Verbose output of running

$ golangci-lint cache clean
$ golangci-lint run -v --disable-all ./...                                             13:23
INFO [config_reader] Config search paths: [./ <REDACTED> /Users/julienschmidt/dev /Users/julienschmidt /Users /]
INFO [config_reader] Used config file .golangci.yaml
INFO [lintersdb] Active 18 linters: [asciicheck bodyclose deadcode durationcheck errcheck errorlint exhaustive exportloopref forcetypeassert gci gocritic gofmt goimports gomoddirectives goprintffuncname gosec gosimple govet]
INFO [loader] Go packages loading at mode 575 (files|types_sizes|compiled_files|exports_file|imports|name|deps) took 2.827652292s
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 31.628005ms
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key printf/facts for package <package1>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key ctrlflow/facts for package <package2>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key buildir/facts for package <package3>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key ctrlflow/facts for package <package4>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key printf/facts for package <package4>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key exhaustive/facts for package <package4>: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
INFO [linters context/goanalysis] analyzers took 8m27.322021697s with top 10 stages: gocritic: 6m18.642849702s, buildir: 58.238175393s, buildssa: 16.741482839s, goimports: 7.129339347s, gosec: 4.520575362s, gofmt: 4.031518176s, exhaustive: 3.735362383s, inspect: 2.677044192s, findcall: 2.519388948s, printf: 2.37948183s
WARN [runner] Can't run linter goanalysis_metalinter: gci: open <REDACTED>.go: too many open files
INFO [runner] processing took 5.016µs with stages: max_same_issues: 2.131µs, exclude: 410ns, skip_dirs: 368ns, nolint: 301ns, max_from_linter: 218ns, path_prettifier: 195ns, uniq_by_line: 147ns, cgo: 145ns, filename_unadjuster: 142ns, source_code: 135ns, skip_files: 127ns, identifier_marker: 124ns, autogenerated_exclude: 121ns, max_per_file_from_linter: 77ns, diff: 65ns, sort_results: 64ns, path_shortener: 64ns, severity-rules: 61ns, path_prefixer: 61ns, exclude-rules: 60ns
INFO [runner] linters took 31.656323073s with stages: goanalysis_metalinter: 31.656273088s
INFO File cache stats: 0 entries of total size 0B
INFO Memory: 314 samples, avg is 2325.8MB, max is 3566.2MB
INFO Execution took 34.525925327s

Code example or link to a public repository

none

[ldez]

Hello,

if someone can provide a way to reproduce the problem, it could help in the diagnosis.

Maybe @sebastien-rosset can take a look.

Note: It can be related to the performance problem that I detected #2041 (comment)

[sebastien-rosset]

You are running on mac. To troubleshoot, you can obtain the current limit of file descriptors:

launchctl limit maxfiles

You might be able to temporarily work around the problem by increasing the limits. For a deeper analysis, we would need to analyze the files that are opened by golangci-lint when running the test with 1.42.1 vs running with 1.43.0

Maximum limits:

sysctl kern.maxfiles
sysctl kern.maxfilesperproc

Ref: too many open files

[julienschmidt]

$ launchctl limit maxfiles                                   
	maxfiles    256            unlimited
$ sysctl kern.maxfiles
kern.maxfiles: 49152
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 24576

And all soft and hard limits via ulimit:

$ ulimit -aS
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2784
-n: file descriptors                256
$ ulimit -aH
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             65532
-c: core file size (blocks)         unlimited
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       2784
-n: file descriptors                unlimited

Using e.g. sudo launchctl limit maxfiles 65536 200000 does not seem to help either.

I can also reproduce those errors when running against github.com/golang/go/src for example:

INFO [config_reader] Config search paths: [./ /Users/julienschmidt/dev/go/src /Users/julienschmidt/dev/go /Users/julienschmidt/dev /Users/julienschmidt /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 24 linters: [asciicheck bodyclose deadcode durationcheck errcheck errorlint exhaustive exportloopref forcetypeassert gci gocritic gofmt goimports gomoddirectives goprintffuncname gosec gosimple govet ineffassign staticcheck structcheck typecheck unused varcheck]
INFO [loader] Go packages loading at mode 575 (compiled_files|deps|exports_file|files|imports|name|types_sizes) took 3.928716524s
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 180.591798ms
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key exhaustive/facts for package socktest: open /Users/julienschmidt/Library/Caches/golangci-lint/30/30a55b31e65f6916eb0e29d948eafb6144c1aad37d35ece1c68a8b8e9e19de9b-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key exhaustive/facts for package internal: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key ctrlflow/facts for package socktest: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key buildir/facts for package norm: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key fact_deprecated/facts for package norm: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key ctrlflow/facts for package internal: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key fact_deprecated/facts for package base64_test: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key fact_deprecated/facts for package suffixarray_test: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key ctrlflow/facts for package suffixarray_test: open /Users/julienschmidt/Library/Caches/golangci-lint/6b/6b0b464bcce5dad4ab3498c832dabc517cc63994dd9d34d65a454d078f7216bc-a: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key fact_deprecated/facts for package heap_test: open /Users/julienschmidt/Library/Caches/golangci-lint/69/69311da9ef155d34762a5141b87f03f12171da8c8354a01f45f927d2962fbf75-a: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key exhaustive/facts for package suffixarray_test: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
WARN [linters context/goanalysis] Failed to persist facts to cache: failed to save data to low-level cache by key exhaustive/facts for package quotedprintable: open /Users/julienschmidt/Library/Caches/golangci-lint/e2/e24a307ec1d4f19e6023639f91a2b1e33a9a2bbadcba08e4b8a7765245b19de7-d: too many open files
INFO [linters context/goanalysis] analyzers took 9m13.851049874s with top 10 stages: gocritic: 7m35.761895762s, buildir: 13.151502555s, goimports: 10.195193525s, gofmt: 5.787985847s, gosec: 5.233750941s, buildssa: 4.262144708s, directives: 2.668971854s, exhaustive: 2.418386236s, varcheck: 1.647772998s, S1002: 1.601944116s
INFO [runner/skip dirs] Skipped 9078 issues from dir builtin by pattern (^|/)builtin($|/)
INFO [runner/skip dirs] Skipped 4 issues from dir ../../../../../usr/local/Cellar/go/1.17.2/libexec/src/vendor/golang.org/x/net/idna by pattern (^|/)vendor($|/)
INFO [runner/skip dirs] Skipped 3168 issues from dir ../../../../../usr/local/Cellar/go/1.17.2/libexec/src/vendor/golang.org/x/text/unicode/norm by pattern (^|/)vendor($|/)
<a whole lot of INFO [runner/max_same_issues] messages>
INFO [runner] Issues before processing: 690278, after processing: 399
INFO [runner] Processors filtering stat (out/in): severity-rules: 399/399, path_prefixer: 399/399, exclude: 677691/677691, nolint: 676787/676787, max_per_file_from_linter: 5320/5320, path_shortener: 399/399, max_from_linter: 399/1859, source_code: 399/399, sort_results: 399/399, filename_unadjuster: 690163/690163, path_prettifier: 690163/690163, identifier_marker: 677691/677691, max_same_issues: 1859/5320, cgo: 690163/690278, skip_files: 690163/690163, autogenerated_exclude: 677691/677913, exclude-rules: 676787/677691, skip_dirs: 677913/690163, uniq_by_line: 5320/676787, diff: 5320/5320
INFO [runner] processing took 13.28361656s with stages: exclude-rules: 6.323364188s, identifier_marker: 5.607051061s, nolint: 600.414047ms, path_prettifier: 262.180771ms, cgo: 139.34577ms, filename_unadjuster: 128.734113ms, autogenerated_exclude: 103.780323ms, skip_dirs: 60.7825ms, uniq_by_line: 36.830929ms, max_same_issues: 14.699194ms, source_code: 5.868345ms, max_per_file_from_linter: 290.04µs, max_from_linter: 115.164µs, sort_results: 101.773µs, path_shortener: 56.569µs, exclude: 605ns, diff: 443ns, skip_files: 312ns, severity-rules: 235ns, path_prefixer: 178ns
INFO [runner] linters took 46.838181455s with stages: goanalysis_metalinter: 33.553645305s
<lint errors>
INFO File cache stats: 165 entries of total size 1.5MiB
INFO Memory: 504 samples, avg is 2199.3MB, max is 3485.6MB
INFO Execution took 50.977960609s

[julienschmidt]

Since I wasn't able to reproduce this issue lately and since this issue is currently not active, I'm going to close it for now.

[julienschmidt]

I was just able to reproduce the issue and I found a fix 🤦‍♂️

If you ended up here via a search: What helped me was increasing the soft limit for file descriptors by setting ulimit -S -n 2048 in my shell (e.g. ~/.zshrc / .bashrc).

[powerman]

I was just able to reproduce the issue and I found a fix 🤦‍♂️

This isn't a "fix", this is a workaround. The real fix is to implement control for amount of simultaneously open files by executed linters. So, please, reopen the issue.

Since update from v1.42.1 to v1.45.2 I often get error from gci linter like this:

ERRO Running error: 1 error occurred:
	* can't run linter goanalysis_metalinter: gci: open /home/…/pkg/grpcx/client.go: too many open files

My project has 376 .go files and I use default limit for max open files (1024).

There is linter-settings.revive.max-open-files option in golangci.yml, but there is no similar global option. I believe it's bad idea to add such option, this value should be auto-detected and used to control parallel execution of linters.

[fgm]

Just to confirm I started having the issue too and the workaround, well... works around it. Seems there is not really a leak, just a very high number of descriptors, which is somehow to be expected when running concurrent code accessing files.