Directory src/scriv used to be processed, now: command not found

[nedbat]

Bug description

In scriv, I was using src/scriv as an argument to pylint:

https://github.com/nedbat/scriv/blob/main/tox.ini#L57

With pylint==3.1.0, this worked, and violations were found in files like src/scriv/config.py.

Updating to pylint==3.2.2, it now seems to be running the scriv command, or one of its files that implements the command?:

Command used

pylint src/scriv tests docs setup.py

Pylint output

quality: commands[3]> pylint src/scriv tests docs setup.py
Usage: scriv [OPTIONS] COMMAND [ARGS]...
Try 'scriv --help' for help.

Error: No such command 'src/scriv'.

Expected behavior

I expect the code in src/scriv to be linted.

Pylint version

pylint 3.2.2
astroid 3.2.2
Python 3.8.19 (default, Mar 21 2024, 07:16:56)
[Clang 15.0.0 (clang-1500.3.9.4)]

OS / Environment

Mac 14.5

[jacobtylerwalls]

I'm eager to reproduce this if I can. I cloned scriv and ran python -m tox -e quality and it passed, without any output suggesting the scriv command ran.

I ran without tox as pylint src/scriv tests docs setup.py after installing pylint_pytest, and with pytest 7.4.2, and the files linted, although they threw lots of errors like:

************* Module scriv.util
src/scriv/util.py:12:0: E0401: Unable to import 'click_log' (import-error)
************* Module scriv.cli
src/scriv/cli.py:6:0: E0401: Unable to import 'click_log' (import-error)
************* Module scriv.linkcheck
src/scriv/linkcheck.py:7:0: E0401: Unable to import 'markdown_it' (import-error)
...
************* Module tests.test_ghrel
tests/test_ghrel.py:1:0: F6401: pylint-pytest plugin cannot enumerate and collect pytest fixtures. Please run `pytest --fixtures --collect-only tests/test_ghrel.py` and resolve any potential syntax error or package dependency issues. stdout: . stderr: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/_pytest/config/__init__.py:324: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
Plugin: helpconfig, Hook: pytest_cmdline_parse
ConftestImportFailure: ModuleNotFoundError: No module named 'responses' (from /Users/jwalls/scriv/tests/conftest.py)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
  config = pluginmanager.hook.pytest_cmdline_parse(
ImportError while loading conftest '/Users/jwalls/scriv/tests/conftest.py'.
tests/conftest.py:10: in <module>
    import responses
E   ModuleNotFoundError: No module named 'responses'
. (cannot-enumerate-pytest-fixtures)

Once I can reproduce, I'm happy to bisect the pylint or astroid changes that could be to blame. I wonder if knowing your installed version of pytest would help. Any other tips for reproducing would be appreciated. Thanks.

[nedbat]

Thanks. I don't see how pytest is involved though.

The main branch of scriv has pylint==3.1.0, and will pass. You can checkout the nedbat/make-upgrade branch, which has pylint==3.2.2, and should fail as shown.

[jacobtylerwalls]

Thanks, I can reproduce now, but also on pylint 3.1 and 3.0. I added a pylint --version to the quality command. I must be missing something else.

quality: commands[0]> pylint --version
pylint 3.0.0
astroid 3.0.0
Python 3.8.10 (v3.8.10:3d8993a744, May  3 2021, 09:09:08) 
[Clang 12.0.5 (clang-1205.0.22.9)]
quality: commands[1]> pylint src/scriv tests docs setup.py
Usage: scriv [OPTIONS] COMMAND [ARGS]...
Try 'scriv --help' for help.

Error: No such command 'src/scriv'.

[nedbat]

Great that you can reproduce it. What did you mean by "I must be missing something else"? Do you mean you don't understand what's causing the behavior yet?

[jacobtylerwalls]

Sorry, I meant that I could reproduce on pylint 3.0 and 3.1, but you report that it was working fine on 3.1, so I'm not sure I've reproduced your specific problem.

[Pierre-Sassoulas]

I've been able to reproduce with:

git clone git@github.com:nedbat/scriv.git
cd scriv
git switch nedbat/make-upgrade
python3 -m venv venv
source venv/bin/activate
pip3 install -e .
pip3 install tox
python -m tox -e quality
git switch main
python -m tox -e quality

(Python 3.10.12, pylint installed/used through tox)

The upgrade for pytest is:

-pytest==8.2.0
+pytest==8.2.1

So probably not linked to pylint-dev/pylint#9576

[Pierre-Sassoulas]

After further investigation and with the following diff:

diff --git a/requirements/quality.txt b/requirements/quality.txt
index 7fdf51c..3ca3d56 100644
--- a/requirements/quality.txt
+++ b/requirements/quality.txt
@@ -8,8 +8,6 @@ alabaster==0.7.13
     # via
     #   -r requirements/doc.txt
     #   sphinx
-astroid==3.2.2
-    # via pylint
 attrs==23.2.0
     # via
     #   -r requirements/doc.txt
@@ -195,7 +193,7 @@ pygments==2.18.0
     #   readme-renderer
     #   rich
     #   sphinx
-pylint==3.2.2
+pylint==2.17.7
     # via
     #   -r requirements/quality.in
     #   pylint-pytest

or

diff --git a/requirements/quality.txt b/requirements/quality.txt
index 7fdf51c..3ca3d56 100644
--- a/requirements/quality.txt
+++ b/requirements/quality.txt
@@ -8,8 +8,6 @@ alabaster==0.7.13
     # via
     #   -r requirements/doc.txt
     #   sphinx
-astroid==3.2.2
-    # via pylint
 attrs==23.2.0
     # via
     #   -r requirements/doc.txt
@@ -195,7 +193,7 @@ pygments==2.18.0
     #   readme-renderer
     #   rich
     #   sphinx
-pylint==3.2.2
+pylint==3.1.1
     # via
     #   -r requirements/quality.in
     #   pylint-pytest

I still have the error, so I suppose the problem come from something else than pylint.

[Pierre-Sassoulas]

By patch adding each dependencies from requirements/quality.txt one by one and bisecting with python -m tox -e quality I blame the pytest upgrade from 8.2.0 to 8.2.1.

[Pierre-Sassoulas]

Thank you for opening the issue though @nedbat ! :)

[nedbat]

What am I missing? How is pytest related to pylint?

[Pierre-Sassoulas]

We also have an issue with pytest's upgrade that I don't understand (I'm bisecting a lot without understanding anything these days 😄) in pylint-dev/pylint#9576 (pytest-dev/pytest#11138 (comment)). So I'm not sure how, but you have a similar issue and the pytest upgrade is what's making your script start failing . Created https://github.com/Pierre-Sassoulas/scriv/tree/pytest-upgrade-to-blame, which reproduce the issue. Here's the log if it makes the way I bisected the issue clearer:
scriv.log

[jacobtylerwalls]

Your repo has the pylint-pytest plugin installed. I don't know or use that plugin, but my assumption is that it uses runtime features of pytest.

[nedbat]

OK, if I remove pylint-pytest, then pylint runs, though it complains about the names of fixtures in the tests, which is the problem the plugin fixes.