allow mypy to use custom stdlib stubs

[Josverl]

in a Q&D test it is possible to allow mypy to use the
new 1.24.1 stubs ( that omit sys)

mypy --custom-typeshed-dir ./typings .
by adding :

• typings\mypy_extensions.pyi

origin : typestub repo : stubs\mypy-extensions\mypy_extensions.pyi

PyPI : types-mypy-extensions

refs:

1. https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-custom-typeshed-dir
2. https://discuss.python.org/t/pep-561-s-module-resolution-order-seems-incorrect/55048
3. Tweak the typing spec's module resolution to more closely emulate Python's runtime semantics python/typing#1772

QA test	code fragment	type check error
feat_aioble\community_code\sample_1.py:53:	await asyncio.sleep_ms(4) # stubs-ignore: linter == "mypy"
feat_aioble\examples_bluetooth\ble_bonding_peripheral.py:140:	key = sec_type, bytes(key) # stubs-ignore: linter == "mypy"
feat_asyncio\check_basics_01.py:31:	tasks[x] = asyncio.create_task(bar2(x)) # stubs-ignore: linter=="mypy"
feat_asyncio\check_basics_03.py:9:	await asyncio.sleep_ms(200 * n) # Pause by varying amounts # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:103:	s = asyncio.StreamReader(sys.stdin) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:106:	hist = [] * _HISTORY_LIMIT # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:173:	cmd # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:180:	sys.stdout.write(b) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:183:	sys.stdout.write(b) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\aiorepl.py:191:	sys.stdout.write(cmd) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:25:	self.swriter = asyncio.StreamWriter(self.uart, {}) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:26:	self.sreader = asyncio.StreamReader(self.uart) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:34:	await self.swriter.awrite( # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:38:	await asyncio.sleep_ms(300) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:51:	self.swriter = asyncio.StreamWriter(self.uart, {}) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:52:	self.sreader = asyncio.StreamReader(self.uart) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\auart_hd.py:68:	await self.swriter.awrite("{}\r\n".format(command)) # stubs-ignore: linter=="mypy"
feat_asyncio\check_demo\roundrobin.py:21:	await asyncio.sleep_ms(0) # stubs-ignore: linter=="mypy"
feat_networking\check_examples\http_client.py:18:	s.write(b"GET / HTTP/1.0\r\n\r\n") # stubs-ignore: linter == "mypy"
feat_networking\check_examples\http_client.py:19:	print(s.read()) # stubs-ignore: linter == "mypy"
feat_stdlib\check_array.py:5:	ar = array.array("I", [0 for _ in range(NUM_LEDS)]) # stubs-ignore: linter == "mypy"
feat_stdlib\check_io.py:7:	buffer_1 = io.StringIO(alloc_size) # stubs-ignore: version<=1.18.0 or linter == "mypy"
feat_stdlib\check_io.py:8:	buffer_2 = io.BytesIO(alloc_size) # stubs-ignore: version<=1.18.0 or linter == "mypy"
feat_stdlib\check_io.py:12:	buf = io.BufferedWriter(stream, 8) # stubs-ignore: linter == "mypy"	Argument 1 to "BufferedWriter" has incompatible type "TextIOWrapper[_WrappedBuffer]"; expected "RawIOBase" [arg-type]
feat_stdlib\check_uio.py:7:	buffer_1 = uio.StringIO(alloc_size) # stubs-ignore: linter == "mypy"
feat_stdlib\check_uio.py:8:	buffer_2 = uio.BytesIO(alloc_size) # stubs-ignore: linter == "mypy"
feat_stdlib\check_sys\check_print_exception.py:3:	from sys import print_exception # stubs-ignore: linter == "mypy"
feat_stdlib\check_sys\check_sys.py:24:	sys.print_exception(exc) # stubs-ignore: linter == "mypy"
feat_stdlib\check_sys\check_sys.py:32:	previous = sys.atexit(byebye) # stubs-ignore: linter == "mypy"

[Josverl]

partially impemented in : 2959e98

[andrewleech]

Hi @Josverl following on from #720 I've done a quick test of the v1.24.1 on my existing project with "patched/hacked stubs for mypu use" but quickly run into blockers that I think are related directly to me using the stubs typing dir as the custom_typing_dir for mypy

tools/typings/_mpy_shed/__init__.pyi:14: error: Invalid "type: ignore" comment  [syntax]
/home/.cache/pre-commit/repo_em78hv0/py_env-python3.10/lib/python3.10/site-packages/mypy/typeshed/stdlib/io.pyi:95: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
If this issue continues with mypy master, please report a bug at https://github.com/python/mypy/issues
version: 1.7.0
/home/.cache/pre-commit/repo_em78hv0/py_env-python3.10/lib/python3.10/site-packages/mypy/typeshed/stdlib/io.pyi:95: : note: please use --show-traceback to print a traceback when reporting a bug

---

I did a terrible job of tracking my changes when I was previously modding my installed stubs typings folder to make it work with mypy, such that now every time I get back to it to attempt to reverse engineer it into clean commits & PR here I don't get very far. Much of the problem is due to me inadvertently allowing vscode and/or black to auto-reformat all the files... so now diff's are very noisy.

I've still got an internally tracked task to clean this up properly and submit it, but little foreseable time to get to it so in liew of that I'm going to dump a snapshot here, just in case it helps you at all. Feel free to ignore it if it's more effort than its worth.

typings_andrewleech_tgz.zip
Note this file is supposed to be a .tar.gz but github wasn't allowing it so I've added a fake .zip to the end of the name that should be removed.

For context I believe this was originally installed with:

pip install -U micropython-stm32-stubs --target ./tools/typings --no-user

and appears to be from micropython_stm32_stubs==1.19.1.post9 & micropython_stdlib_stubs==0.9.0
Then black or ruff have reformatted it and I've made changes since.

I use pre-commit with .pre-commit-config.yaml and pyproject.toml attached here, again with a fake .zip appended to the name.

.pre-commit-config.yaml.zip

pyproject.toml.zip

Additionally, this was another WIP typings folder I had, also for use as a typeshed dir, but trying to resolve #720 in a different way.
typings_mod_tgz.zip
While micropython implements streams as a "protocol" this concept doesn't really map to cpython. It's similar to a base class though, so I added a Stream base class to stdlib/io.pyi and used it as a baseclass for things like io.StringIO. To be honest though I don't remember how close this was to working correctly.

[Josverl]

Thanks for sharing.

I run all all typecheck tests with the same config

exclude = [
    "typings[\\/].*", # TOML basic string 
]

I explicitly exclude the typings folder itself for both pyright and mypy, takinh a hint from typeshed , which also racks up a lot of errors it you turn a typechecker towards it.

wrt to the micropython io ,
I am still trying to better understand where the differences are, and how to express them well enough.
to allow multiple modules to use common micropython (base) classes I have moved these to a _mpy_shed module , that I now include in micropython-stdlib-stubs.
The Micropython io base classes are in _mpy_shed/io_mp.pyi but they will need more work.
but as some of the io methods are special-cased in the type checkers , it is hard/impossible to just alter.

Adding the typings\mypy_extensions.pyi allows mypy to 'accept' changes to the stdlib modules, but i cant add it by default as that then fails other tests.
so that will need to be an extras to allow mypy users to opt in to stdlib

[Josverl]

If you want to submit changes , then the best angle is to target micropython-reference,
as all type info flows from there to the stub packages.
That is also the part that I think should eventually move to the MicroPython repo

(but note that mypy does not like some of the modules there, so be ready to rename sys and collections)

[Josverl]

V1.19.1.post9

compared with :
pip install micropython-stdlib-stubs==0.9.0 micropython-stm32-stubs==1.19.1.post9 --target typings --no-user

Added / changed /:

• asyncio/*.pyi - needed to fix asyncio stubs
• uasyncio/*.pyi - needed to fix asyncio stubs
• remove machine.PWM class
• change pyb.Timer.channel method signature
• change time.time from method to class

V1.21.0 (.mod)

compared with:
pip install micropython-stdlib-stubs==1.0.0 micropython-stm32-stubs==1.21.0.post1 --target typings --no-user

note: the re-install of stdlib is slightly different - while it does have the same version, perhaps changes were copied over from an earlier version

Added / changed /:

• asyncio/*.pyi - needed to fix asyncio stubs
• del VERSIONS.dis file
• stdlib/os
  - from . import path as _path - commented out
  - update statvfs_result ( minor change )
• stdlib/path.pyi - renamed/removed
• stdlib/io.Stream class added
• io
  • add # type: ignore [no-redef] to a lot of methods
  • changes base class from IO to Stream
• machine.IDLE addedd
• machine.ADC class constants added
• uctypes.struct : add __getattr__ method
  def __getattr__(self, a): ...
• _typeshed - slightly newer version
• _codecs - slightly newer version
• contextlib.pyi - slightly newer version
• typing / typing_extensions - slightly newer version
• os - slightly newer version

these are the diffs , ruff format on both sides to get rid of some noise, and leaving out the async patches and some metadata

changes_v1.19.1.patch
changes_v1.21.0.patch

If you can still remember why you made which changes, that would help me decide what and how to integrate these in the stub project.

[Josverl]

io.Stream is a Protocol in MicroPython.

// Stream protocol

so it probably should be typed as a Protocol

class Stream(Protocol):
    """
    MicroPython stream protocol. Due to implementation mechanism
    not all methods are guaranteed to be available on all classes
    based on the stream type / protocol.
    """
    def __init__(self, *argv, **kwargs) -> None: ...
    def __enter__(self: Self) -> Self: ...
    def __exit__(
        self,
        exc_type: type[BaseException] | None,
        exc_val: BaseException | None,
        exc_tb: TracebackType | None,
    ) -> None: ...
    def close(self) -> None: ...
    def flush(self) -> None: ...
    def read(self, __size: int | None = ...) -> bytes: ...
    def read1(self, __size: int = ...) -> bytes: ...
    def readinto(self, __buffer: WriteableBuffer) -> int: ...
    def readline(self, __size: int | None = ...) -> bytes: ...
    def readlines(self, __hint: int = ...) -> list[bytes]: ...
    def seek(self, __offset: int, __whence: int = ...) -> int: ...
    def tell(self) -> int: ...
    def write(self, __buffer: ReadableBuffer) -> int: ...
    def write1(self, __buffer: ReadableBuffer) -> int: ...