What do we need to fix/test/implement to cut a new release.

[apozharski]

A few (8!) months ago we replaced the parsing subsystem to use tree-sitter which provided some new features to handle modern object-oriented MATLAB. It would be good to release this update (I would say probably as 0.23.0) as I think it addresses some of the issues raised in the last few months.

Here is a list of what I have:

• Verify that Parsing fails on methods with nested functions #275 is dealt with current master branch.
• Verify that Document abstract methods #222 is now correct and perhaps write some regression tests for this.
• Verify that Sphinx hangs with block comment and local function #276 is fixed in the current implementation (perhaps also add regression test here).

@joeced @Remi-Gau Let me know if you have any thoughts on this.

[jasonnicholson]

I am testing out the new version via the master branch. I am not having good luck.

tree-sitter-matlab==1.0.6 is corrupt. See stack trace below.

ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\cli\base_command.py", line 105, in _run_wrapper
    status = _inner_run()
             ^^^^^^^^^^^^
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\cli\base_command.py", line 96, in _inner_run
    return self.run(options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\cli\req_command.py", line 68, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\commands\install.py", line 459, in run
    installed = install_given_reqs(
                ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\req\__init__.py", line 83, in install_given_reqs        
    requirement.install(
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\req\req_install.py", line 867, in install
    install_wheel(
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\operations\install\wheel.py", line 729, in install_wheel
    _install_wheel(
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\operations\install\wheel.py", line 587, in _install_wheel
    file.save()
  File "C:\Users\myUserNameRedacted\AppData\Roaming\Python\Python312\site-packages\pip\_internal\operations\install\wheel.py", line 377, in save
    shutil.copyfileobj(f, dest, blocksize)
  File "C:\Program Files\python312\Lib\shutil.py", line 203, in copyfileobj
    while buf := fsrc_read(length):
                 ^^^^^^^^^^^^^^^^^
  File "C:\Program Files\python312\Lib\zipfile\__init__.py", line 989, in read
    data = self._read1(n)
           ^^^^^^^^^^^^^^
  File "C:\Program Files\python312\Lib\zipfile\__init__.py", line 1079, in _read1
    self._update_crc(data)
  File "C:\Program Files\python312\Lib\zipfile\__init__.py", line 1007, in _update_crc
    raise BadZipFile("Bad CRC-32 for file %r" % self.name)
zipfile.BadZipFile: Bad CRC-32 for file 'tree_sitter_matlab/_binding.pyd'

[jasonnicholson]

I went back to tree-sitter-matlab==1.0.5 and it installed. it then failed when I tried to build the Sphinx site.

alabaster==1.0.0
anyio==4.9.0
babel==2.17.0
certifi==2025.6.15
charset-normalizer==3.4.2
click==8.2.1
colorama==0.4.6
docutils==0.21.2
h11==0.16.0
idna==3.10
imagesize==1.4.1
Jinja2==3.1.6
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdit-py-plugins==0.4.2
mdurl==0.1.2
myst-parser==4.0.1
packaging==25.0
Pygments==2.19.2
PyYAML==6.0.2
requests==2.32.4
roman-numerals-py==3.1.0
sniffio==1.3.1
snowballstemmer==3.0.1
Sphinx==8.2.3
sphinx-autobuild==2024.10.3
sphinx-rtd-theme==3.0.2
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jquery==4.1
sphinxcontrib-jsmath==1.0.1
-e git+https://github.com/sphinx-contrib/matlabdomain.git@4b7d0f52b20438b10ab034191fce32279dce135e#egg=sphinxcontrib_matlabdomain
sphinxcontrib-mermaid==1.0.0
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
starlette==0.47.1
tree-sitter==0.22.3
tree-sitter-matlab==1.0.5
typing_extensions==4.14.0
urllib3==2.5.0
uvicorn==0.34.3
watchfiles==1.1.0
websockets==15.0.1

Versions
========

* Platform:         win32; (Windows-11-10.0.26100-SP0)
* Python version:   3.12.4 (CPython)
* Sphinx version:   8.2.3
* Docutils version: 0.21.2
* Jinja2 version:   3.1.6
* Pygments version: 2.19.2

Last Messages
=============

None.

Loaded Extensions
=================

None.

Traceback
=========

    Traceback (most recent call last):
      File "C:\A\Experiments\sphinx\.venv\Lib\site-packages\sphinx\cmd\build.py", line 414, in build_main
        app = Sphinx(
              ^^^^^^^
      File "C:\A\Experiments\sphinx\.venv\Lib\site-packages\sphinx\application.py", line 291, in __init__
        self.setup_extension(extension)
      File "C:\A\Experiments\sphinx\.venv\Lib\site-packages\sphinx\application.py", line 489, in setup_extension
        self.registry.load_extension(self, extname)
      File "C:\A\Experiments\sphinx\.venv\Lib\site-packages\sphinx\registry.py", line 541, in load_extension
        mod = import_module(extname)
              ^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Program Files\python312\Lib\importlib\__init__.py", line 90, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 995, in exec_module
      File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
      File "C:\A\matlabdomain\sphinxcontrib\matlab.py", line 11, in <module>
        from . import mat_documenters as doc
      File "C:\A\matlabdomain\sphinxcontrib\mat_documenters.py", line 11, in <module>
        from .mat_types import (  # noqa: E401
      File "C:\A\matlabdomain\sphinxcontrib\mat_types.py", line 19, in <module>
        from sphinxcontrib.mat_tree_sitter_parser import (
      File "C:\A\matlabdomain\sphinxcontrib\mat_tree_sitter_parser.py", line 46, in <module>
        ML_LANG = Language(tsml.language())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
    TypeError: an integer is required

[jasonnicholson]

I went back to tree-sitter-matlab==1.0.4 and it installed and the site generated. However, only the short description is rendered. The rest of the docstring is not rendered.

[jasonnicholson]

Build #5459 was the last good build. Build #5460 was the first failure. #5459 was tree-sitter-matlab==1.0.4 while #5460 was tree-sitter-matlab==1.0.5. So, at least the error is consistent with what is being seen in CI.

I setup my local system to mirror #5459. I am not able to get the tests back to passing. The biggest difference is I used Sphinx 8.3.0 instead of the master branch.

dependencies:

alabaster==1.0.0
babel==2.17.0
certifi==2025.4.26
charset-normalizer==3.4.2
defusedxml==0.7.1
docutils==0.21.2
idna==3.10
imagesize==1.4.1
iniconfig==2.1.0
Jinja2==3.1.6
MarkupSafe==3.0.2
packaging==25.0
pluggy==1.5.0
Pygments==2.19.1
pytest==8.3.5
requests==2.32.3
roman-numerals-py==3.1.0
snowballstemmer==2.2.0
Sphinx==8.3.0
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
tree-sitter==0.22.3
tree-sitter-matlab==1.0.4
urllib3==2.4.0
-e .

windows 11 version 24H2 (OS Build 26100.4349)
python 3.12.10
pip 25.0.1

I am trying to get to point when I see that the tests work locally.

[jasonnicholson]

I built a docker container with python:3.12.10-alpine3.22 so that I could get closer to a linux environment (WSL). The tests are still failing. That's enough for today. I'll look at the more tomorrow.

[jasonnicholson]

The tests passed on my personal Linux laptop configured the same as GitHub Action run #5459.

alabaster==1.0.0
babel==2.17.0
certifi==2025.4.26
charset-normalizer==3.4.2
defusedxml==0.7.1
docutils==0.21.2
idna==3.10
imagesize==1.4.1
iniconfig==2.1.0
Jinja2==3.1.6
MarkupSafe==3.0.2
packaging==25.0
pluggy==1.5.0
Pygments==2.19.1
pytest==8.3.5
requests==2.32.3
roman-numerals-py==3.1.0
snowballstemmer==2.2.0
Sphinx==8.2.3
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
-e .
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
tree-sitter==0.22.3
tree-sitter-matlab==1.0.4
urllib3==2.4.0

# python==3.12.10
# pip==25.0.1


# NAME="Pop!_OS"
# VERSION="22.04 LTS"
# ID=pop
# ID_LIKE="ubuntu debian"
# PRETTY_NAME="Pop!_OS 22.04 LTS"
# VERSION_ID="22.04"
# HOME_URL="https://pop.system76.com"
# SUPPORT_URL="https://support.system76.com"
# BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
# PRIVACY_POLICY_URL="https://system76.com/privacy"
# VERSION_CODENAME=jammy
# UBUNTU_CODENAME=jammy
# LOGO=distributor-logo-pop-os

The tests fail on m Linux laptop when I configure the same as #5460.

alabaster==1.0.0
babel==2.17.0
certifi==2025.4.26
charset-normalizer==3.4.2
defusedxml==0.7.1
docutils==0.21.2
idna==3.10
imagesize==1.4.1
iniconfig==2.1.0
Jinja2==3.1.6
MarkupSafe==3.0.2
packaging==25.0
pluggy==1.5.0
Pygments==2.19.1
pytest==8.3.5
requests==2.32.3
roman-numerals-py==3.1.0
snowballstemmer==2.2.0
Sphinx==8.2.3
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
-e .
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
tree-sitter==0.22.3
tree-sitter-matlab==1.0.5
urllib3==2.4.0

# python==3.12.10
# pip==25.0.1


# NAME="Pop!_OS"
# VERSION="22.04 LTS"
# ID=pop
# ID_LIKE="ubuntu debian"
# PRETTY_NAME="Pop!_OS 22.04 LTS"
# VERSION_ID="22.04"
# HOME_URL="https://pop.system76.com"
# SUPPORT_URL="https://support.system76.com"
# BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
# PRIVACY_POLICY_URL="https://system76.com/privacy"
# VERSION_CODENAME=jammy
# UBUNTU_CODENAME=jammy
# LOGO=distributor-logo-pop-os

Here is the test failure:

================================================================ test session starts =================================================================
platform linux -- Python 3.12.10, pytest-8.3.5, pluggy-1.5.0
rootdir: /home/jason/Desktop/matlabdomain
configfile: pytest.ini
collected 51 items / 5 errors                                                                                                                        

======================================================================= ERRORS =======================================================================
_______________________________________________ ERROR collecting tests/test_classfolder_class_name.py ________________________________________________
tests/test_classfolder_class_name.py:1: in <module>
    from sphinxcontrib.mat_types import classfolder_class_name
sphinxcontrib/mat_types.py:19: in <module>
    from sphinxcontrib.mat_tree_sitter_parser import (
sphinxcontrib/mat_tree_sitter_parser.py:46: in <module>
    ML_LANG = Language(tsml.language())
E   TypeError: an integer is required
______________________________________________________ ERROR collecting tests/test_matlabify.py ______________________________________________________
tests/test_matlabify.py:3: in <module>
    from sphinxcontrib import mat_documenters as doc
sphinxcontrib/mat_documenters.py:11: in <module>
    from .mat_types import (  # noqa: E401
sphinxcontrib/mat_types.py:19: in <module>
    from sphinxcontrib.mat_tree_sitter_parser import (
sphinxcontrib/mat_tree_sitter_parser.py:46: in <module>
    ML_LANG = Language(tsml.language())
E   TypeError: an integer is required
_____________________________________________________ ERROR collecting tests/test_parse_mfile.py _____________________________________________________
tests/test_parse_mfile.py:3: in <module>
    from sphinxcontrib import mat_types
sphinxcontrib/mat_types.py:19: in <module>
    from sphinxcontrib.mat_tree_sitter_parser import (
sphinxcontrib/mat_tree_sitter_parser.py:46: in <module>
    ML_LANG = Language(tsml.language())
E   TypeError: an integer is required
___________________________________________________ ERROR collecting tests/test_parse_mlappfile.py ___________________________________________________
tests/test_parse_mlappfile.py:3: in <module>
    from sphinxcontrib import mat_types
sphinxcontrib/mat_types.py:19: in <module>
    from sphinxcontrib.mat_tree_sitter_parser import (
sphinxcontrib/mat_tree_sitter_parser.py:46: in <module>
    ML_LANG = Language(tsml.language())
E   TypeError: an integer is required
____________________________________________________ ERROR collecting tests/test_shortest_name.py ____________________________________________________
tests/test_shortest_name.py:1: in <module>
    from sphinxcontrib.mat_types import shortest_name
sphinxcontrib/mat_types.py:19: in <module>
    from sphinxcontrib.mat_tree_sitter_parser import (
sphinxcontrib/mat_tree_sitter_parser.py:46: in <module>
    ML_LANG = Language(tsml.language())
E   TypeError: an integer is required
============================================================== short test summary info ===============================================================
ERROR tests/test_classfolder_class_name.py - TypeError: an integer is required
ERROR tests/test_matlabify.py - TypeError: an integer is required
ERROR tests/test_parse_mfile.py - TypeError: an integer is required
ERROR tests/test_parse_mlappfile.py - TypeError: an integer is required
ERROR tests/test_shortest_name.py - TypeError: an integer is required
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================= 5 errors in 0.65s ==================================================================

[jasonnicholson]

I am going to try again on my Personal Windows desktop later this afternoon to see if I can get things working on Windows.

[jasonnicholson]

The tests fail on my Windows 10 Desktop. The dependencies are the same as run #5459.

alabaster==1.0.0
babel==2.17.0
certifi==2025.4.26
charset-normalizer==3.4.2
defusedxml==0.7.1
docutils==0.21.2
idna==3.10
imagesize==1.4.1
iniconfig==2.1.0
Jinja2==3.1.6
MarkupSafe==3.0.2
packaging==25.0
pluggy==1.5.0
Pygments==2.19.1
pytest==8.3.5
requests==2.32.3
roman-numerals-py==3.1.0
snowballstemmer==2.2.0
Sphinx==8.2.3
sphinxcontrib-applehelp==2.0.0
sphinxcontrib-devhelp==2.0.0
sphinxcontrib-htmlhelp==2.1.0
sphinxcontrib-jsmath==1.0.1
-e .
sphinxcontrib-qthelp==2.0.0
sphinxcontrib-serializinghtml==2.0.0
tree-sitter==0.22.3
tree-sitter-matlab==1.0.4
urllib3==2.4.0

# python==3.12.10
# pip==25.0.1

# Microsoft Windows [Version 10.0.19045.5965]

Attached is the output of pytest -vv
stdout.txt

[watermarkhu]

Hi, great work on migrating the parsing to be tree-sitter based! In fact, when you started that it inspired me to do just that as well, but in the mkdocstrings framework.

Take a look at watermarkhu/mkdocstrings-matlab 😊

[jmgate]

With the changes merged over the weekend, I believe the problems @jasonnicholson ran into should be resolved.

[jmgate]

Is there anything I can do to help move toward the 0.23.0 release?

[jmgate]

I'm afraid while working on #298, I discovered it doesn't look like the switch to tree-sitter-matlab has made it such that #222 is resolved automatically. I'll comment further on that issue.