testing: reorg of test infra for maintainability and scalability

[btmlnsn]

Summary

Restructures the test directory layout in a manner that follows Python packaging best practices, and prepares the codebase for future test expansion.

Why this change?

Problem: tests were located inside the 'LabGym/' application package, which meant:

• Test files were mixed with application code (tests ought to be kept deliberately separate from the application being tested).
• Adding new tests was confusing due to unclear organization.
  Solution: Move tests to a dedicated 'tests/' folder at the repository root, following a standard Python project layout used by major packages.

What Changed

Before:

• Tests existed within '/LabGym/tests' and within '/tests' (at repo root)
• No shared test utilities existed
• Test config was scattered
  After:
• All tests exist within '/tests' at repo root, separate from the LabGym application code; categories exist for integration tests and unit tests.
• New 'tests/conftest.py' with reusable fixtures
• 'pytest.ini' now exists at repo root with markers

I moved some of John's notes into a designated notes folder, and I also aggregated his linting scripts into a 'linting' folder within '/tests/linting/'. I also edited John's linting scripts to reflect the new structure. Feel free to change the scripts if they didn't carry over as expected.

All 59 tests pass on my end via 'pytest -q'.

[ruck94301]

Guys, This PR would benefit from some discussion. So far, I'm the only one producing unit tests. You would want to give me an opportunity to comment before reorganizing the unit tests. 16 hours from PR to merge without asking my input sends a strange message to me. This is not an urgent bugfix that needs to be included in this week's release. I recommend revert the merge of this PR for now, and continue discussion.

…

On Thu, Jan 22, 2026 at 10:47 PM Yujia Hu ***@***.***> wrote: Merged #342 <#342> into master. — Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4CQNDU5QVFVUJCDOES3RD4IGKUBAVCNFSM6AAAAACSSYSSP6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRSGIZTENBQGYZDGMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[yujiahu415]

Guys, This PR would benefit from some discussion. So far, I'm the only one producing unit tests. You would want to give me an opportunity to comment before reorganizing the unit tests. 16 hours from PR to merge without asking my input sends a strange message to me. This is not an urgent bugfix that needs to be included in this week's release. I recommend revert the merge of this PR for now, and continue discussion.
…
On Thu, Jan 22, 2026 at 10:47 PM Yujia Hu @.> wrote: Merged #342 <#342> into master. — Reply to this email directly, view it on GitHub <#342 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4CQNDU5QVFVUJCDOES3RD4IGKUBAVCNFSM6AAAAACSSYSSP6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMRSGIZTENBQGYZDGMQ . You are receiving this because you are subscribed to this thread.Message ID: @.>

I’m really sorry, John. I didn’t realize this. Sure, I’ll revert this PR for your inputs. And I’ll leave more time for possible comments and discussion for a PR before merging it in the future.
Apologies again!

See PR #343

[ruck94301]

Does LabGym --selftest still work?

I recently added a feature where LabGym could run its own unit tests.
I'm concerned that by reorganizing, this feature no longer works.
To use the feature, run LabGym like LabGym --selftest.
There's a top-level menu item that tells how to do it. Can you confirm this selftest feature still works as expected?
My uncertainty here reveals that I should have written unit tests for "selftest" to protect it :-/.
To be fair, if this PR does break "selftest", there is no blame assigned to the Pull "Request" itself.
The first line of defense ideally is the unit test that I should have written, the second line of defense is the discussion phase, where potential side effect issues are raised and addressed.
Thx,
John
BTW, why "selftest"? Happy to discuss that if I didn't sufficiently explain when I introduced it #332, or if you have new questions. My thinking is that selftest is a net positive, but it's decidedly okay to challenge my position.

[ruck94301]

Looking at noxfile... Instead of adding "coverage" to noxfile, should we eliminate "pytest" from noxfile?

I don't understand adding "coverage" to noxfile. There's no nox-driven testing that employs it, is there? But, now that we're looking at it, I'm thinking maybe the nox- install of pytest is no longer appropriate. pytest is in the pyproject.toml, now, so I suspect it is redundant to have nox install it separately. Instead of adding "coverage", I propose removing pytest from noxfile installation.

More broadly, I recommend do not add "coverage" or other developer-only tools without more justification/consensus.

There are developers tools that don't need to be in the project, but that individual developers might use. I'd love to persuade you that coverage analysis is useful and challenge you to expect more of contributors unit tests, but unless and until that's an actual expectation of contributors, I'd argue that developers personal environments -- including mine, of course -- don't belong in the pyproject.toml or noxfile.

We aren't putting pylint in there, we aren't putting mypy in there, we aren't putting mypy stub-files in there.
I imagine you're like me, you have more than one venv: (a) a venv whose content is defined by pyproject.toml, the "standard" user venv... I call mine "3.10+LG". And you also have a separate venv (b) fattened up with your extra developer tools. I call mine "3.10+LG+devtools".
If you ARE using "coverage" in a go/no-go way, during the build process, then I withdraw my objection. But for now, my thinking is that extra non-essential dev tools don't belong in the standard LabGym environment.

BTW, I don't install "coverage" separately/independently like that in my devtools venv, I install a package that has coverage as a prerequisite so pip gets it as part of something else.
If you're adding it because I seem to be using it from my pytest.sh script, then maybe it's better for me to remove my pytest.sh so it doesn't create confusion. I'm leaning towards that. I think I committed some utils early on (before Bobby?) when I thought Henry & I would work more tightly together and I would have a chance to demonstrate some static analysis and see if Henry wanted to adopt some additional metrics targeting code quality, McCabe cyclotomic complexity, unit-test coverage, etc. I should also eliminate pylint.sh and INIT.sh. They ARE useful to me but unless it's part of all developers workflow, it's better that these does not complicate the project, I should keep them private.

[ruck94301]

Can this line be simply eliminated? I think the nox install of pytest was previously necessary, because pytest wasn't in pyproject.toml. But now it is. And nothing depends on "coverage", right?

[ruck94301]

I like using the "markers" feature of pytest. I have some work (not yet PR-ed) where I employ one new marker "wxapp" to mark the tests that require the wx.App object, and I have a use-case for running all unit tests except those that require wx.App object.
I don't have a use case for slow and integration... As I continue to review, I'll look for how you are using them. If you are not using them then I will advise: don't organize for what doesn't have a use yet. The complexity is a type of "cost" that needs to have a reason to absorb it. We all, myself included, need to resist the impulse to create framework that is anticipatory.

[ruck94301]

I like the use of conftest.py for shared pytest fixtures. I use that in a branch that I have not yet PR-ed.
But I'm looking at this and a little puzzled... your wx_app fixture relies on LabGym.ui.gui wxutils.
That doesn't exist right? Does the fixture work?
I don't see it being used, so I'm not sure what to make of it.
I'll look carefully for it and for your use of the mock_argv, sample_grayscale_frame, and sample_video_frames as I continue review.
Followup:
I do not see your fixtures being used and I'm thinking your wx_app is not viable.
One path forward is for me to PR my conftest.py, and sometime in future you add your fixtures when you get to the point you'd use them. I will probably roll exitstatus.py into a shared fixture as well, and eliminate that separate file.

[ruck94301]

I'm prepared for this file to be eliminated, it falls into the category of personal-developer-tooling and shouldn't clutter the project. At one point I thought it would be more generally useful, but, that's looking unlikely.
Also, better to eliminate this directory and its contents. No new "linting" directory. And further, eliminate tmp.pylint/.gitkeep. All of this is very useful to me, but not GENERALLY useful, so it does not belong in project.
Either exclude from your PR and ask me to separately PR their removal, or remove them in your PR.

[ruck94301]

I am uncomfortable with the (excessive?) hierarchicalization. Independent from the issue of the location of tests directory (which I will comment on in a moment),
the EXTRA separation "unit" vs. "notes" vs. "integration" isn't justified. And I've never seen it, TBH... is this something you've seen in open-source projects?
I strongly recommend that we all prefer "flat" until flat becomes unwieldy. There's lots of runway before that might happen and probably it never will. Keep it simple and flat.

[ruck94301]

Now, about the motivation for the movement... locating the tests at package-level vs at project-level.
Either way is a viable and accepted approach with pros & cons.
I don't recognize that adding new tests is "confusing", but if it is, maybe it's easier to clarify with a convo or a GitHub discussion first, instead of reorganizing.

Restructures the test directory layout in a manner that follows Python packaging best practices, and prepares the codebase for future test expansion.

I agree that project-level tests folder is an accepted good practice, but, to be clear, so is package-level tests folder.

Why this change?
Problem: tests were located inside the 'LabGym/' application package, which meant:
Test files were mixed with application code (tests ought to be kept deliberately separate from the application being tested).

Two responses...
(a) yes, it's good to keep test code separated, but I think putting it in a "tests" folder accomplishes that. The tests folder being located in the package folder rather than the project folder doesn't violate the principle.
(b) with a selftest feature, the test code actually becomes part of the application, so more decoupling is even less warranted.

Adding new tests was confusing due to unclear organization.
Solution: Move tests to a dedicated 'tests/' folder at the repository root, following a standard Python project layout used by major packages.

That's an approach, but not the only approach. I'm still not yet actually seeing it as a potential cause of confusion. If you say that it genuinely confused you personally, I'll concede it as possibility, as my imagination is sometimes not strong enough. But if you're just anticipating confusion in others, I'd say let things evolve as needed.
A conversation would be another approach. Documenting the strategy is another approach. If it's more than just me writing unit tests, then it becomes more important, but until then, shrug.

What Changed
Before:
Tests existed within '/LabGym/tests' and within '/tests' (at repo root)
No shared test utilities existed
Test config was scattered

Yes there are two tests folders.
The project-level tests folder (at the repo root) contained two test modules, and now that I look at it, one test module is redundant to the other test module. The redundancy should be eliminated.
I agree there's no reason to keep the one test module (in the project-level tests folder) separate from the dozen or so other test modules (in the package-level tests folder).
When you say test config is scattered, not sure exactly what you mean... Do you mean in two files pytest.sh and pytest.ini versus in two files pytest.ini and conftest.py? I am fine withdrawing pytest.sh, and I support using conftest.py, but I'm not seeing the existing as "scattered" or the PR as being "unscattered", so probably I'm misunderstanding the point here.

I'm getting worn out addressing things point-by-point.
Going forward when there's a significant PR that impacts others and has room for misunderstanding, consider holding a live review meeting if possible.

After:
All tests exist within '/tests' at repo root, separate from the LabGym application code; categories exist for integration tests and unit tests.
New 'tests/conftest.py' with reusable fixtures
'pytest.ini' now exists at repo root with markers
I moved some of John's notes into a designated notes folder, and I also aggregated his linting scripts into a 'linting' folder within '/tests/linting/'. I also edited John's linting scripts to reflect the new structure. Feel free to change the scripts if they didn't carry over as expected.

I'm thinking even better to eliminate these scripts that are not justified.
I'm sorry! Their existence is a "complexity" that led you to spend time reworking them (a "cost"). This is a good example of me making the mistake of committing something, anticipating that it would have general value. I had thought we would be cross-pollinating more and we would have ongoing live discussions and collaborate including topics of static analysis and tools.

[ruck94301]

Now, about three widely-accepted organizations of code and tests:

1. "src layout"
2. "flat layout" (what the PR accomplishes)
3. "in-package layout" (the existing layout for the tests I authored, before merging Bobby's PR 342)

Any of the above three work fine for LabGym.
They have slightly different pros/cons.
I don't object to number two, but, I don't think (yet) that number 2 is better than number 3 in a substantive way that justifies the relocation. Thoughts? Maybe I'm overlooking something?

Pasting from Gemini:

The src layout ... separates importable source code from other project assets.
Structure:

my_project/
├── pyproject.toml
├── src/
│   └── my_package/
│       ├── __init__.py
│       └── app.py
└── tests/
    ├── test_app.py
    └── conftest.py

Key Benefits:
Import Isolation: Prevents accidental imports from the local directory, forcing you to test the code as it would be installed by a user.
Clarity: Makes it obvious which files are part of the distributed package versus development tools.
Packaging: Standard tools like Poetry and Setuptools prefer this layout.

In a flat layout, the package directory sits directly in the project root.
Structure:

my_project/
├── pyproject.toml
├── my_package/
│   ├── __init__.py
│   └── app.py
└── tests/
    └── test_app.py

Key Benefits:
Simplicity: Easier for beginners or small scripts because it doesn't require an "editable install" to run code immediately.
Tooling: Default for tools like uv to reduce friction for new projects.

Location and Naming of Pytest Unit Tests
Pytest uses automatic discovery to find your tests.
Directory Location: Tests are typically placed in a dedicated tests/ directory at the root of the project.
Discovery Rules: Pytest will automatically find and run files that match the patterns test_*.py or *test.py.
Internal Functions: Inside these files, pytest looks for functions prefixed with test or classes prefixed with Test.
Optional Inlining: You can also place a tests/ folder inside your package if you want them distributed with the code, though this is less common for large projects.

3. "in-package layout"
   Placing your tests inside my_package/tests is a valid choice often called the "Integrated Tests" or "In-package" layout.

Structure:

my_project/
├── pyproject.toml
└── my_package/
    ├── __init__.py
    └── app.py
    └── tests/
        └── test_app.py

The Good
Easier Distribution: When you ship your library, the tests go with it. This allows users to run pytest --pyargs my_package after installation to verify the library works in their specific environment Pytest Documentation.
Direct Access: It can make importing internal modules slightly more intuitive since the test folder is part of the package hierarchy.
The Bad
Package Bloat: Users installing your library for production use will also download all your test data and test files, increasing the install size unnecessarily.
Namespace Pollution: If you don't include an init.py in your tests/ folder, some tools might get confused; if you do, your tests effectively become a sub-package (my_package.tests).
Testing against Source vs. Build: It increases the risk of testing against your local files rather than the actually installed package, which can hide packaging bugs Setuptools Guide.
Pro Tip
If you stick with this, ensure your pyproject.toml or setup.py is configured to exclude the tests sub-directory during the build process if you don't want them shipped to PyPI.

[btmlnsn]

I'm finally back from my significant travel fiasco - appreciate the feedback. Definitely some things in my original PR that I'd now like to address by changing in my code (likely via a new mini-PR, rather than a complete reversion of #342 ), and some things I'd like to comment on as well. Will have more tomorrow. Thanks

[btmlnsn]

#346