Local Full Test Suite Workflow?

[Wes2000ley]

Hi all,

I have a fork of NIPA that is currently ahead of upstream/main with a local vmksft-oriented workflow intended to make
broader local testing easier to run and reproduce.

The main thing it adds is a containerized/local harness under local/ that can:

• run a broader local vmksft/selftest flow without the full netdev cloud deployment
• provide a Docker/Compose-based setup for dependencies and execution
• support committed, dirty-tree, and patch-based runs
• keep local state/results in one place and serve browsable results locally
• include some supporting tests and helper scripts around that workflow

Before I spend time cleaning this up for upstreaming, I wanted to ask:

1. Is there any interest in NIPA having an officially supported local/containerized full-test or broader vmksft
   workflow like this?
2. If yes, would a PR for something in this direction be welcome?
3. If yes, would maintainers prefer:
   • a smaller scoped contribution first
   • a docs-first proposal/RFC
   • or the full local harness as a draft PR for discussion?

[matttbe]

Hi Wesley,

Thank you for having started this conversation.

Is there any interest in NIPA having an officially supported local/containerized full-test or broader vmksft workflow like this?

Definitively yes, I was planning to work on that ASAP.

If yes, would a PR for something in this direction be welcome?

I guess yes, but when looking at what you did, I'm not sure whether I understand why there are so many new scripts and modifications for this. Is it not possible to re-use existing code?

Out of curiosity, was adding a container to run ./ingest_mdir.py script not enough for your need?

If yes, would maintainers prefer:

I think starting by mentioning what you want to achieve (and why) might be preferable. For example, I imagine most people only want to validate their patches before submitting them. So I guess they don't need the whole automation system with another web interface to display the results, or do they? :)

Otherwise, the development is similar to the kernel one: documented incremental changes reusing existing blocks as much as possible.

[Wes2000ley]

Hi Matthieu,

The intent of this refactor was to make /local a deterministic, Docker-only vmksft harness instead of continuing to
layer conditionals onto the existing contest/remote path. I did start by trying to reuse more of the shared code, but
once I pushed that further it got messier and less safe than keeping the local service self-contained.

ingest_mdir.py is also not a substitute for this. It runs NIPA patch/series checks against a manually prepared tree,
but it does not run the vmksft selftest service, does not provide a long-lived queue/service model, and does not
support queue-time frozen committed, dirty, and explicit patches inputs. Those three modes are deliberate because
different people work from clean commits, dirty trees, or exported patch directories, and I did not want to force one
workflow on everyone. The point of the local harness is to freeze whichever input was chosen at submission time so the
run stays deterministic.

That was the reasoning behind the split and the extra files: to keep the local path deterministic and support a few
different submission workflows without coupling it more tightly to the existing remote path. I am happy to rework
parts of it if there is a cleaner shape that would make it easier to merge upstream, and if you would rather take it
in a different direction entirely, I understand that as well.

[matttbe]

ingest_mdir.py is also not a substitute for this. It runs NIPA patch/series checks against a manually prepared tree

Out of curiosity, why is it an issue for you?

but it does not run the vmksft selftest service

No because usually there is no need to run the whole test suite, and it should be just a matter of vng --build --config and vng (...) make -C ... TARGETS=...

Maybe a simple script could do that?

does not provide a long-lived queue/service model, and does not support queue-time frozen committed, dirty, and explicit patches inputs.

It looks like you would like to have a fully automated system to test patches. Is it a requirement for your workflow? (e.g. to let AI agent coding and testing? Or having something setup for a team of devs?)

It is just to try to understand what you want to achieve vs. what we have for the moment and the current plans.

[Wes2000ley]

Honestly, I just wanted to run the tests in a way that’s as close as possible to how they run when I push patches.

The main benefit for me is having a repeatable, highly parallelized flow instead of piecing it together by hand every time.

The fact that agents can look at the runs/results is a nice bonus, and it would also be easy enough to make available on a local LAN for a dev team, but those are extra benefits, not really the main motivation.

Not having to manually pull all the needed dependencies for every test on every machine I use is a nice benefit also.

This definitely might be overkill, but most of the work is already done and I'm pretty happy with it in my workflow.