Rethinking the runner configuration interface: integration of the OSVVM methodology into VUnit

[umarcor]

The most popular open source VHDL verification projects are cocotb, OSVVM and UVVM, along with VUnit. As discussed in larsasplund.github.io/github-facts, there are some philosophical differences between them: OSVVM and UVVM are defined as "VHDL verification methodologies", cocotb is for (Python) co-routine co-simulation, and VUnit is a Python-aided framework of HDL utilities. Naturally, there are some overlapping capabilities because all of them provide basic features such as logging and building/simulation. Therefore, methodologies can be seen as bundles of utilities (frameworks), and some users might refer to using the VUnit runner as a methodology. Nonetheless, it is in the DNA of VUnit to be non-intrusive and allow users to pick features one by one, including reusing the methodologies they are used to.

Currently, it is possible to use OSVVM utilities/libraries in a VUnit environment. Although there are still some corner cases to fix (#754, #767, #768), it is usable already. In fact, some of VUnit's features do depend on OSVVM's core. However, it is currently not possible to use the OSVVM methodology as-is within VUnit. The OSVVM methodology uses top-level entities without generics or ports, and the entrypoints are VHDL configurations. Meanwhile, VUnit needs a top-level generic of type string in order to pass data from Python to VHDL.

Most simulators do support calling a configuration instead of an entity as the primary simulation unit. It should, therefore, be trivial to support OSVVM's configurations as entrypoints in VUnit. I am unsure about VUnit's parser supporting configurations in the parser and dependency scanning features; but that should not be the main challenge anyway.

The main challenge we need to address is that passing generics to VHDL Configurations is not supported in the language. If that was possible, the runner string might be forwarded to the entiy within the configuration. For the next revision, we might propose enhancements in this regard, since revisiting the limitations of configurations is one of the expected areas to work on. Nevertheless, that will/would take several months or years until made available in simulators.

Yesterday, I had a meeting with @JimLewis and he let me know that he's been thinking about implementing some mechanism for passing data between the TCL scripts (.pro files) and the VHDL testbenches. We talked about .ini, .yml and .json, and I suggested to use the latter because there is a JSON reader library available already: Paebbels/JSON-for-VHDL. In fact, JSON-for-VHDL is submoduled in VUnit, in order to pass very complex generics to the testbench.

I believe this is a good oportunity to document the syntax of VUnit's runner generic, make a public API from it, write a data model and provide multiple reader/writer implementations. @JimLewis said he did not put much thought into the data model yet, but he would be willing to include integration with VUnit into the scope when he works on it. Maybe there is no need for him to write a VHDL solution from scratch and it can be based on JSON-for-VHDL + VUnit's runner package.

Enhance VUnit's simulator interfaces to support writing runner (CLI) arguments to a file or to an envvar

Currently, VUnit's runner expects to receive an string, which Python passes as a top-level generic. Actually, there is no limitation in VHDL for using an alternative method. The generic might be a path, and users might read the file in the testbench before passing it to the functions from the runner package. By the same token, the generic might point to a JSON file, and users might convert that to the string syntax expected by the runner. Hence, the main challenge is that VUnit's Python simulator interfaces do not support writing runner parameters to a file.

Well, that is not 100% correct: when GHDL is used, option ghdl_e prevents running the simulation and instead writes all the CLI arguments in a JSON file (#606):

vunit/vunit/sim_if/ghdl.py

Lines 303 to 322 in 7879504

	if not ghdl_e:
	cmd += sim
	if elaborate_only:
	cmd += ["--no-run"]
	else:
	try:
	makedirs(output_path, mode=0o777)
	except OSError:
	pass
	with (Path(output_path) / "args.json").open("w", encoding="utf-8") as fname:
	dump(
	{
	"bin": str(Path(output_path) / f"{config.entity_name!s}-{config.architecture_name!s}"),
	"build": cmd[1:],
	"sim": sim,
	},
	fname,
	)
	return cmd

That is used for building a design once (and generating an executable binary if the simulator is based on a compile & link model) and then executing it multiple times for co-simulation purposes: https://github.com/VUnit/cosim/tree/master/examples/buffer. Therefore, we might want to generalise this to all the simulator interfaces and make it optional to specify the name/location of the JSON file.

Similarly, we might want to support passing runner arguments through an environment variable. In the context of integrating VUnit and cocotb, one of the requirements is specifying environment variables per test/testbench. That's because cocotb bootstraps an independent Python instance, and the UUT is the design; so data needs to be passed through the filesystem or envvars. In fact, this is a requested feature: #708. If we used the same mechanism for the runner, cocotb might re-implement VUnit's runner package in Python (which is "just" 1K lines of VHDL code for 100% compatibility). I believe that would allow to plug cocotb's regression management. The remaining functionality would be for VUnit to "discover" the test cases before executing the simulations.

So, we might have an enumeration option to decide passing the runner string as a top-level generic, as an environment variable or as a file. Taking VHDL 2019 features into account, OSVVM and VUnit might end up using the envvar approach indeed.

Enhance JSON-for-VHDL

The current implementation of JSON-for-VHDL is functional and synthesisable, but not optimal for simulation. Some years ago, @Paebbels and @LarsAsplund discussed about writing an alternative implementation for simulation only, which would have less constraints and better performance. They also talked about using VHDL 2019 features. I don't remember if using those was a requirement for the optimised simulation-only implementation, or whether that could be done with VHDL 2008.

If we are to use JSON for sharing configuration parameters between VUnit's Python or OSVVM's TCL and VHDL, I guess we would make JSON-for-VHDL a prioritary dependency in the ecosystem.

Co-simulation

The run package might be enhanced for getting the data from a foreign environment. By encapsulating the interaction with the "runner configuration model" in a protected type, we might provide various implementations. For instance, the VHDL testbench might query a remote service which tests to run, and where to push the results.

Terminology: configurations

OSVVM uses VHDL configurations for composing the test harness and the test cases. At the same time, VUnit use the term "configuration" for referring to a set of values for top-level generics. So, in Python users can generate multiple sets of parameters for each testbench/testcase. That is not very conflictive yet because VHDL configurations are not used much in the VUnit ecosystem. However, if we are to improve the integration of OSVVM and VUnit, we might want to reconsider the naming.

/cc @Paebbels @JimLewis @ktbarrett @jwprice100

[Paebbels]

The advantage of generics over file inputs or similar methods is that such an input is a constant and can be used in elaboration. Other techniques might not be available early enough or might not ne static enough for some use cases.

A JSON parser doesn't need VHDL-2019, but an implementation might be easier and have less repeated code.

While a JSON parser already exists and can be transformed to an improved variant for simulation (no fixed arrays, but dynamic memory allocation and pointer structures instead of index arrays), I consider YAML a better solution, especially as it can be written easier to a file.

If INI is needed, I propose to support TOML, which is a superset.

[LarsAsplund]

The VUnit dependency scanner recognizes configurations as it is today. So an intermediate (?) solution would be to combine our ability to control the testbench from a lower level test control entity (see https://vunit.github.io/run/user_guide.html#distributed-testbenches) with a new feature the recognizes a testbench configuration as a test case for that testbench. All the user would have to do to a standard OSVVM testbench is to add the runner_cfg to the testbench entity and the test controller entity. The test architectures would only have test_runner_startup and test_runner_cleanup. For me the key difference between an OSVVM testbench and a VUnit testbench is that OSVVM uses a configuration to define a test case. The fact that OSVVM puts test control at a lower level or that VUnit requires a runner_cfg poses no conflict

Update:

The main challenge we need to address is that passing generics to VHDL Configurations is not supported in the language.

Ouch, I didn't know that. Well, that is a problem

[JimLewis]

WRT using file inputs, ActiveHDL really has a hard time keeping rooted in the simulation start directory and seems to creep into its library directory any time you turn your back or during certain updates - was just wrestling with an issue in ActiveHDL 12 that was not there in 11.

[LarsAsplund]

@JimLewis In general tests should be given private directories to work from or input and output files from different test cases will interfere/overwrite each other. Just need a way to pass that info Not knowing the testbench is running from becomes another problem.

[JimLewis]

@LarsAsplund For me, all tests in the same test suite (testing similar functionality like a UART core or a particular package) have a unique name and unique files they operate on (often named similar to their test name). Only when I switch to a new test suite do I need to run them out of a different directory.

The only time I have had to worry about name conflicts is when I am reusing a test case on a similar design.

[umarcor]

@LarsAsplund, I see two main challenges in your proposal:

• As you wrote in the update/edit, overriding the top-level generics of an entity when using a configuration as a primary unit is problematic:

  • The language does not support generics for configurations.
  • It is not specified that CLI arguments can/should bypass the configuration and be applied to the entity.
  • Overriding top-level generics through the CLI is non-standard indeed. That's why support is uneven among vendors.
  • Therefore, using files (along with a JSON/YAML reader) or using environment variables (along with VHDL 2019's capabilities) are the only LRM compliant solutions at the moment. I do believe we should keep using the CLI for VUnit, because that is proven to be usable. However, I would be careful about double betting on the CLI for "passing complex objects from Python to VHDL"; we should find more robust solutions for that.

• Neither CoCoTb nor OSVVM are willing to require VUnit's libraries to be imported/included/used in the HDL testbenches.
  @ktbarrett was vocally very explicit and @JimLewis is very implicit about it (by explicit omission).
  I do not agree with their criteria, but I believe we must allow those use cases if we are to integrate multiple methodologies into a common Python infrastructure.

That's why I proposed specifying what is the content/format/syntax of the runner_cfg string and how does test_runner_cleanup communicate the results to Python.
In the cocotb case, that'd allow them to implement the functionality in Python, so that they can keep the HDL sources untouched. That is of critical relevance because a lot of cocotb users are using cocotb to avoid any manipulation of HDL. Hence, the top-level of the simulator is the UUT, not the testbench (which is Python).
In the OSVVM case, users might use VUnit's library themselves, on top of the references/templates from OSVVM. Actually, that is something they can do already, because OSVVM users are expected to manipulate HDL sources. However, that breaks one of the critical concepts of the OSVVM Methodology. By using files or envvars, we can allow OSVVM to keep the essence that "primary units and top-level entities don't have generics". Then, it'd be up to Jim to reuse the "runner specification" in OSVVM's plumbing, or to reinvent the wheel.

In other words, the challenge here is the complement of pyEDAA.Reports. Reports is about the outcome of running testsuites, tests, bins, etc. while Runner is about how to tell which of those to schedule. Hence the Runner API/specification needs to conceptually be, in fact, a subset of Reports.

---

With regard to where to execute the simulations, IMHO that is a completely different topic. VUnit can already run simulations using ActiveHDL and OSVVM. None of that is affected by the proposal in this issue. OSVVM's TCL plumbing uses a different approach to VUnit's Python plumbing in that regard, however OSVVM's TCL is out of scope here because the point is precisely to run/use OSVVM through VUnit (using EDAA to treat .pro files as a DSL, independent to TCL).

[JimLewis]

@umarcor The issue with ActiveHDL is not being able to run simulations within a given environment. It is to get the VHDL to write files in the expected locations when running that is interesting. Especially if the VHDL code does not specify path information.

WRT EDAA, why not run TCL using something like tkinter in python?

[JimLewis]

@umarcor WRT environment variables, you have to convince vendors other than one who has proactively implemented VHDL-2019 to implement the 2019 feature that allows environment variables to be read. That would be my only concern with that path. It would be nice though.

[ktbarrett]

Not sure there needs to be 1 method for discovering testbenches or passing test configurations that everyone has to support. Hell, there might even be multiple ways for passing configurations for each methodology due to tool or language limitations. There should be multiple kinds of testbench objects and the runner should dispatch to the testbench object to decide how to run those tests instead of baking this into the tool classes. I have suggested this previously. Each methodology should provide these classes so they don't have to be included in VUnit, but VUnit will need to be updated to defer decisions to the testbench objects where appropriate. That way VUnit can leave alone it's current testbench discovery, configuration passing, and test result system.