Data CSV Dump Attribute

[electronsandstuff]

This PR adds a new attribute data_dump_file to Xopt which will automatically dump Xopt.data to a user specified CSV file while running. This avoids users having to parse the YAML file generated from dump_file and extracting their data. Full list of changes:

• New attribute data_dump_file for a CSV file dumped during .run()
• Rename dump_file to xopt_dump_file to avoid ambiguous name with new attribute.
• Add legacy support for old name with deprecation warnings to update
  • Before validator for existing config files
  • Properties for user code
• Add environment variable and user home expansion in data_dump_file and xopt_dump_file
• Add tests for new property, legacy support, environment variable expansion

[codecov]

Codecov Report

❌ Patch coverage is 98.95833% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
xopt/base.py	97.36%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[nikitakuklev]

No objections in general.

• Not a fan of the rename + shim because of how many saves it affects. Would keeping dump_file + adding data_dump_file work?

• In general, .data can contain whatever the evaluators return, including dicts - ok via YAML, but will get dumped as strings in CSV. Is that intended behavior? (probably yes)

Couple Claude-inspired-nits:

Silent behavior change in dump(). It previously raised ValueError when no file was specified; now it silently no-ops (and evaluate_data relies on that by calling it unconditionally).

adding xopt_dump_file to an old dumped config (which always contains dump_file: null) raises a ValidationError, precisely when the user follows the deprecation message's advice. The conflict check at xopt/base.py:267 should only fire when data["dump_file"] is not None.

[electronsandstuff]

because of how many saves it affects

Sorry, could you explain what you mean?

Personally, I think having a .dump_file and .data_dump_file is confusing and I prefer the unambiguous names. With the before validator and properties, I think all user code / save files should keep working as-is. The intent would be to keep it that way for a long time with .xopt_dump_file being the official internal name in documentation to avoid having a .dump_file and .data_dump_file that seem to be the same thing.

In general, .data can contain whatever the evaluators return, including dicts

Yeah, I don't see a good way of getting around this, but this appears on the user side of whatever the evaluator function returns, so keep as-is.

[nikitakuklev]

Sorry, could you explain what you mean?

Right, I meant that changing the name will introduce a compatibility shim that should at some point be removed and will break old saves. Since we are churning format sometimes anyways, this is not too bad.

Once the llm-bug-reports are checked, can merge.

[electronsandstuff]

adding xopt_dump_file to an old dumped config (which always contains dump_file: null) raises a ValidationError...

OK, I made it so the check only happens if dump_file isn't None.

Silent behavior change in dump(). It previously raised ValueError when no file was specified; now it silently no-ops (and evaluate_data relies on that by calling it unconditionally).

This was intentional to make it possible to call .dump with both files without doing increasingly complicated checks. However, thinking about it some more, maybe combining the file outputs isn't the best interface. I ended up reverting .dump so the original behavior stays. I don't think anything actually uses the exception, but can preserve. Then I added dump_data in the same format.