Add compute nodes support

[xprilion]

This pull request introduces a comprehensive compute node management system to the backend, including models, capability discovery, workspace isolation, and periodic maintenance tasks. It adds new database models for SSH keys and compute nodes, implements probing and management utilities for compute environments, and integrates workspace management for per-conversation isolation. Additionally, it updates the system prompt and API routing to support these features.

Compute Node Management and Capability Discovery:

• Added ComputeCapabilities and GPUInfo dataclasses to represent compute node hardware/software capabilities, with serialization/deserialization methods.
• Implemented probe_sandbox to asynchronously discover and report compute environment details (CPU, RAM, GPU, Python versions, Docker, Conda, network, etc.) for any sandbox.
• Added ComputeManager for validating node configurations (SSH, local, modal) and integrating with SSH key management.

Workspace Isolation and Maintenance:

• Introduced WorkspaceManager to handle per-conversation workspace directories, archiving, cleanup, and metadata reporting, ensuring filesystem isolation and efficient resource management.
• Configured Celery to include new compute tasks and added periodic jobs for node health checks and workspace cleanup. [1] [2]

Database and API Integration:

• Added new database models: SSHKey and ComputeNode, with relationships to User and relevant fields for configuration, capabilities, and status. [1] [2]
• Updated API routers to include endpoints for compute node and SSH key management.

Prompt and Configuration Enhancements:

• Extended the system prompt template and builder to include compute environment information and guidance for compute planning. [1] [2] [3]

Other:

• Updated .gitignore for the .keys directory to ignore all files except the .gitignore itself.
• Exposed new compute-related classes and functions in the backend/openmlr/compute module.

[github-actions]

Qodana for Python

170 new problems were found

Inspection name	Severity	Problems
Unbound local variables	🔶 Warning	2
Unused imports	🔶 Warning	2
Invalid type hints definitions and usages	🔶 Warning	1
Unreachable code	🔶 Warning	1
Redundant character escape	🔶 Warning	1
Unused local symbols	◽️ Notice	67
Accessing a protected member of a class or a module	◽️ Notice	43
Unclear exception clauses	◽️ Notice	30
Shadowing names from outer scopes	◽️ Notice	10
Method is not declared static	◽️ Notice	6
PEP 8 naming convention violation	◽️ Notice	4
Non-optimal list declaration	◽️ Notice	2
An instance attribute is defined outside init``	◽️ Notice	1

☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

• Or via our issue tracker: https://jb.gg/qodana-issue
• Or share your feedback: https://jb.gg/qodana-discussions