feat: implement needs_types serialization for ubproject.toml

Author: Arnaud Riess

.gitignore

@@ -12,6 +12,7 @@ user.bazelrc
 /_build*
 docs/ubproject.toml
 docs/schemas.json
+docs/needs_types_generated.toml
 
 # Vale - editorial style guide
 .vale.ini

src/extensions/score_sync_toml/README.md

@@ -0,0 +1,132 @@
+# score_sync_toml
+
+A Sphinx extension that configures and drives
+[`needs-config-writer`](https://needs-config-writer.useblocks.com) to produce a
+`ubproject.toml` file at build time.  This file is consumed **statically** by the
+[ubCode VS Code extension](https://ubcode.useblocks.com) to power the language server
+(need indexing, autocompletion, hover, schema validation, etc.).
+
+**The problem it solves:** ubCode cannot run Sphinx.  It needs a static
+`ubproject.toml` to know which RST directives are valid need types, what link types
+exist, what schemas apply, and so on.  `score_sync_toml` bridges this gap by
+serialising the live Sphinx-Needs configuration into that file after every build.
+
+---
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `__init__.py` | Sphinx extension — configures `needs_config_writer` and generates helper TOML |
+| `shared.toml` | Static TOML fragment always merged into the output `ubproject.toml` |
+| `BUILD` | Bazel build definition |
+
+---
+
+## How it works
+
+### 1. `setup(app)` configures `needs_config_writer`
+
+| Setting | Value | Reason |
+|---|---|---|
+| `needscfg_outpath` | `ubproject.toml` | Where to write the final file (relative to `confdir`) |
+| `needscfg_overwrite` | `True` | Always regenerate on each build |
+| `needscfg_write_all` | `True` | Write full config, not just diffs, so the file is self-contained |
+| `needscfg_exclude_defaults` | `True` | Omit default values to keep the file readable |
+| `needscfg_warn_on_diff` | `False` | Don't fail CI when the file changes |
+
+### 2. Excluded variables
+
+Some Sphinx-Needs config values are excluded from serialisation because they either
+cannot be represented in TOML or are managed elsewhere:
+
+| Variable | Reason for exclusion |
+|---|---|
+| `needs_from_toml` / `needs_from_toml_table` | Meta-config consumed by `needs_config_writer` itself |
+| `needs_schema_definitions_from_json` | Pointer handled via `shared.toml` |
+| `needs_schema_definitions` | Generated schema content — managed via `schemas.json` / `score_metamodel` |
+| `needs_render_context` | Contains Python callables (`draw_full_interface` etc.) — not serialisable |
+
+### 3. Merging `shared.toml`
+
+The static `shared.toml` is always merged into the output.  It contributes:
+
+- **`[parse.extend_directives]`** — teaches ubCode to parse third-party RST
+  directives (`grid`, `grid-item-card`, `uml`) without treating them as errors.
+- **`schema_definitions_from_json = "schemas.json"`** — points ubCode to the
+  JSON Schema validation file generated by `score_metamodel`.
+- **`index_on_save = true`** — tells ubCode to re-index whenever a file is saved.
+
+### 4. `_write_needs_types_toml(app)` — generating `[[needs.types]]`
+
+`needs_config_writer` cannot serialise the project's custom `ScoreNeedType` dicts
+(they contain Python-specific data) and silently skips them.  Without
+`[[needs.types]]` entries in `ubproject.toml`, **ubCode does not recognise any
+directives as needs** and the index is empty.
+
+`_write_needs_types_toml` works around this by:
+
+1. Reading `app.config.needs_types` — already populated by `score_metamodel` from
+   `metamodel.yaml` before `score_sync_toml` runs.
+2. Writing `needs_types_generated.toml` next to `conf.py` containing clean
+   `[[needs.types]]` entries with only the fields ubCode requires:
+   `directive`, `title`, `prefix`, and optionally `color` / `style`.
+3. Appending that file to `needscfg_merge_toml_files` so it is merged into the
+   final `ubproject.toml`.
+
+### 5. Relative path handling
+
+Fields such as `needs_external_needs[*].json_path` contain **absolute paths**
+injected by Bazel.  `needscfg_relative_path_fields` converts these to paths
+relative to `confdir` in the output TOML so the file remains portable.
+
+### 6. Suppressed warnings
+
+```
+needs_config_writer.path_conversion
+needs_config_writer.unsupported_type
+```
+
+Suppressed so that Bazel builds running with `-W` (warnings-as-errors) do not fail
+due to known, handled edge cases in serialisation.
+
+---
+
+## Data flow
+
+```
+metamodel.yaml
+     │
+     ▼
+score_metamodel.setup()  ──► app.config.needs_types (Python objects)
+                                         │
+                                         ▼
+                            _write_needs_types_toml()
+                                         │
+                                         ▼
+                            needs_types_generated.toml ──┐
+                                                         │
+shared.toml  ────────────────────────────────────────────┤  merge via
+                                                         │  needs_config_writer
+app.config (all other needs_* settings)  ────────────────┤
+                                                         ▼
+                                               ubproject.toml
+                                                    │
+                                                    ▼
+                                              Sphinx-Needs
+                                          ubCode language server
+```
+
+---
+
+## Generated files
+
+Both files are written to `confdir` (the `docs/` directory) on every Sphinx build
+and are committed to the repository so that ubCode works without needing to run a
+build first.
+
+| File | Generated by | Used by |
+|---|---|---|
+| `ubproject.toml` | `needs_config_writer` + merges above | ubCode language server |
+| `schemas.json` | `score_metamodel.sn_schemas.write_sn_schemas` | ubCode + sphinx-needs schema validation |
+| `needs_types_generated.toml` | `_write_needs_types_toml` | Merged into `ubproject.toml`, not committed |

src/extensions/score_sync_toml/__init__.py

@@ -17,6 +17,44 @@
 from src.helper_lib import config_setdefault
 
 
+def _write_needs_types_toml(app: Sphinx) -> Path:
+    """Serialize ``app.config.needs_types`` as ``[[needs.types]]`` TOML entries.
+
+    ``needs_config_writer`` cannot serialize the complex ``ScoreNeedType``
+    dicts (it emits an ``unsupported_type`` warning and skips them), so
+    ``ubproject.toml`` ends up with no type definitions.  Without those,
+    the ubCode language server does not recognise any RST directives as needs
+    and indexes nothing.
+
+    This function writes only the fields that ubCode requires to identify
+    need directives (``directive``, ``title``, ``prefix``, and optionally
+    ``color``/``style``) to a separate TOML file that is then merged into
+    ``ubproject.toml`` by ``needs_config_writer``.
+
+    Must be called *after* ``score_metamodel.setup()`` has run (i.e. after
+    ``app.config.needs_types`` has been extended with the metamodel types).
+    """
+    lines: list[str] = [
+        "# Auto-generated – do not edit manually.\n",
+        "# Contains [[needs.types]] entries derived from metamodel.yaml.\n",
+        "# Required for the ubCode language server to recognise need directives.\n\n",
+    ]
+    for nt in app.config.needs_types:
+        lines.append("[[needs.types]]\n")
+        lines.append(f'directive = "{nt["directive"]}"\n')
+        lines.append(f'title = "{nt["title"]}"\n')
+        lines.append(f'prefix = "{nt["prefix"]}"\n')
+        if color := nt.get("color"):
+            lines.append(f'color = "{color}"\n')
+        if style := nt.get("style"):
+            lines.append(f'style = "{style}"\n')
+        lines.append("\n")
+
+    output_path = Path(app.confdir) / "needs_types_generated.toml"
+    output_path.write_text("".join(lines), encoding="utf-8")
+    return output_path
+
+
 def setup(app: Sphinx) -> dict[str, str | bool]:
     """
     Extension to configure needs-config-writer for syncing needs configuration to TOML.
@@ -50,6 +88,10 @@ def setup(app: Sphinx) -> dict[str, str | bool]:
         # These are managed via schemas.json / score_metamodel and must not be
         # duplicated in ubproject.toml (ubCode only needs schema_definitions_from_json).
         "needs_schema_definitions",
+        # needs_render_context contains Python callable objects (draw_full_interface,
+        # draw_full_module, etc.) that cannot be serialized to TOML.
+        # needs_config_writer emits unsupported_type warnings for these and skips them.
+        "needs_render_context",
     ]
     """Exclude resolved/generated config values that don't belong in ubproject.toml."""
 
@@ -58,6 +100,14 @@ def setup(app: Sphinx) -> dict[str, str | bool]:
     )
     """Merge the static TOML file into the generated configuration."""
 
+    # Write [[needs.types]] from the metamodel into a separate TOML fragment and
+    # merge it.  needs_config_writer cannot serialise the complex ScoreNeedType
+    # dicts itself (unsupported_type), so ubproject.toml would otherwise contain
+    # no type definitions and the ubCode language server would index nothing.
+    needs_types_toml = _write_needs_types_toml(app)
+    app.config.needscfg_merge_toml_files.append(str(needs_types_toml))
+    """Merge the generated [[needs.types]] TOML into the final ubproject.toml."""
+
     app.config.needscfg_relative_path_fields.extend(
         [
             "needs_external_needs[*].json_path",
@@ -70,10 +120,11 @@ def setup(app: Sphinx) -> dict[str, str | bool]:
     """Relative paths to confdir for Bazel provided absolute paths."""
 
     app.config.suppress_warnings += [
-        "needs_config_writer.unsupported_type",
         "needs_config_writer.path_conversion",
+        # needs_render_context values are Python callables; they are excluded via
+        # needscfg_exclude_vars above, but suppress the warning as a safety net.
+        "needs_config_writer.unsupported_type",
     ]
-    # TODO remove the suppress_warnings once fixed
 
     return {
         "version": "0.1",