#195 support for etl std byte as byte type

changes/195.feature.md

@@ -0,0 +1 @@
+Support 'etl::byte' and 'std::byte' as LotusRPC byte type. Support for definition file overlays

docs/binary.md

@@ -115,6 +115,8 @@ packet
 +8: "0x11"
 ```
 
+LotusRPC uses `lrpc::byte` as underlying byte type for byte arrays. The actual type is configurable with the [byte_type](reference.md#settings) setting.
+
 ### Array
 
 In LotusRPC, an array always has a fixed capacity as specified in the interface definition file. The number of used elements in the array is however determined at runtime so can be less than the capacity. In addition to space for all elements, the encoded array has a single byte size field at the start. The array [12, 13, 14, 15] with capacity of 6 and is encoded as follows

docs/index.md

@@ -58,7 +58,7 @@ This type was added because in the generated C++ code `bytearray` is more effici
 LotusRPC defines the following type aliases and uses them internally.
 
 * `LRPC_BYTE_TYPE`. Type that LotusRPC uses internally to represent a byte. Defaults to `uint8_t` but can be defined to any other single-byte type,
-* `lrpc::bytearray`. Alias for `etl::span<const LRPC_BYTE_TYPE>`
+* `lrpc::bytearray`. Alias for `etl::span<const lrpc::byte>`
 * `lrpc::string_view`. Alias for `etl::string_view`
 * `lrpc::span`. Alias for `etl::span`
 * `lrpc::array`. Alias for `std::array`

docs/reference.md

@@ -202,14 +202,26 @@ services:
 
 LotusRPC allows some level of customization with the following optional properties in the `settings` section in the definition file.
 
-| Property                |
-|-------------------------|
-| rx_buffer_size          |
-| tx_buffer_size          |
-| namespace               |
-| version                 |
-| definition_hash_length  |
-| embed_definition        |
+| Property               | Type/value          |
+|------------------------|---------------------|
+| rx_buffer_size         | At least 3          |
+| tx_buffer_size         | At least 3          |
+| namespace              | String              |
+| version                | String              |
+| definition_hash_length | 0 to 64             |
+| embed_definition       | Boolean             |
+| byte_type              | See byte type table |
+
+| Byte type     | Remark                                            |
+|---------------|---------------------------------------------------|
+| uint8_t       | Default                                           |
+| int8_t        |                                                   |
+| char          |                                                   |
+| char8_t       | Requires at least C++20. Not enforced by LotusRPC |
+| unsigned char |                                                   |
+| signed char   |                                                   |
+| etl::byte     |                                                   |
+| std::byte     | Requires at least C++17. Not enforced by LotusRPC |
 
 `rx_buffer_size` and `tx_buffer_size` define the receive and transmit buffer size in bytes for generated C++ server code. Default is 256. `namespace` defines the C++ namespace to generate server code in. By default the code is generated in the global namespace.
 
@@ -222,10 +234,114 @@ Using the definition that is embedded on the server from the client side can be
 * Using the `from_server` factory method of the `LrpcClient` class
 * Specifying `definition_from_server` as `always` or `once` in the [lrpcc](tools.md#lrpcc) config file.
 
+`byte_type` is used to control the type that LotusRPC uses internally for [lrpc::bytearray](binary.md#bytearray).
+
 ## User settings
 
 Section `user_settings` allows for free-format user settings in the LotusRPC definition file. The user can specify any valid YAML data structure under `user_settings`. The user settings are accessible when visiting a `LrpcDef` object with a visitor deriving from `LrpcVisitor`. This flexible approach has no specific use case for LotusRPC and is ignored by LotusRPC internally, but allows users to include custom data in the LotusRPC definition file and use it to their own benefit. User settings may also be useful for creating anchors that can be referenced in other parts of the definition, e.g. as a common array size. See [visiting LrpcDef](extending_lrpc.md#visiting-lrpcdef) for more information on extending LotusRPC.
 
+## Definition overlays
+
+LotusRPC supports merging overlay definitions on top of base definitions. This technique allows for creating variants of a definition without duplicating the entire base structure. For example, you can create platform-specific or variant-specific overlays that selectively add, remove, or replace properties from a base definition.
+
+### Overlay files
+
+Overlay files are YAML files like the main definition file and typically also have the _.lrpc.yaml_ extension. An overlay file follows the same structure as the main definition file but only for the slice of the main definition that is modified. Additionally it has a `merge_strategy` property to control the merge type.
+
+Overlay files can contain multiple [YAML documents](https://yaml.org/spec/1.2.2/#91-documents) to specify multiple (conflicting) overlay actions in a single file.
+
+### Merge strategy
+
+The merging process is controlled by the `merge_strategy` property. This property can be specified at any level to control how the definition properties are merged. The merge strategy is inherited from parent properties to child properties.
+
+| Strategy | Behavior                    | Precondition                    |
+|----------|-----------------------------|---------------------------------|
+| add      | Add a property to base      | Item does not exist in base[^1] |
+| remove   | Remove a property from base | Item exists in base             |
+| replace  | Replace a property in base  | Item exists in base             |
+
+[^1]: When adding a basic item to a list, the item must not exist in the base definition. When adding a composite, named item to a list, the item is added in its entirety when no item with that name exists in the base. When an item with that name does exist in the base, the merge is done recursively on the sub-properties of the composite item
+
+Add, remove and replace overlays on composite properties are always matched by the `name` property. When adding or removing a basic property from a list, the property is matched by value. It is not possible to replace a basic property in a list directly, but it can be achieved by applying a remove overlay followed by an add overlay.
+
+It is also possible to remove a basic property (e.g. string, bool or int) by assigning `null`. In this case it is not necessary to provide a merge strategy.
+
+A merge fails when an overlay fails to meet the precondition.
+
+### Example 1: Adding a new parameter
+
+Base definition:
+
+```yaml
+name: overlay_example
+settings:
+  namespace: ov_ex
+services:
+  - name: MyService
+    functions:
+      - name: DoWork
+        params:
+          - name: timeout
+            type: uint32_t
+```
+
+Overlay:
+
+```yaml
+services:
+  - name: MyService
+    functions:
+      - name: DoWork
+        params:
+          - name: retries
+            type: uint8_t
+        merge_strategy: add
+```
+
+Result: Function `DoWork` has both `timeout` and `retries` parameters.
+
+### Example 2: Removing a parameter
+
+Overlay:
+
+```yaml
+
+services:
+  - name: MyService
+    functions:
+      - name: DoWork
+        params:
+          - name: timeout
+            merge_strategy: remove
+```
+
+Result: Function `DoWork` no longer has the `timeout` parameter.
+
+### Example 3: Removing a property with null
+
+Overlay:
+
+```yaml
+settings:
+  namespace: null
+```
+
+Result: The `namespace` setting is removed from the settings.
+
+### Example 4: Replacing an entire service
+
+Overlay:
+
+```yaml
+services:
+  - name: MyService
+    functions:
+      - name: NewFunction
+    merge_strategy: replace
+```
+
+Result: Service `MyService` is completely replaced; previous functions are removed.
+
 ## LrpcType
 
 The LRPC definition file uses LrpcType to describe function arguments, function return values and struct fields.

docs/tools.md

@@ -15,8 +15,18 @@ The `cpp` command generates all C++ files needed for the RPC server.
 
 `lrpcg cpp -d example.lrpc.yaml -o output-dir`
 
+Definition overlays can be specified with one or more `-ov` options. A single overlay may contain multiple [YAML documents](https://yaml.org/spec/1.2.2/#91-documents). For more information about definition overlays see [Definition overlays](reference.md#definition-overlays).
+
 For more info type `lrpcg cpp --help`
 
+### Overlay merge
+
+Basic usage: `lrpcg merge -d base.lrpc.yaml -ov overlay1.lrpc.yaml -ov overlay2.lrpc.yaml -o result.lrpc.yaml`
+
+The `merge` command merges the specified overlay files into the base definition and saves the result to a new file. See the [cpp command](#c-server-side-code-generation) for details about the overlay files.
+
+For more info type `lrpcg merge --help`
+
 ### C++ server core code generation
 
 The `cppcore` command generates only the core files of the RPC server. These files are static and do not depend on the definition file. In some cases it may be beneficial to generate them separately from the rest of the RPC server.

src/lrpc/client/lrpc_client.py

@@ -11,7 +11,7 @@
 from lrpc.core.meta import MetaVersionResponseDict, MetaVersionResponseValidator
 from lrpc.types import LrpcType
 from lrpc.types.lrpc_type import LrpcResponseType
-from lrpc.utils.load_definition import load_meta_def
+from lrpc.utils import load_lrpc_def
 
 from .decoder import LrpcDecoder
 from .encoder import lrpc_encode
@@ -49,8 +49,10 @@ def __init__(self, lrpc_def: LrpcDef, transport: LrpcTransport) -> None:
 
     @staticmethod
     def from_server(transport: LrpcTransport, save_to: Path | None = None) -> "LrpcClient":
-        meta_def = load_meta_def()
-        meta_client = LrpcClient(meta_def, transport)
+        # The meta client is constructed from a partial LRPC definition (only a name).
+        # The rest of the definition consists of the meta overlay. This is enough to
+        # retrieve the definition embedded in the server
+        meta_client = LrpcClient(load_lrpc_def("{name: RetrieveDefinition}"), transport)
         # pylint: disable = protected-access
         full_def = meta_client._retrieve_definition(save_to)
 

src/lrpc/codegen/constants.py

@@ -61,7 +61,7 @@ def _constant_definition(self, constant: LrpcConstant) -> str:
             v = f'"{str_value}"'
         elif constant.cpp_type() == "bytearray":
             ba_value = typing.cast(bytes, constant.value())
-            t = f"lrpc::array<LRPC_BYTE_TYPE, {len(ba_value)}>"
+            t = f"lrpc::array<lrpc::byte, {len(ba_value)}>"
             v = ", ".join(hex(b) for b in ba_value)
         else:
             t = constant.cpp_type()

src/lrpc/codegen/meta_service_file_writer.py

@@ -19,7 +19,7 @@
 \t\t\tfinal = (transmitSize != lrpc_meta::DefinitionStreamChunkSize) ||
 \t\t\t\t\t(data.size() == lrpc_meta::DefinitionStreamChunkSize);
 
-\t\t\tdefinition_response(data.take<const LRPC_BYTE_TYPE>(transmitSize), final);
+\t\t\tdefinition_response(data.take<const lrpc::byte>(transmitSize), final);
 \t\t}
 \t}
 \tvoid definition_stop() override {}

src/lrpc/core/definition.py

@@ -3,6 +3,7 @@
 
 # pylint: disable = unused-import
 from collections.abc import Iterable  # noqa: TC003
+from copy import deepcopy
 from pathlib import Path
 from typing import cast
 
@@ -60,7 +61,8 @@ def save_to(compressed: bytes, destination: Path) -> None:
         with destination.open("wt+") as dest:
             dest.write(LrpcDef._decompressed(compressed))
 
-    def __init__(self, raw: LrpcDefDict) -> None:
+    def __init__(self, raw_: LrpcDefDict) -> None:
+        raw = deepcopy(raw_)
         LrpcDefValidator.validate_python(raw, strict=True, extra="allow")
         self._definition_yaml = yaml.dump(raw, sort_keys=False)
 

src/lrpc/core/service.py

@@ -14,15 +14,15 @@ class LrpcServiceDict(TypedDict):
     id: int
     functions: NotRequired[list[LrpcFunOptionalIdDict]]
     streams: NotRequired[list[LrpcStreamOptionalIdDict]]
-    functions_before_streams: bool
+    functions_before_streams: NotRequired[bool]
 
 
 class LrpcServiceOptionalIdDict(TypedDict):
     name: str
     id: NotRequired[int]
     functions: NotRequired[list[LrpcFunOptionalIdDict]]
     streams: NotRequired[list[LrpcStreamOptionalIdDict]]
-    functions_before_streams: bool
+    functions_before_streams: NotRequired[bool]
 
 
 # pylint: disable=invalid-name
@@ -36,7 +36,7 @@ def __init__(self, raw: LrpcServiceDict) -> None:
         functions, streams = self._assign_function_and_stream_ids(
             raw.get("functions", []),
             raw.get("streams", []),
-            functions_before_streams=raw["functions_before_streams"],
+            functions_before_streams=raw.get("functions_before_streams", True),
         )
 
         self._name = raw["name"]

src/lrpc/core/settings.py

@@ -1,6 +1,10 @@
+from typing import Literal
+
 from pydantic import TypeAdapter
 from typing_extensions import NotRequired, TypedDict
 
+LrpcByteType = Literal["uint8_t", "int8_t", "char", "char8_t", "unsigned char", "signed char", "etl::byte", "std::byte"]
+
 
 class RpcSettingsDict(TypedDict):
     version: NotRequired[str]
@@ -9,6 +13,7 @@ class RpcSettingsDict(TypedDict):
     namespace: NotRequired[str]
     rx_buffer_size: NotRequired[int]
     tx_buffer_size: NotRequired[int]
+    byte_type: NotRequired[LrpcByteType]
 
 
 # pylint: disable=invalid-name
@@ -25,6 +30,7 @@ def __init__(self, raw: RpcSettingsDict) -> None:
         self._namespace = raw.get("namespace", None)
         self._rx_buffer_size = raw.get("rx_buffer_size", 256)
         self._tx_buffer_size = raw.get("tx_buffer_size", 256)
+        self._byte_type: LrpcByteType = raw.get("byte_type", "uint8_t")
 
     def version(self) -> str | None:
         return self._version
@@ -43,3 +49,6 @@ def rx_buffer_size(self) -> int:
 
     def tx_buffer_size(self) -> int:
         return self._tx_buffer_size
+
+    def byte_type(self) -> LrpcByteType:
+        return self._byte_type

src/lrpc/resources/cpp/EtlRwExtensions.hpp

@@ -262,8 +262,8 @@ namespace lrpc
 
         readSize = std::min(readSize, streamSize);
 
-        const auto ba = stream.free_data().take<const LRPC_BYTE_TYPE>(readSize);
-        (void)stream.skip<LRPC_BYTE_TYPE>(readSize);
+        const auto ba = stream.free_data().take<const lrpc::byte>(readSize);
+        (void)stream.skip<lrpc::byte>(readSize);
         return ba;
     };
 
@@ -416,7 +416,7 @@ namespace lrpc
         const size_t writeSize = std::min(stream.available_bytes(), ba_size);
         for (size_t i = 0; i < writeSize; ++i)
         {
-            stream.write_unchecked<LRPC_BYTE_TYPE>(value.at(i));
+            lrpc::write_unchecked<lrpc::byte>(stream, value.at(i));
         }
     };
 

src/lrpc/resources/cpp/LrpcTypes.hpp

@@ -1,23 +1,19 @@
 #pragma once
 
+$byte_include
 #include <array>
 #include <cstdint>
+#include <cstddef>
 #include <etl/optional.h>
 #include <etl/span.h>
 #include <etl/string_view.h>
 
 namespace lrpc
 {
-    // Define the fundamental byte type used throughout LRPC.
-    // Can be overridden by defining LRPC_BYTE_TYPE before including this header.
-    // Must be exactly 1 byte in size.
-#ifndef LRPC_BYTE_TYPE
-#define LRPC_BYTE_TYPE uint8_t
-#endif
+    using byte = $byte_type;
+    static_assert(sizeof(byte) == 1, "sizeof(byte) must be exactly 1");
 
-    static_assert(sizeof(LRPC_BYTE_TYPE) == 1, "sizeof(LRPC_BYTE_TYPE) must be exactly 1");
-
-    using bytearray = etl::span<const LRPC_BYTE_TYPE>;
+    using bytearray = etl::span<const byte>;
 
     using string_view = etl::string_view;