Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 45 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@

**Hazelcast Open Binary Client Protocol** definitions and code generator for multiple programming languages.

## Hazelcast Open Binary Client Protocol Definitions
## Hazelcast Open Binary Client Protocol definitions

The protocol is defined in `protocol-definitions/*.yaml` yaml files where each yaml file represents a service like Map, List, Set etc.
The protocol is defined in `protocol-definitions/*.yaml` YAML files where each YAML file represents a service like Map, List, Set etc.
Custom data types that are used in the protocol definitions are defined in `protocol-definitions/custom/Custom.yaml`.

> [!WARNING]
Please wrap since parameters with double quotation mark since yaml's auto typing uses float instead of string conversion. In this case, converting versions like 2.10, 2.0.1 are not done correctly. e.g. 2.10 is converted to 2.1.
Wrap since parameters with double quotation mark because YAML's auto typing uses float instead of string conversion. In this case, converting versions like 2.10, 2.0.1 are not done correctly. e.g. 2.10 is converted to 2.1.

## Service definition
## Service definitions

A service is defined by a separate YAML file, containing all its method definitions.

Expand All @@ -26,7 +26,7 @@ methods:
...
```

A method(aka Remote method call) is defined by a request-response pair and an optional events section.
A method (aka Remote method call) is defined by a request-response pair and an optional events section.

A basic method structure example:

Expand Down Expand Up @@ -80,21 +80,21 @@ A basic method structure example:

```

Please refer to [schema](schema/protocol-schema.json) for details of a service definition.
Refer to the [schema](schema/protocol-schema.json) for detailed service definitions.

## Code Generator
## Code generator

The new protocol generator generates the related language codecs into the configured folder. It does not depend on Hazelcast repo.
The new protocol generator generates the related language codecs into the configured folder. It does not depend on the Hazelcast repo.

### Setup

You need to have python3 configured on your `PATH`. After cloning the repository, install the python library dependencies:
You need to have Python 3 configured on your `PATH`. After cloning the repository, install the Python library dependencies:

```bash
pip3 install -r requirements.txt
```

### Code Generation
### Code generation

You can generate codecs for a specific language by calling:

Expand All @@ -112,7 +112,7 @@ where
* `cs` : C#
* `py` : Python
* `ts` : TypeScript
* `md` : Markdown (Documentation)
* `md` : Markdown (documentation)

`java` is the default value if no language is specified.

Expand All @@ -137,34 +137,57 @@ Default value is inferred from the selected `LANGUAGE`.

* `--no-id-check` flag restrains the generator from checking sequentially of service and method ids of protocol definitions.

If you want to generate the Java codecs into your development repo, and let's assume your local Hazelcast git repo is at
`~/git/hazelcast/` then you can run the following command:
If you want to generate the Java codecs into your development repo, assuming your local Hazelcast Git repo is at `~/git/hazelcast/`, run the following command:

```bash
./generator.py -r ~/git/hazelcast/
```

This command generates the codecs at the `ROOT_DIRECTORY/OUTPUT_DIRECTORY` which is `~/git/hazelcast/hazelcast/src/main/java/com/hazelcast/client/impl/protocol/codec/`.
See that the `OUTPUT_DIRECTORY` is inferred from the language, namely `hazelcast/src/main/java/com/hazelcast/client/impl/protocol/codec/` for `java`.
This command generates the codecs at the `ROOT_DIRECTORY/OUTPUT_DIRECTORY`, which is `~/git/hazelcast/hazelcast/src/main/java/com/hazelcast/client/impl/protocol/codec/`.
The `OUTPUT_DIRECTORY` is inferred from the language, namely `hazelcast/src/main/java/com/hazelcast/client/impl/protocol/codec/` for `java`.

If you want to specify an output directory relative to the root directory, you can run the following command:
If you want to specify an output directory relative to the root directory:

```bash
./generator.py -r ~/git/hazelcast/ -o custom/out
```

This command will generate the codecs at the `~/git/hazelcast/custom/out`.

### Schema Validation
### Documentation

To generate the documentation, run the following command:

```bash
./generator.py -l md
```

This will generate a markdown file at `~/output/md/documentation`.

If you want to convert the markdown file to PDF, you can use the `md-to-pdf` tool.

To install the tool:

```bash
npm i -g md-to-pdf
```

To generate a PDF:

```bash
md-to-pdf documentation.md --pdf-options '{"margin": "10mm", "printBackground": true}'
```

### Schema validation

The protocol definitions should validate against the [schema](schema/protocol-schema.json). You can configure your IDE to
use this schema to validate and provide auto code completion.

The generator also uses this schema during the code generation for validation purposes. It stops and reports any schema violation to the console.

### Custom Types
### Custom types

If you are going to use a custom type,i.e., a complex type that is not defined in the [currently supported types](binary/__init__.py),
If you are going to use a custom type, i.e., a complex type that is not defined in the [currently supported types](binary/__init__.py),
as the type of your parameters in the protocol definitions, you need to define how to encode and decode this in the protocol level.

A custom type definition has the following structure:
Expand Down Expand Up @@ -221,17 +244,17 @@ convert it to your enum type and construct the object with it.
Custom type definitions are also validated against a [schema](/schema/custom-codec-schema.json). See the [Schema Validation](#schema-validation)
section for details of the validation.

### Expanding the Client Protocol
### Expanding the client protocol

Client protocol can be expanded by adding new
The client protocol can be expanded by adding new:
* services
* methods
* parameters to existing method requests, responses or events
* events to existing methods
* custom types
* parameters to existing custom types

While expanding the protocol, one needs to follow these simple guidelines:
While expanding the protocol, follow these guidelines:
* `since` field of the protocol definitions of the newly added parameters, methods, events and custom types should
be equal to the current protocol version.
* New services should have the id of the 1 + the highest id of the existing services.
Expand Down
24 changes: 0 additions & 24 deletions md/__init__.py
Comment thread
Rob-Hazelcast marked this conversation as resolved.
Original file line number Diff line number Diff line change
@@ -1,27 +1,3 @@
"""
This template produces Github-flavored markdown documents.

If you want to convert this markdown document to PDF,
the recommended way of doing it is using the `md-to-pdf` tool.

The documentation of the tool can be seen on
https://www.npmjs.com/package/md-to-pdf.

To install the tool, run the following command.

$ npm i -g md-to-pdf

Since the produced markdown document can be quite large
in width due to some tables, it is recommended to set
the output PDF margin to a lower value than the default setting.

It is recommended that the right and left margins should
be less than or equal to "10mm". One can use the following command
to generate the PDF output.

$ md-to-pdf documentation.md --pdf-options '{"margin": "10mm", "printBackground": true}'
"""

internal_services = {
"MC",
}
16 changes: 4 additions & 12 deletions md/documentation-template.j2
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,6 @@ Map of {{ convert_type(key_type(param_name, type)) }} to {{ convert_type(value_t

# Hazelcast Open Binary Client Protocol

**Revision History**

{# Changelog for each version added. #}
| Date | Document Version | Change Description |
| ---- | ---------------- | ------------------ |
| 01/15/2020 | 2.0 | Release of the Hazelcast Open Binary Client Protocol version 2.0 |
| 11/04/2020 | 2.1 | Release of the Hazelcast Open Binary Client Protocol version 2.1 |

**Table of Contents**
{# List of manually written sections #}
- [1. Introduction](#1-introduction)
Expand Down Expand Up @@ -560,7 +552,7 @@ var-sized-to-fix-sized-map = begin-frame *var-sized-entry fix-sized-list end

single-frame-param = frame-header *OCTET ; For String, Data, ByteArray types. Strings must be encoded as UTF-8
custom-type-param = custom-type-begin-frame *1custom-type-initial-frame *var-sized-param end-frame
custom-type-begin-frame = begin-frame / frame-header *fix-sized-param ; Fix sized params might be pigybacked to begin frame
custom-type-begin-frame = begin-frame / frame-header *fix-sized-param ; Fix sized params might be piggybacked to begin frame
custom-type-initial-frame = frame-header *fix-sized-param

null-frame = frame-header ; IS_NULL_FLAG is set to one
Expand Down Expand Up @@ -1054,7 +1046,7 @@ implementation. (e.g. throw an unknown error code exception)
| ENTRY_PROCESSOR | 16 | An exception to indicate a problem occurred attempting to execute an EntryProcessor against an entry |
| EXECUTION | 17 | Thrown when attempting to retrieve the result of a task that aborted by throwing an exception. |
| HAZELCAST | 18 | General internal error of Hazelcast. |
| HAZELCAST_INSTANCE_NOT_ACTIVE | 19 | The Hazelcast member instance is not active, the server is possibly initialising. |
| HAZELCAST_INSTANCE_NOT_ACTIVE | 19 | The Hazelcast member instance is not active, the server is possibly initializing. |
| HAZELCAST_OVERLOAD | 20 | Thrown when the system won't handle more load due to an overload. This exception is thrown when backpressure is enabled. |
| HAZELCAST_SERIALIZATION | 21 | Error during serialization/de-serialization of data. |
| IO | 22 | An IO error occurred. |
Expand Down Expand Up @@ -1096,7 +1088,7 @@ implementation. (e.g. throw an unknown error code exception)
| TRANSACTION_TIMED_OUT | 58 | Thrown when a transaction has timed out. |
| URI_SYNTAX | 59 | Thrown to indicate that a string could not be parsed as a URI reference |
| UTF_DATA_FORMAT | 60 | Signals that a malformed string in modified UTF-8 format has been read in a data input stream or by any class that implements the data input interface |
| UNSUPPORTED_OPERATION | 61 | The message type id for the operation request is not a recognised id. |
| UNSUPPORTED_OPERATION | 61 | The message type id for the operation request is not a recognized id. |
| WRONG_TARGET | 62 | An operation is executed on the wrong machine. |
| XA | 63 | An error occurred during an XA operation. |
| ACCESS_CONTROL | 64 | Indicates that a requested access to a system resource is denied. |
Expand Down Expand Up @@ -1283,4 +1275,4 @@ Header only response message, no message body exist.

Copyright (c) 2008-{{ copyright_year }}, Hazelcast, Inc. All Rights Reserved.

Visit [www.hazelcast.com](https://hazelcast.com/) for more info.
Visit [www.hazelcast.com](https://hazelcast.com/) for more information.