Advice on writing bazel rule for protoc-gen-dart

[jdebou]

Hello there,

I am looking for some advice on how the rules in this repo could be used to create a bazel rule to generate pb.dart files with protoc.

I am using protobuf and grpc in a flutter app to interact with a c++ engine. Apart from the flutter app, the whole project is managed with bazel.

So far I have been painfully using the method described by the protoc-gen-dart documentation to generate the pb.dart files, but am now looking to fully integrate this step with bazel.

While looking around this repository and the protobuf.dart repository, it appears that this has already been done, but I cannot find any documentation on it.

Any pointer on how to achieve this task would be greatly appreciated.

Cheers.

Joackim

[cbracken]

Hi Joackim,

There's an old dart_proto_library branch that adds a rule. It's probably fairly out-of-date at this point. This is the commit where I landed it at the time.

The real difficulty in implementing this wasn't writing the rules or anything; it was that back when I put it together in 2016 or so, we didn't have a mechanism to create standalone Dart binaries; everything ran from source. The proto rules depend on (a) protoc, which is a standalone binary and (b) the dart protoc plugin, which is itself written in Dart, and has a bunch of dependencies. As a result, this required pulling in a pile of third-party dependencies into the repo, which would have been a pain to maintain.

The other difficulty at the time was that no formal public protoc rules existed publicly at the time. Internally at Google we had a cross-platform proto_library rule and the language-specific dart_proto_library which used Bazel's 'aspect' feature.

I'm not sure what the status is of a standard proto_library target. In terms of the Dart protoc plugin, I suspect you could potentially maintain a repo where you pull down and build a binary protoc plugin, which you'd update occasionally, then use that in the rules.

[mark-dropbear]

@jdebou Just wanted to ask did you end up having any luck with this at all? I was about to go down the same path and that approach @cbracken outlined seemed like a lot to deal with unfortunately.

[jdebou]

@mark-dropbear Sadly nothing yet.

Before @cbracken 's answer I had tried to pull and build the protoc plugin in my repo, which posed the issues he described with third party dependencies. I don't know if the idea of maintaining another repo to build the plugin can really make things easier when you have multiple operating systems as build targets.

If I manage the time to give it a shot I'll let you know.

[mark-dropbear]

Thanks for the update @jdebou I've also reached out to the maintainer of the main proto ruleset at the moment to see what options look like from that perspective because it seems like it was actually on their road map and they have already added a bunch of other languages already as far as I can tell. Perhaps there is an easier path there via a community effort? I'm literally running my first Bazel commands as we speak so I am super unqualified to say if there is any potential there or not but it initially seems promising.

[bshi]

@cbracken would you be open to taking contributions making incremental progress toward supporting protobuf in this rules repository?

we didn't have a mechanism to create standalone Dart binaries

E.g. https://github.com/bshi/rules_dart/blob/master/.github/workflows/protoc_plugin.yml (sample output)

[loeffel-io]

I created a simple rule dart_proto_library which supports substitutions and grpc with prebuilt protoc-gen-dart binaries.
The build path must match the output path (e.g. build path: google/protobuf/timestamp.pb output path: google/protobuf/timestamp.*.dart)

I attach the prebuilt binaries (would be great when google/protobuf.dart provide those). Hope this helps someone

protoc-gen-dart.zip

Examples:

dart_proto_library(
    name = "role_v1_dart_proto",
    proto = ":role_v1_proto",
    substitutions = {
        "../../../../google": "package:global_proto/google",
        "../../../global": "package:global_proto/mindful/global",
    },
    visibility = ["//visibility:public"],
)

dart_proto_library(
      name = "timestamp_dart_proto",
      grpc = False,
      proto = "@com_github_protocolbuffers_protobuf//:timestamp_proto",
      visibility = ["//visibility:public"],
  )

Rule:

load("@bazel_skylib//lib:paths.bzl", "paths")

def _dart_proto_library_impl(ctx):
    descriptor_set_in = []
    for file in ctx.attr.proto[ProtoInfo].transitive_descriptor_sets.to_list():
        descriptor_set_in.append(file.path)

    name = ctx.attr.proto[ProtoInfo].direct_sources[0].basename.split(".")[0]
    outputs = [
        ctx.actions.declare_file("%s.pb.dart" % name),
        ctx.actions.declare_file("%s.pbenum.dart" % name),
        ctx.actions.declare_file("%s.pbjson.dart" % name),
    ]

    dart_options = ""
    if ctx.attr.grpc:
        dart_options = "grpc:"
        outputs.append(ctx.actions.declare_file("%s.pbgrpc.dart" % name))

    proto_file = paths.dirname(ctx.build_file_path) + "/" + ctx.attr.proto[ProtoInfo].direct_sources[0].path.split("/")[-1]  # google/protobuf/timestamp.proto or mindful/global/order/v1/order.proto
    ctx.actions.run(
        executable = ctx.executable.protoc,
        progress_message = "Generating Dart proto files",
        inputs = [ctx.executable.protoc_gen_dart] + [ctx.attr.proto[ProtoInfo].direct_sources[0]] + ctx.attr.proto[ProtoInfo].transitive_descriptor_sets.to_list(),
        tools = [ctx.executable.protoc, ctx.executable.protoc_gen_dart],
        outputs = outputs,
        mnemonic = "DartProtoGen",
        arguments = [
            "--plugin=protoc-gen-dart=%s" % ctx.file.protoc_gen_dart.path,
            "--dart_out=%s" % dart_options + ctx.configuration.genfiles_dir.path,
            "--descriptor_set_in=%s" % ":".join(descriptor_set_in),
            "%s" % proto_file,  # ctx.attr.proto[ProtoInfo].direct_sources[0].path,
        ],
    )

    if len(ctx.attr.substitutions) == 0:
        return [
            DefaultInfo(
                files = depset(outputs),
            ),
        ]

    substitution_outputs = [
        ctx.actions.declare_file("%s.pb.dart.substitution" % name),
        ctx.actions.declare_file("%s.pbenum.dart.substitution" % name),
        ctx.actions.declare_file("%s.pbjson.dart.substitution" % name),
    ]

    if ctx.attr.grpc:
        substitution_outputs.append(ctx.actions.declare_file("%s.pbgrpc.dart.substitution" % name))

    for i in range(len(outputs)):
        ctx.actions.expand_template(
            template = outputs[i],
            output = substitution_outputs[i],
            substitutions = ctx.attr.substitutions,
        )

    return [
        DefaultInfo(
            files = depset(outputs + substitution_outputs),
        ),
    ]

dart_proto_library = rule(
    implementation = _dart_proto_library_impl,
    attrs = {
        "proto": attr.label(
            allow_single_file = True,
            mandatory = True,
        ),
        "grpc": attr.bool(
            default = True,
            doc = "Generate gRPC headers",
        ),
        "protoc": attr.label(
            allow_single_file = True,
            executable = True,
            default = Label("@com_google_protobuf//:protoc"),
            cfg = "host",
        ),
        "protoc_gen_dart": attr.label(
            allow_single_file = True,
            executable = True,
            default = Label("@com_github_mindful_hq_rules//:protoc_gen_dart"),
            cfg = "host",
        ),
        "substitutions": attr.string_dict(
            default = {},
            doc = "Substitutions to apply to the proto. The files will be generated with the .substitution suffix",
        ),
    },
    doc = "Builds dart proto files",
)

[loeffel-io]

Here is an updated version which supports multiple .proto files in your proto_library

load("@bazel_skylib//lib:paths.bzl", "paths")

def _dart_proto_library_impl(ctx):
    descriptor_set_in = []
    for file in ctx.attr.proto[ProtoInfo].transitive_descriptor_sets.to_list():
        descriptor_set_in.append(file.path)

    outputs = []
    proto_files = []
    grpc_files = 0
    for source in ctx.attr.proto[ProtoInfo].direct_sources:
        name = source.basename.split(".")[0]

        outputs.append(ctx.actions.declare_file("%s.pb.dart" % name))
        outputs.append(ctx.actions.declare_file("%s.pbenum.dart" % name))
        outputs.append(ctx.actions.declare_file("%s.pbjson.dart" % name))

        for i in range(len(ctx.attr.grpc)):
            if ctx.attr.grpc[i] == source.basename:
                grpc_files += 1
                outputs.append(ctx.actions.declare_file("%s.pbgrpc.dart" % name))

        proto_files.append(paths.dirname(ctx.build_file_path) + "/" + source.path.split("/")[-1])

    dart_options = ""
    if grpc_files != 0:
        dart_options = "grpc:"

    ctx.actions.run(
        executable = ctx.executable.protoc,
        progress_message = "Generating Dart proto files",
        inputs = [ctx.executable.protoc_gen_dart] + ctx.attr.proto[ProtoInfo].direct_sources + ctx.attr.proto[ProtoInfo].transitive_descriptor_sets.to_list(),
        tools = [ctx.executable.protoc, ctx.executable.protoc_gen_dart],
        outputs = outputs,
        mnemonic = "DartProtoGen",
        arguments = [
            "--plugin=protoc-gen-dart=%s" % ctx.file.protoc_gen_dart.path,
            "--dart_out=%s" % dart_options + ctx.configuration.genfiles_dir.path,
            "--descriptor_set_in=%s" % ":".join(descriptor_set_in),
        ] + proto_files,  # ctx.attr.proto[ProtoInfo].direct_sources[i].path
    )

    if len(ctx.attr.substitutions) == 0:
        return [
            DefaultInfo(
                files = depset(outputs),
            ),
        ]

    substitution_outputs = []
    for i in range(len(outputs)):
        substitution_outputs.append(ctx.actions.declare_file("%s.substitution" % outputs[i].basename))
        ctx.actions.expand_template(
            template = outputs[i],
            output = substitution_outputs[i],
            substitutions = ctx.attr.substitutions,
        )

    return [
        DefaultInfo(
            files = depset(outputs + substitution_outputs),
        ),
    ]

dart_proto_library = rule(
    implementation = _dart_proto_library_impl,
    attrs = {
        "proto": attr.label(
            allow_single_file = True,
            mandatory = True,
        ),
        "grpc": attr.string_list(
            default = [],
            doc = "Proto files that should generate gRPC code",
        ),
        "protoc": attr.label(
            allow_single_file = True,
            executable = True,
            default = Label("@com_google_protobuf//:protoc"),
            cfg = "host",
        ),
        "protoc_gen_dart": attr.label(
            allow_single_file = True,
            executable = True,
            default = Label("@com_github_mindful_hq_rules//:protoc_gen_dart"),
            cfg = "host",
        ),
        "substitutions": attr.string_dict(
            default = {},
            doc = "Substitutions to apply to the proto. The files will be generated with the .substitution suffix",
        ),
    },
    doc = "Builds dart proto files",
)

example:

dart_proto_library(
    name = "global_v1_dart_proto",
    grpc = [
        "version.proto",
    ],
    proto = ":global_v1_proto",
    visibility = ["//visibility:public"],
)

output

bazel build //mindful/global/v1:global_v1_dart_proto
INFO: Analyzed target //mindful/global/v1:global_v1_dart_proto (1 packages loaded, 4 targets configured).
INFO: Found 1 target...
Target //mindful/global/v1:global_v1_dart_proto up-to-date:
  bazel-bin/mindful/global/v1/order.pb.dart
  bazel-bin/mindful/global/v1/order.pbenum.dart
  bazel-bin/mindful/global/v1/order.pbjson.dart
  bazel-bin/mindful/global/v1/version.pb.dart
  bazel-bin/mindful/global/v1/version.pbenum.dart
  bazel-bin/mindful/global/v1/version.pbjson.dart
  bazel-bin/mindful/global/v1/version.pbgrpc.dart
INFO: Elapsed time: 0.122s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action