How to Generate some JSON Friendly Golang Code from Protos with Bazel
My Journey generating the JSONMarshal and JSONUnmarshal methods for Protobuf types in Golang using a Protoc plugin
First some context and caveats. This is not a review of Bazel or an endorsement of the tool. I’m merely documenting something I came across and what I did to generate some additional code using a protoc plugin within the Bazel build system. I’ll save my opinions of any of these tools for another time. I can’t claim to have enough practical experience with them to make a judgement.
Some brief descriptions of the things I’m talking about in this article:
- Protobuf (aka Protocol Buffers): Is a schema system from Google
- Bazel: A language agnostic build system also based on a Google system
- Protoc: A CLI tool for generating code from Protobuf schemas (the link is an example of using the tool with another plugin to generate Go code)
Now I’m sure you’re wondering how we got here? Well I work for one of these high speed low drag startups out there. One day we hired someone who had been working at Google and now we must use all of their tools and systems. Ok, maybe that isn’t fair. We did use Golang and Protobufs before they joined the company. We also use Kubernetes and Google Cloud Platform, so we generally loved things made by Google already. Technically only Bazel is their fault…
I digress, on to the real reason I ended up working with Bazel. Bazel was utilized as a bit of an experiment in a repo containing a new service we would be using to manage our complicated configuration system that we wanted to load dynamically at runtime after updates. Then one day I was slinging some golang code that benefited from an existing proto schema for a config in that new service.
I had this crazy idea to reuse the code generated from the protobuf schema in my own service. I’d heard that was a good thing to do. So I built my Go struct representing the request to my service. The request between the UI and the my service used JSON and consisted of a several fields in addition to a section that was represented by one of the autogenerated types.
Example Proto Schema:
syntax = "proto3";
package config;
option go_package = "github.com/jbgrunewald/bazel-proto-example/proto/config";
enum ConfigType {
unknown = 0;
major = 1;
minor = 2;
}
message Config {
string id = 1;
ConfigType type = 2;
}Example Golang Struct Embedding the Above Type:
type MiscServiceRequestTryingToReuseProtoMessage struct {
something string
config *configs.Config
}At this point I’m patting myself on the back, doing the floss dance, firing my finger pistol at my rubber ducky… Thinking of how proud Robert Martin would be with my clean code. You know… Of course I needed to test it, so I spun everything up locally and made a request. That’s when I saw the problem. The exception when I tried using json.Unmarshal to pull data from the request. It was telling my that I was trying to unmarshal a string into an integer in Go.
So what was the problem? In the UI, we are using plain text strings for a field and in the proto schema that field is represented by an enum. When the Go code is generated that field in the Go struct is really an integer. So even though our code is setup with the decorators to unmarshal and marshal json, the types don’t match up.
Here’s what gets generated using protoc-gen-go:
type ConfigType int32type Config struct {
state protoimpl.MessageState
sizeCache protoimpl.SizeCache
unknownFields protoimpl.UnknownFields Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"`
Type ConfigType `protobuf:"varint,2,opt,name=type,proto3,enum=config.ConfigType" json:"type,omitempty"`
}
The “ConfigType”, which is an enum in the schema is an int32 in Golang.
One solution might be to write your own version of the JSONUnmarshal and JSONMarshal methods for each proto based struct you plan to use a composite type within another struct. This works because you can utilize the protobuf package’s marshal and unmarshal methods for the parts that don’t work with the plain text data. This is also tedious and no sane person should go through that much effort. Another way would be to write your own version of the struct represented from the proto schema. Deep down in our bones, we know that these are the wrong things to do here.
What you do instead, is go whine on the github repo for the above package about how protoc-gen-go doesn’t generate the JSONMarshal and JSONUnmarshal methods for you. Clearly this was a gross oversight by the Googlers who put this thing together. It is still my opinion that this should be done by default in that package. My whining did not result in any changes to the package…yet, but someone in the thread was helpful and pointed me to a protoc plugin that would do what I needed.
If all you cared about was generating this additional code through some other build process, then you can stop here. The repository for the plugin has some examples that will get you what you need. If you want to see how I did this in Bazel, then proceed.
I won’t spend much time explaining the technical details of setting up Bazel. Here’s a link to a repository with an example of the problem above with Bazel wired up. I’ll take few moments to talk about the high level details and some things I didn’t know.
The main thing you need from Bazel to create the generate Go code from the protobuf files is the go_proto_library function. One of the options for the function is to pass in references to different compilers, and Bazel has a few different options it already supports. In our example, we’ll want to provide our own by utilizing the go_proto_compiler function.
One thing that surprised me is how the go_repository function behaves. You can use it to load a github repo that represents a Go module. It will fill in the blanks for any missing details related to Bazel. I was under the impression that it would only expose a “go_default_libary” target, but it does in fact expose a “go_default_binary” as well.
Once everything is wired up for your build, you can run it to generate the extra files from the protoc-gen-go-json plugin. This will enable you to embed Go types generated from a protobuf schema inside other structs and marshal/unmarshal them using the built in json methods.
When I came across this problem, I was new to Bazel and protocol buffers. I wasn’t able to find any good examples of how to accomplish what I wanted with the plugin. The documentation for Bazel is good for the basic use cases, but it breaks down with something a little more advanced. Hopefully this will save someone else sometime setting this up for themselves.
Here’s the repo again, https://github.com/jbgrunewald/bazel-proto-example, in case anyone missed it before. If you enjoyed the article don’t forget to clap below.
