Data versioning with Google Protocol Buffers (GPB)

The protoc binary is the tool used to compile your text-based .proto file into a source file based on the language of your choice (Java in this example). You will need to follow these steps if you plan on being able to compile GPB-related code.

We recommend under the project you wish to define and use GBP you place .proto files under the /src/main/proto directory. You can then make use of the GPB option java_package syntax to control the subdirectory/package structure that will be created for the generated Java code from the .proto file.

The projects pom.xml file requires the following GPB related fields:

		    <dependencies>
		        <dependency>
		            <groupId>com.google.protobuf</groupId>
		            <artifactId>protobuf-java</artifactId>
		            <version>2.5.0</version>
		        </dependency>
		    </dependencies>
		
		    <build>
		        <plugins>
		            <plugin>
		                <groupId>org.apache.maven.plugins</groupId>
		                <artifactId>maven-compiler-plugin</artifactId>
		                <version>2.3.2</version>
		                <configuration>
		                    <source>1.7</source>
		                    <target>1.7</target>
		                </configuration>
		            </plugin>
		 
		            <plugin>
		                <groupId>com.google.protobuf.tools</groupId>
		                <artifactId>maven-protoc-plugin</artifactId>
		                <version>0.3.2</version>
		                <executions>
		                    <execution>
		                        <goals>
		                            <goal>compile</goal>
		                        </goals>
		                    </execution>
		                </executions>
		            </plugin>
		        </plugins>
		    </build>

After running mvn clean install on the pom.xml file GPB’s protoc will be used to:

Have the .proto file and generated .java file displayed properly in your IDE from your project’s root directory, i.e. where the project’s pom.xml file is, execute the following:

Because the resulting Java file is protoc generated code it is not recommended that it be checked in to your local source code management repo but instead regenerated when the application is built. The GPB Java Tutorial link on the official GPB website gives a more in depth walk through of the resulting Java class.

Generated GPB message classes are meant to serve as the versioned definition of data distributed by the Coordination Service. They are not meant to be used directly by the application to read/write to the various Coordination Services. It is recommended that a Model Object be defined for this role. This scheme provides two notable benefits:​​

The recommended procedure for versioning Coordination Service data is shown below and the sections that follow explain each of these steps with examples and best practices.

The GPB .proto message serves as the definition of a versioned message to be distributed by the Coordination Service. The application developer should write the .proto messages with the Model Object in mind when considering the data type of fields, whether they are optional or required. etc. The developer should consider all the GPB versioning rules and best practices mentioned in the previous section. The programmer implements a message per Model Object that will be distributed following the GPB rules and conventions previously discussed. ​​

Below is an example .proto message for the Person class. The field data types and REQUIRED/OPTIONAL tags match the Model Object. Since email was not a field to be set in the constructor it is marked optional while name and id are marked as required. Notice that the lastUpdated field of the Model Object is not included in the .proto message definition. This is considered a transient field, in the serialization sense, for the Model Object and it is not meant to be distributed in any version of the message. With this example the reader can see not all fields in the Person Model Object must be defined and distributed with the .proto message.

option java_outer_classname = "PersonProto;	// Wrapper class name.

		message Person {
				required string name = 1;
				required int32 id = 2;
				optional string email = 3;
		}

The application developer will generate the matching wrapper and builder classes for the .proto message to have a Java class that defines the message using protoc in the .proto Compilation Process section above.

Finally, a custom serializer needs to be defined to translate between instances of the Model Object being used in the Coordination Services and instances of the GPB message that will ultimately be transported by that service. For example, we may wish to write the Person Model Object on the PubSub Message Bus and have it received by another instance of the application which has subscribed to Person messages through its local Coordination Service. ​

In the custom serializer the developer will map the fields between these two objects on transmit (serialization) and receive (deserialization). With data types and naming conventions it should be clear what this 1:1 mapping is in the serializer. The Serializer must implement the Serializer<Model Object> interface as shown in the example below. It is recommended this serializer be kept in the <application>-bl project (if using the provided application project generation script of the SDK). PersonProto is the java_outer_classname we define in the GPB message above and will be the outer class from which inner GPB message classes, and their builders, are defined.

import <your package>.PersonProto;

public class PersonSerializer implements Serializer<Person> {

    @Override
		public byte[] serialize(Person subject) {
		    PersonProto.Person.Builder message = PersonProto.Person.newBuilder();
		    message.setName(subject.getName());
		    message.setId(subject.getId());  
		    return message.build().toByteArray();
		}

		@Override
		public Person deserialize(byte[] serialization) {
		    PersonProto.Person message = null;
		    try {
		         message = PersonProto.Person.parseFrom(serialization);
		    } catch (InvalidProtocolBufferException e) {
		        // Handle the error
		    }

		    Person newPerson = new Person();
		    if (message != null) {
		        newPerson.setName(message.getName());
		        newPerson.setId(message.getId());
		        return newPerson;
		     }
		    return null;
		}
}
		

In the serialize() method the builder pattern of the generated GPB message class is used to create a GPB version of the Person Model Object. After the proper fields are set the message is built and converted to a byte array for transport. In the deserialize() method on the receiver the byte array is converted back to the expected GPB message object. An instance of the Model object is then created and returned to be placed into the Coordination Service for which the serializer is registered.

The application must register this custom serializer with the Coordination Service it wishes to use this Model Object and GPB message combination. Below is an example of that registration process in an OSGI Component of an example application.

@Reference(cardinality = ReferenceCardinality.MANDATORY_UNARY, 
policy = ReferencePolicy.DYNAMIC)
protected volatile CoordinationService coordinationSvc;

@Activate
public void activate() {
	  // Register Message Serializers
		if (coordinationSvc != null) {
		    coordinationSvc.registerSerializer(new PersonSerializer(),Person.class);
		}
}