September 12, 2018

How to use protocol buffers with rest

In the world of microservices, REST over HTTP, is the most commonly deployed setup to enable them communicate with each other. Also JSON is probably the most popular messaging format used in this case. Although such a setup works perfectly fine for most of the scenarios, there might be good reasons to look beyond, for examples in cases where microservices are too chatty, data size being transferred might start taxing too much on the overall performance.

This article is about the alternative of JSON for such scenarios i.e. Protobuf. I am also assuming the basic familiarity with Protobuf (just knowing how to define message in protobuf is enough) and Node.js. You can find the protobuf reference here. If you want to directly jump into the code, you can find that proto-with-rest-sample.

Sample Project

Sample project is build on Node.js. It shows a simple client/server example, where server implements the GET and POST endpoints for Catalog resource. Following is the project structure:

|-- package-lock.json
|-- package.json
|-- src
    |-- client.js
    |-- common
    |   |-- protoResolver.js
    |-- protobuf
    |  |-- catalog.proto
    |-- server.js

Let’s see how these individual modules look like, one by one:

src/protobuf/catalog.proto

As you might have guessed, this proto file defines all the messages used in the app:

syntax = "proto3";

package catalog;

message Product {
    enum Status {
        AVAILABLE = 0;
        OUT_OF_STOCK = 1;
    }
    int32 id = 1;
    string name = 2;
    float price = 3;
    repeated string tags = 4;
    int32 quantity = 5;
    Status status = 6;
}

message Catalog {
    repeated Product products = 1;
}

enum ApiStatus {
    UNKNOWN = 0;
    SUCCESS = 1;
    FAILED = 2;
}

message GetCatalogResponse {
    ApiStatus status = 1;
    repeated Catalog catalogs = 2;
}

message AddCatalogResponse {
    ApiStatus status = 1;
}

Product and Catalog structures are straight forward, the reason of choosing such structures was to accommodate different data types in the demo app.

GetCatalogResponse and AddCatalogResponse are the wrapper messages for GET and POST endpoints respectively.

src/common/protoResolver.js

This is a utility module, which allows us to load the proto files, and also provides helpers functions to encode and decode protobuf messages easily.

const path = require("path");
const protobuf = require("protobufjs");

const PROTO_PATH = path.join(__dirname, "../protobuf/catalog.proto");
let _pkg;

function _init() {
    try {
        _pkg = protobuf.loadSync(PROTO_PATH);
    } catch (err) {
        console.log("failed to load proto files", err);
        process.exit(1);
    }
}

/**
 * searches and returns a type in the loaded proto files
 * @param {string} messageType name of the type to resolve example: "Product", "Catalog"
 */
function resolveType(messageType) {
    const resolved = _pkg.lookupTypeOrEnum(messageType);
    resolved.resolveAll();
    if (resolved.resolved) {
        return resolved;
    }
    return null;
}


/**
 * Serializes plain js objects to Buffer which conforms to protobuf message
 * @param {string} messageType name of the type to resolve example: "Product", "Catalog"
 * @param {object} message js plain object which need to be serialized into protobuf buffers
 */
function encodeMessage(messageType, message) {
    const messageProto = resolveType(messageType);
    const validationErr = messageProto.verify(message);

    if (validationErr) {
        throw new TypeError(validationErr);
    }

    return messageProto.encode(messageProto.create(message)).finish();
}

/**
 * Marshals Buffer to plain js objects
 * @param {string} messageType name of the type to resolve example: "Product", "Catalog"
 * @param {Buffer|Uint8Array} data buffer which needs to be deserialized
 */
function decodeMessage(messageType, data) {
    const messageProto = resolveType(messageType);
    const parsed = messageProto.decode(data);
    const message = messageProto.toObject(parsed);
    return message;
}

_init();

module.exports = {
    resolveType,
    encodeMessage,
    decodeMessage
}

We are using protobufjs npm package to read the protobuf files. Within the _init method, this module loads the protobuf file when required the first time, and later on searches within the loaded proto definitions.

src/server.js

const express = require("express");
const bodyParser = require("body-parser");

const protoResolver = require("./common/protoResolver");

const catalogs = [];
const PORT = 7878;

/********* Handlers and middlewares ***********/

//creating a middleware which can do the protobuf message parsing would be nice
function parseProto(req, res, next) {
    req.body = protoResolver.decodeMessage("Catalog", req.body);
    next();
}

function getCatalogs(req, res) {
    const apiResponse = protoResolver.encodeMessage("GetCatalogResponse", {
        status: 1,
        catalogs
    });
    
    res.setHeader("Content-Type", "application/octet-stream");
    res.write(apiResponse);
    res.end();
}

function addCatalog(req, res) {
    const product = req.body;
    catalogs.push(product);
    const apiResponse = protoResolver.encodeMessage("AddCatalogResponse", {
        status: 1
    });
    
    res.setHeader("Content-Type", "application/octet-stream");
    res.write(apiResponse);
    res.end();
}

/********************************/

//define the express router, which allows as to segregate logical operations against one endpoint nicely 
const router = express.Router();
router.route("/catalog")
    .get(getCatalogs)
    .post([parseProto, addCatalog]);

//create a express server app
const server = express();

//following is important, we do need raw request body stream, since we are gonna parse it ourselves
server.use(bodyParser.raw());
server.use("/", router);

server.listen(PORT, () => console.log("started listening on port", PORT));

Thing to note in the otherwise straightforward example are:

  • we are using server.use(bodyParser.raw()); to ensure we get the raw buffer from the POST requests.
  • parseProto is a handy middleware, which decodes the posted protobuf messages to plain js objects, which can be easily consumed in the request handlers.
  • while sending the response bach, we set the content type to application/octet-stream to help client libraries understand that response should not be attempted to be parsed, rather will directly be dealt with in the client application.
  • for the sake of simplicity, we are not using any databases, rather we simply store posted data in const catalogs = [];.

src/client.js

const axios = require("axios");
const protoResolver = require("./common/protoResolver");

const CATALOG_ENDPOINT = "http://localhost:7878/catalog";

function addCatalog() {
    const catalog = {
        products: [{
            id: 20,
            name: "Diamond Ring",
            price: 102.4,
            quantity: 2,
            tags: ["jewel"],
            status: 1
        }]
    };

    const catalogProto = protoResolver.resolveType("Catalog");
    const validationErr = catalogProto.verify(catalog);

    if (validationErr) {
        console.log("invalid message, error:", validationErr);
        return;
    }

    const data = catalogProto.encode(catalogProto.create(catalog)).finish();

    return axios.post(CATALOG_ENDPOINT, data, {
            headers: {
                "Content-Type": "application/octet-stream"
            },
            responseType: 'arraybuffer'
        })
        .then(res => {
            const apiResponse = protoResolver.decodeMessage("AddCatalogResponse", res.data);
            console.log(apiResponse);
        })
        .catch(err => {
            console.log(err);
        });
}

function getCatalog() {
    return axios.get(CATALOG_ENDPOINT, {
            responseType: 'arraybuffer'
        })
        .then(res => {
            const apiResponse = protoResolver.decodeMessage("GetCatalogResponse", res.data);
            console.log(apiResponse);
        })
        .catch(err => {
            console.log(err);
        });
}

function _init() {
    addCatalog();
    getCatalog();
}

_init();

we are using axios npm package to make HTTP calls. Some points to note in the above example:

  • we set responseType: 'arraybuffer', which sets Accept http header in the requests, this is important otherwise you will get an empty res.Data.
  • while making the post request, we set content type to application/octet-stream.

Conclusion

As you can see, we can replace JSON with Protobuf almost as easily. Data size reduction when using Protobuf, is higher even with gzip compression enabled while using JSON. If you do decide to use Protobuf, hopefully this sample will help you get started.

Thanks for reading. Copyright © notabot.in