Friday, November 18, 2011

Cross-platform communication using Google's Protocol Buffers

NOTE: The C++ portion of this blog entry is incorrect. I've made some pretty stupid mistakes. These are discussed in a newer blog entry, which can be found here.

In this entry I will be discussing using Google's message format, Protocol Buffers, to communicate between a Java client and a C++ server using TCP sockets.

I came across protobuf in my search for a good means of communicating over the network between apps that may not be written in the same language. I wanted an Android phone, an iPhone, or even just a laptop to be able to communicate with the same server program. I feel like this is an increasingly desirable feature for mobile apps, so I'm going to regurgitate the information that I compiled on my search.

The way protocol buffers works is this: You define your message (data types and fields) in protobuf's own language. Then, once you have a properly formed message with all of the fields you need, you find a compiler for your language. Google provides official compilers for C++, Java, and Python. There is also a huge list of 3rd party compilers for many other languages (pretty much any one you can think of, including Objective C for you iPhone guys!).

Then you run the protobuf message through the compiler, and the compiler generates a library (source code/headers) for you to use that includes the message object, parsers, getters and setters, serializers and deserializers, and many other nifty little things that you would normally have to painstakingly code out for your own message. You "include" this in your program, and you're good to go!

Another advantage of protobuf is that Google encodes the messages in to their own binary format rather than ASCII, so the messages are a lot smaller than a message sent via JSON or XML.

I found an unsatisfactory amount of stuff on the internet about using protobuf across platforms. In my case, I wanted to know how to send a message from Java -> C++, and from C++ -> Java. So, I needed to know how to send and receive a message on both sides via TCP.

Here is a code example of a Java client sending a protobuf message to a C++ server, and the C++ server replying with another protobuf message.

Note: I am using Java NIO because it's just way easier to deal with than Streams, and it's more efficient.

Java Send

 //set up socket
 SocketChannel serverSocket;
 serverSocket=SocketChannel.open();
 serverSocket.socket().setReuseAddress(true);
 serverSocket.connect(new InetSocketAddress(servIP,servPort));
 serverSocket.configureBlocking(true);
 
 //create BAOS for protobuf
 ByteArrayOutputStream baos=new ByteArrayOutputStream();
 //mClientDetails is a protobuf message object, dump it to the BAOS
 mClientDetails.writeDelimitedTo(baos);

 //copy the message to a bytebuffer
 ByteBuffer socketBuffer=ByteBuffer.wrap(baos.toByteArray());

 //keep sending until the buffer is empty
 while(socketBuffer.hasRemaining())
  serverSocket.write(socketBuffer);

C++ Receive

    //set up server socket
    sockaddr_in *sa=(sockaddr_in *)malloc(sizeof(struct sockaddr_in));
    memset(sa, 0, sizeof(struct sockaddr_in));
    sa->sin_family = AF_INET;
    sa->sin_addr.s_addr = htonl(INADDR_ANY);
    sa->sin_port = htons(port);
    serverSock=socket(PF_INET,SOCK_STREAM,IPPROTO_TCP);
    int reuse=1;
    setsockopt(serverSock, IPPROTO_TCP, SO_REUSEADDR, &reuse, sizeof(reuse));

    //bind socket to port, listen
    bind(serverSock, (sockaddr *)sa, sizeof(*sa));
    listen(serverSock, 1);

    //create protobuf object
    protobuf::ClientDetails client;       

    //accept a client connection
    sockaddr_in *clientAddr;
    long unsigned int caSize=sizeof(clientAddr);
    int clientSock=accept(serverSock, (struct sockaddr *)&clientAddr, (socklen_t *)&caSize);

    //receive message from the client, where BUFFER_SIZE is large enough to contain your message
    int received=recv(clientSock, buffer, BUFFER_SIZE, 0);

    //read varint delimited protobuf object in to buffer
    //there's no method to do this in the C++ library so here's the workaround
    google::protobuf::io::ArrayInputStream arrayIn(buffer, received);
    google::protobuf::io::CodedInputStream codedIn(&arrayIn);
    google::protobuf::uint32 size;
    codedIn.ReadVarint32(&size);
    google::protobuf::io::CodedInputStream::Limit msgLimit = codedIn.PushLimit(size);
    client.ParseFromCodedStream(&codedIn);
    codedIn.PopLimit(msgLimit);

C++ Send

    //already set up a message object called serverAck
    //make a buffer that can hold message + room for a 32bit delimiter
    int ackSize=serverAck.ByteSize()+4;
    char* ackBuf=new char[ackSize];
            
    //write varint delimiter to buffer
    google::protobuf::io::ArrayOutputStream arrayOut(ackBuf, ackSize);
    google::protobuf::io::CodedOutputStream codedOut(&arrayOut);
    codedOut.WriteVarint32(serverAck.ByteSize());

    //write protobuf ack to buffer
    serverAck.SerializeToCodedStream(&codedOut);

    //send buffer to client
    send(clientSock, ackBuf, ackSize, 0);
    delete(ackBuf);

Java Receive
//receive message from the client, where BUFFER_SIZE is large enough to contain your message
socketBuffer=ByteBuffer.allocate(BUFFER_SIZE);
int bytesRead=serverSocket.read(socketBuffer);

//copy message byte array from socket buffer
socketBuffer.flip();
byte[] ackBuf = new byte[socketBuffer.remaining()];
socketBuffer.get(ackBuf);

//create ByteArrayInputStream from byte[] (this is what protobuf wants)
ByteArrayInputStream ackStream=new ByteArrayInputStream(ackBuf);

//parse message
ServerAck serverAck=ServerAck.parseDelimitedFrom(ackStream);

//done!
socketBuffer.clear();


For further reading on implementing protobuf, I recommend taking a look at Google's tutorials.

12 comments:

  1. Hi Adam, Could you please attach an example project including all.proto files and the generated ones? (The Java side)

    thanks.

    ReplyDelete
  2. In the C++ Receive example. How do you know:

    int received=recv(clientSock, buffer, BUFFER_SIZE, 0);

    actually reads the whole packet? AFAIK one send does not equal one recv. A TCP socket is conceptually a stream and may read any number of bytes regardless of how many bytes are sent. This is my current problem, i.e. how to know how many bytes a varint is.

    ReplyDelete
    Replies
    1. You are correct. The proper implementation of this would first recv the varint and then recv the rest of the message according to the size specified by the varint. Therefore all recv's would be in a loop.

      The only way I can think of to read a varint over TCP is by reading it 1 byte at a time. Here is how you conceptually interpret varints: https://developers.google.com/protocol-buffers/docs/encoding#varints

      I don't have time right now to write a function to do this, but it can't be too hard to implement.

      Delete
    2. Thought I'd mention I've solved the problem. It turned out to be really easy. It's a matter of implementing google::protobuf::io::CopyingInputStream and using google::protobuf::io::CopyingInputStreamAdaptor to create a ZeroCopyInputStream. Here's a quick example:
      https://github.com/Fulkerson/FRTN01/blob/master/common/asio_copy_stream.h

      It's for using boost::asio, but doing any kind of stream is pretty much the same. The only hard part is it's almost never mentioned even though it probably is the best way of doing it. There's amazingly little documentation for doing such a common task as serializing to and from a socket.

      Delete
    3. Thanks for following up! That's quite helpful. I wonder if there is a way to use that without boost. I'll look in to it if I get some free time. For now, thanks for finding this solution!

      Delete
    4. Johan, I wanted to let you know that I've come up with an alternative solution that avoids using boost. I'm not sure if you care! Just thought I would follow up.

      Here: http://blog.ajhodges.com/2013/01/cross-platform-communication-using.html

      Delete
  3. how to compile protobuf for an android phone..??

    can you please send me a sample android.mk file..?

    ReplyDelete
    Replies
    1. Hi Ruchir,

      I have not done any NDK development so I do not have an android.mk file. The process I used to compile protobuf for the Android SDK is detailed on this page (see the section "Compiling Your Protocol Buffers"): https://developers.google.com/protocol-buffers/docs/javatutorial

      If you are doing NDK development you may want to try the same section in the C++ tutorial: https://developers.google.com/protocol-buffers/docs/cpptutorial

      I hope that helps!
      Adam

      Delete
  4. I am trying to understand how to implement it for java (server) and python (client). I am new to python and hence find tutorials online difficult to understand. can you show me basic example?

    ReplyDelete
    Replies
    1. I have never used the protobuf Python library. However it will probably look pretty similar to the code above.

      I found this blog post of someone who found the method for creating a delimited protobuf message: http://zombietetris.de/blog/building-your-own-writedelimitedto-for-python-protobuf/

      Once you have done that, you should be able to send/receive the message to/from Java by using sockets. Remember that TCP makes no guarantee of the amount of data you will receive in a recv(), so read the delimiter byte by byte as I did in this blog post: http://blog.ajhodges.com/2013/01/cross-platform-communication-using.html

      Here is the java code that accomplishes the same thing, you will have to rewrite this in python if you are receiving any packets there: https://gist.github.com/ajhodges/9940869

      Delete
  5. is there a way to communicate with python ?

    ReplyDelete
    Replies
    1. Hi Haifeng,

      Please look at my response to this related question:
      http://blog.ajhodges.com/2011/10/cross-platform-communication-using.html?showComment=1396461447872#c795105530855157701

      Thanks,
      Adam

      Delete