Send Google Protocol Buffer With ZeroMQ Across NetWork to Raspberry Pi.md

Send Google Protocol Buffer With ZeroMQ Across NetWork to Raspberry Pi

1. Overview

When we want to send message across network with fast respond, We know ZeroMQ is one of the best choices. While sending across network and taking care of your message can be a big headache. Our own message can be hard to maintain with little flexibility. That’s where you need to consider a protocol to handler your growing messages. So now we use Google Protocol Buffer which is free and efficient to handle our protocol.

Ubuntu PC ----ZeroMQ Message With Protocol Buffer----> Raspberry Pi

You can check out the entire project from my GitHub.

2. Installation

We need

Qt Creator at Ubuntu PC
Python at Raspberry Pi
Both Ubuntu PC and Raspberry Pi are in same network
Protocol Buffer Library at both Ubuntu PC and Raspberry Pi

You can refer to this link for installation of Protocol Buffer
You can refer to this link to install Python version of ZeroMQ
You can refer to this link to install C++ version of ZeroMQ

Note: Installation instructions can be applied to both Ubuntu and Raspberry Pi. They are basically same since they are both *nix system.

3. Implementation

Since the project is hosted on GitHub, I will just explain necessary implementation.

3.1 Both System

In order to exchange message between Ubuntu and Raspberry, Both system must have protocol message file (.proto) in their own system.

We will decipher the following message protocol as followed. We will be sending as message as addressBook. In the adressBook, there can be many Person as signified by repeated keyword. Similarly, Person must have name as indicated by required and email is optional for a Person since the keyword is optional. The " = 1", " = 2" markers on each element identify the unique tag that field uses in the binary encoding.

package message;
message Person {
    required string name = 1;
    optional string email = 2;
}
message AddressBook{
    repeated Person person = 1;
}

Note: .proto file must be compiled before it can be used in the program. You can looked how it can be compiled at .pro file and python in the project.

You can manually compile the .proto file also. Just use

For Python

protoc message.proto --python_out='.'

For C++

protoc message.proto --cpp_out='.'

3.2 Raspberry Pi

We need to make sure Python, ZeroMQ Python and Protocol Buffer are already installed. Writing program in Python can be very short so it’s good to test in server first.

Here we will compile message.proto file and generate message_pb2.py file. We need to import this ‘message_pb2.py’ file in the program

<...omitted...>
    if (os.path.exists("message_pb2.py") == False):
        subprocess.call("protoc message.proto --python_out='.'", shell=True)
<...omitted...>

Then we will setup ZeroMQ and bind to the Raspberry Pi localhost IP address.

<...omitted...>
    context = zmq.Context()
    socket = context.socket(zmq.REP)
    socket.bind("tcp://*:5556")
<...omitted...>

Now we will parse String from the message that sent from client.

<...omitted...>
    message = socket.recv()
    addressBook = message_pb2.AddressBook()
    addressBook.ParseFromString(message)
<...omitted...>

The complete program of this file can be read from here.

3.3 Ubuntu

In Ubuntu system, we will be using Qt, ZeroMQ and Protocol Buffer so make sure they are already installed. Since we will be using C++, there are more setup required to make it works.

In order to compile *.proto file, we use *.pri file in Qt to compile and generate message.pb.h. Honestly I don’t write the ‘.pri’ file so I can’t explain it. Basically, it generates ‘.ph.h’ and ‘.ph.cc’ files at its own directory. You can read more detail from here.

At Client.cpp, we will setup ZeroMQ and register the ZeroMQ slot.

<...omitted...>
    socket = mContext->createSocket(ZMQSocket::TYP_REQ, this);
    QObject::connect(socket, SIGNAL(messageReceived(QList<QByteArray>)), SLOT(onReceivedFromZqmlToConsumeData(QList<QByteArray>)));
    mContext->start();
<...omitted...>

Still in Client.cpp, we will setup network and connect to Raspberry Pi Server.

<...omitted...>
    QString raspberryPiAddress = "192.168.1.104";
    _initilizeConnection();

    const int portNumber = 5556;
    QString stringPortNumber = QString::number(portNumber);

    QString ipAddressString = "tcp://" + raspberryPiAddress.toLocal8Bit() + ":" +stringPortNumber.toLocal8Bit();
    QByteArray qByteArray = ipAddressString.toLatin1();
    const char *charIpAddress = qByteArray.data();

    socket->connectTo(charIpAddress);
<...omitted...>

While still Client.cpp, we add two Person to Address Book, serialize the data and send to Raspberry Pi server.

<...omitted...>
    message::AddressBook addressBook;

    message::Person *john = addressBook.add_person();
    john->set_name("John ");
    john->set_email("john@abc.com");
    
    message::Person *drake = addressBook.add_person();
    drake->set_name("Drake");
    drake->set_email("drake@abc.com");

    qDebug() << "Number of Person in Address Book: " << addressBook.person_size();
    std::string msg_str;
    addressBook.SerializeToString(&msg_str);
    _sendStringToServer(msg_str.c_str());
<...omitted...>

The complete program of this file can be read from here.

4. Testing Result

We should first run the server program at Raspberry Pi. After that, we will run Qt program. Both of program will generate necessary compiled files from *.proto file.

The output result at Qt client will be like this.

Number of Person in Address Book:  2
Hello  John 
Your email will be  john@abc.com
Hello  Drake
Your email will be  drake@abc.com
Received Msg from Server is  "Hello Client"

And the server side will be displayed like the following.

Number of Person in Address Book:  2
Hello  John 
Your email will be  john@abc.com
Hello  Drake
Your email will be  drake@abc.com
Received Msg from Server is  "Hello Client"

5. Reference

How to Install Google Protocol Buffers on Raspberry Pi and Test with Python.md

How to Install Google Protocol Buffers on Raspberry Pi and Test with Python

1. Installation

Make sure you have Python 2.6 or newer but not Python 3.x yet. If in doubt, then run:

$ python -V

1.1 Install C++ Protocol Compiler

We need C++ Protocol Compiler for python. This is required to compile .proto file

$ cd ~/
$ sudo apt-get install autoconf automake libtool curl python-dev
$ wget https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz
$ tar -zxvf protobuf-2.6.1.tar.gz
$ cd protobuf-2.6.1/
$ ./configure
$ make
$ make check
$ sudo make install
$ sudo ldconfig

Note: You can also refer to the full installation guide from here OR read README.md from pytobuf-2.6.1 folder

1.2 Build Python runtime library

This is required to run Python program. We will use c++ implementation for Python runtime.

$ cd ~/
$ cd protobuf-2.6.1/
$ cd python
$ export LD_LIBRARY_PATH=../src/.libs
$ python setup.py build --cpp_implementation
$ python setup.py test --cpp_implementation
$ sudo python setup.py install --cpp_implementation
$ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
$ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION=2

Note: I cannot run $ python setup.py test --cpp_implementation for some reason. but you should try as it is stated in the full installation guide.

Note: You can refer to full installation guide from here OR read README.txt file in the python folder

2. Testing Protocol Message

2.1 Write .proto file

We need to write a protocol message. This can be use for both server side and client side if you are going to transfer across the network.

$ cd /~
$ mkdir ProtobuffTest
$ cd ProtobuffTest
$ gedit addressbook.proto

Copy the following text and save it as addressbook.proto

package message;

message Person {
    required string name = 1;
    optional string email = 2;
}
message AddressBook{
    repeated Person person = 1;
}

2.2 Compile Protocol Message

Before we use any of the protocol message, we first need to compile once. In the current folder, type

$ protoc addressbook.proto --python_out="."

After the compilation, addressbook_pb2.py will be generated and we will use this file for Python program.

2.3 Compile Python Program

We first create a python program first.

$ gedit AddressBookTest.py

Copy the following text and save it as AddressBookTest.py

#! /usr/bin/python

import addressbook_pb2

address_book = addressbook_pb2.AddressBook()

john = address_book.person.add()
john.name = "John"
john.email = "john@abc.com"

drake = address_book.person.add()
drake.name = "Drake"
drake.email = "drake@abc.com"

for person in address_book.person:
    print " Hello ", person.name
    print " Your email will be ", person.email

Just like any other Python program, you can just run the program by typing

$ python AddressBookTest.py

The output will be like this.

 Hello  John
 Your email will be  john@abc.com
 Hello  Drake
 Your email will be  drake@abc.com