Google Protobuf

Back from holidays !! Did some work on the project over the start of the holidays, so gonna share what I’ve done over the past 2 weeks. First of all, we had a problem of transferring multiple data from the server to the client. This is crucial, because for our real time system, we would want to inform the client about the speed and maybe other things as well. The implementation we’re looking for to solve this problem would be serialization. Based on wikipedia, serialization is the process of converting a data structure or object state into a format that can be stored (for example, in a file or memorybuffer, or transmitted across a network connection link) and “resurrected” later in the same or another computer environment.

Serialization

http://en.wikipedia.org/wiki/Serialization

In our case, we need serialization because we want to recreate the data in our client, which might be of different platform. For example, in our project we need to transfer data from the Pandaboard (ARM) server to a PC client (x86). I have yet to try whether without serialization the image remains the same. However, based on the result I got from trying a PC(x86) to an iPod Touch 3G (ARMv7), there seems to be some error. My guess would be because of different architecture type having different byte size or its just poor coding.

Having looked through multiple serialization, we have decided on using Google Protocol Buffers. There are other serialization frameworks such as Apache Thrift(used by Facebook), Boost Serialization (part of the popular Boost library). The reason why protocol buffer was chose is because it is the fastest and the cleanest amongst all. Apache Thrift would be good for multi-platform support, but there isn’t enough documentation on how to use it.

Google Protocol Buffer (Protobuf)

You can find the main page here ( http://code.google.com/p/protobuf/ ). It is a bit like XML and JSON. After installing you will need to create a .proto file using the compiler. Just use any text editor and create the file by adding a .proto extension when you save it. The .proto file looks a bit like java. You create a class, and you define its variables. For our project we used something like this.

package tutorial;

option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";

message Packet {
  repeated int32 data = 1 [packed=true];
  optional int32 speed = 2;
  optional int32 size = 3;
}

Not very sure the exact definition of package, but to my understanding its just the class. Message is like a new type used in protobuf. In it you can declare fields depending on what you need. You set the fields to either optional or required. Optional is good especially when you want to reuse the buffer but you want to not send certain things. Repeated means you can store data in that field multiple times, which can be accessed through index. It is similar to arrays. The downside is that multi-dimension arrays are not supported natively, so you will need google around to find out how to do it.

How to add it in your code :

	 //Declare protocol buffer
	tutorial::Packet packetSize;

    //Serializes protocol buffer into array

	int arraysize = packetData.ByteSize();
	// Sets a value for the size field in the protofbuf
	packetSize.set_size(arraysize);
	// Create an array of the size of packetSize
	char* id[packetSize.ByteSize()];
	//Serialize it to the array created
	packetSize.SerializeToArray(id, packetSize.ByteSize());
	//Send it over the network
	bytes = send(clientsock, id, sizeof(id), 0);
	packetSize.Clear();

Installing on Xcode for iOS:

http://www.kotancode.com/2012/10/14/using-google-protocol-buffers-in-objective-c-on-ios-and-the-mac/

This is much more complicated than installing on a PC or Mac. I had to fiddle around a little bit.

Results:

I’ve managed to get it working between computers. I have coded for iOS as well. It seems to be slightly slow because I’ve only managed to borrow an old iPod Touch 3G from a friend. Hoping to get something faster. Although it might not improve performance significantly because I’ve yet to implement concurrency on it, but I’m sure it will work faster than now because of the better network card.

Between PC and iOS :

IMG_20130107_143724

Will update more when I’ve implemented more stuff. Happy New Year Everyone !!

Advertisements

PandaBoard Streaming

In order to let us see what the car is doing, we needed a way to stream data between the car and a server. Having googled around, so far, two methods fit the bill.

First – C++ server → Websocket → HTML5 Client

This seems to be the best solution (if working) for the project as the clients can have the comfort of their HTML5 – supported browser ( while using Facebook at the same time) without any extra installation. Decided to follow the method implemented here:

http://systemsdaemon.blogspot.co.uk/2011/03/live-streaming-video-tutorial-for.html

Streaming from a fixed video works well. There wasn’t any frame drop or lag, and it works for mp4 and ogg formats. In order to stream captured videos, I’ve used OpenCV to capture frames, then writing every frame to video container using the built-in OpenCV VideoWriter and writeframe(). Ogg was used as the extension in this case as OpenCV doesn’t have any codec to write MP4 files. As the writing happens, it is streamed through the server mentioned above.

At the receiving end, results doesn’t look good at all. There is a massive lag and delay in the video received. This is expected as there are overhead from file I/Os, encoding and network delays. After some googling, there doesn’t seem to be a better solution in solving this problem at the moment as OpenCV doesn’t allow writing frames to memory instead of files. On the bright side, the recording from the webcam is playable and seems to be real time. We might incorporate this method into the project as part of recording the journey of the car.

Pros

  • Video stream is in colour
  • Can be accessed through a browser
  • Supports many devices
  • Streaming is recorded

Cons

  • Streaming is sluggish, massive delay of 30 seconds

Second Method – C++ Server → Websocket → C++ Client

Anyway, moving on, instead of streaming encoded data, I’ve decided to try streaming OpenCV data instead. The downside of this method is that the client end has to have OpenCV installed instead of using the more available browsers. Managed to stream data using the method from below :

http://nashruddin.com/streaming_opencv_videos_over_the_network

Results:

Pros –

  • Faster speed
  • Much lesser delay (about 5 seconds only)
  • Doesn’t require a server to be setup.

Cons –

  • At the moment, image viewed is in gray-scale. (Working on color streaming)
  • Presence of delay defeats the objective of real-time

Verdict –

At the moment, the second method is better than the first method in displaying video  obtained from OpenCV. More options to be explored, hopefully its better than these 2. Results to be uploaded soonish in the future.

Awaken the Panda (Board)

This is a long overdue post, mainly my fault for being too focused on the technical stuff. Anyway, this post will give you a head start on making the Pandaboard work.

Installing Ubuntu

This part of the post covers on how to get Ubuntu up and running on the Pandaboard.

http://www.omappedia.com/wiki/Ubuntu_Pre-built_Binaries_Guide

The link above covers the installation process step-by-step and is pretty comprehensive in my opinion. However, I’m still going to walk through the installation process.

For our project, the hardware we used to install Ubuntu are covered below:

  • PandaBoard (Upgrading to the ES version when we get it)
  • SanDisk Secure Digital High Capacity Card Extreme Video HD 16GB Class10                                (generally Class 10 SD cards are recommended)
  • SD Card Reader

To get started, I would recommend to download the Ubuntu operation system on your computer and install it. This is recommended for Windows user as it is more convenient when backing up images of the SD card. Virtual Machine should be fine if you don’t want to dual-boot.

After that, download the pre-built binaries from the links given below:

Ubuntu 12.04

http://cdimage.ubuntu.com/releases/12.04/release/ubuntu-12.04-preinstalled-desktop-armhf+omap4.img.gz

Moving on, you need to put the image into the SD card. I had trouble trying to follow the instructions as it seems to be missing on how to prepare the SD card for imaging. All you need to do is to delete any partition in the SD card. I used GParted Disk Editor to delete the partition.

Linux
Steps:
1. Place the SD card at your host computer.
2. Make sure the SD card is not mounted (just umount it if needed)
3. Identify the correct raw device name (like /dev/sde – not /dev/sde1, can be found in GParted)
4. Run the following command to write it:                                                                                                                   (replacing sde with the right card reader id; my card reader was /dev/mmcblk0)

  1.  gunzip ubuntu-12.XX-preinstalled-desktop-armhf+omap4.img.gz
  2. sudo dd bs=4M if=ubuntu-12.04-preinstalled-desktop-armhf+omap4.img of=/dev/sde
  3. sudo sync

Once done, you’re ready to boot up your PandaBoard. Plug the SD card into the SD card slot of the PandaBoard, turn on the power, and prepare to be amazed.

Follow the installation instructions on the screen. The installation is rather straight forward. Therefore, I won’t cover on this.

Backing up SD Card
Installation process can be quite a hassle. Other than that, you might want to backup personal settings or files. Therefore, it is essential to backup especially if you are working on a project. Backing up is a simple and hassle free process.

Instructions as follow:

  1. Open a Terminal in Ubuntu on your PC.
  2. Before that, you have to identify the identity of your SD Card in your card Reader.
  3. In the terminal, key in the following, replacing sde with id of card:
    dd if=/dev/sde of=./[Name of backup].img

Restoring image to SD Card

  1. delete any partition in the SD card.
  2. open terminal, run the following commands, replacing sde with id of card:
    dd bs=4M if=[Name of backup].img of=/dev/sde

That’s all for getting the PandaBoard to work. The only driver you need to install is the PowerVR Graphics driver. You should be notified by Ubuntu to install it.

Good Luck with the Panda :DD

– Chee –