28 September, 2013

Google ProtoBuff + ZeroMq -- C++

In the last series of posts we demonstrated ZeroMq as a technology that supports 'sockets on steroids', supporting multiple platforms as well as multiple languages.  The examples to-date have been transmitting strings between senders and receivers.  While interesting, to effectively create a distributed heterogeneous system we need to be capable of transmitting meaningful messages, preferably complex data structures rather than just strings.  That's where Google's Protobuff comes into play: http://code.google.com/p/protobuf/

Building off our previously created Ubuntu 12.04 32-bit VM, let's start by installing the additional necessary packages;


$ sudo apt-get install libprotoc-dev

With the developer libraries installed, we can now extend our previous C++ example to transmit a ProtoBuff message.

We'll extend our Makefile to add the necessary libraries and a target (e.g. msgs) to generate the C++ files for the message.


$ cat Makefile 
CC=g++
SRCS=main.cpp Messages.pb.cc
OBJS=$(subst .cpp,.o,$(SRCS))
INCLUDES += -I.
LIBS += -lpthread -lrt -lzmq -lprotobuf
.cpp.o:
$(CC) -c $<
main: msgs ${OBJS} 
${CC} ${CFLAGS} -o $@ ${OBJS} ${LIBS}
msgs:
${SH} protoc -I. --cpp_out=. Messages.proto
clean:
${RM} ${OBJS} main *.pb.*

Oh, we should take a look at our simple Protobuff message file:

$ cat Messages.proto 
message Person {
  required int32 id=1;
  required string name=2;
}

Finally, our extended main file:

$ cat main.cpp 
#include
#include
#include
#include
#include
#include
#include "Messages.pb.h"
void* ctx=zmq_init(1);
char* EndPoint="tcp://127.0.0.1:8000";
static const int N=100;
static const int BufferSize=128;
void* sender(void*)
{
  printf("(%s:%d) running\n",__FILE__,__LINE__);
  void* pub=zmq_socket(ctx, ZMQ_PUB);
  assert(pub);
  int rc=zmq_bind(pub,EndPoint);
  assert(rc==0);
  Person p;
  p.set_name("fatslowkid");
  p.set_id(01);
  for(int i=0; i
  {
    zmq_msg_t msg;
    std::string S=p.SerializeAsString();
    char* content=(char*)S.c_str();
    int rc=zmq_msg_init_size(&msg, BufferSize);
    assert(rc==0);
    rc=zmq_msg_init_data(&msg, content, strlen(content), 0,0);
    assert(rc==0);
    rc=zmq_send(pub, &msg, 0);
    assert(rc==0);
    ::usleep(100000);
  }
}
void* receiver(void*)
{
  printf("(%s:%d) running\n",__FILE__,__LINE__);
  void* sub=zmq_socket(ctx, ZMQ_SUB);
  assert(sub);
  int rc=zmq_connect(sub,EndPoint);
  assert(rc==0);
  char* filter="";
  rc=zmq_setsockopt(sub, ZMQ_SUBSCRIBE, filter, strlen(filter));
  assert(rc==0);
  for(int i=0; i
  {
    zmq_msg_t msg;
    zmq_msg_init_size(&msg, BufferSize);
    const int rc=zmq_recv (sub, &msg, 0);
    char* content=(char*)zmq_msg_data(&msg);
    Person p;
    p.ParseFromString(content);
    printf("(%s:%d) received: '%s'\n",__FILE__,__LINE__,p.name().c_str());
    zmq_msg_close(&msg);
  }
}
int main(int argc, char* argv[])
{
  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
  int major, minor, patch;
  zmq_version (&major, &minor, &patch);
  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);
  pthread_t rId;
  pthread_create(&rId, 0, receiver, 0);
  pthread_t sId;
  pthread_create(&sId, 0, sender, 0);
  pthread_join(rId,0);
  pthread_join(sId,0);
  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}

Notice that we now transmit and receive Protobuf messages, serialized as strings.  The value of this is that the serialization mechanism is multi-platform & multi-language support.

Cheers.

21 September, 2013

ZeroMq Installation -- Python

In a previous post I documented the procedure for installing ZeroMq libraries for C/C++ applications. This post will focus on installing ZeroMq for Python development.


Let's get started;

Our target installation machine is Ubuntu 12.04 64-bit Desktop edition; http://www.ubuntu.com/download/desktop.  Your mileage may vary with alternative distributions.  I'm starting with a new default installation and will work through the installation process from here on.


$ sudo apt-get install python-zmq


After the APT package handling utility is done humping package installations you should have a function, albeit dated, version to begin working with.

Let's take a look at what version we're dealing with here.



$ cat go
#!/usr/bin/python
import zmq;
print zmq.pyzmq_version()
Running this beauty will show us the version of Zmq we've got installed.

$ ./go
2.1.11
Let's continue our sandbox by creating a sender and receiver Python script as follows;

$ more sender receiver
::::::::::::::
sender
::::::::::::::
#!/usr/bin/python

import zmq;
import time;

context = zmq.Context();
pub=context.socket(zmq.PUB);
pub.bind("tcp://127.0.0.1:8000");

for i in range(0,20):
  print "iteration",i
  pub.send("some message");
  time.sleep(1);
::::::::::::::
receiver
::::::::::::::
#!/usr/bin/python

import zmq;
import time;

context = zmq.Context();
sub=context.socket(zmq.SUB);
sub.connect("tcp://127.0.0.1:8000");
filter=""
sub.setsockopt(zmq.SUBSCRIBE, filter);


for i in range(0,20):
  print "waiting on msg"
  M=sub.recv();
  print "received",M

Running these two concurrently demonstrates the message delivery from the sender to the receiver;

::::::::::::::
sender.log
::::::::::::::
iteration 0
iteration 1
iteration 2
iteration 3
iteration 4
iteration 5
iteration 6
iteration 7
iteration 8
iteration 9
iteration 10
iteration 11
iteration 12
iteration 13
iteration 14
iteration 15
iteration 16
iteration 17
iteration 18
iteration 19
::::::::::::::
receiver.log
::::::::::::::
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg

What may be of interest you as well is that since the 'endpoint' is defined identically to the one used in the C++ example you can send messages from the Python sender script to the C++ main application.

Enjoy.

14 September, 2013

ZeroMq Installation -- C++

ZeroMq, or 0Mq if you prefer, is considered 'sockets on steroids'.  The framework is aimed at simplifying distributed communications, serves as a concurrency framework, and supports more languages than any normal mortal would ever require.
http://zeromq.org

The topic of this post will be guiding through the installation on Ubuntu 12.04 and writing enough example code to verify all is happy.  Future posts will focus on installation of alternative, and interesting, languages with the overall intent of demonstrating interoperability between the languages.

Let's get started;

Our target installation machine is Ubuntu 12.04 64-bit Desktop edition; http://www.ubuntu.com/download/desktop.  Your mileage may vary with alternative distributions.  I'm starting with a new default installation and will work through the installation process from here on.


$ sudo apt-get install libzmq-dev


After the APT package handling utility is done humping package installations you should have a function, albeit dated, version to begin working with.

While ZeroMq supports a plethora of languages, the focus of this post is on C++.  That said, we'll need to install GCC C++ compiler.


$ sudo apt-get install g++


As I said earlier, the Ubuntu repository has a fairly old version.  At time of this writing, v3.2.3 was the latest stable release but for the purposes of this post we'll continue to use this version.

Let's confirm what version we have by authoring our first C++ application and ensure we can successfully compile/link with the installed libraries.

Start with a simple Makefile;


$ cat Makefile



CC=g++



SRCS=main.cpp

OBJS=$(subst .cpp,.o,$(SRCS))

INCLUDES += -I.

LIBS += -lpthread -lrt -lzmq



.cpp.o:

 $(CC) -c $<



main: ${OBJS}

 ${CC} ${CFLAGS} -o $@ ${OBJS} ${LIBS}



clean:

 ${RM} ${OBJS} main


And now let's look as a trival main program that confirms we can compile/link with Zmq libraries;



$ cat main.cpp



#include 

#include 



int main(int argc, char* argv[])

{

  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);

  int major, minor, patch;

  zmq_version (&major, &minor, &patch);

  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);

  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);

}



Running it confirms the installed Zmq libraries are near ancient, but again ok for our short-term purposes.


$ ./main



(main.cpp:6) main process initializing

(main.cpp:6) zmq version: 2.1.11
(main.cpp:6) main process terminating

Let's extend this to make it a bit more interesting, classic sender/receiver communication model using the PUB/SUB comm model provided by ZeroMq.



$ cat main.cpp



#include 

#include 

#include 

#include 

#include 

#include 



void* ctx=zmq_init(1);

char* EndPoint="tcp://127.0.0.1:8000";

static const int N=100;

static const int BufferSize=128;



void* sender(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* pub=zmq_socket(ctx, ZMQ_PUB);

  assert(pub);

  int rc=zmq_bind(pub,EndPoint);

  assert(rc==0);

  for(int i=0; i  {

    char content[BufferSize];

    sprintf(content, "message %03d",i);

    zmq_msg_t msg;

    int rc=zmq_msg_init_size(&msg, BufferSize);

    assert(rc==0);

    rc=zmq_msg_init_data(&msg, content, strlen(content), 0,0);

    assert(rc==0);

    printf("(%s:%d) sending '%s'\n",__FILE__,__LINE__,content);

    rc=zmq_send(pub, &msg, 0);

    assert(rc==0);

    ::usleep(100000);

  }

}



void* receiver(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* sub=zmq_socket(ctx, ZMQ_SUB);

  assert(sub);

  int rc=zmq_connect(sub,EndPoint);

  assert(rc==0);

  char* filter="";

  rc=zmq_setsockopt(sub, ZMQ_SUBSCRIBE, filter, strlen(filter));

  assert(rc==0);

  for(int i=0; i  {

    zmq_msg_t msg;

    zmq_msg_init_size(&msg, BufferSize);

    const int rc=zmq_recv (sub, &msg, 0);

    char* content=(char*)zmq_msg_data(&msg);

    printf("(%s:%d) received: '%s'\n",__FILE__,__LINE__,content);

    zmq_msg_close(&msg);

  }

}



int main(int argc, char* argv[])

{

  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);

  int major, minor, patch;

  zmq_version (&major, &minor, &patch);

  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);



  pthread_t rId;

  pthread_create(&rId, 0, receiver, 0);



  pthread_t sId;

  pthread_create(&sId, 0, sender, 0);



  pthread_join(rId,0);

  pthread_join(sId,0);



  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);

}


Running this beauty will result in the following;

$ cat /var/tmp/ZmqMb/main.log 
(main.cpp:59) main process initializing
(main.cpp:62) zmq version: 2.1.11
(main.cpp:15) running
(main.cpp:29) sending 'message 000'
(main.cpp:38) running
(main.cpp:29) sending 'message 001'
(main.cpp:52) received: 'message 001'
(main.cpp:29) sending 'message 002'
(main.cpp:52) received: 'message 002'
(main.cpp:29) sending 'message 003'
(main.cpp:52) received: 'message 003'
(main.cpp:29) sending 'message 004'
(main.cpp:52) received: 'message 004'
(main.cpp:29) sending 'message 005'
(main.cpp:52) received: 'message 005'
(main.cpp:29) sending 'message 006'
(main.cpp:52) received: 'message 006'
(main.cpp:29) sending 'message 007'
(main.cpp:52) received: 'message 007'
(main.cpp:29) sending 'message 008'
(main.cpp:52) received: 'message 008'
(main.cpp:29) sending 'message 009'
(main.cpp:52) received: 'message 009'
(main.cpp:29) sending 'message 010'
(main.cpp:52) received: 'message 010'
(main.cpp:29) sending 'message 011'
(main.cpp:52) received: 'message 011'


You'll notice that the first message was lost.  I've never had a sufficient understanding of why this happens, but typically the first message can be lost, primarily due to the pub/sub sockets not necessarily being fully established.  Issuing a 'sleep' between socket creation and between the receiver and sender socket usage doesn't solve the problem, it's more involved than that and I have little to offer as an explanation.

So, there you go; a simple usage of ZeroMq.

Cheers.





07 September, 2013

Youtube Downloader from the Linux Command Line

God Bless YouTube!

Where else can you find entertainment, education and countless kitty/puppy videos?

But what if you wish to preserve the video, downloading for archive purposes or access it from Internet-incapable devices?

Start by installing youtube-dl:

# apt-get install youtube-dl


Next, navigate to your desired video and copy the URL; e.g.

http://www.youtube.com/watch?v=Wtfo43yV8rA

Lastly, execute youtube-dl with specified URL:


$ youtube-dl http://www.youtube.com/watch?v=Wtfo43yV8rA


The result will be a Macromedia Flash Video file.  Convert to your preferred file format using Ffmeg and Bob's Your Uncle.