26 February, 2009

Extracting Raw Video Stream with Ffmpeg

We utilize raw, uncompressed video streams as input to sensor related video processing. Normally, our camera streams are captured directly from the camera source and stored in raw, uncompressed data file (no header or codec). Lately, we've found it useful to extract uncompressed streams from standard video file formats. Here's how:


$ ffmpeg -i ./VideoIn.avi -f rawvideo -pix_fmt gray /tmp/video.raw


This will convert the video stream to 8-bit grayscale and store it in a raw WxH stream, which you can play with:


$ mplayer -demuxer rawvideo -rawvideo w=640:h=352:y8 /tmp/video.raw


Note, the 640x352 image resolution was outputted as part of the ffmpeg extraction command, without the resolution you'll have more trouble viewing the video than 15 minutes of "The View".

Enjoy.

15 February, 2009

VMWare Server Installation

I've been sucessfully running VmWare Server 2.0 on my Debian 4.0 workstation for the past couple months. Since I had a couple hic-ups during the installation I figure I'll document them here for my own personal use and the use of others.

1.Perform fresh installation of Debian 4.0

2.Change Web Browser Preferences to utilize IceWeasel (necessary for web-enabled UI)

3.Install necessary packages
  • xion:/var/tmp# apt-get install binutils-doc make manpages-dev autoconf automake1.9 libtool flex bison gdb gcc g++ linux-headers-`uname -r`
4.Enable sound card channel
  • xion $ alsamixer (analog F channel)
5.Download VMWare Server 2.0 (http://www.vmware.com/download/server/)
6.Untar server archive
  • xion:/var/tmp# tar -zxvf /media/usb0/VMware-server-2.0.0-122956.x86_64.tar.gz
7.Install VMWare Server
xion:/var/tmp/vmware-server-distrib# ./vmware-install.pl
Creating a new VMware Server installer database using the tar4 format.

Installing VMware Server.

In which directory do you want to install the binary files?
[/usr/bin]

What is the directory that contains the init directories (rc0.d/ to rc6.d/)?
[/etc]

What is the directory that contains the init scripts?
[/etc/init.d]

In which directory do you want to install the daemon files?
[/usr/sbin]

In which directory do you want to install the library files?
[/usr/lib/vmware]

The path "/usr/lib/vmware" does not exist currently. This program is going to
create it, including needed parent directories. Is this what you want?
[yes]

In which directory do you want to install the manual files?
[/usr/share/man]

In which directory do you want to install the documentation files?
[/usr/share/doc/vmware]

The path "/usr/share/doc/vmware" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

The installation of VMware Server 2.0.0 build-122956 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/vmware-uninstall.pl".

Before running VMware Server for the first time, you need to configure it by
invoking the following command: "/usr/bin/vmware-config.pl". Do you want this
program to invoke the command for you now? [yes]

Making sure services for VMware Server are stopped.

Stopping VMware autostart virtual machines:
Virtual machines failed
Stopping VMware management services:
VMware Virtual Infrastructure Web Access
VMware Server Host Agent failed
Stopping VMware services:
VMware Authentication Daemon done
Virtual machine monitor done

You must read and accept the End User License Agreement to continue.
Press enter to display it.

NOTICE: BY DOWNLOADING AND INSTALLING, COPYING OR OTHERWISE USING THE
SOFTWARE, YOU AGREE TO BE BOUND BY THE TERMS OF THIS VMWARE MASTER END
USER LICENSE AGREEMENT ("EULA"). IF YOU DO NOT AGREE TO THE TERMS OF
THIS EULA, YOU MAY NOT DOWNLOAD, INSTALL, COPY OR USE THE SOFTWARE, AND
YOU MAY RETURN THE UNUSED SOFTWARE TO THE VENDOR FROM WHICH YOU ACQUIRED
IT WITHIN THIRTY (30) DAYS AND REQUEST A REFUND OF THE LICENSE FEE, IF
ANY, ALREADY PAID UPON SHOWING PROOF OF PAYMENT. "YOU" MEANS THE
NATURAL PERSON OR THE ENTITY THAT IS AGREEING TO BE BOUND BY THIS EULA,
THEIR EMPLOYEES AND THIRD PARTY CONTRACTORS THAT PROVIDE SERVICES TO
YOU. YOU SHALL BE LIABLE FOR ANY FAILURE BY SUCH EMPLOYEES AND THIRD
PARTY CONTRACTORS TO COMPLY WITH THE TERMS OF THIS AGREEMENT.

1. DEFINITIONS

1.1 "Designated Administrative Access" means that access to the
standard user interfaces of a given instance of the Software
(designated in this section) that you may grant to a designated
third party (a) for which you have provided advance written notice
to VMware that you are providing outsourced services and (b) for
whose dedicated benefit you have licensed such instance of the
Software. Designated Administrative Access is applicable only
where you are 1) an IT outsourcing company that is providing
outsourced IT services to a client company and 2) applicable only

Do you accept? (yes/no) yes

Thank you.

None of the pre-built vmmon modules for VMware Server is suitable for your
running kernel. Do you want this program to try to build the vmmon module for
your system (you need to have a C compiler installed on your system)? [yes]

Using compiler "/usr/bin/gcc". Use environment variable CC to override.

What is the location of the directory of C header files that match your running
kernel? [/lib/modules/2.6.18-6-amd64/build/include]

Extracting the sources of the vmmon module.

Building the vmmon module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmmon-only'
make -C /lib/modules/2.6.18-6-amd64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.18-6-amd64'
CC [M] /tmp/vmware-config0/vmmon-only/linux/driver.o
CC [M] /tmp/vmware-config0/vmmon-only/linux/driverLog.o
CC [M] /tmp/vmware-config0/vmmon-only/linux/hostif.o
CC [M] /tmp/vmware-config0/vmmon-only/common/comport.o
CC [M] /tmp/vmware-config0/vmmon-only/common/cpuid.o
CC [M] /tmp/vmware-config0/vmmon-only/common/hashFunc.o
CC [M] /tmp/vmware-config0/vmmon-only/common/memtrack.o
CC [M] /tmp/vmware-config0/vmmon-only/common/phystrack.o
CC [M] /tmp/vmware-config0/vmmon-only/common/task.o
CC [M] /tmp/vmware-config0/vmmon-only/common/vmx86.o
CC [M] /tmp/vmware-config0/vmmon-only/vmcore/moduleloop.o
LD [M] /tmp/vmware-config0/vmmon-only/vmmon.o
Building modules, stage 2.
MODPOST
CC /tmp/vmware-config0/vmmon-only/vmmon.mod.o
LD [M] /tmp/vmware-config0/vmmon-only/vmmon.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.18-6-amd64'
cp -f vmmon.ko ./../vmmon.o
make: Leaving directory `/tmp/vmware-config0/vmmon-only'
The vmmon module loads perfectly into the running kernel.

None of the pre-built vmci modules for VMware Server is suitable for your
running kernel. Do you want this program to try to build the vmci module for
your system (you need to have a C compiler installed on your system)? [yes]

Extracting the sources of the vmci module.

Building the vmci module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmci-only'
make -C /lib/modules/2.6.18-6-amd64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.18-6-amd64'
CC [M] /tmp/vmware-config0/vmci-only/linux/driver.o
CC [M] /tmp/vmware-config0/vmci-only/linux/driverLog.o
CC [M] /tmp/vmware-config0/vmci-only/linux/vmciKernelIf.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciContext.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciDatagram.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciDriver.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciDs.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciEvent.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciGroup.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciHashtable.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciProcess.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciQueuePair.o
CC [M] /tmp/vmware-config0/vmci-only/common/vmciResource.o
LD [M] /tmp/vmware-config0/vmci-only/vmci.o
Building modules, stage 2.
MODPOST
CC /tmp/vmware-config0/vmci-only/vmci.mod.o
LD [M] /tmp/vmware-config0/vmci-only/vmci.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.18-6-amd64'
cp -f vmci.ko ./../vmci.o
make: Leaving directory `/tmp/vmware-config0/vmci-only'
The vmci module loads perfectly into the running kernel.

None of the pre-built vsock modules for VMware Server is suitable for your
running kernel. Do you want this program to try to build the vsock module for
your system (you need to have a C compiler installed on your system)? [yes]

Extracting the sources of the vsock module.

Building the vsock module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vsock-only'
make -C /lib/modules/2.6.18-6-amd64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.18-6-amd64'
CC [M] /tmp/vmware-config0/vsock-only/linux/af_vsock.o
CC [M] /tmp/vmware-config0/vsock-only/linux/driverLog.o
CC [M] /tmp/vmware-config0/vsock-only/linux/util.o
CC [M] /tmp/vmware-config0/vsock-only/linux/vsockAddr.o
LD [M] /tmp/vmware-config0/vsock-only/vsock.o
Building modules, stage 2.
MODPOST
WARNING: "VMCIDatagram_CreateHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_DestroyHnd" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCI_GetContextID" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
WARNING: "VMCIDatagram_Send" [/tmp/vmware-config0/vsock-only/vsock.ko] undefined!
CC /tmp/vmware-config0/vsock-only/vsock.mod.o
LD [M] /tmp/vmware-config0/vsock-only/vsock.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.18-6-amd64'
cp -f vsock.ko ./../vsock.o
make: Leaving directory `/tmp/vmware-config0/vsock-only'
The vsock module loads perfectly into the running kernel.

Do you want networking for your virtual machines? (yes/no/help) [yes]

Configuring a bridged network for vmnet0.

Please specify a name for this network.
[Bridged]

The following bridged networks have been defined:

. vmnet0 is bridged to eth0

All your ethernet interfaces are already bridged.

Do you want to be able to use NAT networking in your virtual machines? (yes/no)
[yes]

Configuring a NAT network for vmnet8.

Please specify a name for this network. [NAT]

Do you want this program to probe for an unused private subnet? (yes/no/help)
[yes]

Probing for an unused private subnet (this can take some time)...

The subnet 172.16.38.0/255.255.255.0 appears to be unused.

The following NAT networks have been defined:

. vmnet8 is a NAT network on private subnet 172.16.38.0.

Do you wish to configure another NAT network? (yes/no) [no]

Do you want to be able to use host-only networking in your virtual machines?
[yes]

Configuring a host-only network for vmnet1.

Please specify a name for this network.
[HostOnly]

Do you want this program to probe for an unused private subnet? (yes/no/help)
[yes]

Probing for an unused private subnet (this can take some time)...

The subnet 192.168.208.0/255.255.255.0 appears to be unused.

The following host-only networks have been defined:

. vmnet1 is a host-only network on private subnet 192.168.208.0.

Do you wish to configure another host-only network? (yes/no) [no]

None of the pre-built vmnet modules for VMware Server is suitable for your
running kernel. Do you want this program to try to build the vmnet module for
your system (you need to have a C compiler installed on your system)? [yes]

Extracting the sources of the vmnet module.

Building the vmnet module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmnet-only'
make -C /lib/modules/2.6.18-6-amd64/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.18-6-amd64'
CC [M] /tmp/vmware-config0/vmnet-only/driver.o
CC [M] /tmp/vmware-config0/vmnet-only/hub.o
CC [M] /tmp/vmware-config0/vmnet-only/userif.o
CC [M] /tmp/vmware-config0/vmnet-only/netif.o
CC [M] /tmp/vmware-config0/vmnet-only/bridge.o
CC [M] /tmp/vmware-config0/vmnet-only/filter.o
CC [M] /tmp/vmware-config0/vmnet-only/procfs.o
CC [M] /tmp/vmware-config0/vmnet-only/smac_compat.o
CC [M] /tmp/vmware-config0/vmnet-only/smac.o
CC [M] /tmp/vmware-config0/vmnet-only/vnetEvent.o
CC [M] /tmp/vmware-config0/vmnet-only/vnetUserListener.o
LD [M] /tmp/vmware-config0/vmnet-only/vmnet.o
Building modules, stage 2.
MODPOST
CC /tmp/vmware-config0/vmnet-only/vmnet.mod.o
LD [M] /tmp/vmware-config0/vmnet-only/vmnet.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.18-6-amd64'
cp -f vmnet.ko ./../vmnet.o
make: Leaving directory `/tmp/vmware-config0/vmnet-only'
The vmnet module loads perfectly into the running kernel.

Please specify a port for remote connections to use [902]

Please specify a port for standard http connections to use [8222]

Please specify a port for secure http (https) connections to use [8333]

The current administrative user for VMware Server is ''. Would you like to
specify a different administrator? [no]

Using root as the VMware Server administrator.

In which directory do you want to keep your virtual machine files?
[/var/lib/vmware/Virtual Machines]

The path "/var/lib/vmware/Virtual Machines" does not exist currently. This
program is going to create it, including needed parent directories. Is this
what you want? [yes]

Please enter your 20-character serial number.

Type XXXXX-XXXXX-XXXXX-XXXXX or 'Enter' to cancel: A8MAJ-FK34X-UCL11-4J3U0

Creating a new VMware VIX API installer database using the tar4 format.

Installing VMware VIX API.

In which directory do you want to install the VMware VIX API binary files?
[/usr/bin]

In which directory do you want to install the VMware VIX API library files?
[/usr/lib/vmware-vix/lib]

The path "/usr/lib/vmware-vix/lib" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

In which directory do you want to install the VMware VIX API document pages?
[/usr/share/doc/vmware-vix]

The path "/usr/share/doc/vmware-vix" does not exist currently. This program is
going to create it, including needed parent directories. Is this what you want?
[yes]

The installation of VMware VIX API 1.6.0 build-122956 for Linux completed
successfully. You can decide to remove this software from your system at any
time by invoking the following command: "/usr/bin/vmware-uninstall-vix.pl".

Enjoy,

--the VMware team

Starting VMware services:
Virtual machine monitor done
Virtual machine communication interface done
VM communication interface socket family: done
Virtual ethernet done
Bridged networking on /dev/vmnet0 done
Host-only networking on /dev/vmnet1 (background) done
DHCP server on /dev/vmnet1 done
Host-only networking on /dev/vmnet8 (background) done
DHCP server on /dev/vmnet8 done
NAT service on /dev/vmnet8 done
VMware Server Authentication Daemon (background) done
Shared Memory Available done
Starting VMware management services:
VMware Server Host Agent (background) done
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines done

The configuration of VMware Server 2.0.0 build-122956 for Linux for this
running kernel completed successfully.

xion:/var/tmp/vmware-server-distrib#


WinXp Setup
1.When configuring a Win VM, configure NIC as bridged device so ip address is on local network, otherwise you won't have visibility to print server
2.Install samba
  • # apt-get install samba
3.xion:/var/tmp# smbpasswd -a user1
4.Configure cdrom to support multimedia demands
  • utilization of ATAPI interface appears too slow to support dvd playback, copy and the like. Configure to use the host device directly

09 February, 2009

Creating Debian Packages

I have to admit, one of the most appealing features of Debian is the ease of package management. While we've authored many tool suites to be used in analysis, field testing, or development I've yet to package them into a distribution package. While I've always wanted to, I've never taken the initiative to investigate how to do it. Today I'll be documenting my journey into authoring my first Debian distribution package.

Debian Packages
A package is a simple collection of files with instructions of what to do with them. While most often packages consist of executables, they may instead contain source code, window themes, or other file collections. Since I'm interested in maintaining executables I'll focus on the binary distribution package. I start my journey by issuing the command 'man deb'.

File Contents
All binary Debian packages consist of 3 basic files: 1 text file, and 2 compressed tarballs. You can examine the contents of a Debian package yourself by using the archive command to list the file contents. For example,


$ ar -t /var/cache/apt/archives/telnet_0.17-34_amd64.deb
debian-binary
control.tar.gz
data.tar.gz


You can examine the content of each file by extracting them from the archive, done by issuing the following command:

$ ar -x /var/cache/apt/archives/telnet_0.17-34_amd64.deb


debian-binary
This file contains the version number for the binary package, which should be 2.0.

control.tar.gz
This gzipped archive contains the control file, a postinst file, a prerm file and may optionally contain info about configuration files for the package and a MD5 checksums for the package. You can examine the contents of this archive by issuing:

$ tar -ztf ./control.tar.gz
./
./postinst
./prerm
./postrm
./md5sums
./control

The only mandatory file in this archive is the control file which contains the core control info for the package.
control
Issuing the command 'man deb-control' documents the contents of the mandatory and optional fields for the control file. The file consists of mandatory and optional fields. The mandatory fields are listed with brief descriptions of each.
Package: This value identifies the name of the package.
Version: This field identifies the version number. The use of a hyphen identifies the Debian revision number so the package author is restricted from using the hyphen the version number.
Maintainer: This should be defined in the format of full name followed by e-mail address 'Joe Bloggs '.
Description: This field should provide a brief description of the package as the first line. This should be followed by a more detailed package description where each line begins with a space.

data.tar.gz
This file contains the actual “payload” of the package. This means that it contains a filesystem with all the relevant files for the program that when installed will be placed in the appropriate spots on the destination systems file system.


$ tar -ztf ./data.tar.gz
./
./usr/
./usr/bin/
./usr/bin/telnet.netkit
./usr/share/
./usr/share/doc/
./usr/share/doc/telnet/
./usr/share/doc/telnet/README.telnet
./usr/share/doc/telnet/README.gz
./usr/share/doc/telnet/BUGS
./usr/share/doc/telnet/copyright
./usr/share/doc/telnet/changelog.gz
./usr/share/doc/telnet/README.telnet.old.gz
./usr/share/doc/telnet/changelog.Debian.gz
./usr/share/man/
./usr/share/man/man1/
./usr/share/man/man1/telnet.netkit.1.gz
./usr/share/menu/
./usr/share/menu/telnet


Building a Package
Now that you know what's in a Debian package building one is pretty simple. Start by creating a working directory where you'll store the contents of the package.

$ mkdir /var/tmp/deb

Next, create a subdirectory where the contents of the control file will be stored.

$ mkdir /var/tmp/deb/DEBIAN


Author the control file and locate it in the DEBIAN directory.

$ cat DEBIAN/control
Package: MyPackage
Version: 1.0
Maintainer: Fat Slow Kid
Description: A brief description.
A more thorough description with lines and charts
pictures and graphs with a 8x10 glossy photo
with descriptions and notations.


Next, create the directory structure of the binary file contents. In our example, we intend on installing a new script foo.sh in /usr/local/bin.

$ mkdir -p usr/local/bin


Create a foo.sh script in this directory, our contents are as follows:

$ cat usr/local/bin/foo.sh
#!/bin/sh

echo "running 'My Script'"

Lastly, move out of this working directory and build it with the Debian provided package utilities.

$ dpkg -b /var/tmp/deb MyPackage.deb


After this, a new MyPackage.deb file will exist in the present working directory. You can examine the intended installed files by issuing the command as follows:

$ dpkg --contents MyPackage.deb
drwxr-xr-x fsk/fsk 0 2009-02-08 23:11 ./
drwxr-xr-x fsk/fsk 0 2009-02-08 23:11 ./usr/
drwxr-xr-x fsk/fsk 0 2009-02-08 23:11 ./usr/local/
drwxr-xr-x fsk/fsk 0 2009-02-08 23:12 ./usr/local/bin/
-rwxr-xr-x fsk/fsk 38 2009-02-08 23:12 ./usr/local/bin/foo.sh


Installing this package will now recreate the file system structure starting at the destination systems root file directory. Namely, foo.sh will be located in /usr/local/bin when installed.

# dpkg -i MyPackage.deb
Selecting previously deselected package mypackage.
(Reading database ... 79280 files and directories currently installed.)
Unpacking mypackage (from MyPackage.deb) ...
Setting up mypackage (1.0) ...
xion:/home/fsk#
xion:/home/fsk# which foo.sh
/usr/local/bin/foo.sh
xion:/home/fsk# foo.sh
running 'My Script'
xion:/home/fsk#

Granted the removal procedures aren't defined, but you've got the start of your very own Debian package.

You've achieved your Debian package yellow-belt.

06 February, 2009

Have You Named Your Pipe?

I am the proud owner of a 13lb Series 2 Tivo. I'm also a Linux advocate leaving it a matter of time before figuring out how to download and playback Tivo shows.

TivoDecode provides the means of converting the Tivo Mpeg-4 video format to a good 'ole fashion Mpeg-4. Then you can play the converted file back with your favorite video player.

TivoDecode operates pretty straightforward, you specify the input file and the MAK key of your Tivo (which is embedded in the TiVo file). If you specify an output file the decrypted video is stored in said output file.

This works pretty well, but as a side-effect you are essentially duplicating the source Tivo file which can be quite large.

I just recently found an alternative to creating this temporary file, created solely to provide to the video player.

Named pipes act just like a file and can be used to stream the output of one application as input to another. They are created by the mkfifo command and removed with the rm command (just like a regular file). Using them, you can go from this:

$ tivodecode --mak=1234567890 ./Desktop/NCIS.TiVo -o /tmp/video.mpg
$ mplayer /tmp/video.mpg

to this:

$ mkfifo /tmp/pipe.0
$ mplayer /tmp/pipe.0 &
$ tivodecode --mak=1234567890 ./Desktop/NCIS.TiVo -o /tmp/pipe.0
$ rm /tmp/pipe.0


BTW, I found this info by what I'd consider an unexpected site....mainly a Mac guy! No disrespect, but this is the first time I've made use of Mac info...OSX is a flavor of BSD so it was just a matter of time.

Take a look, I thought it was quite useful.