ColorImageFrame - Windows Forms

Now we come to the most complicated part of the program as the event handler has to do something useful with the raw bitmap data that the video camera sends to it. 

The raw data is packaged in the event argument. To get at it you need to go through a set of standard steps.

• use the OpenColorImageFrame method to retrieve a ColorImageFrame object.
• use the ColorImageFrame object's CopyPixelDataTo method to retrieve a byte array of pixel data.

The ColorImageFrame also includes some useful data such as the frame number and a time stamp. 

The raw data is in the form of a byte array with the image data packed in according to the format it uses. For the video sensor this takes the form of four bytes per pixel in RGBA format but with the Alpha (transparency) channel ignored.

In general working with bitmap data is made difficult by two considerations. The first is the format used for the data corresponding to each pixel and the second is the order that the data is stored in the array. 

The pixel format has already been described as 32 bit RGBA. The pixels data is stored in the array in row order and as each pixel is four bytes there is no need for any padding at the end of a row.

What this means is that the "stride" i.e. the number of bytes you need to read before you get to the start of the next row is simply

stride=width*bytesperpixel

and the total number of bytes in the array is

size=height*width*bytesperpixel

and you can work out the start of the four bytes that hold the pixel at x,y in the image as:

index=x*bytesperpixel+y*stride

So let's start work on retrieving and displaying the pixel data.

The first task in the event handler is to get the ColorImageFrame object:

void FrameReady(object sender, 
       ColorImageFrameReadyEventArgs e)
{ 
 ColorImageFrame imageFrame = 
                 e.OpenColorImageFrame();

Our next job is to do something with the bit data stored in Image.Bits.

What exactly you have to do depends on the task your application is carrying out and the framework you have selected.

Let's look at how to do it using Windows Forms first - as it is slightly more tricky than WPF and there aren't many examples to look at. WPF is relatively easy using the BitmapSource class and is covered later.

In the case of Windows Forms you have the Bitmap class which is generally used to manipulate and display images in say PictureBox controls or for saving to disk.

Getting the bits from the ColorImageFrame to a Bitmap without making any changes is a bit involved but its fairly standard "boilerplate" code that you can reuse - so lets write a function to do the job:

Bitmap ImageToBitmap(ColorImageFrame Image)
{

You also need to add:

using System.Drawing.Imaging;

at the start of the program.

This function takes a ColorImageFrame and returns a Bitmap with the same pixel data.

Notice that this only works for a ColorImageFrame that used 32 bit RBGA format data - which is the case for the video sensor as we have set it up. You can modify the function as required as you use other sensors.

First we need to retrieve the pixel data as a byte array:

byte[] pixeldata = new byte[
                Image.PixelDataLength];
Image.CopyPixelDataTo(pixeldata);

The PixelDataLength gives the number of bytes needed to store the entire image.

Next we need to create a Bitmap object capable of holding the pixel data:

Bitmap bmap = new Bitmap(Image.Width, 
                         Image.Height, 
             PixelFormat.Format32bppRgb);

This creates a Bitmap object the same size as the PlanarImage and sets it to 32 bit RGB i.e. RGBA with the Alpha channel ignored.

The Bitmap object uses another class, the BitmapData class, to create a memory buffer to store the bit data and this makes it slightly more difficult to use than the equivalent WPF classes.

You first need to use the LockBits method return a BitMapData object which has a locked in place memory buffer containing the pixel data. In general this operation copies the bitmap data from the bit map to the buffer so you can work on it.  However if you set the WriteOnly option then a memory buffer of the correct size is allocated and nothing is transferred from the Bitmap object - this is more efficient.

So to create the buffer and lock it:

BitmapData bmapdata = bmap.LockBits(
 new Rectangle(0, 0, Image.Width, 
                     Image.Height), 
 ImageLockMode.WriteOnly, 
 bmap.PixelFormat);

Now we have a memory buffer waiting for the bits in the byte array to be transferred to it. To do this we need to use a memory-to-memory transfer provided by the InterOp services so add:

using System.Runtime.InteropServices;

to the start of the file. First we need a pointer to the start of the memory buffer and then we use the Copy method to move the data:

Marshal.Copy(pixeldata, 0, ptr, 
             Image.PixelDataLength);

The first instruction stores the address of the start of the image buffer in ptr. The Copy method then proceeds to copy the data in Bits to the buffer that ptr points at. The second parameter is an offset, usually zero and the final parameter gives the total number of bytes to copy.

Now all we have to do is unlock the buffer which also transfers the data to the BitMap object and return it:

 bmap.UnlockBits(bmapdata);
 return bmap;
}

Now if we assign the bitmap that is returned to a PictureBox control on the form you can see the video from the Kinect's camera.

The complete event handler is:

void FrameReady(object sender, 
    ColorImageFrameReadyEventArgs e)
{
 ColorImageFrame imageFrame = 
              e.OpenColorImageFrame();
 Bitmap bmap = ImageToBitmap(imageFrame);
 pictureBox1.Image = bmap;
}

and the ImageToBitmap function is:

Bitmap ImageToBitmap(
                   ColorImageFrame Image)
{
 byte[] pixeldata = 
          new byte[Image.PixelDataLength];
 Image.CopyPixelDataTo(pixeldata);
 Bitmap bmap = new Bitmap(
        Image.Width, 
        Image.Height, 
        PixelFormat.Format32bppRgb);
 BitmapData bmapdata = bmap.LockBits(
   new Rectangle(0, 0, 
              Image.Width, Image.Height),
   ImageLockMode.WriteOnly, 
   bmap.PixelFormat);
 IntPtr ptr = bmapdata.Scan0;
 Marshal.Copy(pixeldata, 0, ptr,
            Image.PixelDataLength);
 bmap.UnlockBits(bmapdata);
 return bmap;
 }

If you put the whole lot together you will see an image displayed in the PictureBox when you run it. You can see a complete listing by downloading the code from the CodeBin.

<ASIN:B002BSA298@COM>

<ASIN:B0036DDW2G@UK>

<ASIN:B0036DDW2G@FR>

<ASIN:B002BSA298@CA>

<ASIN:B003H4QT7Y@DE>

<ASIN:B00499DBCW@IT>

<ASIN:1849690669>