|Real RealSense In C# - Event Streams|
|Written by Harry Fairhead|
|Thursday, 16 April 2015|
Page 1 of 3
Things get a little more complicated when it comes to processing a stream of samples from the RealSense camera. It is quite easy to get it working. but much more difficult to get it working right. We show you how in this second article about RealSense In C#.
Real RealSense In C#
From the first part of this series we know how to get a single sample from the camera, but how do you get a stream of samples?
The answer is fairly obvious as long as you don't dig too deep into the details.
To get a single sample you:
Once the frame is available you can process the data using QuerySample to retrieve the sample:
The sample contains one image object for each of the streams you enabled.
This gives you a single sample. To acquire a stream of samples all you have to do is repeat the AcquireFrame and processing instruction in a loop:
When you have finished with the loop you need to dispose of the processing pipeline:
This is all fine and it works, but notice that there is a problem. The tight loop implemented as for(;;) uses the main thread even though the program is just waiting for AcquireFrame to return.
If you implement things this way using the UI thread then the UI will freeze and your users will be very unhappy. They will be able to see the results of your work but they will not be able to interact with your program at all.
The obvious solution is to take the AcquireFrame, QuerySample, process sample loop and move it onto a separate thread of its own.
This isn't difficult and it is the approach most of the sample programs take. However the API provides a set of methods that will implement the necessary threading for you - why not use it?
Event Based Streaming
Using event based streaming is very easy. You start off with the same steps.
After this things are only slightly different.
First you have to supply a callback function that will be automatically called every time a particular state occurs. To connect the callback to the state you need to create a instance of the Handler object. This has properties designed to allow you to specify any of a number of callbacks:
If we are streaming raw data from one of the cameras we need to use OnNewSample. When you start to use the more sophisticated processing modules to get gesture or hand data then you need to use OnModuleProcessedFrame.
All you have to do is create a Handler:
and set the callback property that you want to use. In the case of our example just the OnNewSample callback which is a standard method with the signature:
The mid parameter is a stream identifier if multiple streams are in use - this can be mostly ignored in simple setups. The sample parameter is a Sample object containing the data from the streams that have been enabled. The Callback returns a status code usually
Once you have the Handler object correctly initialized you can set the processing pipeline read to run using an overloaded version of the init method:
This specifies the Handler object that contains the references to the callbacks that you want the pipeline to call for each state.
Finally you can set the pipeline running using the
method. If the Boolean parameter is true the call blocks until the streaming is canceled. That is the callback will be repeatedly called but the thread that started the process will hang until the streaming is canceled by calling the Close function. If the Boolean parameter is false then the call returns immediately and the thread that called StreamFrames continues to process instructions and events. The callback is called repeatedly until the Close function is used.
A Simple Video Example
Now that we know the theory let's put it into practice. In turns out that there are a number of problems that arise because of the use of two threads - one for the UI and one for the data processing.
Start a new WPF project called WpfCallback and add the necessary references and DLL needed to use the Realsense camera - see Getting Started With RealSense In C# if you need detailed instructions.
Next place a button on the form and define its click event handler as:
You can see that when the button is click we get a SenseManager, create a Handler and set it's onNewSample to reference a method we have to write called OnNewSample. Then we initialize the pipeline specifying the handler and set the whole thing going with a non-blocking call to StreamFrames.
Now all we have to do is write the OnNewSample callback and this is where things get a little subtle.
Place an Image control on the form and name it image1 - this is where we will display the video data.
All we have to do is use the Sample object passed to the callback to extract the video data and then convert it into a WriteableBitmap, as described in the previous part of this series.
So, a direct approach to the problem would be to first get the image data:
Finally we set the WriteableBitmap as the source for the Image control and release the image data:
If you try this you will discover that you get a runtime error when you try to set the Image source to the WriteableBitmap. The problem is that the UI components are not threadsafe and so they detect any attempt to access them from any thread other than the once that created them. In short, you can access a UI component from the UI thread but no other.
|Last Updated ( Friday, 26 January 2018 )|