|Real RealSense In C# - Event Streams
|Written by Harry Fairhead
|Thursday, 16 April 2015
Page 2 of 3
Invoke, Freeze And GC
In this case we have to deal with the problem that the callback is being executed by a thread that the RealSense system has created and not the UI thread as is the common practice for a UI event.
To allow other threads to access a UI component you can use the Dispatcher to schedule a call to a method using the UI thread.
That is, if you write:
then the call to method is executed by the UI thread as soon as it becomes available.
In this form the Invoke method blocks and calling thread only continues after the method as finished execution. You can use the InvokeAsync method if you want a non-blocking call to run the method on the UI thread. However, this brings with it some complications of its own. If the calling thread doesn't wait for the completion of the InvokeAsync then it is possible that the callback will complete and be called again before the InvokeAsync has completed. This would result in an ever-growing backlog of frames to process.
For the moment let's continue with the blocking Invoke which ensures that another frame cannot be processed until this one is completed. Of course, it is important that whatever we do with the frame it is processed as quickly as possible to avoid missed frames.
So now all we have to do is use the Invoke method to set the Image source:
You can define a completely new method to pass to the Invoke method, but it is easier to use an anonymous lambda expression.
If you try this out you will once again discover that things don't work.
The reason is that wbm, the WriteableBitmap, is created on the new thread and on on the UI thread. WPF is set up to detect attempts to work with objects that weren't created using the UI thread. However, it can work with objects that support IFreezable as long as they are frozen first so that no changes can occur. The WriteableBitmap does support IFreezable so our final attempt at making it all work is:
Notice that it is the WriteableBitmap object that is frozen, not the wbm variable that references it.
This does work but there is still an intermittent problem - the program occasionally runs out of memory.
If you think about what is happening it becomes fairly obvious why:
Each time the callback is used it creates a complete new WriteableBitmap object, freezes it and then sets the Image source to reference it.
What happens to the old WriteableBitmap object?
The answer is that when you set the Image source to the new WriteableBitmap the old one no longer has any references to it and so become eligible for garbage collection. This should all work, but we are creating a very large WriteableBitmap object perhaps sixty times per second. It is possible that the garbage collector will not be called in time to avoid running out of memory.
What is the solution?
There are a number of possible ways around this problem. The most obvious is not to create a WriteableBitmap every time you retrieve a sample - use the same one and just change its data. However, this raises the problem of how to do this when you have to freeze its data to pass it to the UI thread. To implement this approach you have to pass the ImageData to the UI thread and get it to perform the update of the WriteableBitmap on the UI thread with no call to freeze being needed.
A simple and direct solution to the occasional running out of memory problem is to place an explicit call to the garbage collector:
This works but with the risk that the garbage collection might take a long time and so result in missed frames. In practice that garbage collector doesn't have too much to do and it all works.
Putting this all together gives us the final version of the callback function:
Depending on what you plan to do with the data, creating big expensive object like WriteableBitmap every time a frame is received might well not be the best way to work.
How to stop the streaming?
Simple just add another button and call the SenseManager's Close method. To make this work we need to keep a reference to the SenseManager.
Aligned and unaligned streams
When you enable multiple streams you can either set things up so that the callback is triggered when any single frame from any of the enabled streams is available or only when a complete sample of all of the enabled frames is available.
If you simply enable streams individually then you get an unaligned stream and you have to test to see which of the frames is available for each call of OnNewSample.
For example, to stream both the color and depth data add a new Image control named image2:
Then you have to process each frame as it arrives but now you have to test to see which which frame has arrived:
If you run this program you will notice that there is a lag before the depth data appears compared to the color data.
If you want to align the data streams then all you have to do is change the way the you enable them. You have to enable them in one go using the streams object:
Replace the two stream enable commands with:
You will discover that now the program runs and the two streams appear in the Image controls at the same time. You don't need to change the the OnNewSample callback, but now the if statements are no longer required as both streams are present in each sample.
|Last Updated ( Friday, 26 January 2018 )