How Kinect tracks people
Tuesday, 09 November 2010

The Kinect hardware is impressive but what about the software? Body tracking is a longstanding problem in computer vision - has it been solved at last?

Banner

When you consider the Microsoft whole body input device - Kinect - you tend to focus on the hardware. Well it is quite impressive - a standard video camera and an infra-red camera working together to provide a depth map of the 3D scene.

kinect2

However the software deserves a mention because its role is to segment the depth map into objects and then track the objects. More specifically it tracks a person in real time without them having to wear sensors and this is a very difficult task - one that has been extensively studied as part of AI and computer vision.

It has now been revealed that the key software was developed by Microsoft Research Cambridge's vision research group. The old way of approaching the problem is to construct an avatar and attempt to find a match in the data provided by the camera.  Tracking is a matter of updating the match by moving the avatar as the data changes.This was the basis of the first Kinect software and it didn't work well enough for a commercial product. After about a minute or so it tended to lose the track and then not be able to recover it. It also had the problem that it only worked for people whe were the same size and shape as the system's developer - because that was the size and shape of the avatar used for matching.

kinect1

The new approach developed by the vision research team makes use of machine learning. They trained a learning system to recognise body parts. This can then be used to identify body parts in the incoming data stream using the GPU on a per pixel basis. The classifications are then pooled across pixels to produce hypothetical 3D body joint positions used by a skeletal tracking algorithm.

Explaining this recently one of the researchers said:

"We train the system with a vast and highly varied training set of synthetic images to ensure the system works for all ages, body shapes & sizes, clothing and hair styles. Secondly, the recognition does not rely on any temporal information, and this ensures that the system can initialize from arbitrary poses and prevents catastrophic loss of track, enabling extended gameplay for the first time."

Who says that AI never delivers on its promises! It is also clear that this sort of approach has many more applications than just a game input device.

Further reading:

Inside Kinect

$2000 bounty for open source Kinect drivers

3D Gesture Input

LightSpace agumented reality the future UI

Banner


Paul Allen's Living Computers Sold For Record Prices
04/10/2024

Auction house Christie's of New York broke existing records for sales of rare and iconic computers when it sold the collection that the late Paul Allen, co-founder of Microsoft, had assembled to  [ ... ]



Cortex Click - Bringing AI To Technical Documentation
19/09/2024

New start-up, Cortex Click, has launched a self-service, LLM-powered content generation platform designed produce high quality content for developer audiences including blog posts, tutorials, landing  [ ... ]


More News

<ASIN:0596518390>

<ASIN:0321643399>

<ASIN:3540436782>

<ASIN:3540669353>

<ASIN:019852451X>

<ASIN:3642125522>

Last Updated ( Tuesday, 09 November 2010 )