Your Phone or Mine? Fusing Body, Touch and Device
Written by Kay Ewbank   
Thursday, 31 May 2012

Another novel use for the Kinect. It can work out which of multiple users is interacting with a multi-user interactive
touch display.

How can you work out who’s touching a multi-user interactive touch display?

It’s all very well writing apps that users can interact with by touching, but at the moment there’s no way your app can automatically tell who’s doing the touching if you allow multiple people to touch the screen.

Researchers at Microsoft Research have published a paper suggesting a way that you could work out who it is.




The method makes use of data from a smartphone’s on-board sensors together with camera-based body tracking. ShakeID can work out who’s touching the screen while holding their mobile phone.

So long as each user is holding a smartphone or other portable device that can sense its own movement, ShakeID matches the motion sensed by the device to motion observed by a Microsoft Kinect camera pointed at the users standing in front of the touch display. By comparing the movements of each phone in the scene with the movements of each user, the system can associate each phone to a specific user’s hand.

It then transforms the 2D space of the display to the 3D camera space, and associates the touches on the display to specific users. This means the app can work out which users are doing the touching, and which devices they’re holding.

The research paper gives the example of two users touching a display simultaneously in different locations to grab content, and ShakeID being able to associate each touch to a specific user and transfer the correct content to each user’s personal device. By using Kinect tracking data, the technique can also work out which hand is holding the device, and can detect when users move away from the screen and log the user out from the session automatically.




In the tests carried out of the technique, the users were told that phone movement was being used to associate devices with people and if they experienced any problems with the system not recognising their touch, they could shake their phone to improve the accuracy - what the researchers call ‘shake to associate’.

The current work matches the phone to the users’ hands, but the researchers suggest the technique may work when matching to body parts other than hands. They suggest it may be possible to successfully associate a device to a user if it is in the user’s pants pocket by matching to users’ hip joints data from the Kinect SDK. Or putting it another way, wiggle your hips to associate.

Not a suggestion often heard in software development, but it may just catch on.

More Information

Your Phone or Mine? Fusing Body, Touch and Device Sensing for Multi-User Device-Display Interaction (pdf)

Related Articles

Kintinuous - Kinect Creates Full 3D Models

Kinect SDK 1.5 - Now With Face Tracking

LightGuide: Kinect Teaches Movement

Kinect 3D Full Body "Hologram"


To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.


Hydraulic Atlas Bows Out, Welcome Electric Atlas

Boston Dynamics dismayed us at the beginning of the week with a video that suggested was discontinuing Atlas, its humanoid robot. Fast forward a day and its successor was unveiled. Designed to be even [ ... ]

GitLab Releases Duo Chat

GitLab has announced that Duo Chat is now generally available in GitLab 16.11, offering a range of AI features in a single natural language chat experience.

More News

raspberry pi books



or email your comment to:

Last Updated ( Friday, 01 June 2012 )