metro app and kinect voice control

Peter Daukintis

Recently I had a need to reacquaint myself with the latest Kinect SDK. v1.6 has just been released and is downloadable from here Kinect SDK. So I downloaded it and started playing around with some code and it occurred to me that what I would really like was an environment in which I could use the Kinect in a metro application. Using the Kinect SDK from a metro app is not a supported scenario and neither is communication between a metro app and a desktop app. Having said that, I recall reading somewhere that someone may have used a wcf service to carry out this communication so I guessed that a local socket connection would probably work (I believe this is an unsupported mechanism and would likely fail store certification). My goal was not to release an app but just to be able to make use of it for experimental/personal usage.

Now, I had previously written some WinRT code for socket communication using StreamSocket (see StreamSocket Example) and I saw a few blog posts about using WinRT libraries on the Windows 8 desktop so I though I’d give it a try…

The first thing was to get a communication path between a metro app and windows 8 desktop app. I followed the instructions here to enable WinRT in a desktop app. Then, I copied the server side of the socket code and the xaml into a WPF app. With a few minor adjustments (replacing OnNavigatedTo event handler and MessageDialog) it was good to go..

 

To test that everything worked ok, I

• Ran the wpf application in the debugger…
• Ran the store ui app in the debugger
• Listen from the desktop
• Connect from the store ui app

And….

It worked!

(side by side apps screenshot…)

So, the next step is to configure the wpf application to respond to Kinect voice commands and ship them over the socket.

First I added references to Kinect SDK and toolkit assemblies, which on my machine were located as follows:

C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.6.0\Samples\bin\Microsoft.Speech.dll

And

C:\Program Files\Microsoft SDKs\Kinect\v1.6\Assemblies\Microsoft.Kinect.dll

Add code to carry out speech recognition using the Kinect with a very simple grammar.

_sre = new SpeechRecognitionEngine(recognizer);

var gb = new GrammarBuilder { Culture = recognizer.Culture };

var choices = new Choices();
choices.Add(new SemanticResultValue("go", "NEXT"));
choices.Add(new SemanticResultValue("back", "PREV"));

gb.Append(choices);
var g = new Grammar(gb);

_sre.LoadGrammar(g);
_sre.SpeechRecognized += sre_SpeechRecognized;

_sre.SetInputToAudioStream(_kinect.AudioSource.Start(),
    new Microsoft.Speech.AudioFormat.SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));

_sre.RecognizeAsync(RecognizeMode.Multiple);

When the speech recognizer recognises the next (go) and previous (back) words I will transmit that text over the socket to the waiting metro app whereupon I will use them to activate commands.

To make a useful (sort of) example I will create a Metro app which loads images from your pictures library into a flip view and use the voice commands to allow navigation of the pictures using ‘next’/"previous".

 

Disclaimer: Please note that I haven’t tested this code beyond trying it in my development environment and it was not written to be robust in any way.

Download the solutions here.

KinectSocketServer

KinectPictureViewer

kinect drums

Whilst exploring the possibilities opened by the Kinect device I made a demo using the OpenNI framework http://www.openni.org/ with PrimeSense NITE components.

 

Kinect Drums

As you can see from the video it allows tracking of multiple ‘players’. I started out by using ManagedNITE and using the hand tracking it provides but had some trouble getting a second hand to track (it worked but seemed less responsive than the first hand and was difficult to start the tracking).

This is roughly how the demo was made:

• modification of the NITE c++ sample which does user tracking (Players) to track multiple users
• wrapping this in a c+/cli component for consumption in a .NET environment
• consumed the .NET component in a WPF 3D desktop application

OpenNI + Kinect (getting set up on Windows)

 

 

I have recently been hacking around with a kinect camera + OpenNI + PrimeSense NITE components and initially it was quite challenging to get the software bits and pieces installed correctly. I got it working but then needed to rebuild my PC, soooo……

Since I didn’t find a package which carries out the whole thing, and the prospect of carrying out these repetitive tasks again prompted me to create a program which does most of the hard work on Windows.

Most of it is based on instructions from

http://www.studentguru.gr/blogs/vangos/archive/2011/01/20/how-to-successfully-install-kinect-windows-openni-nite.aspx

but I couldn’t get this to work so also used

http://pushypanda.blogspot.com/2011/01/kinecting.html

and an installer from here

http://code.google.com/p/kinemote/downloads/detail?name=SensorKinect-Win32-5.0.0.exe&can=2&q=

So, just in case it helps someone else here it is….

http://cid-4f1b7368284539e5.office.live.com/self.aspx/.Public/KinectInstaller/KinectInstall.zip

Instructions:

Please uninstall any related installations before commencing..

Unzip KinectInstall.zip and run KinectInstall.exe.

Then plug your Kinect camera into a USB port on your computer and try to run some of the demos which are installed with OpenNI and NITE. Hopefully they will work!

(Disclaimer : Use at your own risk.)

This is what it does;

This program will automate the installs for setting up a kinect camera with OpenNI and Prime Sense NITE. It will install current (26.01.2011) stable binaries from here

OpenNI framework

http://www.openni.org/downloadfiles/2-openni-binaries

PrimeSense OpenNI-compliant components

http://www.openni.org/downloadfiles/12-openni-compliant-middleware-binaries

and

Sensor Kinect

http://code.google.com/p/kinemote/downloads/detail?name=SensorKinect-Win32-5.0.0.exe&can=2&q=