metro app and kinect voice control

Peter Daukintis

Recently I had a need to reacquaint myself with the latest Kinect SDK. v1.6 has just been released and is downloadable from here Kinect SDK. So I downloaded it and started playing around with some code and it occurred to me that what I would really like was an environment in which I could use the Kinect in a metro application. Using the Kinect SDK from a metro app is not a supported scenario and neither is communication between a metro app and a desktop app. Having said that, I recall reading somewhere that someone may have used a wcf service to carry out this communication so I guessed that a local socket connection would probably work (I believe this is an unsupported mechanism and would likely fail store certification). My goal was not to release an app but just to be able to make use of it for experimental/personal usage.

Now, I had previously written some WinRT code for socket communication using StreamSocket (see StreamSocket Example) and I saw a few blog posts about using WinRT libraries on the Windows 8 desktop so I though I’d give it a try…

The first thing was to get a communication path between a metro app and windows 8 desktop app. I followed the instructions here to enable WinRT in a desktop app. Then, I copied the server side of the socket code and the xaml into a WPF app. With a few minor adjustments (replacing OnNavigatedTo event handler and MessageDialog) it was good to go..

 

To test that everything worked ok, I

• Ran the wpf application in the debugger…
• Ran the store ui app in the debugger
• Listen from the desktop
• Connect from the store ui app

And….

It worked!

(side by side apps screenshot…)

So, the next step is to configure the wpf application to respond to Kinect voice commands and ship them over the socket.

First I added references to Kinect SDK and toolkit assemblies, which on my machine were located as follows:

C:\Program Files\Microsoft SDKs\Kinect\Developer Toolkit v1.6.0\Samples\bin\Microsoft.Speech.dll

And

C:\Program Files\Microsoft SDKs\Kinect\v1.6\Assemblies\Microsoft.Kinect.dll

Add code to carry out speech recognition using the Kinect with a very simple grammar.

_sre = new SpeechRecognitionEngine(recognizer);

var gb = new GrammarBuilder { Culture = recognizer.Culture };

var choices = new Choices();
choices.Add(new SemanticResultValue("go", "NEXT"));
choices.Add(new SemanticResultValue("back", "PREV"));

gb.Append(choices);
var g = new Grammar(gb);

_sre.LoadGrammar(g);
_sre.SpeechRecognized += sre_SpeechRecognized;

_sre.SetInputToAudioStream(_kinect.AudioSource.Start(),
    new Microsoft.Speech.AudioFormat.SpeechAudioFormatInfo(EncodingFormat.Pcm, 16000, 16, 1, 32000, 2, null));

_sre.RecognizeAsync(RecognizeMode.Multiple);

When the speech recognizer recognises the next (go) and previous (back) words I will transmit that text over the socket to the waiting metro app whereupon I will use them to activate commands.

To make a useful (sort of) example I will create a Metro app which loads images from your pictures library into a flip view and use the voice commands to allow navigation of the pictures using ‘next’/"previous".

 

Disclaimer: Please note that I haven’t tested this code beyond trying it in my development environment and it was not written to be robust in any way.

Download the solutions here.

KinectSocketServer

KinectPictureViewer

Streamsocket example c# metro

Peter Daukintis

This is a very basic example of socket communication in a c# metro-style application using StreamSocket and StreamSocketListener. It mirrors the Simple StreamSocket example in msdn here http://code.msdn.microsoft.com/windowsapps/StreamSocket-Sample-8c573931 which is coded in c++ and javascript but no c#. I decided to roughly convert it as a learning exercise – please note I haven’t tested the code in that many scenarios and haven’t handled failure cases and errors much.

A socket provides send and receive over TCP and a StreamSocketListener will listen for incoming TCP connections.

The sample will act as a server or client depending on which buttons you press, i.e. press listen to be a server and connect to be a client. Also, you can set the same program as client and server ad have it communicate with itself.

To start the listener wire up a connection received event handler and bind the service name…

_listener.ConnectionReceived += listenerConnectionReceived;
await _listener.BindServiceNameAsync("3011");

The event handler stores the incoming socket in a list (so it can use it to reply), and waits for incoming data from it. The ‘waiting for incoming data’ bit looks like this…

async private void WaitForData(StreamSocket socket)
{
    var dr = new DataReader(socket.InputStream);
    //dr.InputStreamOptions = InputStreamOptions.Partial;
    var stringHeader = await dr.LoadAsync(4);

    if (stringHeader == 0)
    {
        // disconnected
        return;
    }

    int strLength = dr.ReadInt32();
    uint numStrBytes = await dr.LoadAsync((uint)strLength);
    string msg = dr.ReadString(numStrBytes);
    WaitForData(socket);
}

It uses a DataReader to read from the incoming socket and then calls itself to wait for subsequent incoming message data.

Note that the samples use a protocol which sends the length (number of bytes) of the message first followed by the message data itself. If you want to use a different strategy, for example characters to delimit the start and end of a message then the code will change a little bit. The difference being that you won’t always know how much data to read. You can modify the code like this to accommodate this:

async private void WaitForData(StreamSocket socket)
        {
            var dr = new DataReader(socket.InputStream);
            dr.InputStreamOptions = InputStreamOptions.Partial;
            var stringHeader = await dr.LoadAsync(512);

            // change the rest acccordingly….

Note that setting the InputStreamOptions to Partial will allow you to specify a larger buffer but also the async load operation will complete when a smaller sized buffer comes in. This enables you to read the data coming in and respond to it accordingly.

The pictures show a session between a socket server running on a build slate and a win8 vm both on my local network..

The project can be downloaded from here.

https://skydrive.live.com/redir.aspx?cid=4f1b7368284539e5&resid=4F1B7368284539E5!445&parid=4F1B7368284539E5!123

UPDATE: This project has been updated for the Release Preview

http://sdrv.ms/Nema94

metro c#–simple transform using touch

Well, this turned out to be pretty easy ……

The aim was to carry out a simple image transform with touch; so pinch zoom, translate and rotate. This is how the code turned out.

First, the xaml, which is an Image inside a Canvas…

 

<Canvas>
    <Image x:Name="myImage"
           ManipulationMode="All"
           ManipulationDelta="Image_ManipulationDelta_1"
           RenderTransformOrigin="0.5, 0.5">
        <Image.RenderTransform>
            <CompositeTransform></CompositeTransform>
        </Image.RenderTransform>
    </Image>
</Canvas>

 

Then the event handler for the ManipulationDelta event

 

private void Image_ManipulationDelta_1(object sender, ManipulationDeltaRoutedEventArgs e)
{
    var ct = (CompositeTransform)myImage.RenderTransform;
    ct.ScaleX *= e.Delta.Scale;
    ct.ScaleY *= e.Delta.Scale;
    ct.TranslateX += e.Delta.Translation.X;
    ct.TranslateY += e.Delta.Translation.Y;
    ct.Rotation += 180.0 / Math.PI * e.Delta.Rotation;

    UpdateControls(ct);
}

 

And here’s the sample running on my Build slate…

 

 

The code is available here…

https://skydrive.live.com/redir.aspx?cid=4f1b7368284539e5&resid=4F1B7368284539E5!443&parid=4F1B7368284539E5!123

metro c# webapi client

Peter Daukintis

For my own purposes I wanted to create a test application which I can use as a test-bed to experiment with user interface aspects of Metro. I decided to create a simple REST web service and make create/read/update/delete type calls to it from a c# metro application. Pretty much the next step up from ‘Hello World’ for a web connected application.

First, to create the Web API choose ASP.NET MVC 4 Application from the New Project Dialog and in the ensuing dialog select WebApi. We are using the ASP.NET MVC 4 Beta which now includes the WebApi (http://www.asp.net/web-api); which was formerly known as WCF WebApi.

The template creates a ValuesController to expose the sample CRUD methods (GetCollection/Get/Post/Put/Delete). It uses a string as the payload so I’m going to change that to use a Person class.

public class Person
{
    static int idGen = 123456;
    public Person()
    {
        Id = ++idGen;
    }
    public Person(Person other)
    {
        Id = other.Id;
        Name = other.Name;
        Occupation = other.Occupation;
    }

    public string Occupation { get; set; }
    public string Name { get; set; }
    public int Id { get; private set; }
}

Next, I renamed the ValuesController to PeopleController and modified the template code to query a list of Person objects:

public class PeopleController : ApiController
{
    private List<Person> _list = new List<Person>
        {
            new Person{ Name = "fred", Occupation = "plumber" },
            new Person{ Name = "bob", Occupation = "lawyer" },
            new Person{ Name = "flo", Occupation = "pilot" },
        };

    // GET /api/people
    public IEnumerable<Person> Get()
    {
        return _list;
    }

    // GET /api/people/5
    public Person Get(int id)
    {
        return _list.SingleOrDefault(p => p.Id == id);
    }

    // POST /api/people
    public void Post(Person value)
    {
        _list.Add(value);
    }

    // PUT /api/people/5
    public void Put(int id, Person value)
    {
        var item = _list.SingleOrDefault(p => p.Id == id);
        if (item != null)
        {
            item.Name = value.Name;
            item.Occupation = value.Occupation;
        }
    }

    // DELETE /api/people/5
    public void Delete(int id)
    {
        var item = _list.SingleOrDefault(p => p.Id == id);
        if (item != null)
        {
            _list.Remove(item);
        }
    }

I’m not worried about the correctness or completeness of the http responses and behavior here as this post is about the client side of the code. Making the above code changes is enough to expose my RESTful api. For this example I an running the server on IISExpress on localhost – this enables me to test the GET calls using the browser by typing in http://localhost:60442/api/people to IE10 on the Desktop. (since web api returns JSON by default I made the modification to the registry described here http://stackoverflow.com/questions/2483771/how-can-i-convince-ie-to-simply-display-application-json-rather-than-offer-to-do to easily view the data coming back).

 

(I tested the other HTTP verbs using Fiddler by constructing requests by hand).

With that done it’s over to the Metro client.

I started with a blank c# Metro Style application and added a GridView to which I would bind my ViewModel (I tend to use the MVVM pattern). To support my MVVM architecture I rolled up a few helper classes; a DelegateCommand and a ViewModelBase class to provide INotifyPropertyChanged and an IsDesignMode property. The Expression Blend designer doesn’t work with network calls so the IsDesignMode property is used to facilitate design time data.

Next, I created some design-time data to enable me to design the ui in Blend

if (IsDesignMode)
{
    People = new ObservableCollection<Person>
    {
        new Person { Name = "design fred", Occupation = "Nurse" },
        new Person { Name = "design bob", Occupation = "Artist" },
        new Person { Name = "design flo", Occupation = "Teacher" },
        new Person { Name = "design zak", Occupation = "Fireman" },
        new Person { Name = "design rita", Occupation = "Nurse" },
    };
}

Here’s how it looked in Blend:

With this out of the way I could get to the heart of the problem – making the client-side calls:

Starting with retrieving the initial list

var http = new HttpClient();
var resp = await http.GetAsync(new Uri(apiRoot));
var stream = await resp.Content.ReadAsStreamAsync();
var djs = new DataContractJsonSerializer(typeof(List<Person>));
People = new ObservableCollection<Person>((IEnumerable<Person>)djs.ReadObject(stream));
stream.Dispose();

Then, Deleting a Person from the list

var http = new HttpClient();
var resp = await http.DeleteAsync(new Uri(apiRoot + "/" + Selected.Id));
resp.EnsureSuccessStatusCode();

Then, editing and adding, which I implemented in the same function:

var http = new HttpClient();
var djs = new DataContractJsonSerializer(typeof(Person));
var ms = new MemoryStream();
djs.WriteObject(ms, person);
ms.Position = 0;
var sc = new StreamContent(ms);
sc.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/json");

var resp = original == null ?
        await http.PostAsync(new Uri(apiRoot), sc) :
        await http.PutAsync(new Uri(apiRoot + "/" + original.Id), sc);

ms.Dispose();
resp.EnsureSuccessStatusCode();

In order to invoke the CRUD commands I added an application bar and populated it with app bar command buttons which I data bound to ICommands exposed by the View Model.

 

 

One thing worth mentioning is that in order to edit the Person objects I needed some form of editor page. I wasn’t sure of the best Metro-Style way to implement it, but in the end I went for a modal dialog which looked similar to the built in message box.

 

 

I created this as a full screen UserControl.

 

The solution containing the web api project and the metro c# client is here https://skydrive.live.com/redir.aspx?cid=4f1b7368284539e5&resid=4F1B7368284539E5!442&parid=4F1B7368284539E5!123.

UPDATE: An update for the Release Preview is here http://sdrv.ms/N4kfZT

skydrive upload c# metro

Peter Daukintis

A preview of the latest Live SDK for Metro apps can be downloaded here https://connect.microsoft.com/site1226. This is for usage with the Windows 8 Consumer Preview (Note that the SDK is a preview version with no go-live license).

After installing you can add a reference via ‘add Reference’ > and select Windows > Extensions in the Reference Manager dialog.

In order to configure sign in on Windows Phone a client ID was required; this is different for Metro-style Apps as you now need to register your app package name and publisher identity here http://go.microsoft.com/fwlink/?LinkId=227628.

Once registered you will be provided with a new Package Name – copy this back into your app manifest.

Once this is complete, using the following code you will be able to sign in:

XAML:

add this namespace declaration

xmlns:live="using:Microsoft.Live.Controls"

and a sign in button

<live:SignInButton x:Name="btnSignin" Scopes="wl.signin wl.basic wl.skydrive wl.skydrive_update"
                   SessionChanged="SignInButtonSessionChanged" />

 

The Live id session status is reported by the SessionChanged event so I handle it like this, storing the session object for later use:

private void SignInButtonSessionChanged(object sender, LiveConnectSessionChangedEventArgs e)
{
    if (e.Status == LiveConnectSessionStatus.Connected)
    {
        _liveConnectSession = e.Session;
        UpdateEnabled();
    }
}

Next we need a way to get hold of an image so I just used the File Picker to retrieve one (I added a way to capture an image using the camera but I couldn’t test it so it probably won’t work )

Here’s the code for the file picker:

async private void ChooseFile(object sender, RoutedEventArgs e)
{
    FileOpenPicker openPicker = new FileOpenPicker();
    openPicker.ViewMode = PickerViewMode.Thumbnail;
    openPicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;
    openPicker.FileTypeFilter.Add(".jpg");
    openPicker.FileTypeFilter.Add(".jpeg");
    openPicker.FileTypeFilter.Add(".png");
    StorageFile file = await openPicker.PickSingleFileAsync();

    if (file != null)
    {
        _filename = file.Name;
        BitmapImage bitmapImage = new BitmapImage();
        _stream = (FileRandomAccessStream)await file.OpenAsync(FileAccessMode.Read);
        bitmapImage.SetSource(_stream);
        myImage.Source = bitmapImage;
        UpdateEnabled();
    }
}

and finally, what it was all about, uploading to skydrive:

private void Upload(Stream data, string filename)
{
    var liveConnectClient = new LiveConnectClient(_liveConnectSession);

    liveConnectClient.UploadCompleted += LiveConnectClientUploadCompleted;
    VisualStateManager.GoToState(this, "Busy", true);
    liveConnectClient.UploadAsync("me/skydrive", filename, true, data, data);
}

Here’s the ui whilst uploading an image:

 

and when completed – displaying the json result returned by the upload.

 

 

and finally, the image on skydrive.

 

The project can be found here https://skydrive.live.com/redir.aspx?cid=4f1b7368284539e5&resid=4F1B7368284539E5!441&parid=4F1B7368284539E5!123

metro background audio c# (consumer preview)

Peter Daukintis

Some changes occurred in the consumer preview release of Windows 8 regarding background audio; the steps required to get it working are outlined in this forum post

http://social.msdn.microsoft.com/Forums/en-AU/winappswithcsharp/thread/2d3496df-a145-4d87-be08-aadd5a8098e2

The steps are

• Create a MediaElement and set it’s AudioCategory property to ‘BackgroundCapableMedia’
  <MediaElement x:Name="myMedia"
                Source="/Assets/Sleep Away.mp3"
                AudioCategory="BackgroundCapableMedia"/>

• Update the app manifest to declare Background Tasks of types ‘Audio’ and ‘Control Channel’
• Implement event handlers for
  • MediaControl.PlayPressed
  • MediaControl.PausePressed
  • MediaControl.PlayPauseTogglePressed
  • MediaControl.StopPressed

 

public BlankPage()
{
    this.InitializeComponent();
    MediaControl.PlayPressed += MediaControl_PlayPressed;
    MediaControl.PausePressed += MediaControl_PausePressed;
    MediaControl.PlayPauseTogglePressed += MediaControl_PlayPauseTogglePressed;
    MediaControl.StopPressed += MediaControl_StopPressed;
}

private void MediaControl_StopPressed(object sender, object e)
{
    myMedia.Stop();
}

private void MediaControl_PlayPauseTogglePressed(object sender, object e)
{
}

private void MediaControl_PausePressed(object sender, object e)
{
    myMedia.Pause();
}

private void MediaControl_PlayPressed(object sender, object e)
{
    myMedia.Play();
}

The media transport event handlers which need to be implemented are detailed here http://msdn.microsoft.com/en-us/library/windows/hardware/hh833781.aspx

 

Here’s a working sample project.

https://skydrive.live.com/redir.aspx?cid=4f1b7368284539e5&resid=4F1B7368284539E5!440&parid=4F1B7368284539E5!123‏

UPDATE: I have updated this for the Release Preview (see http://babaandthepigman.wordpress.com/2012/08/12/metro-background-audio-c-release-preview/)

datatemplateselector winrt

by Peter Daukintis

Having used DataTemplateSelectors previously with WPF and similar on Silverlight/Windows Phone I decided to check out whether they work the same in a xaml Metro application. It turns out to be pretty much the same (barring a few things being a bit buggy in the Win8 Developer Preview). Anyway, for reference, here’s a description:

First I created a custom data template selector. This enables you to intercept the decision of which template to use.

Custom Template Selector

1.     public class MyDataTemplateSelector : DataTemplateSelector
2.     {
3.         public DataTemplate PlaceTemplate { get; set; }
4.         public DataTemplate PersonTemplate { get; set; }
6.         protected override DataTemplate SelectTemplateCore(object item, DependencyObject container)
7.         {
8.             if (item is Person)
9.                 return PersonTemplate;
10.             if (item is Place)
11.                 return PlaceTemplate;
12.  
13.             return base.SelectTemplateCore(item, container);
14.         }
15.     }

I then created an instance in my Main Page’s xaml…

Create Instance

1. <UserControl.Resources>
2.     <FlipView:MyDataTemplateSelector x:Key="mySelector" />
3. </UserControl.Resources>

If you’re used to other xaml technologies you need to be careful with your namespace declaration as the syntax has changed for WinRT:

Using declaration

1. xmlns:FlipView="using:FlipView"

One thing we haven’t done is to create and set up the DataTemplates:

DataTemplates

1. <UserControl.Resources>
2.     <DataTemplate x:Key="myPersonTemplate">
3.         <Border Background="CornflowerBlue">
4.             <TextBlock Text="{Binding Name}"
5.                        FontSize="64"
6.                        Foreground="Red"
7.                        HorizontalAlignment="Center"
8.                        VerticalAlignment="Center" />
9.         </Border>
10.     </DataTemplate>
11.     <DataTemplate x:Key="myPlaceTemplate">
12.         <Border Background="Orange">
13.             <Grid>
14.                 <Image Source="{Binding Image}"></Image>
15.                 <TextBlock Text="{Binding PlaceName}"
16.                            FontSize="128"
17.                            Foreground="AliceBlue"
18.                            HorizontalAlignment="Center"
19.                            VerticalAlignment="Center" />
20.             </Grid>
21.         </Border>
22.     </DataTemplate>
23.     <FlipView:MyDataTemplateSelector x:Key="mySelector"
24.                                      PersonTemplate="{StaticResource myPersonTemplate}"
25.                                      PlaceTemplate="{StaticResource myPlaceTemplate}">
26.     </FlipView:MyDataTemplateSelector>
27. </UserControl.Resources>

Note that I have created the DataTemplates with a key in the UserControl Resources. This allows me to access the item as a StaticResource; I do this to set the properties on my custom DataTemplateSelector. Now the above may be used with any ItemsControl since that defines an ItemTemplateSelector property on which we can set our custom DataTemplateSelector. It just remains to mock up some data and try it out on a few different Containers:

Dummy Data

1. private ObservableCollection<object> _collection = new ObservableCollection<object>
2.     {
3.         new Person { Name = "fred", Age = 22 },
4.         new Person { Name = "bob", Age = 22 },
5.         new Person { Name = "doris", Age = 22 },
6.         new Place { PlaceName="Moscow", Image="./Images/350px-St__Basil2.jpg"},
7.         new Person { Name = "mabel", Age = 22 },
8.     };

Note that I have used the instance type to be the criteria under which I swap the template but of course, the criteria could be anything here. To test on a FlipView just add

FlipView

1. <FlipView x:Name="myFlip" ItemTemplateSelector="{StaticResource mySelector}" />

 

and set the ItemSource in the code-behind:

ItemSource

1. myFlip.ItemsSource = _collection;

Just replace the above two steps to try this with a different container.

FlipView seems to suffer from some issues where it rendered unexpected results; sometimes updating the template and sometimes not.

The ListBox gave the expected results: