The Kinectonitor

Suppose you had a some scary-looking hoodlum walking around your house when you were out? You’d want to know about it, wouldn’t you? Take one Kinect, mix in a little Azure Service Bus, sprinkle in some SignalR, and mix it all together with some elbow grease, and you could watch in near-real-time as sinewy folks romp through your living room. Here’s how.

You might not be there (or want to be there) when some maniac breaks in, but it’d be great to have a series of photographs with the dude’s face to aid the authorities in their search for your home stereo equipment. The video below is a demonstration of the code this post will dive into in more detail. I figured it’d give some context to the problem this article will be trying to solve.

I’d really like to have a web page where I could go to see what’s going on in my living room when I’m not there. I know that fancy Kinect I picked up my kids for their XBox can do that sort of thing and I know how to code some .NET. Is it possible to make something at home that’d give me this sort of thing?

Good news! It isn’t that difficult. To start with, take a look at the Kinectonitor Visual Studio solution below.

image

At a high level it’ll provide the following high-level functions. The monitor application will watch over a room. When a skeleton is detected in the viewing range, a photograph will be taken using the Kinect camera. The image will be published to the Azure Service Bus, using a Topic publish/subscribe approach. An MVC web site will subscribe to the Azure Service Bus topic, and whenever the subscriber receives a ping from the service bus with a new image taken by the Kinect, it will use a SignalR hub to update an HTML client with the new photo. Here's a high-level architectural diagram of how the whole thing works, end-to-end. 

The Kinectonitor Core Project

Within the core project will exist a few common areas of functionality. The idea behind the core project is to provide a domain structure, functional abstraction, and initial implementation of the image publication concept. For all intents and purposes, the Kinect will be publishing messages containing image data to the Azure Service Bus, and allow subscribers (which we’ll get to in a moment) their own autonomy. The ImageMessage class below illustrates the message that’ll be transmitted through the Azure Service Bus.

image

A high-level abstraction will be needed to represent consumption of image messages coming from the Azure cloud. The purpose of the IImageMessageProcessorservice is to receive messages from the cloud and to then notify it’s own listeners that an image has been received.

image

A simple implementation is needed to receive image messages and to notify observers when they’re received. This implementation will allow the SignalR hub, which we’ll look at next, to get updates from the service bus; this abstraction and implementation are the custom glue that binds the service bus subscriber to the web site.

image

Next up is the MVC web site, in which the SignalR hub is hosted and served up to users in an HTML client.

Wiring Up a SignalR Hub

Just before we dive into the MVC code itself, take a quick look at the solution again and note the ServiceBusSimplifierproject. This is a super naïve, demo-class wrapper around the Azure Service Bus that was inspired by the far-more-complete implementation Joe Feser shares on GitHub. I used Joe’s library to get started with Azure Service Bus and really liked some of his hooks, but his implementation was overkill for my needs so I borrowed some of his ideas in a tinier implementation. If you’re deep into Azure Service Bus, though, you should totally look into Joe’s code.

image

Within the ServiceBusSimplifierproject is a class that provides a Fluent wrapper abstraction around the most basic Azure Service Bus publish/subscribe concepts. The ServiceBusclass (which could probably stand be renamed) is below, but collapsed. The idea is just to get the ideaof how this abstraction is going to simplify things from here on out. I’ll post a link to download the source code for this article in a later section. For now, just understand that the projects will be using this abstraction to streamline development and form a convention around the Azure Service Bus usage.

image

A few calls are going to be made in calls to the ServiceBus.Setupmethod, specifically to provide Azure Service Bus authentication details. The classes that represent this sort of thing are below.

image

Now that we’ve covered the shared code that the MVC site and WPF/Kinect app will use to communicate via the Azure Service Bus, let’s keep rolling and see how the MVC site is connected to the cloud using this library.

In this prototype code, the Global.asax.cs file is edited. A property is added to the web application to expose an instance of the MockImageMessageProcessor, (a more complete implementation would probably make use of an IoC container to store an instance of the service) for use later on in the SignalR Hub. Once the service instance is created, the Azure Service Bus wrapper is created and the ImageMessage messages are subscribed to by the site MessageProcessorinstance’s Processmethod.

image

When the application starts up, the instance is created and shared with the web site’s server-side code. The SignalR Hub, then, can make use of that service implementation. The SignalR Hub listens for ImageReceivedevents coming from the service. Whenever the Hub handles the event, it turns around and notifies the clients connected to it that a new photo has arrived.

image

With the Hub created, a simple Index view (and controller action) will provide the user-facing side of the Kinectonitor. The HTML/JQuery code below demonstrates how the client responds when messages arrive. There isn’t much to this part, really. The code just changes the srcattribute of an imgelement in the HTML document, then fades the image in using JQuery sugar.

image

Now that the web client code has been created, we’ll take a quick look at the Kinect code that captures the images and transmits them to the service bus.

The Kinectonitor Monitor WPF Client

Most of the Kinect-interfacing code comes straight from the samples available with the Kinect SDK download. The main points to be looked at in the examination of the WPF client is to see how it publishes the image messages into the Azure cloud.

The XAML code for the main form of the WPF app is about as dirt-simple as you could get. It just needs a way to display the image being taken by the Kinect and the skeletal diagram (the code available from the Kinect SDK samples). The XAML for this sample client app is below.

image

When the WPF client opens, the first step is, of course, to connect to the Kinect device and the Azure Service Bus. The OnLoadevent handler below is how this work is done. Note that this code also instantiates a Timer instance. That timer will be used to control the delay between photographs, and will be looked at in a moment.

image

Whenever image data is collected from the camera it’ll be displayed in the WPF Image control shown earlier. The OnKinectVideoReadyhandler method below is where the image processing/display takes place. Take note of the highlighted area; this code sets an instance of a BitmapSourceobject, which will be used to persist the image data to disk later.

image

Each time the Kinect video image is processed a new BitmapSourceinstance is created. Remember the Timerinstance from the class earlier? That timer’s handler method is where the image data is saved to disk and transmitted to the cloud. Note the check being performed on the AreSkeletonsBeingTrackedproperty. That’s the last thing that’ll be looked at next, that’ll tie the WPF functionality together.

image

If the Kinectonitor WPF client just continually took snapshots and sent them into the Azure cloud the eventual price would probably be prohibitive. The idea behind a monitor like this, really, though, is to show only when people enter during unexpected times. So, the WPF client will watch for skeletons using the built-in Kinect skeleton tracking functionality (and code from the Kinect SDK samples). If a skeleton is being tracked, we know someone’s in the room and that a photo should be taken. Given that the skeleton might be continually tracked for a few seconds (or minutes, or longer), the Kinect will continue to photograph while a skeleton is being tracked. As soon as the tracking stops, a final photo is taken, too. The code that sets the AreSkeletonsBeingTrackedproperty value during the skeleton-ready event handler is below.

image

Some logic occurs during the setter of the AreSkeletonsBeingTrackedmethod, just to sound the alarms whenever a skeleton is tracked, without having to wait the typical few seconds until the next timer tick.

image

That’s it for the code! One more note – it helps if the Kinect is up high or in front of a room (or both). During development of this article I just placed mine on top of the fridge, which is next to the kitchen bar where I do a lot of work. It could see the whole room pretty well and picked up skeletons rather quickly. Just a note for your environment testing phase.

IMAG0356

Summary

This article took a few of the more recent techniques and tools to be released to .NET developers. With a little creativity and some time, it’s not difficult to use those components to make something pretty neat at home in the physical computing world. Pairing these disciplines up to create something new (or something old someway different) is great fodder for larger projects using the same technologies later.

If you’d like to view the Kinectonitor GitHub source code, it’s right here.