The Microsoft Kinect for Windows SDK and Toolkit v1.5 does not yet provide built in support for gestures.  I found this to be pretty disappointing since it would seem trivial to insert some basic recognition patterns into the SDK, as they are already present on the Xbox Dashboard.  So I went hunting for reusable gesture library — ultimately finding and updating a library written for an earlier version of the SDK.

The gesture library presented here was originally proposed in Writing a gesture service with the Kinect for Windows SDK, written by Michael Tsikkos and James Glading.  I have updated the code for v1.5 of the Microsoft Kinect for Windows SDK, made a few minor changes to the architecture and placed the code into an easily reusable library.

Credit for the concepts and solutions of this library go to the original authors.  All I did was tweak it.

Fizbin.Kinect.Gestures has been updated!

The library described in this post has been updated since the original release.  To get the latest updates on the library and the latest documentation, please visit the libraries GitHub repository: http://github.com/EvilClosetMonkey/Fizbin.Kinect.Gestures.

Understanding and Creating Gestures

To fully understand what makes up a gesture and how to compose your own, I will leave you to read the original blog post: Writing a gesture service with the Kinect for Windows SDK.  The concepts and solution were unchanged in the update and readers should be able to easily translate the examples from the original post to the code provided here.

Executing the Demo

The gesture library is available on GitHub at the following repository: http://github.com/EvilClosetMonkey/Fizbin.Kinect.Gestures.

It includes the library, a small demo, and support libraries sourced from the Kinect SDK.

  • Fizbin.Kinect.Gestures – A reusable library for recognizing gestures based on Kinect skeleton data.
  • Fizbin.Kinect.Gestures.Demo – A very simple demo showing how to use the gesture library.
  • Microsoft.Kinect.Toolkit – sourced from the SDK.  Used to reference to KinectSensorChooser component, to automatically find and handle updates to the Kinect sensor.
  • Microsoft.Samples.Kinect.WpfViewers – sourced from the SDK.  Used to reference the KinectSensorManager data model class.

Connecting to the Kinect Services

Version 1.5 of the Microsoft Kinect for Windows SDK and Toolkit comes with several helper classes that make connecting to the Kinect sensor a breeze.  The demo application included with the gesture library takes full advantage of this!  The reader should be able to extrapolate their use from this example and the examples included with the SDK.  I also recommend the “ShapeGame” demo application as a great tutorial.

It is beyond of the scope of this post to discuss how to connect to these services.

Executing on Gestures

In order to capture and execute on a gesture we initialize the gesture recognizer’s callback function in InitializeKinectServices():

gestureController = new GestureController();
gestureController.GestureRecognized += OnGestureRecognized;

The gesture recognizer analyzes data from the skeleton.  Each time we receive a new skeleton frame we send it off to the gesture recognizer for review (see line 223):

private void OnSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
	using (SkeletonFrame frame = e.OpenSkeletonFrame())
	{
		if (frame == null)
			return;

		// resize the skeletons array if needed
		if (skeletons.Length != frame.SkeletonArrayLength)
			skeletons = new Skeleton[frame.SkeletonArrayLength];

		// get the skeleton data
		frame.CopySkeletonDataTo(skeletons);

		foreach (var skeleton in skeletons)
		{
			// skip the skeleton if it is not being tracked
			if (skeleton.TrackingState != SkeletonTrackingState.Tracked)
				continue;

			// update the gesture controller
			gestureController.UpdateAllGestures(skeleton);
		}
	}
}

If a gesture is recognized an event will be fired.  We go to our event callback and execute on the type of gesture returned:

private void OnGestureRecognized(object sender, GestureEventArgs e)
{
	Debug.WriteLine(e.GestureType);

	switch (e.GestureType)
	{
		case GestureType.Menu:
			Gesture = "Menu";
			break;
		case GestureType.WaveRight:
			Gesture = "Wave Right";
			break;
		case GestureType.WaveLeft:
			Gesture = "Wave Left";
			break;
		case GestureType.JoinedHands:
			Gesture = "Joined Hands";
			break;
		case GestureType.SwipeLeft:
			Gesture = "Swipe Left";
			break;
		case GestureType.SwipeRight:
			Gesture = "Swipe Right";
			break;
		case GestureType.ZoomIn:
			Gesture = "Zoom In";
			break;
		case GestureType.ZoomOut:
			Gesture = "Zoom Out";
			break;

		default:
			break;
	}
}

That is all that is required.  Update any Kinect enabled application with the above lines, in addition to including the gesture library project, and you can execute on basic gestures!