JGameGrid
 
 

The source code of all examples is included in the JGameGrid distribution.

Kinect Natural User Interface, Skeletal Tracking

(For more information consult the KinectJLib web site.)

The Kinect controller is a natural user interface (NUI) that can track persons by a depth sensing video camera and generate skeletal information for 20 body joints in real-time. It also contains a microphone array to capture sound with direction information. There are numerous potential uses including game control, robotics, surveillance, input device for exhibition stands, people tracking, etc.

In 2010 a non-official Kinect driver with a C++ API became available and is now distributed through Code Laboratories and OpenNI. There are also some Java wrappers, especially with Greenfoot, but the installation and use is pretty tricky. In 2011 Microsoft release an official development system for the Kinect device called Kinect SDK. It's fully based on the Visual Studio IDE and all examples are in C++ and C#. As usual with Microsoft, it is restricted to the Windows OS and runs only under Windows 7 and up. As soon as this official release became available we decided to create a simple Java wrapper library KinectJLib based on our Java API Wrapper (JAW). Moreover we added the class GGKinect to our game development framework JGameGrid to shield the developer completely from graphics animation issues.

One of the usual demonstration examples of Kinect is a skeletal tracker. With KinectJLib and JGameGrid the program is simplified by an order of magnitude compared to the C++ code example distributed with the Kinect JDK.

By instantiating the GGKinect class a window is displayed that shows the video image. Because the window is completely managed in native code, there is no need to transfer video data to Java. On the native side, DirectX is used to enhance the quality of the image. Even with moderate computing power, a refresh rate of more than 20 images per seconds in 640 x 480 pixel resolution is obtained. The skeletal extraction, a rather complicated pattern recognition operation, is also performed in native Kinect SDK code and only the coordinates of the 20 skeletal joints are transferred to Java using the getJoints() method. The joints are instances of class Point3D with coordinates (x,y,z). x and y correspond to the video window (left to right, 0..windowWidth, top to bottom, 0..windowHeight), z is the distance to the camera in mm, not used in this simple demonstration. Finally drawSkeleton() and drawJoints() show the skeleton in the GameGrid window using the graphics methods of the class GGBackground.

Because a KinectCloseListener is registered, the callback notifyClose() is triggered when the close button of the native window is hit.

import ch.aplu.jgamegrid.*;
import
 ch.aplu.kinect.*;
import
 java.awt.*;
import
 ch.aplu.ggkinect.*;
import
 javax.swing.JOptionPane;

public
 class SkeletalViewer extends GameGrid implements KinectCloseListener
{
  
private String dllPath =
    Kinect.
is64bit()? "KinectHandler64" : "KinectHandler";

  
public SkeletalViewer()
  
{
    
super(640, 454, 1, nullfalse);  // GameGrid window
    
setPosition(640, 20);
    GGBackground bg 
= getBg();
    bg.
clear(new Color(128, 255, 128));
    
setTitle("GameGrid Kinect - Waiting for valid skeleton...");

    GGKinect kinect 
= new GGKinect(dllPath, "Video Frame",
      
0, 20, 640, 480, // Position and size
      GGKinect.DecorationStyle.STANDARD
);
    
if (!kinect.isInitialized())
    
{
      kinect.
setVisible(false);
      
JOptionPane.showMessageDialog(null"Initializing of Kinect failed.");
      
System.exit(0);
    
}

    kinect.
addCloseListener(this);
    
show();

    Point3D
[] joints = new Point3D[20]; // Initializes 20 skeleton joints
    
for (int i = 0; i < 20; i++)
      joints
[i] = new Point3D();
    kinect.
getJoints(joints, 0); // Blocks undefinitely
    
int rc = 0;                  // until skeleton is valid

    
// Tracking loop
    
while (true)
    
{
      bg.
clear();
      
if (rc != -1)  // Valid skeleton
      
{
        
setTitle("Valid Skeleton");
        kinect.
drawSkeleton(this, joints, 5, Color.red);
        kinect.
drawJoints(this, joints, 5, Color.black);
      
}
      
else
        
setTitle("Waiting For Valid  Skeleton");
      
refresh();
      rc 
= kinect.getJoints(joints, 20);  // Waits maximum 200 ms
    
}
  
}

  
public void notifyClose()
  
{
    
System.exit(0);
  
}

  
public static void main(String args[])
  
{
    
new SkeletalViewer();
  
}
}



Execute the program locally using WebStart.

System prerequisites:

  • Windows 7 or higher (32- or 64-bit)
  • Installation of the Kinect SDK from http://kinectforwindows.org
  • Kinect for Xbox 360 or Kinect for Windows (both devices are supported)

(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)

 

Sound Detection With Kinect's Microphone Array, Transfer Video To Java

In the following example Kinect's microphone array is used to detect the sound level. The microphone acts as a standard Windows sound device and its sound direction finding capabilities are not used. The sound is digitized and and the values captured by blocks in a byte buffer of selectable length. Every time the buffer is full, the sound level (maximum value) is calculated and the callback soundLevel() of the SoundListener interface is triggered to report the level of the last block.

In the same example we show how simple it is to transfer a captured video image from the native code to Java. Just call getImage() to return a BufferedImage that can be used in Java for any image transformation and display.

import ch.aplu.jgamegrid.*;
import
 ch.aplu.kinect.*;
import
 ch.aplu.ggkinect.*;
import
 java.awt.*;
import
 java.awt.image.*;
import
 javax.swing.JOptionPane;

public
 class SoundDetector extends GameGrid
  
implements KinectCloseListener, GGButtonListener, SoundLevelListener
{
  
private GGKinect kinect;
  
private String dllPath = 
    Kinect.
is64bit()? "KinectHandler64" : "KinectHandler";
  
private final String title = "Kinect Video Frame";
  
private static final int captionHeight = 19;
  
private final int ulx = 0; // Upper left x of native window
  
private final int uly = 20; // Upper left y of nativewindow
  
private static int width = 640;  // Width of native window
  
private static int height = 480 + captionHeight; // Height of native window
  
private GGButton grabBtn = new GGButton("sprites/grab.png");

  
public SoundDetector()
  
{
    
super(640, 520, 1, nullfalse);
    
setSimulationPeriod(20);
    
setPosition(ulx + width, uly);
    GGBackground bg 
= getBg();
    bg.
setPaintColor(Color.black);
    bg.
clear(new Color(128, 255, 128));
    bg.
drawRectangle(new Point(0, 0), new Point(640, 470));
    
addActor(grabBtn, new Location(40, 500));
    grabBtn.
addButtonListener(this);

    
if (Kinect.is64bit())
    
{
      
JOptionPane.showMessageDialog(null"Must use 32-bit JVM.");
      
System.exit(0);
    
}
    
    kinect 
= new GGKinect(dllPath,
      title, 
// Window title
      ulx, uly, 
// Window position
      width, height, 
// Window size
      GGKinect.DecorationStyle.STANDARD, 
// Decoration style
      
1000);  // Sound buffer size
    
if (!kinect.isInitialized())
    
{
      kinect.
setVisible(false);
      
JOptionPane.showMessageDialog(null"Initializing of Kinect failed.");
      
System.exit(0);
    
}
    kinect.
addCloseListener(this);
    kinect.
addSoundLevelListener(this);
    
show();
    
doRun();
  
}

  
public int soundLevel(int level)
  
{
    
setTitle("Sound Level = " + level);
    
if (level > 10)
    
{
      Fish fish 
= new Fish();
      
addActor(fish, new Location(10, 450 - 4 * level));
      
return 200;
    
}
    
return 0;
  
}

  
public void notifyClose()
  
{
    
System.exit(0);
  
}

  
public void buttonClicked(GGButton button)
  
{
  
}

  
public void buttonReleased(GGButton button)
  
{
  
}

  
public void buttonPressed(GGButton button)
  
{
    
BufferedImage bi = kinect.getImage();
    
getBg().drawImage(bi);
  
}

  
public static void main(String args[])
  
{
    
new SoundDetector();
  
}
}

 

The Fish class just moves the clownfishes back and forth.

import ch.aplu.jgamegrid.*;

public class Fish extends Actor
{
  public Fish()
  {
    super("sprites/nemo.gif");
  }

  public void act()
  {
    move();
    if (!isMoveValid())
      turn(180);
  }
}

Execute the program locally using WebStart. Make same noise to create new actors.

System prerequisites:

  • Windows 7 or higher (32- or 64-bit)
  • Installation of the Kinect SDK from http://kinectforwindows.org
  • Kinect for Xbox 360 or Kinect for Windows (both devices are supported)

(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)

jgamegridex26