Aegidius
 Plüss   Aplulogo
 
in examples
Use the asterisk (*) character for wildcard searches
Searches on more than one word will be treated "as a phrase"
 
serverstatus
 www.aplu.ch
 www.java-online.eu
     Print Text 
© 2018, V10.0
 
  TcpJLib
 
 

The source code of all examples is included in the KinectJLib distribution.

Example 2: Kinect gestures to control a robot

In typical gaming application, the Kinect device is used to detect gestures for controlling a game object. You may even steer a real object like a car or a robot. In this example we drive the Lego NXT robot in direct mode by using left and right hand gestures. The commands are transferred to the robot via Bluetooth (see the NxtJLib website for more information).

The following gestures are implemented:

  • Left or right hand up: STOP
  • Left and right hands up: MOVE_BACKWARD
  • Left hand down-left: MOVE_LEFT
  • Right hand down-right: MOVE_RIGHT
  • Left and right hand down-middle: MOVE_FORWARD

Execute the program locally using WebStart.

System prerequisites:

  • Windows 7 or higher (32- or 64-bit)
  • Installation of the Kinect SDK from http://kinectforwindows.org
  • Kinect for Xbox 360 or Kinect for Windows (both devices are supported)
  • Lego NXT robot, Bluetooth enabled
  • Bluetooth link PC-NXT

(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)

If you don't have a Lego NXT at hand or for training purposes, you can drive a simulated robot using the NxtSim library that is based on the JGameGrid framework. We add a touch sensor that detects if the robot touches the channel border. When the border is hit, the game is over. It may be restarted by rasing a hand.

import ch.aplu.kinect.*;
import java.awt.*;
import ch.aplu.jaw.*;
import ch.aplu.nxtsim.*;
import ch.aplu.jgamegrid.*;
import javax.swing.JOptionPane;

public class KinectNxtSim implements TouchListener
{
  enum State
  {
    TURN_LEFT, TURN_RIGHT, MOVE_FORWARD, MOVE_BACKWARD, STOP
  };

  private String dllPath =
    Kinect.is64bit() ? "KinectHandler64" : "KinectHandler";
  private NxtRobot robot;
  private Gear gear;
  private GameGrid gg;
  private boolean isOver = false;
  private State state = State.STOP;

  public KinectNxtSim()
  {
    robot = new NxtRobot();
    gg = robot.getGameGrid();
    gg.setPosition(10, 40);
    gg.addStatusBar(40);

    gear = new Gear();
    robot.addPart(gear);
    gear.setSpeed(50);
    TouchSensor ts = new TouchSensor(SensorPort.S3);
    robot.addPart(ts);
    ts.addTouchListener(this);

    Kinect kinect = new Kinect(dllPath, "Policeman", 550, 40, 640 / 4, 480 / 4,
      NativeHandler.WS_BORDER | NativeHandler.WS_VISIBLE);

    if (!kinect.isInitialized())
    {
      kinect.setVisible(false);
      JOptionPane.showMessageDialog(null"Initializing of Kinect failed.");
      System.exit(0);
    }

    kinect.setWindowScaleFactor(4);
    Point3D[] joints = new Point3D[20]; // 20 skeleton joints
    for (int i = 0; i < 20; i++)
      joints[i] = new Point3D();
    gg.setStatusText("Waiting for valid skeleton...");
    kinect.getJoints(joints, 0);  // Blocks undefinitely
    int rc = 0;
    String msg;
    while (true)
    {
      if (rc != -1)  // Valid
      {
        Point3D leftHand = joints[SkeletonJoint.HAND_LEFT.ordinal()];
        Point3D rightHand = joints[SkeletonJoint.HAND_RIGHT.ordinal()];

        if (isOver && (rightHand.y < 100 || leftHand.y < 100)) // one hand up
        {
          robot.reset();
          isOver = false;
          continue;
        }

        if (rightHand.y < 100 && leftHand.y < 100)  // both hands up
          state = State.MOVE_BACKWARD;
        else if (rightHand.y < 100 || leftHand.y < 100) // one hand up
          state = State.STOP;
        else if (rightHand.x < 320)  // Right hand points to left
          state = State.TURN_LEFT;
        else if (leftHand.x > 320)  // Left hand points to right
          state = State.TURN_RIGHT;
        else  // both hands in neutral position
          state = State.MOVE_FORWARD;
        msg = "(xLeft, xRight, yLeft, yRight) = ("
          + leftHand.x + ", " + rightHand.x + ", "
          + leftHand.y + ", " + rightHand.y + ")";
        msg += "\nStatus: " + state
          + (isOver ? " (Border hit. Game Over. Raise a hand to restart.)" : "");
        gg.setStatusText(msg);
      }
      else
      {
        gg.setStatusText("Waiting for valid skeleton...");
        state = State.STOP;
        gear.stop();
        robot.reset();
      }
      if (!isOver)
        setGear();

      rc = kinect.getJoints(joints, 20);  // Blocks maximum 300 ms
    }
  }

  private void setGear()
  {
    switch (state)
    {
      case MOVE_FORWARD:
        gear.forward();
        break;
      case MOVE_BACKWARD:
        gear.backward();
        break;
      case TURN_LEFT:
        gear.leftArc(0.2);
        break;
      case TURN_RIGHT:
        gear.rightArc(0.2);
        break;
      case STOP:
        gear.stop();
        break;
    }
  }

  public void pressed(SensorPort port)
  {
    isOver = true;
    state = State.STOP;
    setGear();
  }

  public void released(SensorPort port)
  {
  }

  // ------------------ Environment --------------------------
  static
  {
    NxtContext.setLocation(10, 10);
    NxtContext.setStartDirection(0);
    NxtContext.setStartPosition(100, 240);
    NxtContext.useObstacle(NxtContext.channel);
  }

  public static void main(String args[])
  {
    new KinectNxtSim();
  }
}

Execute the program locally using WebStart.

System prerequisites:

  • Windows 7 or higher (32- or 64-bit)
  • Installation of the Kinect SDK from http://kinectforwindows.org
  • Kinect for Xbox 360 or Kinect for Windows (both devices are supported)
  • (No real Lego robot needed)

(If you install the KinectRuntime and not the full SDK, only the Kinect for Windows is supported.)

  kinectnxtsim

 


 

 
 
 
kinectdevice