Working with Kinectv2 in Python

Time has not been kind to the Xbox One Kinect's Python libraries.  If you want depth images or skeletons but don't want to deal with C++/C#, you might be thinking you're out of luck --- you're not.

Interfacing with hardware isn't my specialty, so I opted to dig around for people who knew better.

The documentation for it seems scattered or out of date, but with some tinkering, it works.

I was able to get two methods working:

• PyKinect2
• PrimeSense with OpenNI2 

PyKinect2 is the better option if you want to easily get skeletons (why wouldn't you?).

• PyKinect2 is also the easier of the two to install -- especially if you are using a 32bit installation of Anaconda: it's on pip, but has some other prerequisites: comtypes, pygame (for the example), and the Windows KinectSDK 2.0.
• However, say you need Anaconda 64-bit: it will throw an error about a size not being 72, but 80.
• Personally, I have Anaconda 64 bit and refused to compromise.
• Here are two options: 
• Option 1) Clone the github version, which supposedly added support for 64-bits;
• (I read the GitHub Issues afterwards, and that was mentioned, but I didn't test it).
• Option 2) When you attempt "pip install pykinect2," it may fail (if it still hasn't been updated), 
• However, it will show the link to a zip file that it is caching 
• (you may have to run pip install twice for it to say: using cached file from "link").
• Copy the link and download/extract the zip file.
• Modify the PyKinectV2.py file: change the one and only 72 to 80.
• Using the Anaconda Prompt in Windows (or with the environment activated), navigate to the folder and run the setup.py file:
• python setup.py install
• I was able to run the skeleton example from PyKinect2's github.
• https://github.com/Kinect/PyKinect2/blob/master/examples/PyKinectBodyGame.py

from pykinect2 import PyKinectV2

from pykinect2.PyKinectV2 import *

from pykinect2 import PyKinectRuntime

import ctypes

import _ctypes

import pygame

import sys

if sys.hexversion >= 0x03000000:

import _thread as thread

else:

import thread

# colors for drawing different bodies

SKELETON_COLORS = [pygame.color.THECOLORS["red"],

pygame.color.THECOLORS["blue"],

pygame.color.THECOLORS["green"],

pygame.color.THECOLORS["orange"],

pygame.color.THECOLORS["purple"],

pygame.color.THECOLORS["yellow"],

pygame.color.THECOLORS["violet"]]

class BodyGameRuntime(object):

def __init__(self):

pygame.init()

# Used to manage how fast the screen updates

self._clock = pygame.time.Clock()

# Set the width and height of the screen [width, height]

self._infoObject = pygame.display.Info()

self._screen = pygame.display.set_mode((self._infoObject.current_w >> 1, self._infoObject.current_h >> 1),

pygame.HWSURFACE|pygame.DOUBLEBUF|pygame.RESIZABLE, 32)

pygame.display.set_caption("Kinect for Windows v2 Body Game")

# Loop until the user clicks the close button.

self._done = False

# Used to manage how fast the screen updates

self._clock = pygame.time.Clock()

# Kinect runtime object, we want only color and body frames

self._kinect = PyKinectRuntime.PyKinectRuntime(PyKinectV2.FrameSourceTypes_Color | PyKinectV2.FrameSourceTypes_Body)

# back buffer surface for getting Kinect color frames, 32bit color, width and height equal to the Kinect color frame size

self._frame_surface = pygame.Surface((self._kinect.color_frame_desc.Width, self._kinect.color_frame_desc.Height), 0, 32)

# here we will store skeleton data

self._bodies = None

def draw_body_bone(self, joints, jointPoints, color, joint0, joint1):

joint0State = joints[joint0].TrackingState;

joint1State = joints[joint1].TrackingState;

# both joints are not tracked

if (joint0State == PyKinectV2.TrackingState_NotTracked) or (joint1State == PyKinectV2.TrackingState_NotTracked):

return

# both joints are not *really* tracked

if (joint0State == PyKinectV2.TrackingState_Inferred) and (joint1State == PyKinectV2.TrackingState_Inferred):

return

# ok, at least one is good

start = (jointPoints[joint0].x, jointPoints[joint0].y)

end = (jointPoints[joint1].x, jointPoints[joint1].y)

try:

pygame.draw.line(self._frame_surface, color, start, end, 8)

except: # need to catch it due to possible invalid positions (with inf)

pass

def draw_body(self, joints, jointPoints, color):

# Torso

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_Head, PyKinectV2.JointType_Neck);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_Neck, PyKinectV2.JointType_SpineShoulder);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineShoulder, PyKinectV2.JointType_SpineMid);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineMid, PyKinectV2.JointType_SpineBase);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineShoulder, PyKinectV2.JointType_ShoulderRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineShoulder, PyKinectV2.JointType_ShoulderLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineBase, PyKinectV2.JointType_HipRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_SpineBase, PyKinectV2.JointType_HipLeft);

# Right Arm

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_ShoulderRight, PyKinectV2.JointType_ElbowRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_ElbowRight, PyKinectV2.JointType_WristRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_WristRight, PyKinectV2.JointType_HandRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_HandRight, PyKinectV2.JointType_HandTipRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_WristRight, PyKinectV2.JointType_ThumbRight);

# Left Arm

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_ShoulderLeft, PyKinectV2.JointType_ElbowLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_ElbowLeft, PyKinectV2.JointType_WristLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_WristLeft, PyKinectV2.JointType_HandLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_HandLeft, PyKinectV2.JointType_HandTipLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_WristLeft, PyKinectV2.JointType_ThumbLeft);

# Right Leg

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_HipRight, PyKinectV2.JointType_KneeRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_KneeRight, PyKinectV2.JointType_AnkleRight);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_AnkleRight, PyKinectV2.JointType_FootRight);

# Left Leg

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_HipLeft, PyKinectV2.JointType_KneeLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_KneeLeft, PyKinectV2.JointType_AnkleLeft);

self.draw_body_bone(joints, jointPoints, color, PyKinectV2.JointType_AnkleLeft, PyKinectV2.JointType_FootLeft);

def draw_color_frame(self, frame, target_surface):

target_surface.lock()

address = self._kinect.surface_as_array(target_surface.get_buffer())

ctypes.memmove(address, frame.ctypes.data, frame.size)

del address

target_surface.unlock()

def run(self):

# -------- Main Program Loop -----------

while not self._done:

# --- Main event loop

for event in pygame.event.get(): # User did something

if event.type == pygame.QUIT: # If user clicked close

self._done = True # Flag that we are done so we exit this loop

elif event.type == pygame.VIDEORESIZE: # window resized

self._screen = pygame.display.set_mode(event.dict['size'],

pygame.HWSURFACE|pygame.DOUBLEBUF|pygame.RESIZABLE, 32)

# --- Game logic should go here

# --- Getting frames and drawing

# --- Woohoo! We've got a color frame! Let's fill out back buffer surface with frame's data

if self._kinect.has_new_color_frame():

frame = self._kinect.get_last_color_frame()

self.draw_color_frame(frame, self._frame_surface)

frame = None

# --- Cool! We have a body frame, so can get skeletons

if self._kinect.has_new_body_frame():

self._bodies = self._kinect.get_last_body_frame()

# --- draw skeletons to _frame_surface

if self._bodies is not None:

for i in range(0, self._kinect.max_body_count):

body = self._bodies.bodies[i]

if not body.is_tracked:

continue

joints = body.joints

# convert joint coordinates to color space

joint_points = self._kinect.body_joints_to_color_space(joints)

self.draw_body(joints, joint_points, SKELETON_COLORS[i])

# --- copy back buffer surface pixels to the screen, resize it if needed and keep aspect ratio

# --- (screen size may be different from Kinect's color frame size)

h_to_w = float(self._frame_surface.get_height()) / self._frame_surface.get_width()

target_height = int(h_to_w * self._screen.get_width())

surface_to_draw = pygame.transform.scale(self._frame_surface, (self._screen.get_width(), target_height));

self._screen.blit(surface_to_draw, (0,0))

surface_to_draw = None

pygame.display.update()

# --- Go ahead and update the screen with what we've drawn.

pygame.display.flip()

# --- Limit to 60 frames per second

self._clock.tick(60)

# Close our Kinect sensor, close the window and quit.

self._kinect.close()

pygame.quit()

__main__ = "Kinect v2 Body Game"

game = BodyGameRuntime();

game.run();

If you just want depth, color, or infrared images without skeletons, PrimeSense + OpenNI2 will work.

You need to install OpenNI2 separately: https://structure.io/openni

You can pip install primesense... but for OpenNI2, you have to build the Kinect2 driver and place it into the correct folder.

• This address has the driver and project files you need to build: clone it or download the zip and extract it: https://github.com/occipital/OpenNI2/tree/kinect2
• Open it with Visual Studio (I used 2019, which was the newest version at the time of writing).
• Open the OpenNI.sln file: VS told me the "Install" project was out of date; ignore it -- it doesn't matter.
• Every sub-Project has its own stupid properties: you have to turn off "Treat Warnings as Errors" for the important ones (sometimes, you have to do it for both C++/C# and Linker in the same project):
• Devices/Kinect2
• DepthUtils
• OpenNI
• XnLib
• Build the solution file.
• Errors, errors everywhere, but you just want the Kinect2 driver to build.
• You can find the Kinect2 files under:
• OpenNI2-kinect2/Bin/x64-Debug/OpenNI2/Drivers/
• Copy at least the Kinect2.dll and pdb files to the OpenNI2 that you installed (instead, I did all of the Kinect2.* files because I didn't know any better).
• The correct Driver folder is under Redist/ (there are 3+ in various directories for some stupid reason, ignore them).
• ...OpenNI2/Redist/OpenNI2/Drivers/
• I copied the Kinect2.lib to OpenNI2/Lib -- I don't remember if I did it because it had .lib files or because I had an error otherwise.

I modified the example from PrimeSense to get all three streams sequentially.

I avoided using OpenCV (cv2) for imshow() because it led to an assertion error; matplotlib's imshow() worked.  Be sure to run it in something like VS Code so you can kill it easily.

import numpy as np

from primesense import openni2

from primesense import _openni2 as c_api

import pylab as plt

openni2.initialize("C:/Program Files/OpenNI2/Redist") # can also accept the path of the OpenNI redistribution

dev = openni2.Device.open_any()

print ("IR", dev.get_sensor_info(openni2.SENSOR_IR))

print ("DEPTH", dev.get_sensor_info(openni2.SENSOR_DEPTH))

print ("COLOR", dev.get_sensor_info(openni2.SENSOR_COLOR))

print(dir(dev))

infrared_stream = dev.create_ir_stream()

infrared_stream.start()

depth_stream = dev.create_depth_stream()

depth_stream.start()

color_stream = dev.create_color_stream()

color_stream.start()

def read_stream(stream):

frame = stream.read_frame()

img = None

if frame.sensorType == openni2.SENSOR_COLOR:

# color has 3 separate bytes for rgb

frame_data = frame.get_buffer_as_triplet()

img = np.array(np.frombuffer(frame_data, dtype=np.uint8), dtype=np.uint8)

# the frame width isn't necessarily accurate?

img.shape = (frame.height, -1, 3)

else:

frame_data = frame.get_buffer_as_uint16()

# the depth data is more comprehensive than 0-255 (0 to lots); 0 means it is

# invalid, and otherwise, bigger means farther away.

# returns mm distance, I think

# depth data supposedly has the player index too, which you can remove with

# bit shifting (I think it is 2 bytes?)

# distance = depthData >> DepthImageFrame.PlayerIndexBitmaskWidth

# player = depthData & DepthImageFrame.PlayerIndexBitmask

img = np.array(np.frombuffer(frame_data, dtype=np.uint16), dtype=np.uint16)

# the frame width isn't necessarily accurate?

img.shape = (frame.height, -1)

return img

while True:

infrared = read_stream(infrared_stream)

depth = read_stream(depth_stream)

color = read_stream(color_stream)

plt.imshow(depth, cmap='gray')

plt.show()

plt.imshow(infrared, cmap='gray')

plt.show()

plt.imshow(color)

plt.show()

infrared_stream.stop()

depth_stream.stop()

color_stream.stop()

openni2.unload()

Supposedly, you could use OpenNI2 for more, but I didn't look into the docs because PyKinect2 did what I wanted.

Creating Models From 2D Views

Process:

• Have the camera's relative position and orientation for each snapshot (might not be needed later) for stationary object(s).

• For each snapshot:
• Separate surfaces so you can have a mask for each surface (that's a long post for another day).

• Get the border/edges (white, but touching black in a cardinal direction -- north, south, east, west):

• Obtain the order of the surfaces' vertices (you might have multiple vertex chains if you have a complex shape -- like a torus).  
• You can create chains of vertices that only link pixels when they are within a short distance (1 pixel if x's or y's are equal, sqrt(2) pixels if they aren't).

• Define triangular planes between each pair of sequential vertices that connect to the camera's position.
• Extend the planes farther out from the camera, but it should still be orthogonal to the same normal (move it at least twice as far).

• If vertex chains are a loops, each chain defines an in and out region based on which side is in/out of the mask.
• Counting outside in, you can treat the odd numbered chains as filling on the inside and even numbered chains as filling their outsides.
• Adjusting for movement of the camera between snapshots, rotate the vertices around Z (or whatever your "Up" axis is).

• Get the intersection of the in and out regions to obtain a 3D object.

• If you use Blender's Boolean Intersection, I recommend:
• using "Carve",
• using Dissolve Degenerate between intersection steps,
• Adding faces (F)
• Automating it with Python! (there are a lot of intersection steps!)
• Note: the Boolean modifier can be very finicky if you use it many times on one object -- I ended up using only half of the camera steps just to avoid the shape disappearing completely.
• Alternatively, you can intersect the shapes in pairs to minimize the number of Boolean operators for any one shape (by spreading them out), which let me use all of my camera angles.

• For round objects, more perspectives will lead to more precision (ideally, you could loop around the whole axis of the round object with the camera).
• If you're up for some post processing on the input images, you could smooth the vertex chains to remove the vertices that create that staircase effect in some regions, but you'd have to pick an arbitrary threshold for what constitutes a staircase.
• (what if the object really does look like a staircase? You'd smooth it away)
• Because of the perspective warping, you'll always need to take pictures in either pairs or 4's.
• 2 pictures 180 degrees apart or 4 pictures every 90 degrees (depending on how much symmetry or roundness there is).
• Concave bowls (like a 360 degree lathed U) are practically impossible to recreate with these simple masking images from the outside (you'd have to put the camera inside with many positions instead of orbiting it).
• If you really wanted to get recesses, you can take advantage of:
• Motion video: obtain topology info by observing how distinctively colored marble(s) move along surfaces.
• Spotlights: obtain topology info by seeing how circular lights warp as they move along a surface.
• Projectors: project a grid onto the surface to observe the warping.