OpenCV

Overview

This article shows how to convert OpenNI2 video frames to OpenCV Mat format. Once you have frame in Mat format, you can leverage all OpenCV functions.

Expect Output

Tutorial - C++

• After connect to camera and start each sensor (color, depth and IR). Read the frame into VideoFrameRef

VideoFrameRef colorFrame;
VideoFrameRef depthFrame;
VideoFrameRef irFrame;

color.readFrame(&colorFrame);
depth.readFrame(&depthFrame);
ir.readFrame(&irFrame);

• Convert VideoFrameRef to cv::Mat

// Color frame
cv::Mat colorMat = cv::Mat(colorFrame.getHeight(), colorFrame.getWidth(), CV_8UC3, (void *)colorFrame.getData());

// Depth frame
cv::Mat depthMat = cv::Mat(depthFrame.getHeight(), depthFrame.getWidth(), CV_16UC1, (void *)depthFrame.getData());

// IR frame
cv::Mat irMat = cv::Mat(irFrame.getHeight(), irFrame.getWidth(), CV_16UC1, (void *)irFrame.getData());

• For better visualization, you can do the following conversion

// For color frame
cv::cvtColor(colorMat, colorMat, cv::COLOR_BGR2RGB);

// For depth frame
depthMat.convertTo(depthMat, CV_8U, 255.0 / 1024.0);
cv::applyColorMap(depthMat, depthMat, cv::COLORMAP_JET);

// For IR frame
irMat.convertTo(irMat, CV_8U, 255.0 / 1024.0);

Tutorial - Python

• After connect to camera and start each sensor (color, depth and IR). Read the video frames

colorFrame = colorStream.read_frame()
depthFrame = depthStream.read_frame()
irFrame = irStream.read_frame()

• Convert video frame to OpenCV format which is numpy array

# Color frame
colorMat = np.fromstring(colorFrame.get_buffer_as_uint8(), dtype=np.uint8).reshape(480, 640, 3)

# Depth frame
depthMat = np.fromstring(depthFrame.get_buffer_as_uint16(), dtype=np.uint16).reshape(480, 640)

# IR frame
irMat = np.fromstring(irFrame.get_buffer_as_uint16(), dtype=np.uint16).reshape(480, 640)

• For better visualization, you can do the following conversion

# For color frame
colorMat = cv2.cvtColor(colorMat, cv2.COLOR_BGR2RGB)

# For depth frame
depthMat = np.uint8(depthMat.astype(float) * 255 / 1024)
depthMat = cv2.applyColorMap(depthMat, cv2.COLORMAP_JET)

# For IR frame
irMat = np.uint8(irMat.astype(float) * 255 / 1024)

Full code

C++

Python