ToF cameras for Activity recognition

ToF cameras can be used for efficiently tracking rigid and non-rigid objects. Object detection, segmentation and tracking can be easily performed by inspecting distance and amplitude data. The following scenes show a person filmed from the ceiling (camera pointing downwards, distances are color-coded from green=near to blue=far):

 

Detected person with open hand grabbing   Detected person with fist

The detected blobs can be used in combination with 3D bounding boxes to detect activity in well-defined regions in space.

For activity recognition at finer scale, the blobs can be abstracted further: The following picture show the object outline(red) and an overlay of the object skeleton in white before pruning; the structural differences between open and closed hands are clearly visible.

Detected person with open hand grabbing   Person overlaid with skeleton; skeleton key points belonging to extremities are filled

Keypoints of the skeleton can be classified very efficiently by incorporating thickness and height information of the detected regions.

The object tracking and analysis middleware is hosted on an embedded platform for easy deployment, relevant events can be delivered to a service backend via Wifi. Our development systems are based on a Wandboard platform with optimized Linux firmware and OpenCV 3.0 stacks, so the framework can be tailored to specific needs quickly. The current development kit comes preconfigured with a Yocto-based image with native C++ compiler toolchain.

Person grabbing into bounding box

Possible applications include

  • Activity statistics
  • Geometry-based recommender systems
  • Asset surveillance & security
  • Crowd statistics (height, gait, occupancy)
  • Energy / Smart room control
  • Attention analysis

 

Contact us for more information any time on further details.

Images: 
Person skeleton keypoints classified (hand/arm-related points are filled)
Detected person - stretched arm with fist
Stretched arm with fist - skeleton overlaid
Detected person - stretched arm with open hand
Stretched arm with open hand - skeleton overlaid
Ceiling-mounted ToF sensor with DevKit
Person grabbing into bounding box