Interactive Object Segmentation

Jeannette Bohg, from the Computer Vision and Active Perception Lab, Centre for Autonomous Systems at the Royal Institute of Technology (KTH) in Stockholm, visited us this spring to work on integrating an object segmentation method into the interactive manipulation framework. Object segmentation is the task of identifying and outlining individual objects in a potentially cluttered scene. One of its applications is in the context of grasping and manipulation, where an object is first segmented and potentially recognized before it is manipulated.

The interactive segmentation is available as a plugin for the rviz simulator. A user can click on objects or select a region around them to provide the segmentation with initial information about the number of objects in the scene, their rough position and appearance. From there, the segmentation is autonomously refined producing individual object point clouds.

"Under the hood" of this user interface is an iterative two-stage algorithm that uses RGB and disparity information. In the first stage it performs pixel-wise labeling based on the current model parameters and then updates these parameters in the second stage. The algorithm can simultaneously optimize the segmentation for background, a supporting surface and a set of foreground objects. A CPU as well as a GPU based implementation are available.

Unlike purely geometric approaches, such as Euclidian segmentation, this method can better handle cluttered scenes and correctly separate objects from supporting planes. The method has also been adapted to cope with incomplete color information and tested using the MS Kinect sensor.

The core segmentation method has been developed by Marten Björkman; for more details see Björkman et al., "Active 3D scene segmentation and detection of unknown objects", ICRA 2010. The GPU and CPU implementations of the segmentation algorithm are available in the active_realtime_segmentation ROS package. The interactive segmentation GUI is available in the object_segmentation_gui package. For more details on these packages, see their documentation pages on