Willow Garage Blog

October 4, 2010

We loved seeing our Texai robot (aka Shel-bot) on CBS' Big Bang Theory and hope you did too. It even got to share some screentime with Woz! Here's a clip in case you missed it:

September 28, 2010

Steven Gould from Stanford University interned this summer at Willow Garage. While some companies have their interns spend time cleaning up after their co-workers, Steven spent his time developing computer vision models for detecting objects in cluttered scenes -- helping robots learn how to clean up.

One of the challenges for computer vision is identifying objects when they are partially obscured by another object, such as the items partially obscured behind your laptop. Reality can be messy, but if we expect robots to work with everyday objects in real environments it's a challenge that must be met.

Standard parts-based object detection models directly assign a cost to matching each object part to the corresponding pixels in an image. This is problematic when objects can be partially occluded and the image pixels corresponding to a part's location belong to another object (i.e., the occluding objects).

Steven worked on building an occlusion-aware parts-based model that reduces the penalty for mismatch parts if those parts are obscured. This was accomplished by associating an occlusion variable with each part that indicates whether a part is visible or not. This variable can depend on other objects in the image so that once an object is found at a particular location the model encodes the pixels associated with that object as possibly occluding pixels for other objects. In this way, mismatch object parts are not heavily penalized as they are in the standard approach.

The same reasoning can be applied to truncated objects -- objects that are cut-off by the edge of the image since the parts not within the camera view can be simply treated as occluded.

Steven's work makes progress towards the goal of finding objects in everyday, cluttered environments. For more information, please see his slides below (download as PDF), or read his research on this topic. Code for this work will be integrated in future versions of the open-source OpenCV library.

September 21, 2010


ROS has really taken to the skies, so for this sixth installment it's all about micro aerial vehicles (MAVs):

Previously Part I, Part II, Part III, Part IV, Part V.

September 20, 2010

Marius Muja from University of British Columbia returned to Willow Garage this summer to continue his work object recognition. In addition to working on an object detector that can scale to a large number of objects, he has also been designing a general object recognition infrastructure.

One problem that many object detectors have is that they get slower as they learn new objects. Ideally we want a robot that goes into an environment and is capable of collecting data and learning new objects by itself. In doing this, however, we don't want the robot to get progressively slower as it learns new objects.

Marius worked on an object detector called Binarized Gradient Grid Pyramid (BiGGPy), which uses the gradient information from an image to match it to a set of learned object templates. The templates are organized into a template pyramid. This tree structure has low resolution templates at the root and higher resolution templates at each lower level. During detection, only a fraction of this tree must be explored. This results in big speedups and allows the detector to scale to a large number of objects.

Marius also worked on a recognition infrastructure that allows object detection algorithms to be dynamically loaded as C++ plugins. These algorithms can easily be combined together or swapped one for another. The infrastructure also allows very efficient data passing between the different algorithms using shared memory instead of data copying. This recognition infrastructure is useful for both users and researchers -- they can experiment with different algorithms and combine them together in a system without having to write additional code.

The code and documentation for the binarized gradient grid pyramid object detector (bigg_detector) and the recognition infrastructure (recognition_pipeline) are available on ros.org. Slides from Marius' end-of-summer presentation can be viewed below or downloaded from ROS.org.

September 16, 2010

Robots Using ROS

This fifth edition of Robots Using ROS features companies fully supporting ROS and robots made from basic building blocks.

  • Meka Robots: Meka has integrated their M3 realtime system with ROS so that all of their lightweight, force-controlled robots -- hands, arms, torsos, and complete systems -- can be used with ROS software.
  • Thecorpora Qbo: Qbo is a personal, open-source robot being developed on top of ROS and OpenCV.
  • Mini-PR2: The ModLab at Penn has constructed a "Mini-PR2" out of CKBot modules.
  • Lego NXT: You can now connect Lego Mindstorms NXT robots to ROS and even convert models from Lego Digital Designer.

Previously Part I, Part II, Part III, Part IV

September 15, 2010


Research scientist Matei Ciocarlie was recently honored with the Robotdalen Scientific Award for his 2010 PhD dissertation on robotic grasping at Columbia University. Since graduating, Matei has joined the Willow Garage research team, where he is continuing to contribute to the wider scientific community through his research, publications, and open source ROS code. Together with the rest of the manipulation team, he has released the object manipulation pipeline for the PR2 robot, and is currently working on probabilistic methods for reliable grasp planning and identification of graspable object features in sensor data. 

Matei’s dissertation, “Low-Dimensional Robotic Grasping: Eigengrasp Subspaces and Optimized Underactuation,” focused on the following central question: how can we reduce the complexity of robotic grasping without compromising versatility? The thesis proposes using low-dimensional hand posture subspaces, that can act as planning spaces (for dexterous robotic hands), interaction spaces (for hand prosthetics) and design spaces (for adaptive underactuated hands). 

The purpose of the Robotdalen Scientific Award is to encourage young, innovative people all over the world in all aspects of robotics, finding new and untried approaches for the future. 

Robotdalen Scientific Award

September 15, 2010

Helen Oleynikova, a student at Olin College of Engineering, spent her summer internship at Willow Garage working on improving visual SLAM libraries and and integrating them with ROS. Visual SLAM is a useful building block in robotics with several applications, such as localizing a robot and creating 3D reconstructions of an environment.

Visual SLAM uses camera images to map out the position of a robot in a new environment. It works by tracking image features between camera frames, and determining the robot's pose and the position of those features in the world based on their relative movement. This tracking provides an additional odometry source, which is useful for determining the location of a robot. As camera sensors are generally cheaper than laser sensors, this can be a useful alternative or complement to laser-based localization methods.

In addition to improving the VSLAM libraries, Helen put together documentation and tutorials to help you integrate Visual SLAM with stereo camera data on your robot. To find out more, check out the vslam stack on ROS.org. For detailed technical information, you can also check out Helen's presentation slides below.

September 13, 2010

Jon Scholz, a summer intern from Georgia Tech, spent his summer figuring out how to allow the PR2 to navigate while pushing carts and other wheeled objects. After working with the Cart Hackathon team to quickly prototype a cart-pushing solution, Jon worked on the more difficult task of making these capabilities robust.

The ability to push mobile objects is critical for a large number of service and utility tasks, ranging from healthcare to warehouse automation to custodial work. Although the PR2 has all the physical capabilities to navigate with carts, wheelchairs, stretchers, and tables, all these tasks require careful planning to ensure robust and smooth navigation in cluttered environments. Jon's work focused on the design of components for collision-free maneuvering of these objects with the PR2's arms, and combining this ability with robot navigation for robust planning in indoor environments.

The PR2 can now efficiently push a utility cart throughout the building, and researchers at Willow Garage are continuing to expand the range of objects with which the robot can navigate. Details about the project, including open source code, can be found here. For a technical overview, you can view Jon's slides below.

September 13, 2010

Joe Romano, a PhD student from the Penn Haptics Group, spent the summer at Willow Garage enabling the PR2 to perform fine motor control with its grippers using tactile information, as well as understand a variety of other tactile cues. Joe's approach attempts to recreate the same sensory information that humans use when completing similar tasks. This information has already been used to allow the robot to quickly stop after contact before damaging itself or the environment, naturally pass objects to a person, or detect when an object has been set down on a tabletop.

The goal of this work was to allow the robot to understand the sensory information that arises from contact with the environment, intelligently reason about what is happening in the world, and decide what the appropriate robot response should be.  Using this tactile approach, the PR2 can delicately grasp a wide range of unknown objects, such as raw fruit, eggs, and heavy liquid-filled containers, all without crushing or dropping them. The PR2 can also quickly detect contacts between its arms and objects in the world, as well as contact between hand-held objects and the world.

The sensory-signals that people receive can be mimicked by processing information from pressure-sensors in the fingertips of the PR2 and accelerometers mounted in the palm of the robot. These signals can then be used to do tactile-event-driven control of the robot, such as pushing a button until it clicks, or grasping an object tightly enough until it stops slipping out of the gripper.

To learn more about Joe's work, click here. For more technical information, please see Joe's presentation slides below.

September 9, 2010

Group photo

Our PR2 robots have been out at the PR2 Beta Sites for only a few short months and they have been busy with research projects, developing new software libraries for ROS, and creating Youtube hits. We held our first PR2 Beta Program Conference call to highlight their work, and the list of software that they have released as open source is already impressing us.

We've compiled a partial list of this software below so that all the ROS users and researchers can try it out and be involved. You'll find many more libraries in their public code repositories, and there is much more coming soon.

Georgia Tech

KU Leuven


  • EusLisp: now available under a BSD license
  • ROS/PR2 integration with EusLisp: roseus, pr2eus, and euscollada
  • jsk_ros_tools: includes rostool-alias-generator (e.g. rostopic_robot1) and jsk-rosemacs (support for anything.el)


  • knowrob: tools for knowledge acquisition, representation and reasoning
  • CRAM: reasoning and high level control for Steel Bank Common Lisp (cram_pl) and executive that reasons about locations (cram_highlevel)
  • prolog_perception: logically interact with perception nodes
  • pcl: contributions include pointcloud_registration, pcl_cloud_algos, pcl_cloud_tools, pcl_ias_sample_consensus, pcl_to_octree, mls



  • towel_folding: version of Towel Folding from pre-PR2 Beta Program that relies on two Canon G10 cameras mounted on chest. Uses optical flow for corner detection.
  • LDA-SIFT: recognition for transparent objects
  • Utilities:
    • pr2_simple_motions: Classes for easy scripting in Python of PR2 head, arms, grippers, torso, and base
    • visual_feedback: Streamlined image processing for 3d point extraction and capturing images
    • stereo_click: Click a point in any stereo camera feed and the corresponding 3d point is published
    • shape_window: Provides a highgui-based interface for drawing and manipulating 2D shapes.


  • iSAM: Incremental Smooth and Mapping released as LGPL.


  • OIT: Overhead interaction toolkit for tracking robots and people using an overhead camera.
  • deixis: Deictic gestures, such as pointing


  • articulation: (stable) Fit and select appropriate models for observed motion trajectories of articulated objects.
  • Contributions to pcl, including range image class and border extraction method


  • wviz: Web visualization toolkit to support their PR2 Remote Lab. Bosch has already been able to use their Remote Lab to collaborate with Brown University, and Brown University has released a rosjs to access ROS via a web browser.