CoGAME: Manipulation by Projection

An example of an application enhanced by the "manipulation-by-projection" technique, this cooperative game allows players to visually and intuitively control a robot with projectors. Players interchangeably move and connect their projected images to create a path that leads the robot to its goal.

Enhanced Life

With a handheld projector, players detect robot states and surroundings, deliver commands to the robot, and confirm the commands and robot reactions. At the same time, other players can monitor the information and interact with the robot without conflicts.

CoGAME enables a simple robot with only a few sensors and interfaces to perform many intelligent functions. As the system recognizes robot states and locations, it plans robot movements, and the robot behaves as if it is independently well-adapted to the environment.


To enable users to accurately, intuitively, and visually manipulate a variety of movable objects such as robots and toys and share their manipulation with other users.


Each component of the CoGAME system is based on existing technology, but the application and the combination of components are innovative. It includes two novel components for intuitive, visual robot control:
  • Images are constantly corrected with a tilt sensor to detect the projector's position and orientation, and to enhance the robot's image comprehension.

  • To lead a robot along a path on a projected display, a camera attached to a mobile device detects and estimates its position and orientation. Then the system transforms these data from a camera image plane to a corrected image plane and generates a robot-movement plan.


The CoGAME interface could be used in a wide range of applications in a robot-rich future. For example, a user could project a furniture layout onto a real office environment, and CoGAME-controlled robots could move and place the furniture by following the projected image.


Kazuhiro Hosoi
The University of Tokyo
Hosoi (at)


Vinh Ninh Dao
Akihiro Mori
Masanori Sugimoto
The University of Tokyo