01
Look naturally.
HEAD → HMD
Head motion guides the robot’s gaze. Stereo vision brings the robot’s view back to you.
HUMAN MOTION. HUMANOID POSSIBILITIES.
HITMEN.
A Hybrid Interface for Humanoid Teleoperation
with Multimodal Sensing and Exoskeleton Control
Department of Robotics, Kwangwoon University / Seoul, Republic of Korea
* Corresponding author
01 / THE INTERFACE
Your motion.
The robot’s reach.
Different parts of the body move differently. HITMEN assigns each one an interface that fits—so looking, reaching, leaning, and grasping become part of one continuous action.
02
Move your torso.
CHEST → VR CONTROLLERS
Chest-mounted controllers capture how you turn and lean, independently of your head.
03
Reach directly.
ARMS → LEADER ARMS
Shoulder-mounted leader arms map your joint motion directly to the robot’s arms.
04
Keep your touch.
HANDS → VR + RGB CAMERAS
Palm-facing wrist cameras keep fingers in view when headset hand tracking struggles.
FULL SYSTEM VIDEO
A closer look at HITMEN
Explore the interface and see the full system in action.
02 / ANTICIPATING OCCLUSION
Tracking fades.
Control carries on.
A hand turns. An arm crosses the chest. The headset’s view changes—and so does the quality of its tracking.
HITMEN learns from recent motion to anticipate the next second of hand and controller tracking reliability. When VR hand tracking becomes unreliable, finger control shifts toward the wrist-camera estimate.
Recent motionPredict reliabilityAdapt control
RELIABILITY ESTIMATION
Reading the signs of tracking loss
Watch the estimated reliability change as hand poses affect the headset’s view.
SIMULATION
Beyond a tracking failure
The simulated robot hand stays responsive even when the VR tracking state reports a failure.
03 / INDEPENDENT HEAD & TORSO
Look one way.
Lean another.
Your gaze should not decide every movement of your body.
HITMEN gives the head and chest separate targets. The neck follows the HMD; the waist follows the chest controllers. That separation lets you change your viewpoint while keeping your torso where the task needs it.
HORIZONTAL MOTION
Turn without turning everything
Head and torso rotation follow their own motion targets.
VERTICAL MOTION
Lean in. Keep your perspective.
Coordinate the waist and neck while preserving separate head and chest targets.
04 / HITMEN IN ACTION
Small details.
Wider possibilities.
From lining up a precise insertion to moving a hand beyond the headset’s tracking coverage, the interface makes more of the robot’s upper body available to the operator.
SIMULATION
A precise fit
Guide, align, and insert a peg into a hole in simulation.
BEYOND VR COVERAGE
Out of view. Still in control.
Palm-facing cameras support finger control beyond the headset’s hand-tracking range.
05 / SIDE-BY-SIDE BENCHMARKS
Same task.
Different interfaces.
Four selected manipulation tasks, shown across three interfaces. Watch how each setup handles coordination, occlusion, and delicate contact.
“Play all” starts the three clips together. Clips retain their original timing and are not aligned by task phase. Swipe each row to compare all three interfaces.