MR-GLi: Mixed Reality-Based Gripper-Linked Overlays for Underwater Robot Arm Teleoperation via Bilateral Control

Visual torque feedback supports underwater bilateral teleoperation, but the benefit of mixed reality (MR) over conventional monitor presentation remains unclear. We present MR-GLi, an MR interface that spatially registers a reaction torque indicator and wrist-camera image to the robot gripper. Twenty participants performed lift and pick-and-place tasks with rigid and compliant objects in a counterbalanced within-subject comparison with a 2D monitor, using identical visual-feedback content and four-channel bilateral control. MR-GLi provided gripper-linked access to visual feedback while maintaining a similar level of torque-regulation performance to the 2D monitor. Subjective evaluation further indicated reduced perceived burden associated with shifting attention between the workspace and visual feedback. These results demonstrate the feasibility of gripper-linked MR overlays for underwater bilateral teleoperation and highlight the importance of considering information access in addition to task performance.

The operator teleoperates an underwater robot arm through bilateral control while wearing a Quest 3 headset. Inside the headset, the reaction torque indicator (RTI) and the wrist-camera image are registered to the gripper and follow the arm as it moves (inset), so the grasping torque can be read without looking away from the manipulation site. In the 2D Monitor condition, the same visual information is shown on a display beside the tank.

The submerged follower arm and the leader arm operated by the user are coupled by four-channel bilateral control at about 1 kHz, which returns haptic feedback and estimates the gripper’s reaction torque. A PC converts that torque into the RTI and combines it with the wrist-camera stream. In MR-GLi, both are placed on a panel anchored to the gripper and viewed through the headset’s video passthrough; in the 2D Monitor condition, they appear on a monitor beside the leader device. The controller, RTI encoding and camera image are identical in both conditions.

Method

Robot Platform and Bilateral Control

Four-channel leader–follower bilateral control enforces position tracking and the action–reaction law at approximately 1 kHz on all four joints. Joint torque is estimated sensorlessly with a reaction torque observer (RTOB), and the displayed quantity is the gripper-joint reaction torque — the same signal used inside the bilateral loop.

Reaction Torque Indicator

The RTI encodes torque with two cues: bar length for magnitude and hue for state (blue below, green inside, red above the optimal band). The optimal band is 0.20–0.40 N·m around a target of 0.30 N·m, with linear blends at the thresholds. The encoding is identical in both conditions.

Gripper-Linked Overlays and Calibration

Panel poses come from the follower arm’s joint encoders: forward kinematics gives the gripper pose, and both panels sit at a fixed offset in the gripper frame, so the visual information follows the gripper as the arm moves. The canvas is turned toward the operator each frame so it stays legible at any gripper orientation. The robot-to-MR transform is set once per session by aligning a virtual arm model to the physical arm, then stored relative to the headset’s play-area boundary so it can be restored without repeating the alignment.

Experiments

A counterbalanced within-subject study with 20 participants (9 female, 11 male; mean age 22.0, SD 2.7). Each condition comprised Lift and Pick&Place on a rigid polyurethane block and a compliant cellulose sponge — 160 trials in total (20 × 2 conditions × 4 task–object combinations), logged at approximately 1 kHz.

TABLE I. Comparison of the two display configurations.

Torque Regulation

TABLE II. Experimental results. *Trial time was evaluated for Pick&Place only.

No significant difference in time within the optimal torque range (p = .546, r = .14) or in any other torque measure (all p ≥ .409). Torque regulation is preserved. Trial time was shorter with MR-GLi but not significantly (p = .053); because it also depended on condition order (p = .021), the difference may partly reflect familiarization with the task.

Subjective Measures

TABLE III. Subjective measures at participant level (n = 20), median [IQR]. Higher is better for SUS and the two five-point items, lower for NASA-TLX and its sub-scales.

Gaze-shift ease improved significantly (2.0 → 5.0, p = .002, r = .68), and mental demand also favored MR-GLi (p = .034, r = .48). Usability and overall workload showed no significant difference but numerically favored MR-GLi (SUS 77.5 vs 76.2; NASA-TLX 41.7 vs 49.5), and 14 of 20 participants found MR-GLi easier to use.

Summary

• MR-GLi registers the reaction torque indicator and wrist-camera image to the gripper, so visual feedback travels with the manipulation site instead of sitting on a separate monitor.
• Against a strong 2D-monitor baseline with identical bilateral control, RTI encoding and camera content, torque regulation stayed at a comparable level. Workload and usability showed no significant differences but numerically favored MR-GLi (SUS 77.5 vs 76.2; NASA-TLX 41.7 vs 49.5), indicating a tendency toward higher usability and lower perceived workload.
• The gain is in information access: perceived gaze-shift burden dropped significantly, suggesting that gripper-linked MR overlays can improve access to visual feedback without degrading torque-regulation performance.

Citation

@misc{sasago2026mrglimixedrealitybasedgripperlinked,
      title={MR-GLi: Mixed Reality-Based Gripper-Linked Overlays for Underwater Robot Arm Teleoperation via Bilateral Control}, 
      author={Masashi Sasago and Masato Kobayashi and Yuki Uranishi},
      year={2026},
      eprint={2609.16041},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2609.16041}, 
}

Contact

Masato Kobayashi (Assistant Professor, The University of Osaka, Kobe University, Japan)

• X (Twitter)
  • English : https://twitter.com/MeRTcookingEN
  • Japanese : https://twitter.com/MeRTcooking
• Linkedin https://www.linkedin.com/in/kobayashi-masato-robot/