MR-UBi: Mixed Reality-Based Underwater Robot Arm Teleoperation System with Reaction Torque Indicator via Bilateral Control

We present a mixed reality-based underwater robot arm teleoperation system with a reaction torque indicator via bilateral control (MR-UBi). The reaction torque indicator (RTI) overlays a color and length-coded torque bar in the MR-HMD, enabling seamless integration of visual and haptic feedback during underwater robot arm teleoperation. User studies with sixteen participants compared MR-UBi against a bilateral-control baseline. MR-UBi significantly improved grasping-torque control accuracy, increasing the time within the optimal torque range and reducing both low and high grasping torque range during lift and pick-and-place tasks with objects of different stiffness. Subjective evaluations further showed higher usability (SUS) and lower workload (NASA–TLX). Overall, the results confirm that \textit{MR-UBi} enables more stable, accurate, and user-friendly underwater robot-arm teleoperation through the integration of visual and haptic feedback.

---

Why MR-UBi?

Underwater manipulation suffers from ambiguous force perception (drag, poor visibility, limited training). Bilateral teleoperation helps—but precisely feeling torque underwater is still difficult. MR-UBi closes this gap by putting torque into sight while you feel it in hand.

---

Related Work Comparison

The table below summarizes representative teleoperation interfaces for both land and underwater environments, highlighting differences in control architecture, feedback modality, and visualization strategies.
MR-UBi uniquely integrates mixed-reality visual augmentation with bilateral haptic feedback for underwater teleoperation.

What’s New

• MR + Bilateral Control: 3-DoF leader–follower arm with gripper and 4-channel bilateral control at 1 kHz.
• Reaction Torque Indicator (RTI): A bar in the MR passthrough view encodes torque magnitude (length) and state (hybrid continuous–discrete color).
• Practical & low-cost: Each underwater arm ≈ USD 3,400 (Dynamixel XW540-T260, IP68).
• Validated with users (n=16): Significant gains in torque regulation, higher SUS, lower NASA–TLX.

---

System Overview

MR-UBi consists of:

1. Underwater robot arm (3-DoF + gripper)
2. Leader arm with 4-channel bilateral control (position + force control) @ 1000Hz
3. RTI module integrated via the Unity-based MR-HMD passthrough pipeline

Underwater Robot Arm (MR-UBi Arm)

The MR-UBi system employs a 3-DoF underwater robotic arm with a two-finger gripper as both the leader and follower system.
Each joint is driven by waterproof Dynamixel XW540-T260 actuators (IP68), allowing direct operation in submerged environments without additional sealing hardware.

The mechanical structure combines aluminum profiles and 3D-printed components, achieving both rigidity and buoyancy balance.
This modular design ensures quick assembly and maintenance, while maintaining smooth backdrivability essential for bilateral control.
The per-arm material cost is approximately USD 3,400, offering a low-cost yet high-performance platform for underwater manipulation research.

---

Bilateral Control @ 1 kHz

We use four-channel bilateral control:

• Position tracking: θₗ − θ_f = 0
• Action–reaction torque: τₗ + τ_f = 0
• Sensorless torque via a Reaction Torque Observer (RTOB)

As shown above, the leader–follower pair maintains tight position synchronization while satisfying the action–reaction torque relationship.
This demonstrates that MR-UBi’s bilateral control enables precise and stable haptic transmission even in the underwater domain.

---

Reaction Torque Indicator (RTI)

The Reaction Torque Indicator (RTI) is the core visual–haptic interface of MR-UBi.
It augments the operator’s haptic perception by rendering underwater reaction torque directly in the MR field of view, allowing intuitive and continuous awareness of grasping force.

Unlike conventional numeric or discrete-color indicators, MR-UBi introduces a hybrid continuous–discrete encoding scheme that integrates quantitative torque magnitude and qualitative torque state into a single, glanceable bar.

---

🔹 Design Concept

• Bar length represents the normalized torque magnitude in real time.
  The visual filling ratio increases proportionally to the sensed torque, enabling the user to intuitively gauge relative strength.

• Color hue encodes grasping state (hybrid continuous–discrete encoding):

  • Blue → under-grasp (low torque region)
  • Green → optimal grasp (safe torque range)
  • Red → over-grasp (excessive torque)
    Smooth gradients interpolate between these hues, creating a natural transition around range boundaries and preventing abrupt perception shifts.

• Spatial anchoring:
  The RTI is fixed at the bottom-center of the MR-HMD passthrough, minimizing gaze shift and maintaining alignment with the real-world scene.
  This ensures the operator can perceive torque variations without breaking situational awareness.

---

🔹 Visual–Haptic Synergy

By combining bilateral haptic feedback (felt torque) and visual torque cues (RTI), this multimodal fusion helps operators:

1. Rapidly identify when grasping torque deviates from the optimal band,
2. Smoothly modulate applied force during contact transitions, and
3. Maintain stability across objects of varying stiffness.

---

🔹 Key Advantage

The RTI achieves perceptual reinforcement:
it strengthens the user’s proprioceptive and haptic sense through a synchronized visual mapping, enabling significantly more stable and confident manipulation.
As demonstrated in our user study, operators using MR-UBi spent over twice as much time within the optimal torque range and exhibited lower mean torque error compared with the baseline bilateral system.

---

Experimental Procedure

We conducted a within-subject experiment comparing the Baseline and MR-UBi systems.

Objectives

1. Evaluate torque control accuracy and stability
2. Evaluate usability and workload (SUS, NASA–TLX)

Conditions

Component	Baseline	MR-UBi
Bilateral control	✅	✅
MR-HMD passthrough	✅	✅
RTI visual overlay	❌	✅

---

Experimental Environment

• Leader: Desk-mounted 3-DoF arm + Quest 3 MR-HMD
• Follower: Underwater arm fixed in a water tank
• Control Loop: 1 kHz bilateral control, Unity for visualization

RTI Parameters:
T_min = 0.00 Nm, T_max = 0.60 Nm
Optimal band = [0.20, 0.40] Nm (target 0.30 Nm), margin 0.1 Nm
Colors = Blue → Green → Red

---

Experimental Phases

1. Preparation Phase (15 min)

• Participants receive task explanation and safety briefing.
• They practice bilateral operation without MR-HMD using a training object (different stiffness).
• Ensures safe operation and baseline familiarity.

2. Task Phase (30 min × 2 conditions)

Participants perform both Baseline and MR-UBi conditions in counterbalanced order.

Tasks:

1. Lift – grasp and lift a target object vertically.
2. Pick & Place – lift, move horizontally, and place the object.

Objects:

• Rigid polyurethane block
• Compliant cellulose sponge

During Trials:

• In Baseline, numeric torque is shown only during calibration.
• In MR-UBi, the RTI remains visible throughout the task.
• After each condition, participants complete SUS and NASA–TLX questionnaires.

3. Final Phase (5 min)

• Participants complete an overall comparison survey and give free comments.

---

Participants

Sixteen participants (3 female, 13 male) took part in the study.
Four had no MR experience, eleven had limited experience, and one was a frequent MR user.
Only one participant had previous experience operating another robotic arm.

---

Quantitative Results: Torque Distribution

MR-UBi consistently compressed torque variance and maintained values within the optimal band (0.20–0.40 Nm),
indicating more stable and precise force modulation compared with Baseline.

Across all conditions, MR-UBi reduced both under- and over-grasping occurrences, halving MAE on average.

---

Subjective Evaluation

Participants reported higher confidence, stability, and comfort with MR-UBi, describing the RTI as “intuitive” and “visually reassuring.”

---

Citation

@misc{nishi2025mrubi,
    title={MR-UBi: Mixed Reality-Based Underwater Robot Arm Teleoperation System with Reaction Torque Indicator via Bilateral Control},
    author={Kohei Nishi and Masato Kobayashi and Yuki Uranishi},
    year={2025},
    eprint={2510.20407},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Contact

Masato Kobayashi (Assistant Professor, The University of Osaka, Kobe University, Japan)

• X (Twitter)
  • English : https://twitter.com/MeRTcookingEN
  • Japanese : https://twitter.com/MeRTcooking
• Linkedin https://www.linkedin.com/in/kobayashi-masato-robot/
  
  * Corresponding author: Masato Kobayashi