Gesture-Based Human-in-the-Loop Control of Space Exploration Vehicle Convoys

Abstract

The deployment of multi-robot systems for planetary surface operations is essential as space exploration initiatives grow in scale and complexity. Coordinated convoys of rovers hold promise for a range of critical missions, including scientific exploration, material transport, infrastructure construction, and emergency response. However, enabling robust inter-rover communication and dynamic coordination in uncertain, unstructured environments presents technical and human factors challenges. This research is aimed at the development of a gesture-based control architecture tailored for human-in-the-loop space exploration vehicle convoys. The system demonstrates a framework in which a lead vehicle is driven by the human operator using a conventional steering wheel, while a trailing rover receives high-level commands via hand gestures detected through a wearable technology. This approach leverages intuitive interactions to minimize cognitive load while maintaining operational efficiency. When gesture input is not actively provided, the semi-autonomous rover defaults to a reactive mode, synchronizing its speed and trajectory with the lead vehicle to preserve formation and maintain convoy integrity. A Microsoft HoloLens 2 Heads-Up Display (HUD) provides contextual information to the operator via augmented reality (AR) visualizations. A real-time cognitive workload estimation module integrates cardiovascular physiological signals-specifically heart rate and heart rate variability-into a fuzzy logic framework for shared control in convoy operations. This enables real-time adaptation of vehicle control authority based on the driver’s cognitive state. Effectiveness of this novel interface design has been validated in simulation and hardware utilizing small scale autonomous vehicles.