Artificial-Intelligence-Assisted Multi-Modal Terahertz Sensing and Environment Reconstruction

Abstract

Multi-modal sensing is an important enabler for future environment-aware wireless systems, since a single sensing modality is generally insufficient to provide accurate metric geometry, material awareness, and semantic interpretability in complex environments. This paper presents a measurement-based multi-modal THz sensing and vision framework for indoor environment reconstruction. A three-dimensional monostatic THz channel sounding system operating at 290-310 GHz is integrated with an omnidirectional fisheye camera to acquire radio-frequency and visual observations from a common sensing viewpoint. From the measured THz data, a signal processing pipeline extracts multipath components and infers geometryand material-consistent structural primitives through trajectory tracking-assisted parameter estimation, graph-based structure discovery, planar reconstruction, and reflection-loss analysis. In parallel, AI-based visual perception modules extract object-level semantic masks and depth priors from panoramic images. To associate these heterogeneous representations, an agentic-AI-based task-driven THz-agent module is developed to select appropriate integration tools according to the attributes of the modality-specific outputs. Through angular alignment and consistency analysis, THz-derived metric geometry and material information are associated with vision-derived semantic regions and depth priors, enabling geometry-consistent and semantically interpretable environment reconstruction directly from measurements. Experimental validation in the indoor L-shaped hallway demonstrates that the proposed framework reconstructs dominant structural elements with centimeter-level accuracy while identifying semantic categories and material attributes of representative indoor objects.