AI summaryⓘ
The authors created MyoMechanix, a large dataset and system that looks at weightlifting actions using videos, 3D poses, and muscle activity signals together, which is more detailed than previous work that only used video or poses. They also built a Fitness Knowledge Graph to organize expert advice on actions and errors, helping the system understand and explain mistakes better. Their method, called CUBIST, uses these tools to give detailed feedback and achieved top performance in assessing action quality. They also explored using video to estimate muscle signals, which could reduce the need for expensive sensors. Overall, the authors aim to improve how machines understand and evaluate physical activities using multiple types of data and structured knowledge.
Action Quality Assessment (AQA)Surface Electromyography (sEMG)3D Pose EstimationMultimodal SensingFitness Knowledge GraphCompositional ReasoningVideo Question Answering (VideoQA)Muscle MechanicsBiomechanicsPhysical AI
Authors
Hao Yin, Paritosh Parmar, Lijun Gu, Lin Xu, Tianxiao Guo, Xiujin Liu, Tianyou Zheng, Yang Zhang, Weiwei Fu
Abstract
Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/