MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching

Noosaga Placements

• Knowledge Graphsframework95%
  The paper constructs the Fitness Knowledge Graph (FKG) to structure expert knowledge about actions, phases, errors, and feedback, which is a direct application and extension of knowledge graph techniques.
  We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedbackFKG recasts fitness actions as structured procedures, enabling fine-grained error reasoning, feedback generation, and interpretable scoring
• Computer Visionsubfield95%
  The paper's primary focus is on action quality assessment from video, a core computer vision task, and it introduces a new video-based benchmark and model (CUBIST) for this purpose.
  Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and posewe introduce MyoMechanix, a multimodal ecosystem for weight-loaded actionsCUBIST achieves state-of-the-art results
• Deep Learning and End-to-End Representation Learningframework90%
  The paper's CUBIST model and baselines are deep learning models for video understanding, trained end-to-end for AQA, which falls under deep learning and end-to-end representation learning.
  CUBIST (Compositional Ontological Reasoning Engine), a framework that performs decomposition–analysis–recompositionwe selected 7 representative SOTA heterogeneous network architectures—including I3DMLP, CoRe, TPT, HGCN, MCoRe, T2CR, and DAE
• Knowledge Representation Reasoningsubfield85%
  The paper constructs the Fitness Knowledge Graph (FKG) to structure expert knowledge about actions, phases, errors, and feedback, and uses it for compositional scoring and reasoning, which is a key contribution.
  We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedbackFKG recasts fitness actions as structured procedures, enabling fine-grained error reasoning, feedback generation, and interpretable scoring
• Supervised Deep Learningframework80%
  The paper trains models for AQA, VideoQA, and Video2EMG using supervised learning with expert-annotated labels, including error types, scores, and QA pairs.
  Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjectsStage II introduces an asymmetric loss function to guide gradient backpropagationwe perform supervised fine-tuning of Qwen2.5-VL using the LLaMA-Factory framework
• Deep Learningsubfield80%
  The proposed CUBIST model and baselines are deep learning architectures (e.g., vision transformers, LSTM, graph convolutional networks) trained end-to-end for AQA and related tasks.
  CUBIST (Compositional Ontological Reasoning Engine), a framework that performs decomposition–analysis–recompositionwe selected 7 representative SOTA heterogeneous network architectures—including I3DMLP, CoRe, TPT, HGCN, MCoRe, T2CR, and DAEThe backbone partitions the input video into non-overlapping spatiotemporal tubelets
• Attention Mechanisms and Transformersframework70%
  The CUBIST model uses attention mechanisms, including multi-head attention for pooling and cross-attention for phase parsing and error query decoding, which are core components of transformers.
  we introduce an attention-based pooling mechanism with a learnable queryit utilizes a cross-attention mechanism to perform selective focusing on the temporal features Zeach error query vector uses a cross-attention mechanism to perform adaptive retrieval on this joint context
• Natural Language Processingsubfield60%
  The paper introduces MyoMechanix-VideoQA, a benchmark for language-grounded action understanding, and evaluates vision-language models (VLMs) on it, involving natural language generation and reasoning.
  MyoMechanix-VideoQA, a dataset supporting tasks ranging from action recognition to fine-grained error diagnosis and feedback generationVideoQA enhances language-grounded action understandingwe use MyoMechanix-VideoQA dataset to assess whether domain-specific fine-tuning can equip current vision-language models (VLMs)
• Generative and Multimodal Modelingframework60%
  The paper introduces a cross-modal task (Video2EMG) that predicts sEMG signals from video, which involves multimodal modeling and translation between visual and physiological modalities.
  we introduce Video2EMG, a novel cross-modal task that aims to infer muscle activation patterns from visual inputwe design sound baseline models for the Video2EMG taskthe models were trained and evaluated on 18 muscles according to the collected sEMG signals
• Artificial Intelligencesubfield50%
  The paper discusses applications for Physical AI and introduces a compositional reasoning paradigm that aligns with broader AI goals of interpretable and structured reasoning.
  MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applicationsCUBIST (Compositional Ontological Reasoning Engine), a framework that performs decomposition–analysis–recomposition

Abstract

Existing action quality assessment (AQA) datasets and methods rely primarily on visual inputs such as RGB and pose, overlooking physiological dynamics such as muscle mechanics and often modeling actions as monolithic patterns. These limitations hinder fine-grained, biomechanically grounded feedback. We introduce MyoMechanix, a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity. Expert-annotated, it contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals, forming the largest multimodal AQA benchmark to date. We further construct the Fitness Knowledge Graph (FKG), which organizes expert annotations into structured relationships among actions, phases, key steps, errors, and corrective feedback, enabling compositional scoring and interpretable assessment. Building on these representations, we develop CUBIST (Compositional Ontological Reasoning Engine), which performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation. We also establish MyoMechanix-AQA, MyoMechanix-VideoQA, and a novel MyoMechanix-Video2EMG task. Experiments show that multimodal sensing and structured representations improve performance, interpretability, and error attribution, with CUBIST achieving state-of-the-art results; VideoQA enhances language-grounded action understanding; and Video2EMG suggests video-based alternatives to costly EMG sensing. MyoMechanix advances skilled activity understanding toward biomechanically grounded, multimodal, and compositional reasoning for Physical AI applications in fitness, rehabilitation, healthcare, and machine learning. Project page: https://haoyin116.github.io/MyoMechanix/

Paper Context