The Large Movement Model

1. Introduction

Human movement is one of the richest natural signals available. It encodes health status, task intent, emotional state, motor skill, and physical capability. Yet no unified AI framework has treated motion as a general-purpose sequence domain — one where the same foundational model can serve rehabilitation assessment, sports analytics, robotics control, and human–computer interaction.

The Large Movement Model (LMM) addresses this gap. It applies the same core insight that drives large language models — that sequence-learning architectures, given consistent tokenization and sufficient data, discover generalizable latent structure — to human pose sequences. Where language models learn grammar and semantics from words, LMM learns biomechanical structure, temporal coordination, and movement intent from the evolution of body-joint positions over time.

This document describes the LMM pipeline, model designs, and Phase I results. The work demonstrates that a multi-resolution attention architecture and a diffusion-based generative model both substantially outperform conventional baselines on motion forecasting — establishing feasibility for a new class of embodied foundation models.

2. Data Pipeline

The pipeline transforms raw video into structured, normalized, machine-learning-ready motion tokens. It is designed to ingest heterogeneous sources — different cameras, frame rates, resolutions, skeleton formats, and recording conditions — and produce one unified representation.

2.1 Pose estimation

Video is processed through OpenPose to extract BODY_25 skeletal keypoints: 25 major joints with per-joint confidence. For datasets provided as pre-extracted skeletons (COCO-17, Kinect V2, Vicon), format converters map alternative schemas to BODY_25 through documented correspondence tables, synthesizing derived joints (neck, mid-hip) where needed. BODY_25 balances representational fidelity with efficiency at roughly 1/100th the cost of mesh-based representations like SMPL.

2.2 Normalization

• Translation invariance: every skeleton is centered on the hip midpoint.
• Scale invariance: skeletons are scaled to a standardized torso length.
• Temporal uniformity: sequences are resampled to 15 FPS regardless of source frame rate.
• Derived features: per-joint velocity, acceleration, and jerk are computed alongside positions.
• Quality control: each clip receives QC metadata — per-joint confidence statistics, detected artifacts, and a usability classification.

2.3 Multi-resolution tokenization

Motion is structured into three simultaneous token views. All three reconstruct losslessly to the original — no information is discarded, only reorganized.

The frame level gives the finest temporal grain. The window level captures meso-scale patterns (a step, an arm swing). The body-part level captures spatial coordination between regions. Together they give the model access to movement structure at multiple scales without committing to which scale matters most.

2.4 Pipeline performance

The pipeline has been validated on over 17,000 clips from five independent sources — multi-camera studio video, clinical rehabilitation video, depth-sensor skeleton data, and optical motion capture. Heterogeneous inputs (60 FPS dance video, 30 FPS clinical footage, Kinect V2 depth skeletons, Vicon optical mocap) all converge to the same 15 FPS BODY_25 normalized representation with zero pipeline failures.

3. Model Architectures

Two architectures were developed and evaluated, each taking 120 frames (8 seconds) of context and producing 30 predicted frames (2 seconds).

3.1 Flat transformer baseline

A standard encoder–decoder transformer on single-resolution frame tokens. Six encoder layers, six decoder layers, sinusoidal positional encodings based on physical time (seconds, not frame index). This is the controlled baseline.

3.2 Hierarchical Temporal Transformer (HTT)

The HTT processes all three token levels simultaneously through three parallel encoder streams connected by bidirectional cross-attention:

• Frame encoder (4 layers): fine-grained temporal dynamics, frame to frame.
• Window encoder (3 layers): meso-scale patterns via a learned 1D convolution that compresses each 5-frame chunk — self-attention over 24 window tokens instead of 120 frame tokens.
• Body-part encoder (3 layers): spatial coordination across 5 anatomical regions, with learned part-type embeddings.

A gated linear combination fuses the three encoder outputs, and a 4-layer decoder generates the forecast autoregressively. Scheduled sampling addresses autoregressive drift by gradually transitioning the decoder from ground-truth to self-predicted inputs during training (0% → 50% over 25 epochs).

3.3 Motion diffusion model

Instead of predicting frame by frame, the diffusion model generates the full 30-frame forecast in a single pass — iteratively denoising random noise into a plausible trajectory, conditioned on the context.

• Context encoder: 6-layer transformer encoder producing a latent representation of the input.
• Denoiser: 6-layer transformer encoder, cross-attending to context, predicting clean motion from noise.
• Schedule: cosine noise schedule (T=100 training steps, DDIM 20-step inference).

Because diffusion generates all frames simultaneously, it does not suffer from compounding autoregressive error within a block. As §4.4 shows, extending it beyond a block surfaces a different failure — which motivated the third and final Phase I architecture.

3.4 Diffusion forcing (streaming)

The third architecture removes the context/forecast boundary entirely. A single causal transformer (12 layers, 9.8M parameters — smaller than either predecessor) processes one sequence in which every frame carries its own noise level. A frame at noise level zero is clean conditioning; a frame at maximum noise is pure uncertainty. "Context" is no longer a separate input handled by a separate encoder — it is simply the frames that happen to be clean. Continuation is inpainting by construction: there is no boundary at which a discontinuity can exist.

• Training: each frame in a training window is independently noised to a random level; the model learns to predict every frame's clean pose from its noisy neighborhood. Half of training uses the inference layout directly — a clean prefix followed by a noisy future.
• Streaming inference: a sliding pyramid. Each future slot sits at a noise level proportional to its distance from the present; every model pass denoises each slot one step; the slot reaching zero is committed as the next output frame, and a fresh pure-noise slot enters at the tail. One forecast frame per model pass, at any horizon.
• Robustness property: because the model trains on corrupted history, its own generation errors at rollout time look like noise it was trained to remove — the mechanism that prevents compounding-error collapse.

4. Experiments & Results

4.1 Setup

Models were trained on 12,483 clips (~2.5M frames) from a multi-camera dance corpus spanning 10 genres, 30 dancers, and 9 synchronized 60 FPS camera angles. Identical hyperparameters throughout: AdamW, lr 1e-4, batch 32, early stopping (patience 10), 90/10 split by clip. Task: 120 frames of context → predict 30 frames. HTT and baseline evaluated autoregressively; diffusion as the sample mean over 10 DDIM trajectories.

4.2 Results

Table 1. Motion-forecasting MSE (lower is better). 12,483 clips, 53,903 training windows.

4.3 Analysis

Autoregressive drift is the central challenge. The flat baseline predicts the next frame well (0.087) but degrades to 0.900 by frame 30 — a 10× increase. Even with a large, diverse corpus, single-resolution attention cannot maintain multi-second coherence.

Multi-resolution attention helps most at the longest horizons. The HTT reduces frame-30 error by 13% (0.900 → 0.785), concentrated at the 1.5–2 second horizon where cross-region coherence matters most. At short horizons the simpler baseline is competitive.

Diffusion substantially outperforms both deterministic models. Overall MSE is 30% lower than the baseline, and frame-30 error (0.621) is 31% lower than the HTT's. Generating all 30 frames at once avoids the compounding error that is the dominant failure mode of autoregressive architectures.

Beyond the training horizon, the difference becomes decisive. The 2-second forecast is only the start of the story. Extend it — roll a model forward on its own output — and the autoregressive HTT collapses toward a static pose: per-frame motion decays to near-zero within about a second and never recovers, the classic regression to the mean. Short-horizon error hides this, because predicting "hold still" scores acceptably on MSE while being dynamically inert. The diffusion model, generating each block in a single pass with no autoregressive feedback, keeps producing coherent, in-range motion far past the training window — sustaining roughly an order of magnitude more per-frame movement across 10+ seconds of rollout.

That makes diffusion the usable forecasting family. A deterministic rollout gives a sharp next frame and a serviceable first second, but the forecast cannot be extended; it coasts to a near-static pose and stays there. The diffusion model produces motion that stays alive across and well beyond the 2-second horizon — which is what any downstream use of a forecast actually requires. But rolling a block model has a cost of its own, one that only became measurable when we pushed rollouts onto real captured motion (§4.4).

What is the frozen pose hiding? The autoregressive forecast is not motionless — it is amplitude-collapsed. Scaling its output by a constant factor — a linear transform that changes each motion's magnitude but not its direction — reveals smooth, coherent movement beneath the stillness. The limitation is one of magnitude, not structure: minimizing squared error against an uncertain future rewards hedging toward the average pose, so the motion is predicted but shrunk. This is a diagnostic, not a fix — amplification makes the structure visible, it does not make the forecast more accurate.

4.4 The seam problem — and diffusion forcing

Extending the block diffusion model past 2 seconds means rolling: generate a block, commit its head, slide the context, regenerate. Measured on real captured motion, this rolling has a signature defect: every commit boundary pops. On a 12-second rollout, all 14 block seams showed a per-frame displacement spike of 2.6–5.6× the surrounding motion (mean 3.6×), and the worst discontinuity of all was the very first handoff from real context into the forecast. A second defect compounds it: averaging multiple diffusion samples — the variance-reduction step behind Table 1 — damps the rolled forecast to roughly a quarter of the clip's real motion energy. The amplitude-collapse video above is, in part, this damping made visible.

The cause is structural, not a tuning problem: each block is denoised from fresh noise, and nothing ties its first frame to the last committed one. Rather than patch it at inference — crossfades, boundary clamping, continuity-selected samples — we rebuilt on the architecture of §3.4, where the boundary does not exist.

Table 2. Block diffusion vs. diffusion forcing, identical corpus and validation split. The −42% at frame 1 is native boundary conditioning showing up exactly where the theory predicts.

4.5 Real-world captures: an honest scoreboard

Phase I closes the loop on live capture: clips recorded on a phone with our MovementModeler app (Apple-Vision pose, 15 of 25 joints) are normalized, tokenized, and streamed through the forecaster — the LLM analogy made literal, with the captured clip as the prompt. Captures longer than the 8-second context window come with their own ground truth: the model forecasts 12 seconds while reality plays out in parallel.

Not every capture scores that well, and the failures are as informative as the success. Across six captures, forecast quality tracked two variables that have nothing to do with the architecture: capture stability (a hand-held or moving camera injects whole-skeleton warp the model dutifully continues — clips shot while the operator walked showed torso-scale fluctuations of 14–18%) and corpus coverage.

This is the cleanest possible statement of where the work stands. The streaming architecture is validated — seam-free, energy-correct, robust to its own feedback — on every capture including the ones it forecasts poorly. What fails is the prior: a 10M-parameter model trained on 10 genres of dance cannot know that a slow gait stays slow. Forecast fidelity is now a data problem, not an architecture problem — and a specialized large-scale corpus is precisely the Phase II plan (§7).

5. What the Model Learns

Temporal coordination across timescales. The window encoder captures meso-scale patterns the frame encoder misses — stride cadence, the arc-and-return of reaching, weight-transfer timing. Cross-attention constrains fine-grained predictions within these broader patterns, which is what reduces drift.

Spatial coordination across body regions. The body-part encoder learns how regions move relative to each other — contralateral arm–leg coordination in gait, trunk stabilization during limb movement, bilateral symmetry. This is exactly the structure needed to detect compensatory patterns.

Biomechanical plausibility. Predicted sequences generally respect physical constraints — joints stay in plausible ranges, limb lengths stay approximately constant, center-of-mass trajectories stay continuous — and these emerge from the data distribution without explicit loss terms.

6. Cross-Domain Transfer

The pipeline and architectures are domain-agnostic by design — the skeleton, normalization, and tokenization make no assumption about what kind of movement is analyzed. But does the model actually transfer, or just memorize the training distribution?

6.1 Held-out generalization test

We evaluated trained models against a held-out corpus from a fundamentally different capture modality: optical motion capture of physical-therapy exercises (Vicon, 10 exercises, 10 subjects). The model never saw this data in training, and its characteristics are visually distinct — no camera perspective, no detection jitter, no confidence variation.

6.2 Scheduled sampling as a transfer mechanism

An unexpected result: scheduled sampling — introduced to reduce autoregressive drift — turns out to be a powerful cross-domain transfer mechanism. Models trained with it showed 54% lower teacher-forced error on held-out data than identical models trained without it. The interpretation: scheduled sampling forces the model to predict from imperfect inputs, and cross-domain inputs are inherently imperfect; models that learn to handle imperfect inputs generalize better.

7. Current Limitations

• 2D projection. Phase I uses 2D pose estimation; depth-dependent movements are geometrically compressed. 3D pose or multi-view triangulation would improve coverage.
• Training-data diversity. The primary corpus is 10 dance genres — fast, diverse, full-body, but not specifically clinical. Domain-specific fine-tuning on rehabilitation exercises is a Phase II objective.
• Clinical alignment not yet validated. Engineering metrics are strong; correlation with expert-assigned movement-quality ratings has not yet been formally evaluated.
• Autoregressive forecasts do not extend. Deterministic models predict a sharp next frame, but their motion decays toward a static pose within about a second, and rolling them forward only deepens the collapse — frame-30 error remains ~6–10× frame-1 error. This is an architectural property of autoregressive generation, not a tuning problem.
• Block-diffusion rollouts seam. Rolled past its 2-second block, the block diffusion model pops at every commit boundary (3.6× interior motion) and its sample-averaged output runs far under real motion energy. Resolved by diffusion forcing (§4.4), which is now the forecasting architecture of record; the block model's numbers are retained above as the honest baseline it was.
• The prior is dance. Trained on 10 dance genres, the forecaster over-animates slow, deliberate movement — on an elderly-gait capture it streams flawlessly while losing to a freeze-frame baseline, because it has never seen a person move slowly. The most clinically relevant motion is the least represented in the training distribution. A large specialized corpus — slow gait, rehabilitation exercise, activities of daily living — is the single highest-leverage Phase II investment, and the streaming recipe carries over unchanged.
• Capture egomotion. A moving or hand-held camera injects whole-skeleton scale and sway that survives hip-centering (torso-scale fluctuations of 14–18% on operator-walking captures vs. ~5% hand-held standing). Planned mitigations: a per-clip egomotion QC gate, and low-pass per-frame torso scaling in normalization.
• Evaluation methodology. HTT (single rollout) and block diffusion (mean of 10) are not evaluated on identical terms; diffusion's Table 1 advantage includes a variance-reduction benefit. Table 2 compares the two diffusion models on identical terms.

8. Architecture Summary

9. Conclusion

The Large Movement Model demonstrates that human motion can be treated as a structured sequence domain analogous to text, and that sequence architectures learn meaningful motion structure from video-derived pose data. Phase I traversed three architecture generations, each fixing the measured failure of the last: autoregressive attention forecasts sharply but freezes when rolled; block diffusion sustains motion but pops at every commit seam and damps to a fraction of real energy; diffusion forcing — per-frame noise levels on a single causal transformer — eliminates the boundary itself, streaming seam-free at correct motion energy for arbitrary horizons, with a 42% reduction in boundary error over block diffusion at 9.8M parameters, the smallest model of the three.

The loop is closed end to end on live capture: a phone-recorded clip is the prompt, and the model streams its continuation one frame per pass — validated against real futures, where it beats a freeze-frame baseline by 40% on in-scope motion and, just as informatively, fails on motion its dance-only corpus never contained. Cross-domain evaluation confirms the learned dynamics transfer across capture modalities; the real-capture scoreboard shows forecast fidelity is now bounded by data, not architecture. That is the Phase II thesis in one line: the streaming foundation is built — what it needs next is a corpus of the movement that matters.

Read the thesis →