Learning-based control of physical systems — manipulation, locomotion, vision-language-action models, and simulation-to-real transfer for autonomous robots.
Robotics: learn what matters, recover from failure, and test physical behaviorLearning-based robotics couples perception, prediction, and control to a world that changes under action. A system can recognize the scene yet attend to the wrong object, generate a plausible-looking future with broken contact physics, or complete a task while violating force, state, or termination requirements.Use this guide by failure mode. Start with dynamics-aware perception when manipulation breaks under object or scene shifts. Turn to recovery and protocol tests when nominal success hides brittle execution, and use contact-aware screening when interaction quality matters as much as completion. For predictive controllers, distinguish action calibration, physical rollout fidelity, pretraining for transfer, and evaluation under interventions.
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided RepresentationStart here for a mechanism-level account of dynamics-aware visual pretraining, ablations that isolate joint image–language–flow alignment, and diagnostic evidence that the encoder can cause manipulation failures under distribution shift.Results span simulated suites and a UR3 setup; gains are largest on author-designed shift tests, while in-distribution LIBERO gains with LoRA are smaller.Read or listen · 7 min audio
What are you trying to solve?
Encode what changes under actionStatic recognition features can emphasize appearance or background rather than the object and contact region that determine control. Dynamics-aware pretraining moves motion understanding into the visual encoder by aligning image transitions, instructions, and estimated 3D flow; downstream policies then use the resulting image-only representation.
DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided RepresentationStart here for a mechanism-level account of dynamics-aware visual pretraining, ablations that isolate joint image–language–flow alignment, and diagnostic evidence that the encoder can cause manipulation failures under distribution shift.Results span simulated suites and a UR3 setup; gains are largest on author-designed shift tests, while in-distribution LIBERO gains with LoRA are smaller.Read or listen · 7 min audio
When should I suspect the visual backbone rather than the policy head?Suspect perception when a policy works on its training bench but degrades after object-position changes, distractors, or unseen object–instruction combinations. DynaFLIP reports its largest gains under these shifts, while its in-distribution LoRA gain on LIBERO is modest. That pattern makes the encoder a useful controlled variable before enlarging the controller.DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
How can a single-image encoder learn action-relevant dynamics?During pretraining, DynaFLIP aligns an image transition, its instruction, and camera-compensated 3D point flow. A joint geometric objective, a cosine term, and contrastive negatives address weak pairwise alignment and representation collapse. At deployment only the image encoder remains, so dynamics provide training supervision rather than an extra runtime sensor.DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation
Test recovery and protocol discipline, not only nominal successCompletion from ideal initial states does not show whether a robot can continue after a failed grasp, unintended object motion, or a disturbed scene. Nor does reaching a target prove that an agent maintained per-robot state and issued the correct terminating action. Recovery suites and prompt-defined control benchmarks expose these separate failure layers.
LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation ModelsRead this to turn vague robustness concerns into four recovery levels and four reasoning capabilities grounded in execution failures rather than clean initial states.The 1,000+ scenarios come from embodied-model executions on LIBERO; the abstract specifies benchmark structure, not a recovery algorithm.
How should manipulation recovery tests be organized?Vary how much correction the changed state requires. LIBERO-Recover separates action retry, action adaptation, object-state recovery, and environmental recovery, then probes spatial, structural, interaction, and topological reasoning. This distinguishes a transient control miss from a state change that invalidates the original plan.LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models
Screen contact for location and force safetyIn assistive robotics, task completion can coexist with contact on the wrong body region or with excessive force. Evaluation therefore needs a responsive human model, contact-quality measures, and an observer/scorer boundary that prevents privileged simulator state from leaking into the policy.
A Physics-Consistent Benchmark for Contact-Rich Human-Robot Interaction in Assistive CareRead this to design contact-rich evaluation that gates task completion by correct contact region and force safety, with simulated response calibrated against physical measurements.The setting is robot-assisted bathing with a deformable passive human model and manikin-based calibration under a fixed T1–T7 protocol.
What should contact-rich evaluation measure beyond task completion?Measure whether contact occurs on the intended region and satisfies force criteria, then report safety-gated success beside raw completion. In robot-assisted bathing, one state-machine method dropped from 72.9% task success to 56.4% after correct-region and force-safety screening, showing how completion alone can overstate readiness.A Physics-Consistent Benchmark for Contact-Rich Human-Robot Interaction in Assistive Care
How can simulated human contact be tied to physical measurements?The benchmark calibrates region-wise deformation responses against force–indentation measurements from Franka impedance pushes on a medical-care manikin. It combines that calibration with a frozen vision-only, scorer-only protocol so evaluated policies do not receive privileged physical state.A Physics-Consistent Benchmark for Contact-Rich Human-Robot Interaction in Assistive Care
Make predicted worlds respond correctly to actionA useful world model must preserve state across a rollout, interpret setup-specific commands, and represent contact rather than merely generating attractive video. The readings cover four distinct choices: broad behavioral evaluation, visual calibration for a new setup, physics-focused rollout supervision, and manipulation-specific world-action pretraining.
HappyWorld-BenchRead this for an evaluation framework that checks generated worlds under exploration, revisits, edits, multi-step action, and altered conditions instead of scoring visual quality alone.The abstract reports 1,138 video prompts, 300 spatial scenes, and 254 embodied cases across independent evaluation tracks.
SyncWorld: Visual Calibration Enables World Models as Zero-Shot SimulatorsRead this when an action-conditioned model must handle a changed camera, environment, placement, or embodiment. It treats visual calibration as an in-context specification of action meaning.The abstract reports unseen-setting simulation and test-time policy improvement but omits calibration-episode construction details and deployment limits.
PhysisForcing: Physics Reinforced World Simulator for Robotic ManipulationRead this for objectives aimed at physical instability in manipulation video, linking point trajectories and semantic interaction relations to closed-loop planning and downstream-policy results.Evidence covers R-Bench, PAI-Bench, EZS-Bench, and WorldArena; no real-robot safety validation is described.
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic ManipulationRead this for a from-scratch manipulation world-action model that separates compressed state, visual planning, and inverse dynamics while testing how data and skill coverage relate to transfer.Results cover 100 tasks, 20 skill groups, two embodiments, and held-out visual conditions without per-task fine-tuning.
What should world-model evaluation test beyond visual quality?Test whether state survives long rollouts and revisits, whether edits and placements take effect, and whether outcomes change correctly after actions, interventions, or altered physical rules. HappyWorld-Bench organizes these checks across video, spatial, and embodied tracks, combining automated behavioral measures with human A/B comparisons.HappyWorld-Bench
How can a world model interpret actions in an unseen robotic setup?SyncWorld conditions on a calibration episode containing paired frames and actions across controllable degrees of freedom. The episode specifies how numerical commands appear under the current camera, placement, environment, and embodiment; interaction history can serve as context when explicit calibration is absent.SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators
How can training target implausible manipulation rollouts?PhysisForcing focuses supervision on moving and interacting regions. It aligns internal features with reference point trajectories at the pixel level and with inter-region relations from a frozen video encoder at the semantic level, targeting discontinuous motion, deformation, and inconsistent contact rather than appearance alone.PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation
What does manipulation-specific world-action pretraining add?GE-Act 2.0 separates a compressed control-oriented state, one-pass visual planning, and inverse dynamics, then selectively trains on predicted futures compatible with recorded actions. Its evaluation across 100 tasks, 20 skill groups, held-out visual conditions, and two embodiments connects broader co-training coverage with higher zero-shot success.GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation