Embodied.cpp is a portable C++ inference runtime for robot policies that reorganizes Vision-Language-Action model and World-Action Model models into a five-layer pipeline with independently-clocked modules, letting one quantized Transformer block shrink from 312.2 MiB to 88.1 MiB while keeping cosine similarity above 0.9997.
You’ve trained a robot policy that eats camera frames, proprioception, and a language instruction, and emits action chunks at 20 Hz. To actually run it on a Jetson bolted to the robot, today you glue together Python research code, a vendor inference backend, hand-rolled sensor wrappers, and a ROS-side control loop. Every new model family (a hierarchical planner-plus-controller, a world model that predicts futures) means rewriting that glue.
General serving runtimes like llama.cpp or ONNX Runtime don’t help much: they assume request-response traffic with uniform token I/O and optimize throughput across big batches. Robot inference is the opposite. It runs batch-1 inside a closed control loop, mixes modules that want different refresh rates, and has to accept tactile signals or emit predicted future frames, not just tokens. The closest prior effort, vla.cpp, ports seven Vision-Language-Action model architectures to portable C++ but stays Vision-Language-Action model-only.
The authors first do a taxonomy pass across recent embodied models and notice they all walk roughly the same path: adapt sensors, build a sequence, run a Transformer backbone, then branch into a task-specific head. The differences are concentrated in the heads (autoregressive action tokens vs. continuous action chunks vs. predicted future frames) and in how components are scheduled relative to each other.
That observation becomes a five-layer runtime. Input adapters normalize cameras, language, proprioception, and force/tactile streams into typed embodied inputs. Sequence builders assemble the token or feature sequence the backbone expects. Backbone execution is the shared Transformer path. Head plugins are the divergent part: an action expert, a world-model head, a hierarchical planner. Deployment adapters connect outputs to a simulator or a real robot stack.
The runtime then adds three cross-cutting capabilities. Multi-rate scheduling lets a perception encoder run slowly while an action head ticks fast, coordinating through shared state buffers rather than a single synchronous forward pass. Latency-first fused execution targets stable batch-1 performance on heterogeneous edge chips (Jetson, RK-based boards, x86 edge, workstations) via graph replay, buffer reuse, and operator fusion. And an embodied kernel warehouse holds reusable and model-specific operators, including GGUF quantized blocks, so new models plug in without rebuilding the substrate.
# Rough control-loop shape enabled by the runtime
state = SharedFeaturePool()
while robot.alive():
if perception.due(): # slow rate
state.vision = perception(cam.read())
if world_model.due(): # medium rate, optional head
state.future = world_model(state)
action = action_head(state) # fast rate, batch-1
robot.execute(action) # via deployment adapter
The paper doesn’t specify the internal scheduling algorithm or how refresh policies are declared beyond calling them “configurable.”
The prevailing assumption when porting a new robot policy is that each model family needs its own inference stack, because architectures look different on the surface. This paper argues the opposite. Across Vision-Language-Action model and World-Action Model families, the execution path is already shared; the real divergence lives in a small set of heads and in the timing relationships between modules, so the runtime should fix the substrate and make heads plus schedules the plugin surface. The evidence is that two quite different Vision-Language-Action model models (HY-VLA and pi0.5) and a World-Action Model Transformer block from LingBot-VA all drop into the same five-layer path with working closed-loop control or matching numerics.
•
The load-bearing result is the World-Action Model microbenchmark: swapping a BF16 WanTransformerBlock for the runtime’s GGUF Q4_K quantized version cuts resident weight memory from 312.2 MiB to 88.1 MiB, with mean absolute error below 3.3×10⁻² and cosine similarity above 0.9997 on 100 random inputs. That’s the evidence the shared substrate can host World-Action Model-style components without meaningful output drift.
•
On the RoboTwin place_empty_cup task, HY-VLA runs closed-loop through the C++ path at 100.0% success; pi0.5 reaches 91.0% in its C++ deployment configuration. Both work end-to-end through the same runtime.
•
HY-VLA shows higher per-step latency than pi0.5, which the authors attribute to a larger Hunyuan-VL backbone, three-view visual input, and a video-history vision path, versus pi0.5’s lighter PaliGemma backbone and longer action chunks. The runtime doesn’t erase model-cost differences; it just makes them measurable in one place.
•
Full LingBot-VA closed-loop numbers are not reported. The authors say the complete model isn’t yet stable on their constrained edge device, so only the first Transformer block is benchmarked.
Reach for this when you’re deploying a multimodal robot policy on an edge board and are tired of maintaining a bespoke Python-plus-glue stack per model. Concretely: if your policy has a slow vision encoder and a fast action head that need to tick at different rates against the same robot, the multi-rate execution path plus typed input adapters replace hand-written buffering and threading code, and the GGUF quantized kernels give you a memory lever without rewriting the model.
Code is released at GitHub. Evaluated models are HY-VLA (on RoboTwin) and pi0.5; the World-Action Model side is exercised through a single LingBot-VA Transformer block. The paper doesn’t specify supported backends in detail beyond mentioning CPUs, GPUs, NPUs, Jetson, and RK-based platforms, and it doesn’t quantify latency improvements against vla.cpp or other runtimes.
Treat the Transformer backbone as fixed infrastructure and the heads, schedules, and I/O adapters as the plugin surface. Embodied models look diverse from the outside, but the runtime contract that actually matters (multi-rate execution, batch-1 latency, typed sensor and action interfaces) is stable enough to standardize once and reuse across Vision-Language-Action model and World-Action Model families.
•
The World-Action Model evidence is a single-block microbenchmark, not a closed-loop robot run. Quantization error that looks tiny per block can compound across a full autoregressive world-model rollout, and the paper acknowledges the full LingBot-VA model isn’t stable on their edge setup yet.
•
The runtime is compared to general serving stacks like llama.cpp and vla.cpp mostly via a capability table, not head-to-head latency or memory numbers. If your current stack already runs your one model well, the concrete win here is portability across future models, not a measured speedup on today’s.
•
“Multi-rate execution” and “latency-first fused execution” are described as design principles with an architecture diagram, but the paper doesn’t spell out the scheduler, the refresh-policy configuration language, or which operator fusions are actually implemented. Real-world jitter behavior on a specific robot is not measured.