LeRobot SONIC Humanoid Integration: Motion Tokens Are Not Motor Joints

What LeRobot's humanoid integration actually adds

The LeRobot SONIC humanoid integration is interesting because it does not ask a task policy to drive every motor on a Unitree G1 directly. That is the right instinct. A humanoid policy can predict a compact motion representation, while a local controller handles whole-body motion and supplies joint targets for low-level tracking. This separation does not establish safe balance in an untested setup.

For the can-placement checkpoint, the task policy predicts 64 motion-token values plus two gripper fields. Those 64 values are not 64 joint commands. They are latent coordinates. An onboard decoder combines that token with recent robot-state history, then produces joint targets for the robot's proportional-derivative control loop.

That split is the whole point. It is also where careless integration can go wrong.

A useful integration note, not a fresh benchmark

The September 25 LeRobot humanoid integration article explains how teleoperation, policy learning and Unitree G1 control fit together. The can-placement checkpoint and dataset were already August artifacts, so the announcement should not be read as a new model release with new weights. The useful part is the wiring diagram, especially the way it exposes the boundary between policy output and onboard motion control.

The first practical question is compatibility

The release describes a workflow. The controller documentation describes deployment. The checkpoint describes a trained policy. Reading all three together raises a simple question: does the runtime supply the observations the checkpoint was trained to use?

That question is still unresolved. The can policy expects 31 state values. The documented SONIC controller exposes 64 token-echo values. Shared action dimensions do not settle the issue.

This is separate from the storage handoff covered in our LeRobot LanceDB integration analysis. A readable dataset and a loadable checkpoint are useful, but they do not prove that a live robot runtime is feeding the same state representation the policy learned from.

Why the task policy is not the balance controller

The token is not a motor list

The release describes a 64-dimensional SONIC latent representation. The can checkpoint adds two gripper fields. Those coordinates encode motion intent. They are not joint positions, torque targets or a direct list of actuator commands.

The G1 documentation describes SonicWholeBodyController decoding a token plus recent proprioception into a residual action. That residual is scaled and added to default standing angles, producing targets for 29 joints. The robot's PD control then tracks those targets.

The decoder loads ONNX weights and deployment constants from lerobot/sonic_decoder. The pinned implementation reads PD gains, default angles, action scale and a neutral token from ONNX metadata. It also separates IsaacLab joint order from MuJoCo deployment order. Those mappings are not bookkeeping. They are part of the motion contract.

For a broader view of hardware and controller assumptions, our BRIDGE morphology-control analysis is useful context. It does not prove that BRIDGE and SONIC share an interface.

Remote inference should not own the local loop

For physical deployment, the documentation describes a GPU-side policy_server, an onboard robot_client running the SONIC controller on the G1's Jetson, and the run_g1_server bridge. Token chunks move over the network. The onboard controller is designed to run its documented 50 Hz loop using local proprioception; actual deadline compliance must be measured.

This diagram is a conceptual interface map, not a verified deployment. The observation processor is left unresolved on purpose. The local feedback cycle should not need a new GPU inference result at every controller tick. Buffering can reduce dependence on remote latency, but it does not prove safe balance or useful behavior after network loss.

Async placement and guided RTC are different knobs

Asynchronous deployment decides where inference runs and how action chunks reach the robot. Guided real-time chunking, or RTC, addresses prediction across action-chunk boundaries. The can checkpoint card recommends guided RTC, while also saying this checkpoint was not trained for trained RTC or piR2.

Do not collapse those ideas into one claim. Dataset sampling at 50 fps, a 50-action chunk, publishing cadence and a 50 Hz controller are separate quantities. Matching numbers can be a clue. They are not proof that the clocks match in a live system.

The Token-to-Motion Interface Map

Start with semantics, then check shapes

The checkpoint configuration declares observation.state with shape [31] and action with shape [66]. The dataset schema names the state fields: 29 joint positions, then left and right gripper values. Its action fields are motion_token_0 through motion_token_63, followed by the grippers.

Part 5 of the current main G1 documentation instead says SonicWholeBodyController returns a 64-dimensional token-echo observation state. The pinned implementation's observation_state() confirms that it exposes the last decoded token, not the checkpoint's measured joint-and-gripper vector.

That is the main integration gap. Drop-in compatibility is not established. A team needs to resolve the observation processor, physical-state mapping, normalization and code revision before connecting these artifacts. Padding a joint vector or slicing a latent vector only makes the tensor pass a shape check. It does not make the values mean the same thing.

The model-card table also labels the action as "64 joints + 2 grippers." That conflicts with the named configuration fields, dataset schema and release explanation. For this interface, the typed artifacts carry more weight than the table wording.

Compatibility matrix for the full motion path

The Token-to-Motion Interface Map below is an editorial synthesis of the inspected artifacts. It is not a standard, and it is not an Optijara-tested deployment recipe. Its job is to make missing evidence visible before anyone chooses a runtime.

The checkpoint card records LeRobot code 8bf6056d1. The controller source inspected here is pinned to e595b7902714ba51f91e47523f66f89c5181b649. Current main documentation requires a source installation and distinguishes stable v0.6.1. Its SONIC examples use nepyope/sonic_walk, not the can checkpoint. Treat those as related references, not interchangeable instructions.

{
  "contract": {
    "checkpoint_state": "29 joint positions plus 2 grippers",
    "checkpoint_action": "64 SONIC coordinates plus 2 grippers",
    "documented_controller_state": "64-value token echo"
  },
  "proposedchecks": ["Resolve observation processor", "Verify names and normalization", "Pin compatible runtime revisions"],
  "limitations": ["Drop-in compatibility unresolved", "No inference, simulation or hardware tests performed for this article"]
}

What the can checkpoint shows, and what it does not

Read the task, cameras and timing as a contract

The can_clean_final dataset card describes 105 episodes and 212,290 frames at 50 fps. The three 480 by 640 camera streams are ego_view, left_wrist and right_wrist. The single task is Bring the can to the white table. Those details define the checkpoint's input contract. They do not establish performance in another room, with another camera mount or under a different task instruction.

The dataset documents action[t] as the command that produces observation.state[t+1]. Preserve that relationship when building replay examples. Same-index state should not quietly become the outcome attributed to the action.

The card also reports that observation.state[29] and action[64], the left-gripper fields, are always zero. In practical terms, the recordings were one-handed. Two gripper output fields do not provide evidence of bimanual skill.

Normalization needs the same care. The configuration specifies quantile normalization for state and action. The dataset card says the left-gripper quantile bounds were set manually to avoid division by zero. A processor migration has to preserve that handling instead of reading useful left-hand behavior into a valid numeric range.

Loss is not a task-success claim

The dataset was reviewed and failures removed. The checkpoint card reports final training loss of 0.025, explicitly without a held-out split. That is useful training metadata. It is not autonomous task success, and it is not evidence of generalization.

The fair reading is narrower: these artifacts document one training setup and expose constraints that a compatible evaluation must respect. Capability claims need separate closed-loop evidence. Budget explicitly for checking interface assumptions and evaluating behavior, rather than treating artifact availability as proof of compatibility.

A simulation-first adoption playbook for the G1 motion stack

Stage 1: reconcile the offline contract

Every check below is proposed. Optijara has not run inference, simulation, training or physical robot tests for this article.

Start with artifact inspection. Pin revisions. Compare state names and units. Confirm joint ordering. Trace both grippers separately. Compare camera identities and preprocessing with the checkpoint, not just image dimensions. Inspect state/action normalization and decoder constants. Treat the unresolved 31-versus-64 observation mapping as a stop condition.

This is also the place to check temporal alignment. A replay pipeline that supplies the right fields from the wrong timestep has not reproduced the training interface. Record the processor decisions so another engineer can tell deliberate transformations from accidental coercion.

Stage 2: separate policy timing from controller timing

After compatible tooling and mappings are resolved, run offline replay or simulation of token-to-target interpretation before judging learned task behavior. The documentation supplies simulation examples, but that does not show that an unchanged simulator reproduces this exact can-policy stack.

Define acceptance thresholds for the intended setup before testing. This article does not supply universal safety defaults. Record publishing cadence and queue consumption alongside inference latency; a decent average can hide interruptions. For complementary discussion of behavioral evidence, see our video-to-task robotics evaluation map.

Stage 3: decide whether hardware preparation is justified

Adopt interface inspection now. Migrate processors only after their meanings are settled. Do not transplant a checkpoint-card command into current main hardware code just because both mention Unitree G1.

Before any physical trial, require manufacturer safety procedures, a controlled test area and a working emergency stop. Review stale-queue and network-loss behavior explicitly. This article does not establish a default watchdog or recommend a synchronous hardware smoke test.

Common mistakes and limitations to keep visible

Mistakes that make the interface look safer than it is

• Treating latent coordinates as motor joints skips the decoder's role. Trace token decoding and joint-order conversion.
• Matching action dimensions while ignoring observation meaning leaves the policy input unresolved. Check both sides of the contract.
• Calling 50 Hz the policy speed confuses local control with inference throughput. Measure them separately.
• Inferring bimanual skill from two output fields ignores the inactive recorded left gripper.
• Treating guided RTC as safe async deployment confuses chunk prediction with runtime placement and failure handling.

Licensing, safety and operating trade-offs

The inspected SONIC decoder file listing exposes decoder artifacts but no visible model card or license file. That absence is not a legal conclusion. Review base-model, decoder, controller software, dataset and hardware terms separately; a library or dataset license does not grant permission for every component.

Main-branch drift and checkpoint-era processors complicate reproducibility. Camera placement, proprioception conventions and GPU execution differences also need review. If camera observations leave the robot for remote inference, assess access controls and privacy for the operating environment.

Budget for integration and evaluation work without assuming production readiness. The immediate decision is whether a team can establish a consistent observation-to-token-to-motion contract, with unresolved assumptions visible before hardware preparation starts.