From Tokens to Torque: Understanding and Experimenting Physical AI

VLA evolution and architectures, followed by hands-on robot assembly, data collection, training, and inference.

Authors:

Mahtab Newaz | Spectrum Software & Consulting (Pvt.) Ltd. AI/ML Team
Published: 21st Sept, 2026
Reading Time: ~60 min
Tags: Physical AI VLA Embodied AI Robotics Imitation Learning

How do foundation models turn visual observations and language instructions into physical actions and what still makes this difficult? This article begins with the foundations and challenges of Physical AI, tracing the evolution of robot learning into modern Vision-Language-Action (VLA) models. It then examines how these models work internally: how they represent observations, incorporate language and robot state, and generate actions within a broader robotics stack.

The second part brings these ideas into practice through implementation and experimentation with an open-source SO-101 robot and LeRobot. We follow the complete workflow from hardware assembly, motor control, calibration, and teleoperation to demonstration recording, dataset preparation, model training, and running inference on the real robot.

Table of Contents

• 1. From Tokens to Torque: What Physical AI Is, and Why Now
  • 1.1 Physical AI: A Working Definition
  • 1.2 The Lineage
  • 1.3 Why Now: Four Convergences
  • 1.4 The Honest Counterweight
• 2. What Is Inside a Modern Vision-Language-Action Model?
  • 2.1 The Common Anatomy of a VLA
  • 2.2 Why Action Chunking Became So Important
  • 2.3 The Major Architectural Families
  • 2.4 Architectural Topology Is Also Changing
• 3. Common Hardware Platforms for AI and Teleoperation Research
• 4. Actuators and Motors used in Robots (AI Research)
  • 4.1 Hardware Considerations for AI-Controlled Robots
  • 4.2 Summary Comparison Table
• 5. Perception and Compute Hardware for Vision–Language–Action Systems
  • 5.1 Camera Systems: The Robot’s Visual Interface
  • 5.2 Compute Hardware: From Robot Control to Model Training
  • 5.3 Practical Hardware Configurations
• 6. The SO-101 Robotic Arm: Hardware Deep Dive
  • 6.1 STL Files & 3D Printing
  • 6.2 Parts Anatomy: What's Called What
  • 6.3 Assembly Guide
• 7. Motors, Drivers & Feedback Systems
  • 7.1 Motor Data & Feedback Signals
  • 7.2 Motor Driver Board: What It Does
• 8. Calibration
• 9. Leader-Follower Teleoperation
• 10. Camera Integration
• 11. LeRobot Setup Pipeline
• 12. Dataset Recording & Management
  • 12.1 Recording Principles
  • 12.2 Data Processing & HuggingFace Upload
  • 12.3 Dataset Visualization & Quality Control
• 13. Imitation Learning with ACT
  • 13.1 ACT Model Architecture
  • 13.2 Training Pipeline
  • 13.3 Inference & Results
• 14. Vision-Language-Action Models (VLAs) Experimentation
  • 14.1 Our Implementation: Five Tasks, Four Architectures
• 15. References & Resources

1. From Tokens to Torque: What Physical AI Is, and Why Now

The Capability Inversion

Between November 2022 and today, a class of systems went from novelty to infrastructure. GPT-3.5 shipped as ChatGPT in November 2022; GPT-4 followed in March 2023; multimodal models that read images, video, audio and code became ordinary. These systems now write production software, prove theorems, and pass professional licensing exams.

None of them can reliably pick up a mug.

The reason is not that manipulation is intellectually harder. It is that the sensorimotor problem has four properties the language problem does not:

1. The output space is continuous, high-dimensional, and physically consequential. A language model that emits a bad token produces a bad sentence. A policy that emits a bad joint command produces a collision, a stripped gear, or a dropped part. There is no regenerate button on a physical action.
2. The data does not exist and cannot be scraped. Every text token on the internet was produced for free by humans doing something else. Every robot trajectory must be deliberately collected, on specific hardware, by a human operating that hardware. This is the central constraint of the field, and The Honest Counterweight returns to it.
3. The loop is closed and the errors compound. Language generation is open-loop with respect to the world; the model's output does not change the distribution of its future inputs. A robot's action becomes its next observation. Small errors move the system into states no demonstration ever covered , the covariate-shift problem that has haunted behavioral cloning since Ross et al.'s DAgger work in 2011.
4. Evaluation is expensive and non-reproducible. You can benchmark an LLM on MMLU for the price of some inference. Evaluating a manipulation policy means physically resetting a scene hundreds of times, and the result depends on your lighting, your table height, and how worn your gripper pads are.

Physical AI is the research field aimed at closing this inversion.

1.1 Physical AI: A Working Definition

Physical AI denotes AI systems that perceive, reason about, and act upon the physical world, where the system's output ultimately produces physical consequence , motion, force, contact, navigation , rather than tokens or pixels.

The load-bearing idea is the closed interaction loop. A vision model that labels a bin of parts is not, by itself, Physical AI. A robot arm replaying a fixed hand-coded trajectory is not necessarily Physical AI either. The system must use observations of its environment to select or adapt its actions, and those actions must in turn affect its own subsequent observations.

The vocabulary, commonly used

These terms are used loosely and often interchangeably in industry writing. For a technical audience it is worth being precise, because the distinctions correspond to real differences in method.

What Physical AI is not

This matters for anyone evaluating vendor claims. A six-axis industrial arm welding car bodies at 0.05 mm repeatability is a triumph of engineering and is not Physical AI. It executes a taught trajectory in a fixtured environment; remove the fixture and it fails completely. The distinction is where the intelligence lives: in classical automation the intelligence is in the environment (jigs, fixtures, feeders, fixed lighting), and the robot is a precise but blind executor. Physical AI moves the intelligence into the robot so the environment can be unstructured.

That is the entire economic argument, and it is why the field cares about generalisation over precision. Industrial robots already beat learned policies on precision by two orders of magnitude and will continue to.

1.2 The Lineage

The modern learned-policy lineage is much shorter and starts, for practical purposes, with Levine et al. (2016), End-to-End Training of Deep Visuomotor Policies (JMLR), which showed a single CNN mapping raw pixels directly to motor torques , no hand-designed state estimator, no separate perception module. Google's grasping "arm farm" (Levine et al., 2018) then demonstrated that self-supervised data collection at scale produces generalisable manipulation.

Rather than reading the last four years as a list of model releases, it is more useful to read it as six architectural phases:

robot transformers → large-scale cross-embodiment learning → VLA → generative action policies → reasoning + action → world/action models

The Four-Year Sprint Major models (2022–2026)

1.3 Why Now: Four Convergences

The "why now" argument is usually told as three converging trends. We think it is four, and the fourth is the one most commonly omitted.

(i) VLM backbones turned out to be transferable action priors.
This was not obvious a priori. There is no reason internet image-text pretraining should help you decide joint velocities. It does, because the backbone supplies open-vocabulary object grounding, spatial relations, and affordance-adjacent semantics that would otherwise require robot data to learn. RT-2 demonstrated the transfer; a large fraction of subsequent generalist policies took it as given. Practically: for that branch of the field, you are no longer training a policy from scratch, you are adapting one.

(ii) The action-decoding problem got a workable answer.
Three ideas, stacked, did most of the work. Action chunking (ACT) , predict k future actions at once, which shortens the compounding-error horizon and removes the jitter of per-step prediction. Generative action heads (Diffusion Policy, then flow matching in π₀/SmolVLA) , model p(a|o) as a distribution rather than a point estimate, which is the correct fix for multimodal demonstrations where two different valid actions average to one invalid one. Continuous outputs , avoid the precision loss of tokenised action bins. This stack is now the most common default, and it is why a 450M model can control an arm competently.

(iii) Hardware cost collapsed.
Many open sourced hardware systems both low costs and medium costs are available for making leader-follower teleoperation and local data collection accessible at a small fraction of the cost of conventional research arms. (Exact cost depends on electronics, cameras, fabrication method and sourcing).

(iv) A shared data format and an open training stack appeared.
Before LeRobot, a manipulation project meant stitching together a driver layer, a recording format, a dataset loader, a policy implementation and an inference harness from four incompatible research codebases. LeRobot provides standardised dataset tooling (LeRobotDataset: synchronised video + state + action), Hub-backed storage and versioning, and reference implementations of several modern policies , substantially reducing the engineering required to get from data collection to training and evaluation.

1.4 The Honest Counterweight

Physical AI has progressed rapidly, but larger models and impressive demonstrations do not remove the practical limits of robot learning. Four issues matter when interpreting research results or planning a deployment.

Data diversity remains a bottleneck

Robot learning cannot rely on internet-scale text alone: learning physical control requires observations paired with actions and their consequences. Collecting those demonstrations takes hardware, operator time, and repeated interaction with the environment.

Dataset size is only part of the problem. A million trajectories collected on similar robots in controlled spaces may still provide limited coverage of unfamiliar objects, lighting, layouts, and robot bodies. What matters is whether the training data covers the variation the deployed system will encounter.

A useful approach is the data pyramid: web and human video provide broad visual knowledge, simulation and synthetic data expand training coverage, and real robot demonstrations ground the policy in executable actions. These sources complement one another; adding more data from one source does not automatically replace the others.

Evaluation depends on the conditions

Two systems reporting a high success rate on “pick and place” may have faced very different tests. Object placement, camera views, hardware wear, lighting, and scene-reset procedures all affect the result. A success rate is meaningful only alongside its evaluation conditions.

Performance also varies substantially across tasks. Picking up an object can tolerate positioning errors that would cause an insertion task to fail. Recognizing an object, manipulating it precisely, and confirming successful completion are separate capabilities.

Useful evaluations should therefore report task-level results, repeated trials, failure modes, and performance under changed conditions. An overall average can hide the weaknesses that matter most in deployment.

Lab success is not production readiness

An 80% success rate may demonstrate research progress, but it also means roughly one failure in five attempts. Whether that is acceptable depends on the task, the consequences of failure, and the cost of human intervention.

Production requires more than successful demonstrations: consistent cycle times, repeatability, safety, maintenance, and recovery from mistakes. A robot must handle a slipped object, an obstructed view, or an incomplete action without turning every failure into a manual reset. These requirements should shape the system design and evaluation from the beginning.

World models must prove real-world transfer

World models can support prediction, synthetic data generation, and simulated practice, potentially reducing the need for repeated physical trials. Their usefulness depends on whether the generated experience captures the details that affect real actions—especially contact, friction, deformation, and unexpected changes.

The key test is reliable transfer to physical robots in unfamiliar environments. Plausible generated video alone does not establish this. World models should also be considered alongside approaches that separate high-level reasoning from motor execution; these approaches can address different parts of the same robotics problem.

2. What Is Inside a Modern Vision-Language-Action Model?

A VLA is not simply a vision-language model with a robot attached to it. At a high level, most modern systems can be decomposed into a multimodal encoder/backbone, a mechanism for incorporating robot state and embodiment information, and an action-generation module that converts the resulting representation into executable robot behavior.

The interesting architectural question is therefore not only how large is the model? but:

Where does perception end, where does reasoning happen, and how is a continuous robot trajectory produced from those representations?

A useful abstraction is:

Observation → Multimodal Representation → Reasoning / Conditioning → Action Generation → Action Chunk

The exact boundaries vary substantially across model families. Some models make the action tokens part of an autoregressive language-model stream; others attach a dedicated continuous action expert; newer systems increasingly separate high-level embodied reasoning from low-level motor execution.

2.1 The Common Anatomy of a VLA

Although implementations differ, a modern VLA can usually be understood through four architectural layers.

Perception: turning the physical world into representations

The first stage converts camera observations,and sometimes depth, audio, or other sensors,into features that the rest of the model can process.

Typical components include:

• Vision transformers / visual encoders for RGB or multi-view observations
• Multimodal projectors that map visual features into the backbone's representation space
• Temporal or multi-view fusion when several frames or cameras are used
• Robot-state encoders for joint positions, velocities, gripper state, or other proprioception

The visual backbone is often pretrained independently and then connected to the language or policy backbone. For example, OpenVLA uses a fused SigLIP + DINOv2 visual encoder followed by a projector into a Llama 2 7B language-model backbone. SmolVLA similarly uses a pretrained vision-language backbone, combining a SigLIP vision encoder with SmolLM2.

The important point is that the VLA does not operate directly on pixels. It operates on learned representations of the scene, instruction, and robot state.

Language and multimodal reasoning backbone

The next component provides the semantic and contextual representation needed to connect an instruction such as "pick up the red cup" with what the robot sees.

This backbone may be:

• a pretrained VLM,
• an LLM augmented with a vision encoder,
• a multimodal transformer,
• or, in newer physical-AI systems, a more specialized world/physical reasoning model.

Historically, the simplest recipe was to start from a strong VLM and extend its output space to include robot actions. RT-2 is the canonical example: robot actions were represented as text tokens, allowing the same transformer to model language and action sequences in a unified autoregressive format.

OpenVLA follows a related but open architecture: a 7B language backbone receives visual features and predicts tokenized actions, which are subsequently decoded into continuous robot commands.

Action representation: the critical architectural choice

This is where modern VLA architectures diverge most visibly.

A robot ultimately needs continuous-valued control signals, but neural networks can represent those signals in several ways.

Discrete action tokens

The model converts an action vector or action chunk into a sequence of discrete tokens and predicts them autoregressively.

Examples: RT-2, OpenVLA

This approach is attractive because it fits naturally into the autoregressive language-model training paradigm. The downside is that generating a long sequence of action tokens can introduce inference latency and quantization limits.

Continuous action experts

Instead of forcing actions through the language-token vocabulary, the model attaches a dedicated action-generation module that predicts continuous trajectories.

π₀ is an important example. It uses a pretrained PaliGemma VLM backbone together with a roughly 300M-parameter action expert, for a total of about 3.3B parameters. The action expert is trained with conditional flow matching to generate continuous action chunks.

SmolVLA follows the same broad direction at a much smaller scale: its 450M-parameter architecture combines a VLM with a roughly 100M-parameter flow-matching action expert that generates continuous action chunks.

Diffusion / flow-matching action transformers

Another major family uses a transformer-based generative action module to model a distribution over future trajectories.

GR00T N1.6, for example, combines a Cosmos-family VLM with a 32-layer diffusion transformer action head. Its continuous actions are generated through a flow-matching formulation implemented with a DiT-style architecture.

The important distinction is therefore not simply "VLA vs diffusion." A more useful abstraction is:

Discrete autoregressive action modeling vs. continuous generative action modeling.

The latter includes closely related flow-matching and diffusion-transformer designs.

2.2 Why Action Chunking Became So Important

Modern robot policies commonly predict a short horizon of future actions rather than only the next control command.

Instead of:

observation → one action

the policy can produce:

observation → [aₜ, aₜ₊₁, aₜ₊₂, ... , aₜ₊ₕ]

This is called action chunking.

It gives the policy a short temporal horizon and allows the action generator to model coordinated motion rather than making every motor decision independently.

This also changes the systems problem. A learned policy may run at a much lower frequency than the robot's low-level controller. The policy can generate a chunk while the robot's controller executes and interpolates the resulting trajectory.

Modern systems therefore increasingly care about:

• inference latency,
• chunk horizon,
• asynchronous inference,
• overlapping action chunks,
• temporal consistency,
• and how quickly a new observation can replace or modify an existing trajectory.

SmolVLA explicitly provides asynchronous inference, while LeRobot's Real-Time Chunking work addresses the same fundamental issue for flow-matching policies such as π₀, π₀.₅, and SmolVLA.

The key architectural insight is:

The policy is not necessarily the servo controller. It is often a trajectory generator operating above a faster low-level control loop.

2.3 The Major Architectural Families

The current landscape can be organized into a few recurring patterns.

These are architectural families, not mutually exclusive categories. A model can combine several of them,for example, a VLM backbone with discrete action pretraining and a continuous flow-matching action expert for post-training and inference.

The Important Shift: From "Action Head" to "Action Expert"

Early VLA designs could be summarized as:

VLM + action head

That description is still useful, but it is increasingly too coarse.

In newer architectures, the action module can be a substantial generative model in its own right. It may have:

• its own transformer blocks,
• separate attention patterns,
• dedicated action embeddings,
• time/noise conditioning,
• embodiment-specific state encoders,
• and a generative objective such as diffusion or flow matching.

π₀ makes this separation particularly explicit: the observation and language context form the conditioning prefix, while a dedicated action expert models the continuous action chunk.

SmolVLA follows the same conceptual pattern while aggressively reducing the size of the action module for efficient inference.

This suggests a useful mental model:

VLM = "What is happening and what should be done?"

Action expert = "Given that context, what continuous trajectory should the robot execute?"

The boundary is not universal, but it captures a major direction in current VLA design.

2.4 Architectural Topology Is Also Changing

The next question is whether all of this intelligence should live inside one model.

There are at least three important answers.

Monolithic VLA

The model directly maps: vision + language + state → action

This is the cleanest end-to-end formulation. RT-2 is an early landmark: the same transformer represents both language and robot actions as tokens. OpenVLA continues the general VLM-to-action formulation with a 7B open model.

Decoupled reasoning and action

A different design separates:

high-level reasoning / planning → low-level VLA execution

This is now a particularly important architecture.

Google's Gemini Robotics ER 2 is explicitly positioned as a high-level embodied reasoning model that understands the physical environment, plans multi-step tasks, tracks progress, and hands motor execution to a lower-level VLA. Gemini Robotics 2 provides the corresponding VLA layer for motor control, while Gemini Robotics On-Device 2 targets efficient local execution.

NVIDIA's GR00T family also uses a dual-system concept in which a vision-language component provides higher-level contextual reasoning and a generative action component produces continuous motor actions.

The attraction is clear: planning and motor control have different computational and temporal requirements.

World-action models

A more ambitious direction is to make the model represent not only the current observation, but also how the physical world evolves under actions.

Cosmos 3 is an important example of this direction. NVIDIA describes it as an open omnimodal foundation model that unifies physical reasoning, world generation, and action generation through a Mixture-of-Transformers (MoT) architecture. The family spans 4B, 16B, and 64B variants and also includes policy-specific variants, so it is better understood as a foundation-model family for physical AI, rather than simply a single large policy that replaces a VLA.

That distinction matters. Cosmos 3 is not merely an "action head attached to a VLM"; its design is moving toward a shared representation of reasoning, generation, prediction, and action.

The seven patterns discussed so far , autoregressive action-token VLAs, VLM-plus-continuous-action-expert designs, diffusion/flow-matching action transformers, decoupled reasoning-and-execution architectures, world/action foundation models, the common chunked-action representation, and the two-rate control loop that deploys it , can be laid out side by side:

Mid-2026: Two Different Answers to "Where Should Robot Intelligence Live?"

By mid-2026, two particularly visible architectural directions had emerged.

Cosmos 3: broaden the model into a physical-AI foundation model

NVIDIA's Cosmos 3, released on May 31, 2026, pushes toward an omnimodal physical-AI model that connects understanding, generation, simulation, and action. Its MoT design allows different capabilities and modalities to be handled within a shared architecture, while policy-specific variants extend the family toward robot control.

The important architectural idea is: world understanding ↔ world generation ↔ action
rather than simply: VLM → action head

Gemini Robotics 2: separate embodied reasoning from motor execution

On July 30, 2026, Google introduced Gemini Robotics 2 together with Gemini Robotics ER 2 and Gemini Robotics On-Device 2. Google describes ER 2 as the high-level embodied-reasoning model and Gemini Robotics 2 as the VLA that converts visual and language information into motor control. ER 2 can plan multi-step tasks lasting several minutes and hand execution to the VLA.

This is a different architectural bet:

slow, deliberative reasoning → fast, specialized execution

rather than putting every level of intelligence into one monolithic policy.

Smaller Policies, Larger Foundations

It is tempting to describe the field as simply moving toward smaller models. The evidence is more nuanced.

Several practical VLA policies are indeed much smaller than early systems:

• RT-2 / RT-2-X: up to roughly 55B parameters
• OpenVLA: 7B
• π₀: about 3.3B
• GR00T N1.6: 3B
• SmolVLA: 450M

These numbers illustrate a strong engineering trend: the model running the control policy does not have to be the largest model in the overall robotics stack.

At the same time, large foundation models remain important for world understanding, reasoning, synthetic data generation, and policy development. Cosmos 3, for example, ranges up to 64B parameters while also providing smaller variants and policy-specific models.

So the more accurate conclusion is:

Robotics is becoming heterogeneous in scale: large models can provide world knowledge and reasoning, while smaller specialized policies can provide low-latency motor execution.

The Architectural Trend

The evolution can be summarized across three stages.

Early VLA , a simple, static pipeline. Vision and language are encoded, a transformer maps them to a fixed set of discrete action labels, and that goes straight to the robot. No continuous control, no explicit physical reasoning.

Modern generalist VLA , continuous control from multiple inputs. Vision, language, and proprioceptive state feed a shared multimodal backbone, which hands off to a dedicated continuous action expert. The output is an action chunk, not a fixed label set.

Emerging physical-AI stack , reasoning splits from acting. Vision and language branch into three parallel streams: world/physical reasoning, high-level planning, and a VLA action expert. Only the action stream continues down to an action chunk and low-level control , the other two branches inform what to do, not how to move.

The important evolution is not simply "bigger VLMs."

It is the increasing specialization of the model into:

perception → representation → reasoning → trajectory generation → control

while some architectures deliberately collapse several of these stages back into a single end-to-end model.

What to Look for When Reading a New VLA Paper

When a new VLA is announced, the model name alone tells you very little. Ask these questions:

1. What is the perception backbone?
   • Which vision encoder?
   • Single image, multi-view, or video?
   • How are visual tokens fused?
2. What is the language/reasoning backbone?
   • LLM?
   • VLM?
   • Specialized physical/world model?
3. How is robot state represented?
   • Proprioceptive tokens?
   • MLP projection?
   • Embodiment-specific conditioning?
4. How are actions represented?
   • Discrete tokens?
   • Continuous vectors?
   • Latent actions?
5. How are actions generated?
   • Autoregressive decoder?
   • Diffusion?
   • Flow matching?
   • Transformer action expert?
6. Does it predict one action or an action chunk?
7. Is reasoning coupled to motor control or separated into another model?
8. Is the model a policy, a foundation model, or both?
9. Where does inference actually run?
   • Datacenter?
   • Edge GPU?
   • On-robot compute?
10. What part is pretrained and what part is robot-specific?

Those questions reveal the real architecture much faster than parameter count or benchmark tables.

The Big Picture

The modern VLA is converging on a surprisingly simple conceptual decomposition:

A multimodal backbone builds a representation of the world and instruction; a state/embodiment interface grounds that representation in a particular robot; and an action generator turns the resulting context into a continuous trajectory.

From there, the field is exploring three major directions:

• better action generators , discrete tokens → continuous experts → diffusion/flow-matching trajectory models;
• better architectural separation , monolithic policies → reasoning + execution stacks;
• richer physical representations , VLMs → world models and multimodal physical-AI foundations.

The result is that "VLA" no longer describes one architecture. It describes a family of design choices around the same fundamental problem:

How do we transform perception and language into reliable, temporally coherent action in the physical world?

3. Common Hardware Platforms for AI and Teleoperation Research

While modern Vision-Language-Action (VLA) policies and robot learning frameworks are designed to be hardware-agnostic meaning the underlying representations are not strictly tied to a single rigid machine standardized physical setups remain essential. The platforms below are widely adopted across academic and industry research labs. Crucially, many are open-sourced and low-cost, lowering the barrier for community-driven data collection, teleoperation recording, and standardized real-world model testing and benchmarking.

Robotic Hands & End-Effectors

Focused on fine-motor dexterity, cost-effective replication, and end-effector manipulation.

• SO-100 / SO-101: Ultra-low-cost, 3D-printable open-source arms widely used for accessible teleoperation and imitation learning.
• Koch v1.1: Compact, open-hardware training and teleoperation arm.
• LeKiwi: Lightweight, mobile learning platform designed for accessible multi-agent or edge experiments.
• HOPEJr Arm: Dexterous articulated platform for manipulation research.

Humanoid & Teleoperation Platforms

Kinematically aligned with human demonstration, enabling intuitive whole-body teleoperation and data collection.

• Reachy 2: Open-source, expressive dual-arm torso platform designed for interactive AI and telemanipulation.
• Unitree G1: Compact bipedal humanoid designed for dynamic locomotion and whole-body teleoperation.
• OpenArm: Open-hardware, modular teleoperation setup for dexterous manipulation.

Research Robot Arms

Benchmark manipulators standard across manipulation datasets (e.g., BridgeData, Open X-Embodiment) for testing policy robustness.

• ALOHA: Open-source dual-arm teleoperation workstation optimized for fine bimanual data collection.
• WidowX 250 Robot Arm: Standard benchmark platform across foundational robot learning datasets.
• OMX & reBot B601-DM: Accessible desktop robot arms for fast iteration and policy validation.
• Franka Emika Panda: Industry-standard 7-DoF research arm featuring torque sensing and compliant control.
• Universal Robots (UR5 / UR5e / UR10): High-payload collaborative arms standard in industrial and lab-scale benchmarking.

Mobile & Legged Platforms

Unifies locomotion with manipulation to test spatial navigation, generalization, and loco-manipulation in unstructured environments.

• Mobile ALOHA: Mobile, untethered dual-arm platform enabling whole-body household task learning.
• Google Everyday Robots & TIAGo (PAL Robotics): Mobile manipulators that formed the basis of foundational VLA models (e.g., RT-1, RT-2).
• Unitree Go2 / B2 & Boston Dynamics Spot: Quadrupedal platforms equipped with manipulator arms for traversing rough terrain and agile loco-manipulation.

4. Actuators and Motors used in Robots (AI Research)

When designing or studying embodied AI, the hardware that executes the policy is just as important as the neural network itself. Motors commonly used in robots for AI, RL, and teleoperation research must be capable of translating high-level commands into physical actions flawlessly.

4.1 Hardware Considerations for AI-Controlled Robots

When an AI policy controls a physical robot, the physical actuation layer dictates success or failure. Several critical hardware and feedback factors must be considered:

• Feedback Latency & Control Loops: AI policies typically output actions at 10–50 Hz, but the low-level motor controllers must execute these at 1 kHz to 20 kHz. Any latency in the encoder feedback loop causes the robot to miss critical contact events or become unstable.
• Jitteriness & PID Issues: Standard high-gain PID controllers often cause micro-oscillations ("jitter") when holding a static pose. For AI setups relying on wrist-mounted cameras, this jitter severely degrades the visual data being fed into the neural network.
• Teleoperation Smoothness: For imitation learning, humans must teleoperate the robots to collect data. High backdrivability and low friction are required so the "leader" arm feels perfectly smooth and the "follower" arm mirrors it without latency.
• Torque & Gear Ratios: High-reduction gears (like harmonic drives) provide massive torque and zero backlash but make the joint rigid. Low-reduction gears (used in Quasi-Direct Drives) sacrifice some torque to provide mechanical transparency, allowing the robot to safely absorb physical impacts.
• Encoder Mechanisms & Calibration: High-end robots use absolute optical encoders for the highest resolution, while budget setups use magnetic encoders. Proper calibration methods such as homing routines, zeroing absolute encoders on boot, and accounting for gear slop are mandatory to ensure the AI's internal coordinate system perfectly matches physical reality.

Below is a detailed breakdown of the most common feedback-enabled motors and actuators used across modern AI robotics research.

Smart Serial Bus Servos

Overview: Smart serial bus servos pack a DC motor, gear train, microcontroller, and an encoder (often magnetic) into a single block. They communicate over a daisy-chained serial bus (like TTL or RS-485), meaning you can read the exact position, velocity, and sometimes temperature/load of every joint simultaneously.

• Common Examples: Dynamixel (XL430, MX, XW series), Hiwonder bus servos, Feetech (STS series).

Closed-Loop Stepper Motors

Overview: Traditional stepper motors operate "open-loop" by counting electrical pulses, meaning if they hit an obstacle, they lose their position. Closed-loop steppers solve this by adding a rear-mounted optical or magnetic encoder.

• Common Examples: NEMA 17 or NEMA 23 motors equipped with encoders and a driver board.

Highly Integrated BLDC Actuators

Overview: These modules combine a Brushless DC (BLDC) motor, a high-reduction gearbox (often a strain wave/harmonic drive for zero backlash), a motor driver running Field Oriented Control (FOC), and high-speed dual encoders.

• Common Examples: T-Motor (AK series), Cubemars (AK series), MyActuator, Maxon modular setups.

Quasi-Direct Drive (QDD) Actuators

Overview: QDD actuators utilize a large-diameter "pancake" BLDC motor paired with a very low-ratio planetary gear (e.g., 6:1 to 10:1) to minimize inertia.

• Common Examples: Motors inspired by MIT Mini Cheetah, Unitree (Go-M8010), TMotor (QDD series).

Frameless Torque Motors

Overview: Frameless torque motors are supplied simply as a bare rotor and stator. Engineers design the joint housing directly around them to save weight and space.

• Common Examples: Kollmorgen TBM series, Maxon EC Frameless, Mosrac U-series.

4.2 Summary Comparison Table

5. Perception and Compute Hardware for Vision–Language–Action Systems

A Vision–Language–Action (VLA) model closes the loop between visual observation, language instruction, and robot action. The hardware supporting this loop can be divided into two main components:

1. Perception hardware, primarily cameras that observe the workspace, robot, and manipulated objects.
2. Compute hardware, which processes camera streams, executes the VLA policy, and sends actions to the robot controller.

5.1 Camera Systems: The Robot’s Visual Interface

Most current manipulation-oriented VLA models receive one or more RGB images rather than raw point clouds or depth maps. This is partly because their visual encoders—such as SigLIP, DINOv2, CLIP, and other Vision Transformers—are pretrained primarily on ordinary RGB images.

Common Camera Positions

Manipulation platforms commonly combine several viewpoints:

• Third-person or base camera: Provides a wide view of the robot, objects, receptacles, and surrounding workspace.
• Wrist camera: Moves with the end effector and provides close-range information for grasping, insertion, and fine manipulation.
• Overhead camera: Reduces occlusion and provides a relatively stable view of object positions.
• Head or mast camera: Common on humanoids and mobile manipulators, where it provides a human-like view of the environment.
• Navigation cameras: Wide-angle, stereo, fisheye, or RGB-D cameras used for mapping, localization, and obstacle detection.

Multiple views are particularly useful because robot arms, grippers, and objects frequently occlude one another.

RGB Cameras

Commodity USB webcams remain surprisingly common in robot-learning research because they are inexpensive, easy to replace, and well supported by Linux, OpenCV, and ROS.

Logitech C920/C922 and Similar USB Webcams

The Logitech C920/C922 family is widely used in low-cost imitation-learning and ALOHA-style systems. The C922 can capture 1080p at 30 FPS or 720p at 60 FPS and has autofocus and a 78° diagonal field of view.

Advantages

• Low cost and easy availability
• Simple USB connection
• Good Linux, OpenCV, and ROS compatibility
• Adequate image quality for controlled indoor environments
• Useful for scalable multi-camera data collection

Limitations

• No native depth measurement
• Autofocus and automatic exposure can cause visual inconsistency
• Rolling shutters can distort images during rapid motion
• Multi-camera synchronization is usually limited
• Consumer cameras are not designed for vibration, dust, or industrial temperature ranges

For robot learning, it is often useful to disable automatic exposure, white balance, and focus after selecting suitable values. Otherwise, the same scene may appear statistically different between demonstrations.

Higher-Resolution RGB Cameras

Models such as the Logitech Brio or equivalent 4K USB cameras can provide higher-quality source images, better digital cropping, and improved low-light performance. However, 4K input is rarely passed directly into a VLA because image-token count, encoder latency, memory use, and data-storage requirements increase rapidly with resolution.

Higher native resolution is most beneficial when:

• Small objects must remain visible after cropping.
• One camera covers a large workspace.
• Recorded data may later be reused with different crops.
• Images are used for annotation, inspection, or conventional computer vision in addition to the VLA.

RGB-D and Stereo Depth Cameras

An RGB-D camera supplies an RGB image together with an aligned or alignable depth map. Stereo cameras estimate depth from the disparity between two spatially separated image sensors. Some systems add an infrared projector to improve correspondence on textureless indoor surfaces.

Depth is valuable for:

• Estimating object-to-camera distance
• Creating point clouds or occupancy maps
• Robot-camera and hand-eye calibration
• Collision avoidance and safety monitoring
• Classical grasp planning and pose estimation
• Navigation, mapping, and scene reconstruction
• Generating auxiliary training labels

However, owning an RGB-D camera does not mean that depth is necessarily passed into the VLA. Many VLA policies use only its RGB stream while other modules use depth for mapping, safety, grasp verification, or motion planning. Feeding depth directly into a VLA generally requires an architecture and training dataset designed for that modality.

RealSense D435i

The RealSense D435i combines active stereo depth, a wide 87 degree 58 degree depth field of view, global-shutter depth sensors, and an integrated IMU. Its wide view and short practical working distance make it suitable for tabletop manipulation, wrist mounting, and mobile robotics.

The IMU can assist visual-inertial estimation, although using it effectively requires accurate timing and calibration. The D435i’s RGB sensor is separate from its stereo depth pair, so RGB-to-depth alignment must be handled through the camera SDK.

RealSense D455

The D455 uses a longer 95 mm stereo baseline than the D435-series cameras, improving depth accuracy at longer distances. It also uses a global-shutter RGB sensor with a field of view better matched to the depth sensors. RealSense specifies depth error below 2% at 4 m under its stated test conditions.

It is generally more appropriate than the D435i when the camera observes a larger workspace or is mounted farther from the manipulation area.

RealSense D405

The D405 is designed for short-range, high-accuracy depth perception and is physically smaller than many other RGB-D cameras.

This makes the D405 particularly suitable for:

• Wrist-mounted perception
• Close-range grasping
• Small-object manipulation
• Tabletop demonstrations

Stereolabs ZED 2/2i and ZED Mini

The ZED family uses passive stereo cameras and GPU-accelerated depth estimation. Unlike active infrared depth sensors, passive stereo can operate indoors and outdoors, although it requires sufficient texture and illumination.

ZED cameras are useful when:

• Longer-range depth is required.
• Indoor and outdoor operation is needed.
• The platform already contains an NVIDIA GPU.
• High-quality stereo video and visual odometry are important.

Their main trade-offs are greater bandwidth and compute requirements than ordinary webcams, especially when neural depth, tracking, or point-cloud generation is enabled.

Luxonis OAK-D Family

OAK-D cameras combine RGB imaging, global-shutter stereo sensors, depth estimation, and an onboard vision processor. Depending on the model, they can execute depth processing and compact neural networks inside the camera, reducing the workload and USB traffic reaching the host computer. OAK-D Pro variants add active infrared illumination for low-texture indoor scenes.

OAK-D is attractive for modular robot systems that need:

• On-camera object detection or tracking
• Stereo depth without consuming the main GPU
• Multiple synchronized image sensors
• A compact RGB, depth, and edge-inference unit

Camera Selection Criteria for Performance Measurements

The camera should be chosen according to the policy and task, rather than simply by maximum resolution.

Camera calibration remains important even when the VLA consumes only RGB images. Intrinsic calibration models focal length, optical center, and lens distortion, while extrinsic calibration defines the camera pose relative to another coordinate frame.

5.2 Compute Hardware: From Robot Control to Model Training

A VLA robot rarely relies on one processor for every task. A practical system often contains several computing layers:

1. A microcontroller or real-time controller for motors, encoders, and safety-critical control.
2. A CPU-based robot computer for ROS 2, device drivers, networking, logging, and motion-control interfaces.
3. A GPU or AI accelerator for image encoding and VLA inference.
4. An optional off-board workstation or server for large-model inference.
5. A separate multi-GPU server or cloud cluster for pretraining and large-scale fine-tuning.

VLA inference combines several computationally expensive operations:

• Encoding one or more camera images with a Vision Transformer
• Processing language and visual tokens using a large transformer
• Predicting discrete actions, continuous actions, diffusion trajectories, or flow-matching action chunks
• Repeating inference frequently enough for closed-loop control

The critical hardware constraint is often memory capacity and bandwidth, not only advertised TOPS or FLOPS. Model weights, image tokens, activations, attention caches, and middleware must all fit in memory.

Numbers such as INT8 TOPS, FP4 TFLOPS, FP16 TFLOPS, and FP32 TFLOPS are based on different numerical precisions and assumptions. They should therefore not be compared as if they were the same metric. Real VLA performance must ultimately be measured using:

• End-to-end action latency
• Achievable policy frequency
• Number and resolution of camera inputs
• Model precision and quantization
• Thermal throttling
• Power consumption
• Robot-task success rate

Consumer Desktop GPUs

NVIDIA GeForce RTX 4090

The RTX 4090 has 24 GB of VRAM and remains an important reference platform for VLA research. It is suitable for:

• Inference of many 3B–7B parameter VLAs
• Quantized inference of larger models
• LoRA or QLoRA adaptation
• Training smaller robot policies
• Simulation and data processing
• Off-board real-time control

The RTX 4090 is therefore highly capable for laboratory inference, but its size, approximately 450 W power requirement, cooling needs, and lack of embedded interfaces make it difficult to mount directly on a compact mobile robot.

RTX 5090 and Similar Consumer GPUs

Newer consumer GPUs offer higher throughput and, in the RTX 5090’s case, 32 GB of VRAM. They are useful for local VLA fine-tuning and off-board inference, but still have the power, cooling, and mechanical constraints of desktop hardware.

Consumer GPUs provide excellent performance per dollar, but workstation GPUs are generally preferable when a project requires larger memory, ECC, long-duration operation, or professional support.

Professional and Workstation GPUs

NVIDIA RTX A6000 and RTX 6000 Ada

The RTX A6000 provides 48 GB of GDDR6 ECC memory in a 300 W workstation form factor. The newer RTX 6000 Ada also provides 48 GB of ECC memory.

Their larger VRAM is valuable for:

• Full-precision inference of larger VLAs
• Larger training batches
• Multi-camera, high-resolution models
• Full or parameter-efficient fine-tuning
• Running simulation and policy inference on the same workstation
• Long research experiments requiring ECC memory

The RTX A6000 is older than the RTX 6000 Ada but remains relevant in laboratories because of its 48 GB memory and availability in used or existing workstations.

RTX PRO 6000 Blackwell

The RTX PRO 6000 Blackwell Workstation Edition provides 96 GB of GDDR7 ECC memory. This makes it considerably better suited to large-model inference and memory-intensive fine-tuning than consumer GPUs, although its maximum power consumption can reach 600 W.
It is best viewed as a workstation or server-class device, not as an embedded robot computer.

Data-Center Training GPUs

Large general-purpose VLAs are usually pretrained on GPU clusters using accelerators such as:

• NVIDIA A100
• NVIDIA H100/H200
• NVIDIA B200/B300
• NVIDIA L40S
• Cloud TPU systems

These platforms provide high-bandwidth memory, fast multi-GPU interconnects, and distributed-training support. They are appropriate for:

• Training on hundreds of thousands or millions of robot demonstrations
• Full fine-tuning of large VLM backbones
• Large effective batch sizes
• Distributed data and model parallelism
• Extensive ablation studies

A research group does not necessarily need such infrastructure for deployment. A model can be pretrained on a cluster, fine-tuned on a workstation, and deployed on an embedded computer.

Embedded NVIDIA Jetson Platforms

Jetson AGX Orin

Jetson AGX Orin is one of the most common embedded computers for advanced robotics. The 64 GB version provides up to 275 TOPS of INT8 AI performance and has a configurable power range of approximately 15–60 W. It integrates a 12-core Arm Cortex-A78AE CPU, an Ampere GPU, video encoders and decoders, and embedded camera and I/O interfaces.

It is suitable for:

• Small and medium VLA inference
• Quantized or optimized 7B-class policies
• Vision encoders and conventional perception models
• ROS 2, Isaac ROS, camera processing, and navigation
• Onboard inference where power efficiency matters

Its 64 GB unified memory is substantially larger than the VRAM of many consumer GPUs, but its raw transformer throughput is much lower than that of an RTX 4090. A model fitting in memory does not guarantee that it will produce actions quickly enough for the task.

Jetson Orin NX and Orin Nano provide smaller and lower-power alternatives. They are well suited to conventional perception, compact policies, or client-side robot control, but large VLA inference often requires aggressive quantization, model distillation, or a remote server.

Jetson AGX Thor

Jetson AGX Thor is designed specifically for high-end physical AI and robotics. The Jetson T5000 module combines a Blackwell GPU with 128 GB of LPDDR5X memory and delivers up to 2,070 FP4 TFLOPS within a configurable 40–130 W power envelope.

Compared with AGX Orin, Thor is better positioned for:

• Larger multimodal and generative robot models
• Multiple high-resolution camera streams
• Humanoid and mobile-manipulator inference
• Concurrent VLA, perception, mapping, and planning workloads
• Transformer inference using low-precision Blackwell Tensor Cores

Thor does not remove the need for optimization. Camera preprocessing, model architecture, action decoding, control frequency, and thermal design still determine whether a VLA can operate in real time.

Compact Blackwell AI Computers

NVIDIA DGX Spark and GB10 Systems

The NVIDIA GB10 Grace Blackwell Superchip combines a 20-core Arm CPU, Blackwell GPU, and 128 GB of coherent LPDDR5X unified memory. DGX Spark provides up to 1 PFLOP of sparse FP4 AI performance, 273 GB/s of memory bandwidth, and a 140 W GB10 thermal design power.

GB10-based systems are useful as:

• Compact off-board VLA inference servers
• Laboratory development machines
• Large-model prototyping systems
• Local LoRA/QLoRA fine-tuning platforms
• Compute units for robots large enough to accommodate their power and cooling requirements

The 128 GB unified memory allows models that cannot fit on a 24 or 48 GB discrete GPU to be loaded locally. However, unified-memory capacity should not be confused with the bandwidth of high-end discrete GPU VRAM or data-center HBM. GB10 is optimized for capacity-efficient desktop AI, while large discrete GPUs may provide higher throughput for models that fit inside their VRAM.

Dell Pro Max with GB10

Dell Pro Max with GB10 is an OEM implementation of the same general compact-AI concept. It includes the GB10 Grace Blackwell platform, 128 GB LPDDR5X unified memory, DGX OS, and NVMe storage in a small desktop chassis.

For VLA laboratories, such a system can sit near the robot and provide wired or wireless inference without requiring a full tower workstation. It is more naturally treated as a compact inference server than as a hard real-time motor controller.

AMD Compact AI PCs

Systems based on the AMD Ryzen AI Max+ 395 combine:

• 16 Zen 5 CPU cores
• Radeon 8060S integrated graphics with 40 RDNA 3.5 compute units
• An XDNA 2 NPU
• Up to 128 GB LPDDR5X unified memory

These systems are attractive for portable development, data processing, simulation, and large quantized models whose weights benefit from unified memory. AMD has also expanded ROCm support for the platform.

Nevertheless, NVIDIA remains more common in VLA research because most released robotics repositories, optimized attention kernels, TensorRT deployments, and Isaac ROS pipelines are developed and tested first with CUDA. An AMD computer should therefore be selected only after confirming that the intended VLA repository, numerical precision, attention implementation, and camera stack work correctly with ROCm.

Standard Intel and AMD CPUs

Ordinary x86 processors remain essential even when a robot contains a powerful GPU. Intel Core/Core Ultra, AMD Ryzen, Xeon, and EPYC CPUs commonly handle:

• ROS 2 nodes and message passing
• Camera acquisition and decoding
• Robot and gripper drivers
• Dataset recording and compression
• Language-command interfaces
• Network communication
• Motion planning and inverse kinematics
• Safety monitoring and system supervision

Modern CPUs can run small or heavily quantized vision and language models, but CPU-only inference is generally too slow for large, closed-loop VLA policies. A CPU-only workstation remains useful for data collection if policy inference is performed by a remote GPU server.

The CPU should provide sufficient:

• Single-thread performance for control and middleware
• Core count for multi-camera decoding and logging
• PCIe lanes for GPUs and capture cards
• USB controllers for multiple cameras
• RAM and NVMe bandwidth for dataset recording
• Ethernet performance for remote inference

Raspberry Pi 5

Raspberry Pi 5 uses a quad-core Arm Cortex-A76 CPU running at 2.4 GHz and is available with up to 16 GB RAM.

By itself, it is better suited to:

• Robot startup and supervision
• Sensor and actuator interfaces
• ROS 2 nodes
• Camera streaming
• Teleoperation
• Dataset collection
• Sending images to a remote VLA server
• Executing received action commands

It should not be considered a direct replacement for Jetson AGX Orin or a desktop GPU for billion-parameter VLA inference.

The Raspberry Pi AI HAT+ adds a 13 or 26 TOPS Hailo accelerator for conventional vision networks. The newer AI HAT+ 2 uses a Hailo-10H accelerator, provides 40 TOPS at INT4, and includes 8 GB of dedicated memory for supported generative-AI workloads.

These accelerators can run supported detection, segmentation, pose-estimation, or compact VLM components, but compatibility depends on Hailo’s compiler and supported operator set. Most open VLA repositories cannot simply be moved from CUDA to a Hailo device without model conversion or architectural changes.

5.3 Practical Hardware Configurations

Onboard Versus Off-Board Inference

Onboard inference reduces network dependence and normally gives more predictable latency. It is preferable for mobile robots operating beyond reliable network coverage or around people. Its limitations are battery consumption, heat, weight, and available memory.

Off-board inference allows a robot to use an RTX workstation or server-class GPU. Images and robot state are transmitted to the server, and action chunks are returned. This approach is common in research, but the system must measure network latency, reject stale actions, and provide a safe fallback when communication is interrupted.

A practical hybrid design runs motor control, emergency stopping, collision limits, and basic perception onboard while executing the large VLA on a nearby workstation. Action chunking can partially hide model and network latency, although long open-loop chunks reduce the robot’s ability to react to unexpected changes.

Recommended Design Principle

The correct hardware stack is not necessarily the camera with the highest resolution or the processor with the largest advertised TOPS value. A reliable VLA platform should be designed around the complete perception-to-action loop:

Exposure→Image transfer→Preprocessing→VLA inference→Action decoding→Action 

The total latency of this pipeline not GPU inference time alone determines how quickly the robot can respond. Camera placement, synchronization, calibration, memory capacity, software compatibility, network delay, thermal limits, and control safety are therefore as important as nominal AI compute.

6. The SO-101 Robotic Arm: Hardware Deep Dive

6.1 STL Files & 3D Printing

The SO-101 is a fully 3D-printable robotic arm designed by TheRobotStudio. All structural components are available as STL files.

Printing Specifications:

Key Printing Tips:

• [Tip 1: orientation matters for strength , describe which parts should be printed in which orientation]
• [Tip 2: tolerance adjustments for your specific printer]
• [Tip 3: post-processing , sanding, cleaning support material]

6.2 Parts Anatomy: What's Called What

Every part of the SO-101 has a specific function. Here's the complete breakdown:

Degrees of Freedom & Arm Design Principles

What is a Degree of Freedom (DoF)?

A degree of freedom represents one independent axis of motion. The SO-100 has 6 DoF (+ 1 for the gripper), which matches the human arm's primary movement capabilities:

1. Base Rotation (Joint 1) , Yaw: Rotates the entire arm left/right
2. Shoulder Pitch (Joint 2) , Pitch: Lifts the arm up/down
3. Elbow Pitch (Joint 3) , Pitch: Bends the forearm
4. Wrist Roll (Joint 4) , Roll: Rotates the forearm along its axis
5. Wrist Pitch (Joint 5) , Pitch: Tilts the end effector up/down
6. Wrist Yaw/Roll (Joint 6) , Provides final orientation adjustment
7. Gripper , Open/close (binary or proportional)

Why 6 DoF?

• 3 DoF gives you position (reach any point in 3D space)
• 6 DoF gives you position + orientation (reach any point AND orient the gripper however you want)
• This is the minimum needed for general-purpose manipulation tasks

6.3 Assembly Guide

Assembly requires patience and a systematic approach. Here's the high-level process:

Tools Needed:

• Screwdriver set
• Allen key set (M2, M2.5, M3)
• Wire strippers / cutters
• Soldering iron (for motor connections if needed)
• Digital calipers (helpful for troubleshooting fit)

Assembly Sequence:

1. Pre-check all printed parts , dry-fit joints, check tolerances, sand if needed
2. Install servos into motor mounts , start from the base, work upward
3. Connect the base assembly , secure base servo, attach base plate
4. Build the shoulder , mount shoulder servo, attach upper arm link
5. Attach elbow joint , connect upper arm to forearm via elbow servo
6. Wire management , route servo cables through the arm channels
7. Wrist assembly , install wrist roll and pitch servos
8. Gripper mounting , attach gripper mechanism to wrist
9. Cable routing finalization , secure all cables, add strain relief
10. Repeat for the second arm (leader or follower)

7. Motors, Drivers & Feedback Systems

STS3215 Servo

The SO-101 uses the Feetech STS3215 Serial Bus Servo. This is a critical choice because:

• Feedback capability: The motor reports its current position, speed, load, voltage, and temperature back to the controller , this is what makes leader-follower teleoperation possible.
• Daisy-chain wiring: All servos share one serial bus (only 3 wires: VCC, GND, Data), massively simplifying wiring.
• 12-bit resolution: 4096 positions per revolution = 0.088° precision.

STS3215 Key Specifications:

7.1 Motor Data & Feedback Signals

Each STS3215 servo provides these data streams, which are essential for both teleoperation and recording training data:

┌─────────────────────────────────────────────────┐
│           STS3215 Feedback Data Packet          │
├─────────────────┬──────────┬────────────────────┤
│ Field           │ Range    │ Why It Matters     │
├─────────────────┼──────────┼────────────────────┤
│ Present Position│ 0 - 4095 │ Joint angle state  │
│ Present Speed   │ 0 - 4095 │ Velocity feedback  │
│ Present Load    │ 0 - 1023 │ Torque estimation  │
│ Present Voltage │ mV       │ Power monitoring   │
│ Present Temp    │ °C       │ Safety monitoring  │
│ Moving Flag     │ 0/1      │ Motion complete?   │
└─────────────────┴──────────┴────────────────────┘

For imitation learning, the key signals are:

• Position , this becomes the action space (what we record and predict)
• Speed , useful for smooth trajectory generation
• Load , can indicate contact/grasping force (advanced use)

7.2 Motor Driver Board: What It Does

The motor driver board (Feetech serial bus adapter / waveshare servo driver) acts as the bridge between your computer and the servo chain:

What the driver interprets/translates:

1. USB to TTL conversion , Your computer sends USB serial data; the board converts to 3.3V/5V TTL logic levels
2. Half-duplex management , The serial bus is shared; the driver handles switching between TX and RX
3. Packet framing , Wraps commands in Feetech protocol headers (0xFF 0xFF, ID, length, instruction, params, checksum)
4. Power distribution , Provides regulated power from external supply to all servos
5. ID-based addressing , Routes commands to the correct servo on the shared bus

8. Calibration

Calibration is the process of establishing a shared reference frame between the leader and follower arms. Without calibration, "position 2048 on the leader" and "position 2048 on the follower" could mean completely different physical angles.

Why Calibration is Critical:

• Each servo's zero-position depends on how it was physically mounted
• Gear teeth mesh at arbitrary angles during assembly
• Even 1-2° error per joint compounds across the chain

Calibration Process:

1. Set each arm to a known physical pose (usually a "home" position , all joints at defined angles)
2. Read the raw encoder values at this known pose for each joint on both arms
3. Compute offset values , the difference between expected and actual readings
4. Store calibration data , saved to a YAML/JSON file for each arm
5. Verify , move leader, check follower matches physically

# Example calibration file structure
# calibration/follower_arm.yaml
joint_1:
  id: 1
  homing_offset: -124
  drive_mode: 0
  range: [0, 4095]
joint_2:
  id: 2
  homing_offset: 87
  drive_mode: 0
  range: [1024, 3072]
# ... and so on for each joint

Key lesson: Recalibrate whenever you disassemble/reassemble any joint. We lost two hours of debugging because a servo cable got unplugged and replugged, shifting the home offset.

9. Leader-Follower Teleoperation

The Twin Concept: Zero-Latency Mirroring

The leader-follower (also called master-slave) paradigm is elegant in its simplicity:

 HUMAN HAND                                        ROBOT ACTION
      │                                                  │
      ▼                                                  ▼
┌──────────┐  Read   ┌──────────┐  Write   ┌──────────────┐
│  Leader  │───────► │  Python  │────────► │   Follower   │
│   Arm    │  pos    │  Script  │   pos    │     Arm      │
│ (no load)│◄─────── │  (loop)  │◄──────── │ (does task)  │
└──────────┘  human  └──────────┘  mirror  └──────────────┘
              moves                         copies motion
              leader                        in real-time

How it achieves near-zero latency:

• Sync read grabs all 7 joint positions from the leader in a single bus transaction (~2-4ms)
• Sync write sends all 7 target positions to the follower in one transaction (~2-4ms)
• Loop runs at 60-200Hz , each cycle takes 5-15ms total
• No processing overhead , it's direct position mapping, no IK, no planning
• The human perceives this as instantaneous

The key insight: Both arms are mechanically identical. So position = angle = physical pose. The follower simply copies the leader's joint angles at every timestep.

LeRobot Python Functions Explained

LeRobot provides a clean abstraction over the hardware. Here are the core functions that make teleoperation work:

# ======================================================
# Core LeRobot Teleoperation Functions (Simplified)
# ======================================================

# 1. Robot Initialization
# Creates the robot object, connects to both arms via serial ports
robot = ManipulatorRobot(
    leader_arms={"main": leader_arm_config},
    follower_arms={"main": follower_arm_config},
    cameras={}  # We'll add cameras later
)
robot.connect()

# What this does under the hood:
# - Opens serial ports for leader and follower
# - Pings all servos to verify connection
# - Loads calibration offsets
# - Sets follower servos to position-control mode
# - Sets leader servos to read-only (no motor drive)

# 2. Teleoperation Loop
# The core loop that makes leader-follower work

def teleoperate(robot, fps=60):
    """
    Reads leader arm positions, writes them to follower arm.
    This is the fundamental loop that everything else builds on.
    """
    while True:
        # Step A: Read leader joint positions
        # Returns a dict: {"joint_1": 2048, "joint_2": 1536, ...}
        leader_pos = robot.leader_arms["main"].read("Present_Position")
        
        # Step B: Apply calibration offsets
        # Converts raw encoder values to normalized angles
        calibrated_pos = apply_calibration(leader_pos)
        
        # Step C: Write to follower
        # Follower mirrors the leader's pose
        robot.follower_arms["main"].write("Goal_Position", calibrated_pos)
        
        # Step D: Maintain loop rate
        time.sleep(1.0 / fps)

# 3. The key data structures

# Observation (what the robot "sees" at each timestep):
observation = {
    "observation.state": tensor([j1, j2, j3, j4, j5, j6, gripper]),
    # shape: (7,) , the 7 joint angles of the follower arm
    
    "observation.images.cam_top": tensor([H, W, 3]),
    # shape: (480, 640, 3) , RGB image from top camera
    
    "observation.images.cam_wrist": tensor([H, W, 3]),
    # shape: (480, 640, 3) , RGB image from wrist camera
}

# Action (what the robot should do):
action = {
    "action": tensor([j1, j2, j3, j4, j5, j6, gripper]),
    # shape: (7,) , target joint angles (= leader's current position)
}

Summary: Think of teleoperation as a live streaming system. The leader "broadcasts" its pose, the follower "plays" it back in real-time. When we add cameras and start recording, we're essentially adding a DVR to this stream.

10. Camera Integration

Camera Types & Selection

Our setup uses two cameras:

1. Top-down camera , Mounted above the workspace, provides scene overview
2. Wrist/side camera , Mounted near the arm, provides close-up task view

OpenCV Integration: RAW vs MJPEG

Understanding camera data formats is crucial for reliable recording:

# ======================================================
# Camera Format: RAW (YUYV) vs MJPEG
# ======================================================

# RAW / YUYV Format:
# - Uncompressed pixel data
# - Higher bandwidth requirement (640x480@30fps ≈ 18 MB/s)
# - Lower latency (no decode step)
# - Multiple cameras may overwhelm USB bandwidth
# - cv2.CAP_PROP_FOURCC = cv2.VideoWriter_fourcc(*'YUYV')

# MJPEG Format:
# - JPEG-compressed frames
# - Lower bandwidth (640x480@30fps ≈ 3-5 MB/s)
# - Slight compression artifacts
# - Multiple cameras work fine on one USB controller
# - cv2.CAP_PROP_FOURCC = cv2.VideoWriter_fourcc(*'MJPG')

# In LeRobot, you configure this in the camera config:
camera_config = OpenCVCameraConfig(
    camera_index=0,
    fps=30,
    width=640,
    height=480,
    color_mode="rgb",
    # MJPEG is recommended for multi-camera setups
)

Practical recommendation: Use MJPEG. The bandwidth savings allow you to run 2+ cameras without frame drops. The compression artifacts at reasonable quality are negligible for learning.

# Quick test to verify camera works before recording
import cv2

cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG'))
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
cap.set(cv2.CAP_PROP_FPS, 30)

while True:
    ret, frame = cap.read()
    if ret:
        cv2.imshow('Camera Test', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()

11. LeRobot Setup Pipeline

Here's the step-by-step setup pipeline we followed to get LeRobot running:

┌─────────────────────────────────────────────────────────┐
│                  LeRobot Setup Pipeline                 │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  1. Environment Setup                                   │
│     └─► Python 3.10+, conda env, install lerobot        │
│                                                         │
│  2. Hardware Connection                                 │
│     └─► Connect USB serial adapters for both arms       │
│     └─► Connect cameras via USB                         │
│     └─► Identify device ports (/dev/ttyUSB0, etc.)      │
│                                                         │
│  3. Motor Configuration                                 │
│     └─► Set unique IDs for each servo (1-7)             │
│     └─► Verify all servos respond to ping               │
│                                                         │
│  4. Calibration                                         │
│     └─► Run calibration script for leader arm           │
│     └─► Run calibration script for follower arm         │
│     └─► Save calibration YAML files                     │
│                                                         │
│  5. Teleoperation Test                                  │
│     └─► Run teleoperation to verify leader-follower     │
│     └─► Check all joints move correctly                 │
│     └─► Verify camera feeds are live                    │
│                                                         │
│  6. Ready for Recording                                 │
│     └─► Configure dataset paths                         │
│     └─► Set episode parameters                          │
│     └─► Begin data collection                           │
│                                                         │
└─────────────────────────────────────────────────────────┘

# Installation commands
conda create -n lerobot python=3.10 -y
conda activate lerobot

git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e ".[feetech]"

# Verify installation
python -c "import lerobot; print(lerobot.__version__)"

# Find connected devices
ls /dev/ttyUSB*  # Linux
# or
ls /dev/tty.usb*  # macOS

# Set servo IDs (run for each servo individually)
python lerobot/scripts/configure_motor.py \
    --port /dev/ttyUSB0 \
    --brand feetech \
    --model sts3215 \
    --baudrate 1000000 \
    --ID 1  # Set to 1, 2, 3, ... for each servo

# Calibration
python lerobot/scripts/control_robot.py \
    --robot.type=so100 \
    --control.type=calibrate \
    --robot.leader_arms.main.port=/dev/ttyUSB0 \
    --robot.follower_arms.main.port=/dev/ttyUSB1

# Teleoperation test
python lerobot/scripts/control_robot.py \
    --robot.type=so100 \
    --control.type=teleoperate

12. Dataset Recording & Management

12.1 Recording Principles

The quality of your dataset directly determines the quality of your learned policy. Garbage in, garbage out , but in robotics, it's more like "inconsistent demonstrations in, confused robot out."

Golden Rules of Data Recording:

1. Consistency is king. Same start position, same object placement, same grasp strategy across episodes.
2. Smooth motions. Jerky teleoperation recordings produce jerky policies. Move deliberately.
3. Sufficient coverage. Record enough variation to handle edge cases (object at slightly different positions).
4. Clean transitions. Start and end each episode cleanly , no accidental motions at boundaries.
5. Quality over quantity. 50 excellent episodes beat 200 sloppy ones.

What gets recorded per timestep:

{
    "observation.state": [7 joint angles],         # Follower arm position
    "observation.images.cam_top": [H, W, 3],       # Top camera frame
    "observation.images.cam_wrist": [H, W, 3],     # Wrist camera frame  
    "action": [7 joint angles],                     # Leader arm position (= target)
    "timestamp": float,                             # Time in seconds
    "episode_index": int,                           # Which episode
    "frame_index": int,                             # Frame within episode
}

View Adjustments & Camera Angles

Camera placement dramatically affects learning performance:

Recommendations:

• Top camera: Mount 40-60cm directly above the workspace, tilted slightly toward the arm. This gives the model a global view of the scene.
• Wrist/side camera: Position to capture the gripper-object interaction zone. This provides fine-grained detail for manipulation.
• Lighting: Consistent, diffused lighting. Avoid shadows that change between episodes.
• Background: Clean, consistent background. A solid-color mat on the workspace helps.

Dataset Recording Tool

# Start recording a dataset
python lerobot/scripts/control_robot.py \
    --robot.type=so100 \
    --control.type=record \
    --fps=30 \
    --repo-id=your-hf-username/so100-pick-place \
    --num-episodes=50 \
    --warmup-time-s=3 \
    --episode-time-s=10 \
    --reset-time-s=5

Parameter Breakdown:

During recording:

• Press Enter to start an episode
• Episode records for episode-time-s seconds
• You get reset-time-s to reset the scene (put object back, etc.)
• Repeat until num-episodes is reached

Rerun Visualization

Rerun is the visualization tool integrated into LeRobot for real-time and post-hoc data inspection.

# Visualize during recording (real-time)
python lerobot/scripts/control_robot.py \
    --robot.type=so100 \
    --control.type=record \
    --enable-rerun=true \
    ...

# Visualize a saved dataset
python lerobot/scripts/visualize_dataset.py \
    --repo-id=your-hf-username/so100-pick-place \
    --episode-index=0

What Rerun shows you:

• Camera feeds (synchronized with joint data)
• Joint position traces over time (each joint as a separate line)
• 3D visualization of the arm pose (if configured)
• Playback controls: scrub through episodes, adjust speed

Why Rerun matters: It's your quality control tool. Before training, scrub through episodes to spot bad data , dropped frames, inconsistent motions, or camera obstructions.

12.2 Data Processing & HuggingFace Upload

After recording, the dataset needs to be processed and can be uploaded to HuggingFace Hub for storage and sharing:

Data Format , Parquet Files:

LeRobot stores datasets in the HuggingFace Datasets format using Parquet files:

dataset/
├── data/
│   ├── train-00000-of-00001.parquet   # Tabular data (states, actions, metadata)
│   └── ...
├── videos/
│   ├── cam_top/
│   │   ├── episode_000000.mp4         # Video for episode 0, top camera
│   │   ├── episode_000001.mp4
│   │   └── ...
│   └── cam_wrist/
│       ├── episode_000000.mp4
│       └── ...
├── meta/
│   ├── info.json                      # Dataset metadata
│   ├── episodes.jsonl                 # Episode-level metadata
│   └── stats.json                     # Statistics (mean, std for normalization)
└── README.md                          # Dataset card

Parquet file contents (each row = one timestep):

# Upload to HuggingFace Hub
huggingface-cli login  # One-time authentication

python lerobot/scripts/push_dataset_to_hub.py \
    --repo-id=your-hf-username/so100-pick-place \
    --local-dir=data/so100-pick-place

12.3 Dataset Visualization & Quality Control

Before training, always inspect your dataset:

# Visualize specific episodes
python lerobot/scripts/visualize_dataset.py \
    --repo-id=your-hf-username/so100-pick-place \
    --episode-index=0

# Check dataset statistics
python -c "
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
ds = LeRobotDataset('your-hf-username/so100-pick-place')
print(f'Total episodes: {ds.num_episodes}')
print(f'Total frames: {ds.num_frames}')
print(f'FPS: {ds.fps}')
print(f'Features: {list(ds.features.keys())}')
"

What to look for:

• Smooth action curves , no sudden jumps or spikes
• Consistent episode length , outliers may indicate problems
• Camera alignment , objects visible in frame throughout the task
• State-action alignment , actions should lead states by 1 timestep

Episode Management: Repeat, Delete, Curate

Not every episode will be good. LeRobot provides tools to manage your dataset:

# During recording:
# - Press 'r' to re-record the last episode (replaces it)
# - Press 'q' to stop recording early

# After recording , delete bad episodes:
# You can manually remove episodes from the dataset
# by editing the Parquet files or using LeRobot utilities

Curation Strategy:

1. Record 10-20% more episodes than you plan to use
2. Review all episodes in Rerun after recording
3. Flag bad episodes , mark for deletion
4. Delete and reindex , remove bad episodes, renumber remaining ones
5. Verify final dataset , check total counts and visualize a sample

Common reasons to delete an episode:

• Object fell or wasn't placed correctly
• Accidental jerky movements
• Camera feed dropped frames
• Task wasn't completed (missed the object, dropped it)
• Recording started before you were ready

13. Imitation Learning with ACT

What is Imitation Learning?

Imitation learning (also called learning from demonstrations or behavioral cloning) is conceptually simple:

Instead of defining rules for what the robot should do, you show it what to do , and it learns to mimic you.

Traditional Programming:        Imitation Learning:
  IF object at (x,y)              SHOW the robot 50 demos
  THEN move arm to (x,y)          TRAIN a neural network
  THEN close gripper              ROBOT learns the mapping:
  THEN lift to (x2,y2)              observation → action
  ... (hundreds of rules)         ... (handles edge cases it's seen)

Key Concepts:

• Behavioral Cloning (BC): The simplest form. Train a supervised learning model where input = observation, output = action. Treat it as a regression problem.
• The distribution shift problem: BC models can drift , small errors compound because the robot encounters states it never saw in training. ACT addresses this.
• Chunk prediction: Instead of predicting one action at a time, predict a sequence (chunk) of future actions. This provides temporal consistency.

13.1 ACT Model Architecture

ACT = Action Chunking with Transformers (Zhao et al., 2023 , from the ALOHA team at Stanford)

ACT addresses the core challenges of behavioral cloning for manipulation:

Key design choices in ACT:

1. Action Chunking: Predicts k future actions at once (typically k=100). This temporal consistency prevents the compounding error problem of step-by-step prediction.
2. CVAE (Conditional Variational Autoencoder): Handles multimodality in demonstrations. When the same observation could lead to multiple valid actions (e.g., grasp from left or right), the style variable z captures which mode to follow.
3. Temporal Ensembling: During inference, overlapping action chunks are averaged with exponential weighting, producing smoother trajectories.
4. CNN + Transformer: ResNet-18 extracts visual features from camera images; Transformer encoder-decoder processes the sequence.

13.2 Training Pipeline

# Training command
python lerobot/scripts/train.py \
    --dataset.repo_id=your-hf-username/so100-pick-place \
    --policy.type=act \
    --output_dir=outputs/act_so100_pick_place \
    --training.num_epochs=2000 \
    --training.batch_size=8 \
    --training.lr=1e-5 \
    --policy.chunk_size=100 \
    --policy.n_obs_steps=1 \
    --wandb.enable=true

Model Saving & Checkpoints

# Checkpoints are saved automatically during training
# outputs/act_so100_pick_place/
# ├── checkpoints/
# │   ├── 000500/  
# │   │   ├── pretrained_model/   # Model weights
# │   │   └── training_state.pth  # Optimizer state
# │   ├── 001000/
# │   ├── 001500/
# │   └── 002000/  (final)
# ├── config.yaml                  # Full training config
# └── wandb/                       # Training logs

# Load a specific checkpoint for inference
from lerobot.common.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("outputs/act_so101_pick_place/checkpoints/002000/pretrained_model")

Tip: Don't just use the final checkpoint. Test intermediate checkpoints , sometimes an earlier checkpoint generalizes better.

13.3 Inference & Results

# Run inference (the robot executes the learned policy)
python lerobot/scripts/control_robot.py \
    --robot.type=so100 \
    --control.type=record \
    --policy.path=outputs/act_so100_pick_place/checkpoints/002000/pretrained_model \
    --fps=30 \
    --num-episodes=10 \
    --repo-id=your-hf-username/so100-pick-place-eval

What happens during inference:

1. Camera captures current frame(s)
2. Joint encoders report current arm position
3. ACT model takes images + state → predicts action chunk (100 future joint angles)
4. Temporal ensembling blends with previous chunk predictions
5. First action in the blended chunk is sent to the follower arm
6. Repeat at 30Hz

14. Vision-Language-Action Models (VLAs) Experimentation

The VLA Formula:

Vision (camera images) + Language (task instruction) → Action (robot joint commands)

SmolVLA

SmolVLA is HuggingFace's compact VLA model, designed to be accessible for research and deployable on consumer hardware.

Architecture Overview:

NVIDIA GR00T N1.5

GR00T N1.5 is NVIDIA's foundation model for humanoid and manipulation robots.

Architecture Overview:

π₀.5 (Pi-Zero.5 π0.5:)

π₀ (Pi-Zero and Zero.5) from Physical Intelligence is a general-purpose robot foundation model.

Architecture Overview:

14.1 Our Implementation: Five Tasks, Four Architectures

Setup

Robot. SO-101 leader–follower pair, 6-DoF + gripper.

Cameras. [fill: e.g. 1× overhead scene camera @ 640×480, 1× wrist camera @ 640×480, both 30 FPS]

Datasets. 60–100 teleoperated episodes per task, recorded in LeRobot format at 30 FPS, curated per the discipline (failed episodes removed, task strings standardised, camera keys normalised). Task labeled with different prompt designs.

Compute and this split is the point of the experiment:

The asynchronous split is not a workaround, it is the deployment pattern these models are designed for. The robot keeps executing the current action chunk while the next chunk is computed on the server, with a new request triggered before the queue drains. Without this, every model in the second row stalls between chunks and the measured success rate drops for reasons that have nothing to do with the policy. One can design their own WebRTC for the inference with high speed internet connectivity.

The five experiments

File stack open the folder and extract the paper

The slight hard fine-tuned task: it is to open the black file and take the white file stack and place it.

White tape pick and place

The control condition. A rigid, high-contrast, Colored Box, separate tape color.

Pencil into basket and the goal-swap test

The task is pick the pencil and place it on black basket and testing after fine-tuning models and also changing the goal and destination places.

Pointing Example Gemini Robotics ER, zero-shot

The only experiment with no training whatsoever. A natural-language prompt, a frontier embodied-reasoning model, and a pen.

GEMINI ROBOTICS ER did not control the robot. It grounded a referent and returned a spatial reference. Turning that into motion for just pointing not pick and place (Calibrated Camera distance to pixel difference through ArucoTag Camera Calibration). The reasoning layer knows where the pen is and what "the pen" means, and knows nothing about our servo limits.

Game Experimentation

The most architecturally interesting experiment, because it is the only one that is a system rather than a policy. We are currently working on experimenting different models on this tasks, sub tasks, zero shot fine tuning, one shot and related experiments on this board game pick and placing in the desired place.

We split it deliberately into three parts that do not share a mechanism:

Model Result Analysis

Core Takeaways for Decision Making

• Choose ACT if your tasks require extreme geometric precision in a heavily controlled, static environment where natural language instruction is unnecessary.
• Choose SmolVLA if you are operating on a strict compute budget and require an on-device edge deployment capable of basic visual-semantic instruction following.
• Choose pi_0 if your use case demands continuous fine-motor dexterity, contact-rich manipulation, and the ability to dynamically recover from physical slipping or errors mid-execution.
• Choose NVIDIA GR00T N1.5 if you need robust zero-shot spatial reasoning and dynamic adaptation to semantic goal changes within known domains, leveraging strong language grounding to cut down on retraining.
• Choose Gemini Robotics ER if you want to completely separate high-level logic from physical execution. As a frontier reasoning VLM, it acts as a semantic brain identifying objects and generating exact spatial coordinates (e.g., pointing) which are then handed off to your existing camera-calibrated low-level controllers for movement.

What we actually learned

Data quality and coverage matter together. More demonstrations help when they cover meaningful variation in object poses, lighting, backgrounds and grasp approaches. Repeating the same successful trajectory is not enough to establish robustness.

Small errors compound during execution. A slight positioning error changes the next observation and can move the robot into unfamiliar states. Recovery demonstrations matter because the robot must learn what to do after an imperfect action.

Lower loss does not always mean a better robot. Intermediate checkpoints need consistent real-world evaluation. Task success, completion time and intervention rate reveal improvements that training curves alone cannot establish.

The robot can learn shortcuts. A policy may associate a task with an object’s usual position or the background instead of the intended instruction. Changing these conditions helps reveal what the model actually learned.

Language conditioning must be tested through behaviour. Changing the requested object, destination or task should produce the corresponding action. Following familiar prompts does not by itself demonstrate generalization to unseen tasks.

The training–deployment gap is often practical. Camera placement, calibration, action normalization and control frequency must remain consistent. Small mismatches can undermine an otherwise capable policy.

Inference is part of the control system. Action chunking and asynchronous prediction affect responsiveness, but long execution horizons and stale observations can delay corrections. Smooth movement alone does not imply reliable control.

Failures should guide the next experiment. Distinguish perception errors, grasp failures, timing issues and missing training coverage before choosing a fix. The useful loop is
train → evaluate → diagnose → collect targeted corrections → retrain.

Simple baselines keep progress honest. ACT and compact VLAs help measure whether added complexity delivers better reliability and generalization. Repeatable success across varied conditions is a stronger result than a single impressive demonstration.

Long tasks need explicit progress and recovery. Break complex goals into verifiable subtasks, check completion at intermediate stages, and define when to retry or replan. A single policy may support multiple skills, while an agentic planner coordinates their execution. Capable language models can help design these workflows, but physical execution still requires grounded feedback and validation.

Robot learning appears to be approaching an inflection point: rapidly evolving VLAs, broader pretrained policies and agentic frameworks are opening a path toward more general-purpose robots. The next challenge is turning that growing capability into reliable execution across unfamiliar tasks and environments.

15. References & Resources

Sources and Further Reading

These readings trace complementary contributions to modern Physical AI: learning from demonstrations, grounding language in perception, generating actions, sharing experience across robots, and coordinating longer tasks. Together, they provide a useful reading path through the field’s development.

Key Contributions to Modern Robot Learning

• ALOHA / ACT Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Combines affordable teleoperation hardware with Action Chunking with Transformers, learning sequences of actions from demonstrations for precise manipulation. Paper
• PaLM-E An Embodied Multimodal Language Model. Integrates visual and continuous sensor representations into a language model’s input sequence, connecting language with physical observations for embodied reasoning and planning. Paper
• RT-2 Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. Represents robot actions as text tokens and jointly fine-tunes on robot trajectories and vision-language tasks. Its central contribution is transferring useful semantic knowledge from web training into robotic behaviour. Paper
• Diffusion Policy Visuomotor Policy Learning via Action Diffusion. Models distributions over action sequences through conditional denoising, allowing policies to represent multiple valid behaviours. Combines visual conditioning with receding-horizon execution; reports a 46.9% average improvement across 12 benchmark tasks in the linked paper. Paper
• Open X-Embodiment Robotic Learning Datasets and RT-X Models. Unifies data from 22 robot embodiments and investigates shared policy training across different bodies. Demonstrates positive transfer in its evaluated settings, helping establish the case for reusable robot datasets and generalist policies. Paper

Language-Guided Planning and Control

• SayCan Do As I Can, Not As I Say. Combines language-model planning with estimates of which robot skills are feasible in the current situation. Project and paper
• Code as Policies Language Model Programs for Embodied Control. Uses language models to generate programs that compose perception and control APIs into robot behaviours. Project and paper

Generalist Policies and Physical World Models

• OpenVLA An Open-Source Vision-Language-Action Model. An open VLA model and adaptation framework for studying generalist manipulation and fine-tuning on new robot setups. Project and paper
• π₀ / π₀.₅ Physical Intelligence. Explore flow-matching action generation, heterogeneous training data and generalization to unfamiliar environments. π₀ · π₀.₅
• SmolVLA A Vision-Language-Action Model for Affordable and Efficient Robotics. A compact VLA emphasizing accessible training, community robot datasets and asynchronous execution. Technical overview
• NVIDIA Isaac GR00T. Foundation models and supporting tools for generalist robot learning. Official repository
• Gemini Robotics / Robotics ER. Google DeepMind’s work on robotic action generation and embodied reasoning. Official overview
• Cosmos 3 — Omnimodal World Models for Physical AI. A complementary direction focused on modelling the physical world across modalities. Technical report

Hands-On Learning and Implementation

• Hugging Face Robot Learning: A Tutorial. A companion resource for exploring robot-learning concepts and practical workflows. Interactive tutorial
• LeRobot. Open-source tools for teleoperation, demonstration recording, dataset management, policy training and evaluation. Documentation · GitHub

Agentic Orchestration

General-purpose reasoning models are relevant to experiments in task decomposition, tool use and progress monitoring. Their use as robot planners requires integration with perception, executable skills and physical feedback.

• GPT-6 Astra. Official release
• Claude Fable. Official documentation
  
  

Mahtab Newaz
Machine Learning Engineer (Robotics | Computer Vision)
Under Supervision of Dr. Faizul Bari (Faiz) PhD (Director & CTO)
Spectrum Software & Consulting (Pvt.) Ltd. (AI/ML Team)
Website: https://sscl.tech/