In-Context Robot Learningwith VLM Agents

Dongzhou Cheng*, Taoran Yi*, Ye Fang*, Xingwu Zhang*, Fan Feng*, Yixuan Li*, Gengxiong Zhuang*, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu†

* Co-first authors. † Corresponding author.

Can a fixed general-purpose VLM learn from demonstrations, examples, and interaction feedback and turn it into executable, verifiable behavior from a new initial state?

Paper Code X
In-context robot control with GPT-Policy: five context families
Figure 1. A general-purpose VLM combines the task instruction, initial state, and contextual information to guide robot actions. Context includes human videos, robot videos with recorded actions, goal images, human–robot interaction, and self-interaction history.

Overview

Robotic in-context learning, without parameter updates

Abstract

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations and interaction feedback, then translate that information into executable robot behavior without gradient updates or persistent changes to task-specific parameters?

We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy equips a VLM agent with a context compiler that organizes task-relevant information and an execution harness that supports geometric grounding, action chunking, and closed-loop decision-making through execution feedback. We evaluate GPT-Policy on ten real-world tasks spanning five context types and ten RoboDojo simulation tasks. Across four real-world demonstration tasks, mean success rate increases from 10% without demonstrations to 80% with human videos or robot demonstrations that include action references. Videos provide procedural guidance, while action references can clarify motion details. Targeted ablations suggest that calibrated localization and action chunking improve success and execution efficiency. Across ten RoboDojo tasks, one-shot GPT-Policy achieves 72% mean success. These results suggest that general-purpose VLMs can use deployment-time context to adapt robot behavior, while reliable physical execution remains a challenge.

Central question

How much robotic in-context learning is already accessible through general VLMs that were not trained as dedicated robot policies?

Context

Five sources of task information

Robotic ICL adapts behavior using demonstrations, examples, or interaction experience at test time, without gradient updates or persistent task-specific parameter changes. Each context family supplies information that the current instruction and observation may omit.

Procedure

01

Human video

Ordered visual keyframes specify how a person carries out a task, without expert robot actions.

  • Procedural order
  • No robot actions

Fine contact

02

Robot demonstration

Aligned visual evidence and optional action references support contact-rich manipulation.

  • Embodied motion
  • Optional actions

Goal state

03

Target image

A reference image gives the final arrangement and spatial relations without prescribing a sequence.

  • Desired end state
  • Sequence-free

Memory

04

Self-interaction history

Earlier observations, actions, failures, and outcomes remain available for exploration and recovery.

  • Closed-loop memory
  • Failure recovery

Coordination

05

Human interaction

Pointing, turn history, and live feedback can change target selection and action timing online.

  • Online cues
  • Turn-aware

Method

From context to constrained action

GPT-Policy connects a general-purpose VLM to robot tools through a shared context-to-action interface. Embodiment-specific adapters translate tool requests into executable commands. Model parameters remain fixed; updated observations and execution or rejection feedback inform each subsequent decision.

GPT 6 Astra (direct)

Images · Proprioception · Instruction ot · pt · ℓ
Execution history ht Tools · Persistent notes
GPT 6 Astra (ot, pt, ℓ, ht) → AtAstra
Bimanual EEF targets (x, R, g) Position · Orientation · Gripper; 1–5 steps
Inverse kinematics → Execute New images + state ↺

GPT-Policy (Ours)

Demonstrations · Goal image Video · Optional actions · Target state
Context compiler → R Keyframes · Subtasks · Aligned action records
GPT 6 Astra (R, ot, ℓ, Ht) → at
Execution harness Grounding · EEF Chunks · Validation
Execute tool call New observations + execution feedback ↺
Direct and GPT-Policy. Both use a frozen model in an observation–action loop. Direct generates end-effector targets from images, robot state, instructions, and execution history. GPT-Policy compiles task references into R and combines them with live observations (images and state), the instruction, and interaction history. Its execution harness supports geometric grounding, action chunking, and trajectory validation.
Overview of the GPT-Policy context-to-action framework
Figure 2. A context compiler builds structured references from the task instruction and demonstrations. A frozen VLM uses these references, live observations, and history to choose one tool call per turn. Geometric grounding triangulates stationary features from calibrated wrist-camera views; action chunking plans multiple end-effector waypoints. Execution adapters validate and retime trajectories, then return feedback for the next decision.

From demonstrations to context

GPT-Policy context construction: demonstrations, keyframes, and aligned action records
Figure 3. A VLM selects keyframes, annotates subtasks and arm roles, and reviews the demonstration. Context assembly aligns measured states with keyframes, attaches historical actions between them, and combines the records with camera images. The plug-reinsertion example shows the resulting structured context. Adapted from paper Figure 6.

Experiments

Real-world experiments

We evaluate a frozen GPT-6 Astra on ten real-robot tasks spanning five forms of context. The π₀.₅ baseline is fine-tuned on 30 demonstrations per task.

Success is reported as successful/total trials. Decisions and time are mean ± sample SD, including failures. GPT-Policy reports action-target/chunk decisions and wall-clock time; π₀.₅ uses logged counts and top-view recording time. Human demonstrations play in real time; robot playback speeds are labeled.

Human video demonstrations

A human demonstration raises success from 20% to 90% on towel pickup and from 20% to 80% on notebook pickup, without robot action labels.

TaskMethodContext conditionSuccessDecisionsTime (min)
Pick Red Towelπ₀.₅None7 / 1019.3 ± 12.80.94 ± 0.53
Pick Red TowelGPT-Policy (Astra)None2 / 1082.3 ± 26.721.1 ± 7.2
Pick Red TowelGPT-Policy (Astra)Human video9 / 1044.9 ± 27.711.7 ± 6.7
Pick Up Notebookπ₀.₅None2 / 1031.4 ± 16.81.35 ± 0.63
Pick Up NotebookGPT-Policy (Astra)None2 / 1076.5 ± 15.420.8 ± 5.0
Pick Up NotebookGPT-Policy (Astra)Human video8 / 1053.5 ± 24.014.6 ± 6.6

Robot video and action references

Robot video alone achieves 70% success on bottle opening but 0% on plug reinsertion. Adding aligned action references brings success to 80% and 70%, respectively.

TaskMethodContext conditionSuccessDecisionsTime (min)
Unscrew Bottle Capπ₀.₅None2 / 1087.4 ± 22.23.04 ± 0.72
Unscrew Bottle CapGPT-Policy (Astra)None0 / 1071.0 ± 50.216.1 ± 9.9
Unscrew Bottle CapGPT-Policy (Astra)Robot video7 / 1053.4 ± 21.010.6 ± 4.3
Unscrew Bottle CapGPT-Policy (Astra)Robot video + action8 / 1049.1 ± 16.613.0 ± 4.9
Remove and Reinsert Plugπ₀.₅None0 / 1065.4 ± 38.62.35 ± 1.28
Remove and Reinsert PlugGPT-Policy (Astra)None0 / 1043.0 ± 38.59.1 ± 7.8
Remove and Reinsert PlugGPT-Policy (Astra)Robot video0 / 1032.0 ± 9.77.5 ± 2.4
Remove and Reinsert PlugGPT-Policy (Astra)Robot video + action7 / 1043.5 ± 7.29.4 ± 2.1

These comparisons span prompt and demonstration revisions.

Goal-image following

Goal images specify the desired arrangement without a demonstrated procedure. Both block and fruit arrangement succeed in all reported trials.

TaskMethodContext conditionSuccessDecisionsTime (min)
Arrange T ShapeGPT-Policy (Astra)Target image3 / 366.7 ± 4.015.8 ± 1.8
Arrange FruitGPT-Policy (Astra)Target image3 / 349.0 ± 14.212.4 ± 2.9

Self-interaction history

Earlier observations and execution feedback support search and recovery. Both tasks succeed, although mobile exploration takes 25.5 minutes on average.

TaskMethodContext conditionSuccessDecisionsTime (min)
Lemon To Pink PlateGPT-Policy (Astra)Self history3 / 335.3 ± 4.08.1 ± 1.0
Movable ExplorationGPT-Policy (Astra)Self history3 / 340.3 ± 1.525.5 ± 4.0

Online human–robot interaction

Live human moves and pointing gestures guide action selection. Wins and draws both count as success in tic-tac-toe.

TaskMethodContext conditionSuccessDecisionsTime (min)
Tic-Tac-Toeπ₀.₅None0 / 318.7 ± 1.50.94 ± 0.03
Tic-Tac-ToeGPT-Policy (Astra)Human-robot interaction3 / 369.7 ± 13.313.6 ± 2.6
Pointed Fruit Pickupπ₀.₅None1 / 320.0 ± 6.61.07 ± 0.34
Pointed Fruit PickupGPT-Policy (Astra)Human-robot interaction3 / 367.3 ± 17.515.0 ± 3.4

Simulation experiments

On RoboDojo, GPT-Policy uses one fixed demonstration per task across ten tasks and five scenes per task. The released results achieve a mean score of 80.20/100 and a success rate of 72%.

Results combine recorded versions and explicitly selected replacement attempts. These task/layout families informed development; the panel is not a held-out evaluation. Baselines retain their published protocols.

Execution and token usage

MetricGPT 6 Astra
(direct)		π₀.₅ + GPT 6 Astra
(hybrid)		GPT-Policy
(Ours)
Success rate26%48%72%
Control steps38,22142,75039,692
Executed action segments7,7293,7765,340
Mean physical duration (s)30.5834.2031.75
Total tokens (including cached input)1,132,343,772624,762,828855,443,119
Cached input tokens1,107,349,760607,555,840815,661,184
Uncached input tokens22,907,44815,901,96336,906,672
Output tokens2,086,5641,305,0252,875,263

Physical duration excludes model waits. Reference results use different settings and were not rerun. GPT-as-Policy ↗

Ablation studies

Real-robot ablations examine context composition, calibrated localization, and EEF chunking.

Context composition

Remove and Reinsert Plug

SettingSuccessDecisionsTime (min)
Subtask + action7 / 1043.5 ± 7.29.40 ± 2.12
Action only4 / 1084.3 ± 14.115.44 ± 3.15
Subtask only0 / 1032.0 ± 9.77.49 ± 2.43
Neither1 / 1091.0 ± 12.018.32 ± 2.24

Calibrated localization

Pick Up Notebook

SettingSuccessDecisionsTime (min)
Calibrated localization8 / 1053.5 ± 24.014.61 ± 6.62
Without calibration / localization3 / 1078.5 ± 23.922.78 ± 6.88

EEF chunking

Remove and Reinsert Plug

SettingSuccessDecisionsTime (min)
With chunking7 / 1043.5 ± 7.29.40 ± 2.12
Without chunking3 / 1081.4 ± 24.216.69 ± 4.84

Mean ± sample SD over all included trials. Conditions span different batches; removing calibration also disables localization. One multi-waypoint chunk counts as one decision.

What the current evidence shows

Takeaways

Finding 01

A Little Context Goes a Long Way.

VLM Agents are surprisingly good at learning on the fly. Human videos reveal how a task unfolds, while action references provide more precise motion cues. In these early trials, such context helps Agents step outside their usual comfort zone and tackle fine manipulation and deformable-object handling.

Finding 02

Good Plans, Tricky Execution.

Context can sharpen task understanding, planning, and adaptation. Still, knowing what to do is not the same as doing it reliably. Precise pose generation, contact execution, outcome verification, and physical safety remain challenges of their own.

Finding 03

Moves Take Time.

VLM Agents can already generate useful actions, but each decision can still be slow and expensive. Specialized VLA/WAM models may therefore have an edge in fast, low-level control, while VLM Agents focus on reasoning, adaptation, and replanning.

Discussion

A promising interface, with a clear boundary

In selected red-towel runs, human video raises task progress from 55% to 100%, with 35.6% shorter run time and 58.9% lower estimated token usage. These are individual-run comparisons, not a model ranking. Across the updated evaluation, GPT-Policy with human video succeeds in 9/10 towel trials and 8/10 notebook trials.

The ablations test subtask/action references, calibrated localization, and action chunking. Context quality, observation–action alignment, and inference latency still constrain performance. Reactive recovery does not ensure collision avoidance; joint action validation, collision checks before execution, and independent stopping mechanisms remain necessary.

Future directions

  • Physical safetyCheck both arms together, monitor separation and contact, and interrupt unsafe commands independently of the VLM.
  • Contact-aware executionExpose grasp stability, slip, and object motion; add force-aware limits and local recovery.
  • Fast control and deliberate reasoningPair a deliberative agent with a fast controller for pose refinement and bimanual coordination.
  • Mobile manipulationCombine active perception, persistent spatial memory, and coordinated base–arm control.
  • Compositional contextCompose subskills from multiple demonstrations while retaining completed subgoals and discarding obsolete context.
  • Adaptation to physical dynamicsUse recent interactions to update predictions of friction, compliance, and object response before failures accumulate.

Citation

@article{cheng2026context,
  title  = {In-Context Robot Learning with VLM Agents},
  author = {Cheng, Dongzhou and Yi, Taoran and Fang, Ye and
            Zhang, Xingwu and Feng, Fan and Li, Yixuan and
            Zhuang, Gengxiong and Wang, Rongze and Yang, Shuai and
            Song, Wei and others},
  journal = {arXiv preprint arXiv:2609.19138},
  year    = {2026}
}