In-Context Robot Learningwith VLM Agents

Dongzhou Cheng*, Taoran Yi*, Ye Fang*, Xingwu Zhang*, Fan Feng*, Yixuan Li*, Gengxiong Zhuang*, Rongze Wang, Shuai Yang, Wei Song, Weizhi Xue, Minyan Wu, Jie Gui, Jiaqi Wang, Tong Wu†

* Co-first authors. † Corresponding author.

Can a fixed general-purpose VLM learn from demonstrations, examples, and interaction feedback and turn it into executable, verifiable behavior from a new initial state?

Paper Code
In-context robot control with GPT-Policy: five context families
Figure 1. A general-purpose VLM combines the task instruction, initial state, and contextual information to guide robot actions. Context includes human videos, robot videos with recorded actions, goal images, human–robot interaction, and self-interaction history.

Overview

Robotic in-context learning, without parameter updates

Abstract

Enabling robots to adapt to unfamiliar environments as readily as humans remains a moonshot goal of embodied AI. No finite collection of demonstrations can cover every task and situation a robot will encounter, making the ability to learn from context at deployment essential for generalization. Such in-context learning (ICL), however, remains largely beyond the reach of existing robotic policies. The broad agentic capabilities of commercial vision-language models (VLMs), such as GPT-6 Astra, raise a compelling question: can these models learn from demonstrations, examples, and interaction feedback, then translate that information into executable and verifiable robot behavior from a new initial state without gradient updates or persistent changes to task-specific parameters?

We introduce GPT-Policy, a general-agent framework for in-context robot learning. GPT-Policy integrates a context compiler that preserves task-relevant visual transitions, a VLM that proposes robot-tool actions, and a constrained controller that verifies and executes each action and reports its outcome. We evaluate its reliability and limitations through task success and efficiency metrics, matched comparisons across models, and controlled context ablations. In real-robot trials, human video demonstrations improve task completion even without robot action labels, while aligned action references yield further gains on contact-sensitive tasks. These findings position GPT-Policy as a step toward robot adaptation through in-context learning, providing an empirical foundation for translating the general-purpose capabilities of VLMs into physical behavior and clarifying the challenges that must be overcome for reliable deployment.

Central question

How much robotic in-context learning is already accessible through general VLMs that were not trained as dedicated robot policies?

Context

Five sources of task information

Robotic ICL adapts behavior using demonstrations, examples, or interaction experience at test time, without gradient updates or persistent task-specific parameter changes. Each context family supplies information that the current instruction and observation may omit.

Procedure

01

Human video

Ordered visual keyframes specify how a person carries out a task, without expert robot actions.

  • Procedural order
  • No robot actions

Fine contact

02

Robot demonstration

Aligned visual evidence and optional action references support contact-rich manipulation.

  • Embodied motion
  • Optional actions

Goal state

03

Target image

A reference image gives the final arrangement and spatial relations without prescribing a sequence.

  • Desired end state
  • Sequence-free

Memory

04

Self-interaction history

Earlier observations, actions, failures, and outcomes remain available for exploration and recovery.

  • Closed-loop memory
  • Failure recovery

Coordination

05

Human interaction

Pointing, turn history, and live feedback can change target selection and action timing online.

  • Online cues
  • Turn-aware

Method

From context to constrained action

GPT-Policy connects a general-purpose VLM to robot tools through a shared context-to-action interface. Embodiment-specific adapters translate tool requests into executable commands. Model parameters remain fixed; updated observations and execution or rejection feedback inform each subsequent decision.

Overview of the GPT-Policy context-to-action framework
Figure 2. The task instruction, current state, task references, and interaction history are combined with shared instructions and tool schemas to form the VLM input. A fixed VLM selects tool requests, which constrained robot tools execute. Returned observations and execution feedback update the context for the next decision.
1

Construct context

Combine task references, current views, state, and retained interaction history.

2

Assemble input

Interleave text and images with stable instructions, constraints, and tool schemas.

3

Select action

A fixed general VLM chooses one structured robot-tool request.

4

Validate and execute

The Cartesian adapter samples poses, checks inverse-kinematics residuals, times joint references, and returns execution or rejection feedback.

GPT-Policy execution harness
Figure 3. The VLM generates tool requests from interleaved image and text inputs. The Cartesian adapter resolves targets, samples the pose path, checks inverse-kinematics residuals, times joint references, and executes the motion. Execution or rejection feedback returns to the next decision.

Experiments

Five experiment families

Human-video tasks compare runs with and without the human demonstration. Robot-demonstration tasks compare no video, video, and video + action. Other tasks show one representative run. Human demonstrations play at real time; robot runs are accelerated as labeled. Success rates and mean costs summarize all trials.

5experiment families
10robot tasks
3trials per condition

GPT-6 Astra on real robots; three trials per condition. Success requires the final scene to satisfy the task-specific geometric and semantic criteria. Decisions and time are averaged over all trials, including failures. Tic-Tac-Toe wins and draws both count as success.

01

Human video demonstrations

Towel and notebook pickup test whether human demonstrations improve execution without robot action labels.

TaskContext conditionSuccessDecisionsTime (min)
Pick Red TowelNone0 / 396.324.6
Pick Red TowelHuman video2 / 376.718.9
Pick Up NotebookNone0 / 394.024.6
Pick Up NotebookHuman video2 / 366.716.1
02

Robot demonstrations and actions

Bottle opening and plug reinsertion compare no demonstration, robot video, and robot video with aligned actions.

TaskContext conditionSuccessDecisionsTime (min)
Unscrew Bottle CapNone0 / 371.016.1
Unscrew Bottle CapRobot video2 / 374.315.2
Unscrew Bottle CapRobot video + action3 / 354.717.9
Remove and Reinsert PlugNone0 / 324.05.3
Remove and Reinsert PlugRobot video0 / 333.77.9
Remove and Reinsert PlugRobot video + action2 / 348.310.8
03

Goal-image following

Block and fruit arrangement test whether the robot can reproduce a desired spatial configuration.

TaskContext conditionSuccessDecisionsTime (min)
Arrange T ShapeTarget image3 / 366.715.8
Arrange FruitTarget image3 / 349.012.4
04

Self-interaction history

Lemon placement and mobile exploration test how earlier observations and failures support search and recovery.

TaskContext conditionSuccessDecisionsTime (min)
Lemon To Pink PlateSelf history3 / 335.38.1
Movable ExplorationSelf history3 / 340.3325.53
05

Human–robot interaction

Tic-tac-toe and pointed-fruit pickup test adaptation to human turns and gestures.

TaskContext conditionSuccessDecisionsTime (min)
Tic-Tac-ToeHuman-robot interaction3 / 369.713.6
Pointed Fruit PickupHuman-robot interaction3 / 367.315.0

What the current evidence shows

Takeaways

Finding 01

A Little Context Goes a Long Way.

VLM Agents are surprisingly good at learning on the fly. Human videos reveal how a task unfolds, while action references provide more precise motion cues. In these early trials, such context helps Agents step outside their usual comfort zone and tackle fine manipulation and deformable-object handling.

Finding 02

Good Plans, Tricky Execution.

Context can sharpen task understanding, planning, and adaptation. Still, knowing what to do is not the same as doing it reliably. Precise pose generation, contact execution, outcome verification, and physical safety remain challenges of their own.

Finding 03

Moves Take Time.

VLM Agents can already generate useful actions, but each decision can still be slow and expensive. Specialized VLA/WAM models may therefore have an edge in fast, low-level control, while VLM Agents focus on reasoning, adaptation, and replanning.

Discussion

A promising interface, with a clear boundary

In individual red-towel runs, human video raises GPT-6 Astra task progress from 55% to 100%, with approximately 35.6% shorter run time and 58.9% lower estimated token usage. Fable 5.1 and Kimi K3 reach 30% and 20% progress. These examples do not establish a reliable model ranking: task progress is distinct from success rate, which is 2/3 for GPT-6 Astra with human video.

The evidence covers small task series under selected platforms, models, and context conditions. Incomplete ablations and unobserved pretraining limit causal and novel-skill claims. Context quality, observation–action alignment, execution latency, and human intervention remain constraints. Observed inter-arm collisions show that existing safeguards alone are insufficient for safe autonomous deployment.

Future directions

  • Physical safetyCheck both arms together, monitor separation and contact, and interrupt unsafe commands independently of the VLM.
  • Contact-aware executionExpose grasp stability, slip, and object motion; add force-aware limits and local recovery.
  • Fast control and deliberate reasoningPair a deliberative agent with a fast controller for pose refinement and bimanual coordination.
  • Mobile manipulationCombine active perception, persistent spatial memory, and coordinated base–arm control.
  • Compositional contextCompose subskills from multiple demonstrations while retaining completed subgoals and discarding obsolete context.
  • Adaptation to physical dynamicsUse recent interactions to update predictions of friction, compliance, and object response before failures accumulate.

Citation

@misc{cheng_incontext_robot_learning,
  title  = {In-Context Robot Learning with VLM Agents},
  author = {Cheng, Dongzhou and Yi, Taoran and Fang, Ye and
             Zhang, Xingwu and Feng, Fan and Li, Yixuan and
             Zhuang, Gengxiong and Wang, Rongze and Yang, Shuai and
             Song, Wei and Xue, Weizhi and Wu, Minyan and
             Gui, Jie and Wang, Jiaqi and Wu, Tong},
  url    = {https://cheng-haha.github.io/GPT-Policy/}
}