中文版

I am excited to share our new work, VisualPatchWorld (VPW), with Jiaxuan Xiong.

VisualPatchWorld asks a simple question: can we learn a world model that is not only useful for planning, but also readable and editable as code?

• Paper: arXiv:2607.25236
• Code: HKBU-KnowComp/VisualPatchWorld

VisualPatchWorld represents dynamics as code: data-driven like a learned model, but explicit, inspectable, and editable like a simulator. The render–simulate–plan framing follows Fei-Fei Li’s blog, A Functional Taxonomy of World Models.

From Latent Dynamics to Executable Programs

World models are usually realized in one of two ways. Neural world models learn powerful dynamics in continuous latent spaces, but their internal transition rules remain implicit. Physics engines expose explicit states and laws, but building a simulator for every new environment requires substantial manual effort.

VisualPatchWorld explores a third possibility: representing world dynamics as executable Python code. The learned program accepts a structured scene state and an action, predicts the next state, and can be rolled forward inside a planner like a small simulator. Because the transition law is explicit, we can inspect it, edit it, and diagnose why a rollout failed.

This work extends the idea behind PatchWorld. PatchWorld induces belief-state programs for partially observable text environments through counterexample-guided repair. VisualPatchWorld moves this research direction into visual control, where the model must recover continuous dynamics and rank action sequences for closed-loop planning.

How VisualPatchWorld Works

The complete VisualPatchWorld pipeline. Image-derived scene graphs provide the live state, while the induced program predicts candidate futures for planning.

The system contains four stages:

1. Visual abstraction. An observation is converted into an object-centric scene graph containing quantities such as object positions, orientations, distances, and contact relations. We study oracle state, computer-vision tools, and vision-language models as alternative interfaces.
2. Transition export. Interaction trajectories are represented as structured triples of the form \((g_t, a_t, g_{t+1})\).
3. Two-level program induction. The system first identifies the qualitative form of the dynamics, then fits its numerical parameters from multi-step rollouts.
4. Model-predictive planning. The induced program scores candidate action sequences inside a cross-entropy method model-predictive controller (CEM-MPC).

The central idea is the two-level induction process.

Level 1 uses active probes to select a qualitative dynamical sketch. Level 2 fits the sketch’s numerical parameters from rollout data and validates the resulting program.

Level 1: Discover the Right Structure

Fitting parameters is not enough when the assumed dynamics are wrong. A moving object might follow first-order motion or second-order dynamics; manipulation might be action-driven or contact-driven; a grasped object might move only when a grip condition is satisfied.

VisualPatchWorld uses short active probes to distinguish among a small family of qualitative hypotheses. These probes test behaviors such as step response, momentum, and contact. The result is a dynamical sketch appropriate for the environment.

Level 2: Fit the Parameters

After selecting the structure, VisualPatchWorld estimates its free parameters by minimizing multi-step rollout error. Multi-step fitting matters because a model that looks accurate for one transition can still drift badly or rank action sequences incorrectly over a planning horizon.

The final output is an executable transition function:

1
2
3
4

def step(state, action):
    """Roll the structured world state forward by one step."""
    ...
    return next_state

This program becomes the predictive model used by the planner.

Building on Existing Code World Models

The current world-model landscape contains two broad families.

Neural world models, including LeWM, DINO-WM, and PLDM, learn dynamics in continuous representations from visual data. They are flexible and achieve strong results, but their transition laws remain implicit in network parameters. Executable code world models, including PatchWorld, WorldCoder, POMDP-Coder, PoE-World, GIF-MCTS, and CWM-Game, instead produce programs that can be executed and inspected.

VisualPatchWorld builds on several important ideas. WorldCoder learns executable models through interaction. PatchWorld repairs Python dynamics with counterexamples. POMDP-Coder induces compact probabilistic programs. PoE-World composes programmatic experts. GIF-MCTS and CWM-Game improve code models through search and test feedback.

We are grateful to these authors for creating the foundations, algorithms, and evaluation questions on which our study builds.

Our goal is not to declare one formulation universally better. We ask a complementary question: when a code world model must support continuous model-predictive control, can ideas from system identification and symbolic regression help recover a planner-usable dynamics program?

This question continues a research direction we have pursued for several years: discovering a compact, executable hypothesis from observations. In our ACL 2024 work on abductive reasoning in knowledge graphs (paper), the input is a set of observed entities and the output is a logical hypothesis that explains why they occur together. Because the hypothesis is a formal knowledge-graph query, it can be executed to test whether the explanation is consistent with structured knowledge.

We later studied the same inverse problem in scientific discovery through NewtonBench (paper). Instead of recovering a logical explanation from a knowledge graph, an agent performs experiments in an interactive physics environment and proposes a scientific law that explains its observations. Counterfactual changes to familiar laws test whether the agent has genuinely inferred the governing mechanism rather than recalled a textbook formula.

VisualPatchWorld extends this progression from logical hypotheses, to scientific laws, and now to executable dynamics programs. All three settings share the same basic loop: observe evidence, propose a compact structure, execute or test it, and revise it when its predictions fail. The program-regression perspective is our attempt to turn this common pattern into a more general methodology for learning world models.

Many existing code methods use flexible program search: an LLM generates, composes, or edits programs, and candidates are evaluated through replay accuracy, unit tests, interaction, or one-step prediction. VPW studies a more constrained alternative:

1. Which qualitative dynamics are present? Active probes select a compact dynamical sketch.
2. What are the numerical parameters of those dynamics? Multi-step optimization fits the selected sketch over the planning horizon.

This separation tests whether explicitly identifying structure can complement open-ended code generation. Local correctness and planner utility are different objectives: a program may pass local tests yet still need a better kinematic or contact model for long-horizon action ranking.

Contributions We Build On, Chronologically

The methods are complementary rather than mutually exclusive. Open-ended synthesis offers flexibility; probabilistic programs represent uncertainty; expert composition supports modularity; tree search provides broad exploration; and VPW tests whether a symbolic-regression-style inductive bias can make dynamics fitting more reliable for planning.

Program Regression: Extending the Idea of Symbolic Regression

Symbolic regression searches for a mathematical expression that explains observed data—for example, selecting an equation structure and estimating its constants. We use program regression as a guiding perspective that extends this idea from a single expression to an executable transition program.

A world model often needs more than one equation. It may contain typed object state, conditional contact, kinematic transforms, grip gates, and different update laws for different entities. Under the program-regression view, learning a world model means:

1. discovering the reusable program structure that explains how actions change state;
2. estimating its continuous parameters from trajectories;
3. validating the program over multi-step rollouts and downstream planning; and
4. retaining executable code that can later be inspected, composed, or repaired.

VPW is an initial, deliberately restricted test of this idea. Its current hypothesis families are compact and partly environment-specific, so the paper does not yet demonstrate a universal program-regression system or out-of-domain generalization. Our longer-term goal is to expand the sketch library, learn reusable program components, and combine structural search with code generation so that the same induction process can transfer to a much broader range of environments.

Two Complementary Examples

• WorldCoder lets an agent gather experience, synthesize a Python model, and improve it with counterexamples. Its original method benefits from online exploration. For our shared offline comparison, every method receives the same traces. WorldCoder reaches a strong 74% on Cube.

• PoE-World takes a modular approach. It synthesizes several experts, learns their weights, and prunes the ensemble. It reaches 84% on Cube, close to VPW’s 86%, and obtains the lowest Reacher open-loop error among the adapted baselines.

These strengths suggest promising combinations: WorldCoder’s interaction, PoE-World’s modularity, and VPW’s rollout-aware structure fitting could complement one another.

What We Learn from the Comparison

The detailed rollouts reveal two recurring challenges:

1. Local prediction is not the same as planning. A model can predict the next state well but rank longer action sequences poorly.
2. Valid code may still need the right dynamics. Robot arms require the right kinematics, while pushing requires a useful contact model.

These observations motivate VPW’s structure-then-fit design. They do not diminish the contributions of the baselines; instead, they identify opportunities to combine flexible code generation with structured dynamics fitting.

Evaluation

We evaluate VisualPatchWorld on four environments from the LeWM benchmark suite:

Under a shared induced-model planner, VisualPatchWorld reaches 69.0% mean success, outperforming the strongest evaluated code-world-model baseline by 23.5 percentage points. The largest gains appear when selecting the correct qualitative dynamics is important: VPW reaches 72% success on Reacher, 22% on PushT, and 86% on Cube.

Comparison with Executable Code Baselines

For a controlled comparison, all methods receive the same simulator-state traces, use oracle state during replanning, and share the same CEM planner. We evaluate them on the same 50 held-out starts.

These are adaptations to a common LeWM interface, not full reproductions of the original systems. Features such as online exploration, specialized game planners, and hidden-state machinery are not included.

We also study where learned code still falls short of a ground-truth engine. The induced model approaches the engine ceiling on navigation and grasp-rich control, while contact-rich pushing remains harder. A hybrid strategy—using the induced model to search broadly and the physics engine to verify a short list of promising plans—raises mean success to 95.0%, compared with 97.5% for full engine scoring.

Induced, Hybrid, and Simulator Scoring

Hybrid scoring evaluates every candidate with the induced program, then re-scores only the top 30% with MuJoCo. This preserves broad, inexpensive search while using the engine where precise contact evaluation matters most.

What the Results Mean

The main comparison is with programmatic code-world-model baselines under matched state-trace supervision and a shared planner. Neural visual world models remain strong references and achieve higher overall results in this benchmark. Our goal is therefore not to claim that code replaces neural latent models.

Instead, the results show that automatically induced programs can become useful planning models when they capture the right qualitative structure. They also expose the remaining bottlenecks:

• Dynamics are currently induced from structured simulator-state traces rather than directly from pixels.
• Image-derived scene graphs are used for live state estimation at replanning time.
• Active structure discovery is most fully demonstrated in the contact-rich PushT setting; other environments use restricted hypothesis families.
• Precise contact dynamics remain difficult for a compact induced program.

These limitations point toward the next step: learning both visual abstractions and executable dynamics from interaction while retaining an interface that humans and agents can inspect and repair.

Why Code as a Structured Latent?

A neural latent state compresses experience into vectors. VisualPatchWorld instead treats code as a structured latent representation: a compact, operational description of objects, interactions, and state transitions.

This perspective connects world modeling to a broader question that also motivates my work on knowledge graphs and structured knowledge: how can we turn raw observations into abstractions that are not only predictive, but also executable, verifiable, and useful for reasoning?

VisualPatchWorld is one step toward world models that can be read like programs, run like simulators, and used directly for planning.

Citation

1
2
3
4
5
6
7
8
9

@misc{bai2026visualpatchworld,
  title={VisualPatchWorld: Code World Models as Latent Structured Representations for Planning},
  author={Jiaxin Bai and Jiaxuan Xiong},
  year={2026},
  eprint={2607.25236},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.25236}
}

中文版

English Version

很高兴和大家分享我与 Jiaxuan Xiong 合作的最新工作 VisualPatchWorld（VPW）。

VisualPatchWorld 试图回答一个简单的问题：我们能否学习出一种不仅可用于规划，而且能够像代码一样被阅读和修改的世界模型？

• 论文： arXiv:2607.25236
• 代码： HKBU-KnowComp/VisualPatchWorld

VisualPatchWorld 用代码表示动力学：它像学习模型一样由数据驱动，同时又像模拟器一样显式、可检查、可修改。图中“渲染—模拟—规划”的划分来自李飞飞教授的博客 A Functional Taxonomy of World Models。

从隐空间动力学到可执行程序

世界模型通常有两种实现方式。神经世界模型在连续隐空间中学习强大的动力学，但其内部状态转移规则隐含在网络参数中。物理引擎则提供显式的状态和物理规律，但为每个新环境构建模拟器需要大量人工工作。

VisualPatchWorld 探索了第三种可能：将世界动力学表示为可执行的 Python 代码。学习得到的程序接收结构化场景状态和动作，预测下一个状态，并像一个小型模拟器一样在规划器中向前运行。由于状态转移规律是显式的，我们可以检查和修改它，也可以诊断 rollout 失败的原因。

这项工作延续了 PatchWorld 的思路。PatchWorld 通过反例引导的修复，为部分可观测文本环境归纳 belief-state 程序。VisualPatchWorld 将这一研究方向推进到视觉控制场景，在这里，模型需要恢复连续动力学，并为闭环规划比较不同动作序列。

VisualPatchWorld 如何工作

VisualPatchWorld 的完整流程。由图像提取的场景图提供实时状态，归纳出的程序则预测候选未来，用于规划。

系统包含四个阶段：

1. 视觉抽象。 将观测转换为以物体为中心的场景图，其中包含物体位置、朝向、距离和接触关系等信息。我们分别研究了模拟器真实状态、计算机视觉工具和视觉语言模型三种接口。
2. 状态转移导出。 将交互轨迹表示为 \((g_t, a_t, g_{t+1})\) 形式的结构化三元组。
3. 两层程序归纳。 系统先识别动力学的定性形式，再利用多步 rollout 拟合数值参数。
4. 模型预测控制。 归纳出的程序在交叉熵方法模型预测控制器（CEM-MPC）中为候选动作序列打分。

其中最核心的是两层程序归纳过程。

第一层通过主动探测选择定性的动力学草图；第二层利用 rollout 数据拟合草图中的数值参数，并验证最终程序。

第一层：发现正确的结构

仅仅拟合参数是不够的，因为预设的动力学形式本身可能不正确。一个运动物体可能遵循一阶或二阶动力学；物体操控可能由动作直接驱动，也可能由接触驱动；被抓取的物体也可能只有在满足抓握条件时才会移动。

VisualPatchWorld 使用简短的主动探测来区分一小组定性假设。这些探测考察阶跃响应、动量和接触等行为，最终得到适合当前环境的动力学草图。

第二层：拟合参数

选择结构之后，VisualPatchWorld 通过最小化多步 rollout 误差来估计其中的自由参数。多步拟合非常重要：一个模型即使能准确预测单步状态转移，也可能在较长时间范围内不断累积误差，或者错误地排序候选动作序列。

最终输出是一个可执行的状态转移函数：

1
2
3
4

def step(state, action):
    """将结构化世界状态向前推进一步。"""
    ...
    return next_state

这个程序随后成为规划器使用的预测模型。

建立在现有代码世界模型之上

当前的世界模型大致可以分为两类。

神经世界模型，例如 LeWM、DINO-WM 和 PLDM，从视觉数据中学习连续表示中的动力学。它们灵活且性能强大，但状态转移规律隐含在网络参数中。可执行代码世界模型，例如 PatchWorld、WorldCoder、POMDP-Coder、PoE-World、GIF-MCTS 和 CWM-Game，则生成可以运行和检查的程序。

VisualPatchWorld 建立在许多重要思想之上。WorldCoder 通过环境交互学习可执行模型；PatchWorld 使用反例修复 Python 动力学；POMDP-Coder 归纳紧凑的概率程序；PoE-World 组合多个程序专家；GIF-MCTS 和 CWM-Game 则利用搜索与测试反馈改进代码模型。

我们非常感谢这些工作的作者。他们提出的基础思想、算法和评测问题，是本研究得以开展的重要前提。

我们的目标并不是宣称某一种形式在所有场景中都更好，而是研究一个互补的问题：当代码世界模型需要支持连续模型预测控制时，系统辨识和符号回归中的思想能否帮助我们恢复出真正适合规划的动力学程序？

这个问题延续了我们多年来持续探索的一条研究主线：如何从观测中发现紧凑、可执行的假设。在我们关于知识图谱溯因推理的 ACL 2024 工作（论文）中，输入是一组被观测到的实体，输出则是一个能够解释它们为何共同出现的逻辑假设。这个假设是一个形式化的知识图谱查询，因此可以直接执行，用来检验该解释是否与结构化知识一致。

随后，我们在科学发现方向通过 NewtonBench（论文）研究了同一个逆向问题。与从知识图谱中恢复逻辑解释不同，NewtonBench 中的智能体需要在交互式物理环境里主动实验，并提出能够解释观测结果的科学规律。基准还会反事实地改变人们熟悉的物理定律，以检验智能体究竟是真正推断出了控制环境的机制，还是仅仅回忆了教科书公式。

VisualPatchWorld 将这条研究路线从逻辑假设推进到科学规律，再推进到可执行动力学程序。这三个问题共享同一个基本循环：观测证据、提出紧凑结构、执行或检验该结构，并在预测失败时进行修正。“程序回归”这一视角，正是我们将这一共同模式发展为更通用世界模型学习方法的尝试。

许多现有方法采用灵活的程序搜索：让 LLM 生成、组合或修改程序，再通过轨迹回放准确率、单元测试、环境交互或单步预测评估候选程序。VPW 则研究一种约束更强的方法：

1. 环境中存在什么定性动力学？ 主动探测从候选空间中选择一个紧凑的动力学草图。
2. 这些动力学的数值参数是什么？ 多步优化在规划时间范围内拟合选定的草图。

这种分解用于检验：显式识别结构能否与开放式代码生成形成互补。局部正确性和规划效用并不是同一个目标。一个程序可能通过局部测试，却仍然需要更准确的运动学或接触模型，才能正确比较长时间范围内的动作序列。

我们所借鉴的工作：按时间排序

这些方法之间是互补关系，而不是互相排斥。开放式程序合成具有灵活性，概率程序可以表示不确定性，专家组合提供模块化能力，树搜索能够扩大探索范围。VPW 则检验一种来自符号回归的归纳偏置，能否让面向规划的动力学拟合更加可靠。

程序回归：对符号回归思想的扩展

符号回归的目标是搜索能够解释观测数据的数学表达式，例如同时选择方程结构并估计其中的常数。我们将程序回归作为一种指导性视角：把这个思想从单个数学表达式扩展到可执行的状态转移程序。

一个世界模型通常不只需要一条方程。它可能包含带类型的物体状态、条件式接触、运动学变换、抓握开关，以及针对不同物体的不同更新规律。从程序回归的角度看，学习世界模型意味着：

1. 发现能够解释动作如何改变状态的可复用程序结构；
2. 从交互轨迹中估计连续参数；
3. 通过多步 rollout 和下游规划验证程序；以及
4. 保留可以被检查、组合或修复的可执行代码。

VPW 是对这一思想的初步且有意受限的验证。当前的候选动力学空间较小，并且部分依赖具体环境，因此论文尚未展示通用的程序回归系统或跨环境泛化能力。我们的长期目标是扩大动力学草图库，学习可复用的程序组件，并将结构搜索与代码生成结合起来，让同一套归纳过程能够迁移到更广泛的环境中。

两个互补的例子

• WorldCoder 让智能体收集经验、生成 Python 模型，并通过反例不断改进模型。原始方法受益于在线探索。为了进行统一的离线比较，我们让所有方法使用相同的轨迹。WorldCoder 在 Cube 上取得了很强的 74% 成功率。

• PoE-World 采用模块化方法：生成多个专家，学习专家权重，再对整个组合进行剪枝。它在 Cube 上达到 84%，接近 VPW 的 86%；同时，它在所有适配后的基线中取得了最低的 Reacher 开环误差。

这些优势说明，WorldCoder 的交互能力、PoE-World 的模块化能力和 VPW 面向 rollout 的结构拟合可以相互补充。

从比较中得到的启发

详细的 rollout 揭示了两个反复出现的挑战：

1. 局部预测不等同于规划。 模型可能准确预测下一步状态，却无法正确排序更长的动作序列。
2. 代码能够运行，不代表动力学已经正确。 机械臂需要合适的运动学模型，推动任务则需要有效的接触模型。

这些观察促使我们采用“先选择结构、再拟合参数”的设计。这并不会削弱现有基线工作的贡献；相反，它指出了将灵活代码生成与结构化动力学拟合结合起来的机会。

实验评估

我们在 LeWM 基准中的四个环境上评估 VisualPatchWorld：

在共享的归纳模型规划器下，VisualPatchWorld 的平均成功率达到 69.0%，比表现最强的代码世界模型基线高 23.5 个百分点。当正确的定性动力学非常重要时，提升最为明显：VPW 在 Reacher、PushT 和 Cube 上分别达到 72%、22% 和 86%。

与可执行代码世界模型比较

为了进行受控比较，所有方法使用相同的模拟器状态轨迹，在重新规划时使用真实状态，并共享同一个 CEM 规划器。我们在相同的 50 个测试起点上进行评估。

这些结果是将各方法适配到统一 LeWM 接口后的比较，而不是对原始系统的完整复现。在线探索、专用游戏规划器和隐状态推断等功能没有包含在统一接口中。

我们也研究了学习得到的代码与真实物理引擎之间还存在多大差距。归纳模型在导航和抓握相关控制任务上接近物理引擎上限，但接触丰富的推动任务仍然更困难。混合策略先利用归纳模型进行大范围搜索，再使用物理引擎检查少量候选计划，使平均成功率提高到 95.0%；完整使用物理引擎打分的结果为 97.5%。

归纳模型、混合验证与模拟器打分

混合打分先使用归纳程序评估全部候选动作序列，再仅使用 MuJoCo 对排名前 30% 的候选进行重新打分。这既保留了低成本的大范围搜索，也能在精确接触建模最重要的地方利用物理引擎。

如何理解这些结果

主要比较对象是使用相同状态轨迹监督和共享规划器的程序化代码世界模型基线。神经视觉世界模型仍然是非常强的参照，并且在该基准上取得了更高的整体结果。因此，我们的目标并不是声称代码能够取代神经隐空间模型。

实验结果说明，当自动归纳出的程序包含正确的定性结构时，它可以成为有用的规划模型。同时，实验也暴露了目前仍然存在的瓶颈：

• 动力学目前从结构化模拟器状态轨迹中归纳，而不是直接从像素中端到端学习。
• 图像提取的场景图用于重新规划时的实时状态估计。
• 主动结构发现主要在接触丰富的 PushT 环境中得到完整展示；其他环境使用受限的候选假设空间。
• 对于紧凑的归纳程序而言，精确接触动力学仍然很困难。

这些局限指出了下一步方向：在保留可检查、可修复接口的同时，直接从交互中同时学习视觉抽象和可执行动力学。

为什么将代码看作结构化隐变量？

神经隐状态将经验压缩为向量。VisualPatchWorld 则把代码视为一种结构化隐表示：它是对物体、交互关系和状态转移的紧凑、可操作描述。

这种视角也连接到一个更广泛的问题，而这个问题同样驱动着我在知识图谱和结构化知识方向的研究：我们如何将原始观测转化为一种抽象，使它不仅具有预测能力，而且可以被执行、验证，并真正用于推理？

引用

1
2
3
4
5
6
7
8
9

@misc{bai2026visualpatchworld,
  title={VisualPatchWorld: Code World Models as Latent Structured Representations for Planning},
  author={Jiaxin Bai and Jiaxuan Xiong},
  year={2026},
  eprint={2607.25236},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2607.25236}
}