We present CLIMB, a continual learning framework for robot task planning that leverages foundation models and execution feedback to guide domain model construction.
CLIMB incrementally builds a PDDL model of its operating environment while completing tasks, creating a set of world state predicates that function as a representation of the causal structure present in the environment. CLIMB's continual learning approach enables it to solve classes of problems it has previously encountered without needing to relearn task-specific information, endowing it with the ability to expand its environment representation to novel problem formulations.
We show CLIMB to be a capable zero-shot planner for simple tasks. For complex tasks with non-obvious predicates, we demonstrate CLIMBs ability to self-improve through iterative executiong and feedback, resulting in superior performance once a PDDL model has been established.
When attempting to build a pyramid, CLIMB is initially missing a precondition that the two base blocks must be adjacent. The resulting plan results in an observation that the block was placed between, but not on top of, the base blocks.
Through iterative reprompting and refinement, CLIMB adds an additional precondition that 2 blocks must be adjacent for a third block to be spanned across them. Adding base adjacency solves the pyramid stacking problem.
We demonstrate CLIMB's ability to learn from evaluations on real hardware. We have created a demonstration platform which mirrors the IsaacLab Blocksworld environment to evaluate the sim to real translation of CLIMB's observations and plans. To address the challenge of perception, we place AprilTag2 markers on the cubes and perceive the scene using a wrist-mounted camera. By returning to a home position after each action, we can gather full pose information for all objects in the scene. The object and robot pose data gathered in this manner mirrors the ground truth state information available in IsaacSim, resulting in similar performance across the two platforms.
Below are demonstrations of CLIMB solving problems and learning from failures in our BlocksWorld++ on real hardware.
We have developed a simulation environment in IsaacLab to evaluate BlocksWorld and BlocksWorld++ problem cases. The IsaacLab environment mirrors the physical robot system but abstracts away perception by directly accessing object state information. The information provided to the LLM for error evaluation and improvment directly matches the physical robot embodiment.
We would like to highlight some other work on language-guided robot planning:
@misc{byrnes2024climblanguageguidedcontinuallearning,
title={CLIMB: Language-Guided Continual Learning for Task Planning with Iterative Model Building},
author={Walker Byrnes and Miroslav Bogdanovic and Avi Balakirsky and Stephen Balakirsky and Animesh Garg},
year={2024},
eprint={2410.13756},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2410.13756},
}