Long-Horizon Planning and Execution with Functional Object-Oriented Networks

Featured in IEEE Robotics and Automation Letters

*Equal Contribution 1Brown University, 2Universit├Ąt Innsbruck, 3Technische Universit├Ąt Wien 4DLR
This work began when all authors were members of the Human-centered Assistive Robotics group at the Technische Universit├Ąt M├╝nchen (TUM).


Following work on joint object-action representations, functional object-oriented networks (FOON) were introduced as a knowledge graph representation for robots. A FOON contains symbolic concepts useful to a robot's understanding of tasks and its environment for object-level planning.

Prior to this work, little has been done to show how plans acquired from FOON can be executed by a robot, as the concepts in a FOON are too abstract for execution. We thereby introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution.

Our approach automatically transforms FOON into PDDL and leverages off-the-shelf planners, action contexts, and robot skills in a hierarchical planning pipeline to generate executable task plans. We demonstrate our entire approach on long-horizon tasks in CoppeliaSim and show how learned action contexts can be extended to never-before-seen scenarios.

Summary Video

Overview of Webpage

Below you will find some videos for experimental results from Section V of our paper.

Other materials can be found in our Google Drive.

Qualitative Examples: Micro-Plan Variety
(Section V-B)

In Section V-B, our goal is to show how our approach produces micro-plans that adapt to the configuration of the environment. It is important to note here that these plans are derived directly from the Fast-Downward planner, and we do not handcraft these plans as would be needed for hierarchical task networks (HTN).

Below are videos corresponding to Figure 7 in the paper. Each video is a solution to a single functional unit or macro-planning operator. Notably, each execution varies due to the state of the robot's environment.

Micro-Plan Variations

Qualitative Examples: Long-Horizon Execution
(Section V-C)

We show some examples of whole recipe execution and partial recipe execution for the Bloody Mary cocktail and Greek salad recipes. A simulated robot uses a total of 703 action contexts (635 from the cocktail scenario + 68 from the salad scenario) learned from demonstration.

In the cocktail task, robot execution was 96% successful for whole execution and 92% for partial execution; in the salad task, robot execution was 80% successful for whole execution and 84% for partial execution.

Note: Micro-plans can be opened with a text editor.

Whole Recipe (Salad)

[Whole Salad Micro-plan]

Whole Recipe (Cocktail)

[Whole Cocktail Micro-plan]

Partial Recipe (Cocktail)

[Partial Cocktail Micro-plan]


    title={{Long-Horizon Planning and Execution with Functional Object-Oriented Networks}},
    author={Paulius*, David and Agostini*, Alejandro and Lee, Dongheui},    
    journal={IEEE Robotics and Automation Letters},