Long-Horizon Planning and Execution with Functional Object-Oriented Networks

Featured in IEEE Robotics and Automation Letters

*Equal Contribution 1Brown University, 2Universität Innsbruck, 3Technische Universität Wien 4DLR
This work began when all authors were members of the Human-centered Assistive Robotics group at the Technische Universität München (TUM).

Abstract

Following work on joint object-action representations, functional object-oriented networks (FOON) were introduced as a knowledge graph representation for robots. A FOON contains symbolic concepts useful to a robot's understanding of tasks and its environment for object-level planning.

Prior to this work, little has been done to show how plans acquired from FOON can be executed by a robot, as the concepts in a FOON are too abstract for execution. We thereby introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution.

Our approach automatically transforms FOON into PDDL and leverages off-the-shelf planners, action contexts, and robot skills in a hierarchical planning pipeline to generate executable task plans. We demonstrate our entire approach on long-horizon tasks in CoppeliaSim and show how learned action contexts can be extended to never-before-seen scenarios.



Summary Video



Overview of Webpage

Below you will find some videos for experimental results from Section V of our paper.

Other materials can be found in our Google Drive.



Qualitative Examples: Micro-Plan Variety
(Section V-B)

In Section V-B, our goal is to show how our approach produces micro-plans that adapt to the configuration of the environment. It is important to note here that these plans are derived directly from the Fast-Downward planner, and we do not handcraft these plans as would be needed for hierarchical task networks (HTN).

Below are videos corresponding to Figure 7 in the paper. Each video is a solution to a single functional unit or macro-planning operator. Notably, each execution varies due to the state of the robot's environment.

Micro-Plan Variations



Qualitative Examples: Long-Horizon Execution
(Section V-C)

We show some examples of whole recipe execution and partial recipe execution for the Bloody Mary cocktail and Greek salad recipes. A simulated robot uses a total of 703 action contexts (635 from the cocktail scenario + 68 from the salad scenario) learned from demonstration.

In the cocktail task, robot execution was 96% successful for whole execution and 92% for partial execution; in the salad task, robot execution was 80% successful for whole execution and 84% for partial execution.

Note: Micro-plans can be opened with a text editor.


Whole Recipe (Salad)


[Whole Salad Micro-plan]

Whole Recipe (Cocktail)


[Whole Cocktail Micro-plan]

Partial Recipe (Cocktail)


[Partial Cocktail Micro-plan]



BibTeX

@article{paulius2023_foonlhpe,
    title={{Long-Horizon Planning and Execution with Functional Object-Oriented Networks}},
    author={Paulius*, David and Agostini*, Alejandro and Lee, Dongheui},    
    journal={IEEE Robotics and Automation Letters}, 
    year={2023},
    volume={8},
    number={8},
    pages={4513-4520},
    doi={10.1109/LRA.2023.3285510}
}