End to end learning for robotic manipulators

Can we learn from scratch an end-to-end, vision-to-joint robot controller for complex manipulation tasks?

Our progress of the project

2017: End-to-end learning of a motor policy with off-the-shelf vision components

2018: End-to-end learning of vision and robot control components

2019: Learning to perform robust manipulation in the presence of physical disturbances

2020: Learning to perform manipulation in clutter with task-specified objects

Publications

• P. Abolghasemi and L. Bölöni. Accept Synthetic Objects as Real: End-to-End Training of Attentive Deep Visuomotor Policies for Manipulation in Clutter. In Proc. of International Conference on Robotics and Automation (ICRA-2020), pp. 6506–6512, May 2020.   BibTeX   Download Video 
• P. Abolghasemi, A. Mazaheri, M. Shah, and L. Bölöni. Pay attention!-Robustifying a Deep Visuomotor Policy through Task-Focused Attention. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR-2019), pp. 4254–4262, 2019.   BibTeX   Download Video 
• R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration. In Proc. of International Conference on Robotics and Automation (ICRA-2018), pp. 3758 – 3765, May 2018.   BibTeX   Download Video 
• R. Rahmatizadeh, P. Abolghasemi, A. Behal, and L. Bölöni. Learning real manipulation tasks from virtual demonstrations using LSTM and MDN. In Proc. of Thirty-Second AAAI Conf. on Artificial Intelligence (AAAI-2018), February 2018.   BibTeX   Download Video