GSoC 2021

JdeRobot is a software toolkit for developing robotics and computer vision applications. These domains include sensors (for instance, cameras), actuators, and intelligent software in between. It is mostly written in C++, Python and JavaScript languages and provides a distributed component-based programming environment where the application program is made up of a collection of several concurrent asynchronous components. They may run in different computers and are connected using ICE communication middleware or ROS messages. Components interoperate through explicit interfaces. Each one may have its own independent Graphical User Interface or none at all. It is ROS-friendly and full compatible with ROS-Melodic (and Gazebo9)

Ideas list

JdeRobot simplifies the access to hardware devices from the control program. Getting sensor measurements is as simple as calling a local function, and ordering motor commands as easy as calling another local function. The platform attaches those calls to the remote invocation on the components connected to the sensor or the actuator devices. They can be connected to real sensors and actuators or simulated ones, both locally or remotely using the network. Those functions build the API for the Hardware Abstraction Layer. The robotic application get the sensor readings and order the actuator commands using it to unfold its behavior. Several driver components have been developed to support different physical sensors, actuators and simulators. The drivers are used as components installed at will depending on your configuration. They are included in the official release.

This open source project needs further development in these topics:

Project #1: Robotics Academy: new computer vision exercises

Brief Explanation: JdeRobot’s Robotics-Academy is a framework for learning robotics and computer vision. It consists of a collection of robot programming exercises using Python language where the student has to code the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision. It uses standard middleware and libraries as ROS and openCV, so the student learn the state of the art tools.

Currently there are many robotics exercises but few Computer Vision ones. Just a Color Filter and a 3D reconstruction exercise. The goal of this project is to design and program several Computer Vision exercises in JdeRobot Academy which cover an image processing introductory course using real cameras. It may take the available OpenCVdemo component as starting point.

ROS2 use for these exercises will be explored.

  • Expected results: ROS-based exercises involving Computer Vision
  • Knowledge Prerequisite: Python programming skills, OpenCV library.
  • Mentors: Eduardo Perdices (edupergar AT gmail.com) and Ignacio Arranz (n.arranz.agueda AT gmail.com)

Project #2: Robotics Academy: new drone based exercises

Brief Explanation: JdeRobot’s Robotics-Academy is a framework for learning robotics and computer vision. It consists of a collection of robot programming exercises using Python language where the student has to code the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision. It uses standard middleware and libraries as ROS and openCV, so the student learn the state of the art tools.

Currently there are 9 drone exercises available in Gazebo-9 and ROS-melodic. The goal of this project is to design and program new drone exercises such as drone inspection or drone package delivery, developing new gazebo models if needed. Furthermore, some of the available exercises might be revisited and updated.

You will work with Gazebo-11 and ROS-noetic. Also, ROS2 use for these exercises will be explored.

  • Expected results: ROS-based drones exercises
  • Knowledge Prerequisite: Python, OpenCV library and ROS.
  • Mentors: Pedro Arias (pedroariasperez96 AT gmail.com)

Project #3: Robotics-Academy: Robotic arm manipulation with moveit!

Brief Explanation: JdeRobot’s Robotics-Academy is an open-source framework for learning robotics and computer vision. It consists of a collection of robot programming exercises using Python language, where the student has to code the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision. It is compatible with the ROS middleware and the Gazebo simulator.

At the moment, Robotics Academy is almost fully oriented to service robotics. Then, the idea for this project is to expand it towards autonomous robotic manipulation using robotic arms and cobots (collaborative robots designed to interact with humans). For that purpose, the GSoC student will develop a collection of exercises of progressive complexity, ranging from the classical pick-n-place using standard ABB, Kuka or Fanuc industrial robotic arms to more complex sorting and manipulation fully-automated tasks, including 3D perception and collision checking. These last exercises will contain vision-assisted cobots and/or arm manipulators attached to mobile AGVs. Realistic environments including industrial work cells and kitting warehouse scenarios will also be developed for the Gazebo simulator.

Together with ROS and Gazebo, MoveIt! will be used as main working framework for the exercises. MoveIt! is an open source software running on top of ROS that provides interaction, visualization, planning and control functionalities for robotic manipulators. As it is already being ported to ROS2 (as MoveIt! 2), at the end of the project we will explore the migration of the exercises to ROS2 using MoveIt!2.

This proposal for diversifying our Robotics-Academy matches well with the increasing interest in extending the ROS capabilities to advanced manipulation and manufacturing. Clear examples of it are the ROS Industrial project, ROS-based competitions as the ARIAC series or the RoboND project by udacity . Our preliminar developments in robotic manipulation with ROS and MoveIt! can also be found in the Final Degree thesis of Ignacio Malo.

  • Expected results: a set of exercises for robotics arms and gazebo (description + code + worlds).
  • Knowledge Prerequisites: python, ROS and gazebo
  • Mentors: Diego Martín (diego.martin.martin AT urjc.es)

Project #4: Robotics-Academy: exercise using Deep Learning for Visual Detection

Brief Explanation: JdeRobot’s Robotics-Academy is a framework for learning robotics and computer vision. It consists of a collection of robot programming exercises. Up until now, the students had to code in Python the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision.

The idea for this project is to develop a new deep learning exercise within the Robotics-Academy context. More specifically, we will be building a template for visual detection exercises (e.g. human pose estimation or traffic sign recognition) that must be solved by providing a trained model for the particular task. In that way, instead of asking the user to code the solution in a web-based editor, he or she will have to upload a deep learning model that matches our defined set of inputs (e.g. single or multiple images) and outputs (e.g. 2D coordinates). In this project, we will be:

  • Updating our web interface for accepting models trained with PyTorch/Tensorflow as input.
  • Building new widgets for monitoring results for the particular exercise.
  • Coding the core application that will feed the trained model with input data and return the result.
  • Training a naïve model that allow us to show how the exercise can be solved.

In the following videos examples of one of our current web-based exercises and a visual detection task solved using deep learning are shown:

  • Expected results: a web-based exercise for solving a visual detection task using deep learning
  • Knowledge Prerequisite: Python, OpenCV, PyTorch/Tensorflow
  • Mentors: David Pascual ( d.pascual AT gmail.com ) and - (- AT gmail.com)

Project #5: Robotics-Academy: exercise using Deep Learning for Visual Control

Brief Explanation: JdeRobot’s Robotics-Academy is a framework for learning robotics and computer vision. It consists of a collection of robot programming exercises. Up until now, the students had to code in Python the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision.

The idea for this project is to develop a new deep learning exercise for visual robotic control within the Robotics-Academy context. In that sense, we will be building a web-based interface that allows the user to input a trained model that matches our defined input (e.g. a camera installed on a drone) and outputs (e.g. linear speed and angular velocity). The controlled robot and its environment will be simulated using Gazebo. In this project, we will be:

  • Updating our web interface for accepting models trained with PyTorch/Tensorflow as input.
  • Building new widgets for monitoring results for the particular exercise.
  • Getting a simulated environment ready.
  • Coding the core application that will feed the trained model with input data and send back the results.
  • Training a naïve model that allow us to show how the exercise can be solved.

In the following videos examples of one of our current web-based exercises and a visual control task solved using deep learning are shown:

  • Expected results: a web-based exercise for robotic visual control using deep learning
  • Knowledge Prerequisite: Python, OpenCV, PyTorch/Tensorflow, Gazebo
  • Mentors: David Pascual ( d.pascual AT gmail.com ) and - (- AT gmail.com)

Project #6: Python robot applications in the web browser: extending the PyOnBrowser converter to JS

Brief Explanation: WebSim2D is a web-based robot simulator which accepts JavaScript programming of its robots. We are working on allowing the programming of such robots in Python, directly from the browser. In GSoC-2019 we developed the first release of the PyOnBrowser converter which translates from Python to JavaScript and supports main Python language features such as Arithmetic expressions, Variable declarations, Conditional statements, While and For loops and Functions. It is based in Abstract Syntax Trees library. More details of such project can be found at Srinivasan’s blog. For 2020 we aim to extend the Python coverage of such converter including support for strings, classes, standard math, iterators, etc. In addition, automatic tests should be included to improve the quality of each accepted contribution.

Running Python robot applications in the web browser is awesome! and opens the door to learn robotics in a multiplatform way (Linux, Windows, MacOS).

  • Expected results: Python applications running inside the Websim2D robot simulator in the browser.
  • Knowledge Prerequisite: Python, JavaScript, compilers.
  • Mentors: Gorka Guardiola Múzquiz (gorka.guardiola AT urjc.es) and Srinivasan Vijayraghavan (Srinivasan.Vijayraghavan AT iiitb.org )

Project #7: Shifting VisualCircuit to a web server

Brief Explanation: VisualCircuit allows users to program robotic intelligence using a visual language which consists of blocks and wires, as electronic circuits. We want to shift this tool from a desktop application to an online web server so anyone can try it out without going through the process of installation. The main area of focus would be on the frontend. The current desktop release is based on NodeJS and Electron. We want to shift that to a web server using Django. So basically you will need to develop a UI similar to the one in the current release which synthesizes that circuit into a .vc file (JSON file) which is then read by a local machine using the Python backend (already developed). You can read further about the tool on the website. In this way, VisualCircuit will be cross-platform, attract new users and be very convenient to use.

Requirements:

  • A Dockerfile for installation of required dependencies on a local machine.
  • A Small UI wrapper around the current Python backend that can execute updated circuits on a local machine.
  • Shifting VisualCircuit to the web.
  • Users can create an account which provides online storage to save their design files.

Automated Build on Local Machine through the Website: Web server → Downloaded circuit file → Check for updated .vc file (WatchDog) → Execute updated file

  • Expected results: An online working release on VisualCircuit on a Web Server.
  • Knowledge Prerequisite: Python, Django and some basic JavaScript knowledge..
  • Mentors: JoséMaría Cañas (josemaria.plaza AT urjc.es) and Muhammad Taha (mtsg09 AT gmail.com).

Project #8: Autonomous Driving drone with Gazebo using Deep Learning techniques

Brief Explanation: Behavior Studio is a tool used to compare different autonomous driving architectures. It uses Gazebo, a well-known really powerful robotics simulator and ROS Noetic.

Currently we are capable of autonomously driving using deep learning or reinforcement learning techniques using a F1 robot on different circuits (check out the videos below).

In this project, we would like to expand the current state with a new scenario for a drone that can learn how to drive following a line over the environment. For this purpose, the drone will be using its vision capabilities and with the help of deep learning algorithms that we will be implementing using PyTorch, it should be capable of driving autonomously across the scenario under different conditions. We will be comparing different deep learning approaches based on CNN and LSTMs. Additionally, the current Behavior Studio environment should be upgraded to ROS2.

  • Expected results: New drone scenario with an autonomous driving drone using Gazebo and PyTorch.
  • Knowledge Prerequisites: Python and Deep Learning knowledge (specially PyTorch).
  • Mentors: Sergio Paniego Blanco (sergiopaniegoblanco AT gmail.com)

Project #9: Robotics-Academy: exercise on Localization using Particle Filters

Brief Explanation: JdeRobot’s Robotics-Academy is a framework for learning robotics and computer vision. It consists of a collection of robot programming exercises using Python language where the student has to code the behavior of a given (either simulated or real) robot to fit some task related to robotics or computer vision. It is ROS and Gazebo based.

Autonomous navigation and obstacle avoidance are the two most important problems in mobile robotics. Navigation consists of 3 fundamental subproblems: Self-Localization, Path-Planning and Map Generation. The idea for this project is to develop and update this exercise draft and get a ready to use exercise, using the current Web Template Design.

The goal of the exercise would be to implement the logic that would allow a robot to self locate in space by means of a laser or a camera sensor, in order to estimate its position in a given space (of which a map would be provided).

The first video shows the current web-template design and the second video describes particle filters for localization.

  • Expected results: exercise on particle filters
  • Knowledge Prerequisite: Python, Javascript and ROS
  • Mentors: Sakshay Mahna ( sakshum19 AT gmail.com ) and ()

Project #10: Robotics Academy: multirobot version of the Amazon warehouse exercise in ROS2

Brief Explanation: Swarm robots have gained the importance in the domain of Intelligent Systems with their applications being extended to warehouses. One such popular multi-robot systems could be found in Amazon warehouses, from where this project borrows its inspiration. The goal of this project is to build a multi-robot navigation exercise extending the already existing Amazon warehouse exercise (demonstrated in video) to a fleet of robots and migrating its infrastructure to ROS2.

ROS2 has brought out several improvements over ROS with changes in middleware and software architecture in many aspects and one of its target applications is multi-robot agents. In this project, we would focus on developing new exercises with ros2 for multi robots (2 or 3 robots for the beginning). The Navigation stack of ROS2 brings with it, better features to implement control of a swarm of robots. It now supports having multiple robots in the same ROS domain navigating independently. The idea is to use the ROS2 navigation stack for the control system and planning and create new robotics exercises for JdeRobot Academy. It will involve using functionalities like collision avoidance between robots, global path planning, and Multi-robot coordination for developing application based exercises.

  • Expected results: A new multirobot exercise in JdeRobot-Academy
  • Knowledge Prerequisite: C++, Python programming skills, experience with ROS.
  • Good to know:* ROS2
  • Mentors: Pankhuri Vanjani (16uec069 AT lnmiit.ac.in) and David Pascual (d.pascualhe AT gmail.com).

Application instructions for GSoC-2021

We welcome students to contact relevant mentors before submitting their application into GSoC official website. If in doubt for which project(s) to contact, send a message to jderobot AT gmail.com We recommend browsing previous GSoC student pages to look for ready-to-use projects, and to get an idea of the expected amount of work for a valid GSoC proposal.

Requirements

  • Git experience
  • C++ and Python programming experience (depending on the project)

Programming tests

Project #1 #2 #3 #4 #5 #6 #7 #8 #9 #10
Academy (A) X X X X X X X X * X
C++ (B) O O O X X O O X * X
Python (C) X X X X X X X X * X
Industrial Robots (D) * * O * * * * * * *


Where:  
* Not applicable
X Mandatory
O Optative

Before accepting any proposal all candidates have to do these programming challenges:

Send us your information

AFTER doing the programming tests, fill this web form with your information and challenge results. Then you are invited to ask the project mentors about the project details. Maybe we will require more information from you like this:

  1. Contact details
    • Name and surname:
    • Country:
    • Email:
    • Public repository/ies:
    • Personal blog (optional):
    • Twitter/Identica/LinkedIn/others:
  2. Timeline
    • Now split your project idea in smaller tasks. Quantify the time you think each task needs. Finally, draw a tentative project plan (timeline) including the dates covering all period of GSoC. Don’t forget to include also the days in which you don’t plan to code, because of exams, holidays etc.
    • Do you understand this is a serious commitment, equivalent to a full-time paid summer internship or summer job?
    • Do you have any known time conflicts during the official coding period?
  3. Studies
    • What is your School and degree?
    • Would your application contribute to your ongoing studies/degree? If so, how?
  4. Programming background
    • Computing experience: operating systems you use on a daily basis, known programming languages, hardware, etc.
    • Robot or Computer Vision programming experience:
    • Other software programming:
  5. GSoC participation
    • Have you participated to GSoC before?
    • How many times, which year, which project?
    • Have you applied but were not selected? When?
    • Have you submitted/will you submit another proposal for GSoC 2017 to a different org?

Previous GSoC students

How to increase your chances of being selected in GSoC-2021

If you put yourself in the shoes of the mentor that should select the student, you’ll immediately realize that there are some behaviors that are usually rewarded. Here’s some examples.

  1. Be proactive: Mentors are more likely to select students that openly discuss the existing ideas and / or propose their own. It is a bad idea to just submit your idea only in the Google web site without discussing it, because it won’t be noticed.

  2. Demonstrate your skills: Consider that mentors are being contacted by several students that apply for the same project. A way to show that you are the best candidate, is to demonstrate that you are familiar with the software and you can code. How? Browse the bug tracker (issues in github of JdeRobot project), fix some bugs and propose your patch submitting your PullRequest, and/or ask mentors to challenge you! Moreover, bug fixes are a great way to get familiar with the code.

  3. Demonstrate your intention to stay: Students that are likely to disappear after GSoC are less likely to be selected. This is because there is no point in developing something that won’t be maintained. And moreover, one scope of GSoC is to bring new developers to the community.

RTFM

Read the relevant information about GSoC in the wiki / web pages before asking. Most FAQs have been answered already!