The METRICS project is designed to organize robotics competitions as reproducible and objective evaluation campaigns. It aims to structure the European robotics and Artificial Intelligence (AI) community sustainably around the four priority areas: Healthcare, Inspections & Maintenance, Agri-food, and Agile Production. Funded by the European Union’s Horizon 2020 framework, the METRICS project is led by the National Metrology and Testing Laboratory (LNE), bringing together 17 partners with extensive experience in a robotics competition and metrology with highly complementary testing facilities and networks from all over Europe.
At the very beginning of the METRICS project, the pandemic outbreak was declared. Thus, the consortium meetings, first campaigns, and dry runs were brought online. While physical conferences, public events, and workshops have been cancelled throughout the year 2020 and the beginning of 2021, the METRICS project website and social media became the main channels for the project information share.
Pave the way for new robotics and artificial intelligence competition paradigm
Despite the pandemic and its aftermath, the METRICS consortium continued working on the projects and campaigns. The METRICS project has developed the evaluation frameworks that allow assessing intelligent robotic systems in a rigorous and reliable way. Each competition includes evaluation campaigns in physical environments with physical testbeds, and evaluation campaigns based on testing datasets. Robots participating in the field evaluations generate data. These datasets are collected, annotated, and qualified. To evaluate the robots participating in the field competitions, METRICS compares the data annotated by human experts with the automatic annotations generated by robots. Rigorous evaluation helps the potential users to identify the most promising solution for their needs through objective and reliable benchmarks. These benchmarks could, for example, feed the conformity assessment related to the AI Act, as well as generate a very strong appeal for the general public, increase the visibility of the technical solutions, maximizing marketing benefits. More about the METRICS competitions approach paving the way for new robotics and artificial intelligence competition paradigm could be found in “AI evaluation campaigns during robotics competitions: the METRICS paradigm”by Guillaume Avrin, Virginie Barbosa, Agnes Delaborde.
Metrics methodology. Photo credits: METRICS consortium
Impact the user’s well-being
The year 2021 HEART-MET (Healthcare Robotics Technologies – Metrified) cascade campaign consisted of four different challenges: activity recognition, gesture recognition, object detection, and handover failure detection. The challenges were based on datasets collected from robots performing each of the functionalities in labs set up as homes. The robots require these functionalities when operating in an assistive capacity for persons who have care needs at home or in a care facility. The evaluation campaign consisting of dataset-based challenges was conducted entirely online. Data presented for two challenges (Activity Recognition and Gesture Recognition) allowed identification of the winning entries and it was demonstrated by the AcRec@UniKoblenz from the University of Koblenz.
In the upcoming months, we will continue sharing success stories of the METRICS competitions. Stay tuned for the euRobotics newsletters and the upcoming METRICS news.
Alina Parshukova
Project manager
Robotex | METRICS project
alina@robotex.ee
metricsproject.eu