Overview

“It works really well!” is the expected conclusion of academic publications, technical reports, and project outcomes that seek to demonstrate a designed system (i.e. robot, autonomous vehicle, controller) is functionally sound, safe and/or trustworthy. In this workshop, we aim to dissect and examine under a critical eye each component of the phrase: the “It”, namely how the system-under-test has been defined for verification and validation purposes, and how this meets the needs for robotics in sectors such as autonomous vehicles and medical robots; the “works”, namely the definition of the system’s tasks or behaviours, including systems that might learn and at industrial scale; the “really well”, namely the formulation of metrics, evaluation tools, reproducibility and replicability frameworks, and extensibility to an “industrial grade”; and, furthermore, the ability to openly report “failures”. The workshop will bring together experts in robotics, verification and validation, end-user experience, regulations and standards, formal methods, assurance and certification, and industries including healthcare, transportation, aviation, military, and automotive. We will have three sessions with invited talks followed by Q&A, and a fourth session with an interactive panel. We aim to open new avenues for future interdisciplinary collaboration among participants and expect that this will significantly advance the field of verification of autonomous systems.

This will be a hybrid workshop.

Intended Audience

We invite anyone interested in verification of autonomous systems to participate, including

• The new generations of roboticists, new to the field of verification and validation.
• Experts in verification and validation who are exploring challenges in emerging robotics fields, developing state-of-the-art tools and techniques
• Experts in human-robot collaboration, who are studying shared workspace domains and their implications to human safety.
• Experts in evaluation of robots and systems, who can provide insights on opportunities for the development of metrics, methods and tools applied to robotics.
• The community writing norms and standards to guide the development of safe and trustworthy robots and autonomous systems.
• The community working on field robotics and industrial applications, to provide feedback to the academy and to foster new collaborations between industry-academy to tackle challenges.
• Researchers interested in exploring the space of research challenges in the verification of autonomous systems.

We also invite all members from the following IEEE RAS Technical Committees:

• Verification of Autonomous Systems (all co-chairs are workshop organizers)
• Software Engineering for Robotics and Automation (co-chair is workshop organizer)
• Software Engineering for Robotics and Automation (co-chair is workshop organizer)
• Performance Evaluation and Benchmarking of Robotic and Automation Systems (co-chair is invited speaker)
• Cognitive Robotics (workshop organizers are TC members)

Content and Schedule

The workshop is a joint collaboration between the IEEE RAS Technical Committees on Verification of Autonomous Systems and Software Engineering for Robotics and Automation. The workshop will bring together experts beyond robotics, in the areas of verification and validation, end-user experience, regulations and standards, formal methods, assurance and certification, and industries including healthcare, transportation, aviation, military, and automotive. Our main goal is to discuss, with a highly self-reflective angle, the state-of-the-art in the domain of verification of autonomous systems, in terms of requirements, techniques, tools, architectures, frameworks, performance and evaluation metrics, reproducibility and replicability, and real-life use-case needs. A major goal is to build common ground on how verification and validation have been applied successfully and unsuccessfully to date, what lessons have been learned from it, and what is yet to be addressed. Additionally, we aim to open new avenues for future interdisciplinary collaboration among participants and expect that this will significantly advance the field of verification of autonomous systems.

The workshop will consist of four sessions. The first three sessions will start with four short invited talks that will position different perspectives of academy, industry, government and end-users, related to the session’s main topic. After the invited talks, each session will have a Q&A from the audience. The workshop organizers will moderate the sessions, for time keeping and inclusivity.

The last session will consist of a panel of three experts, who will address topics regarding key challenges in the field of verification in the context of robotics. This will be an open-ended discussion with the audience, and pre-selected key questions.

The diversity of expertise will encourage a cross-domain perspective in challenges and applications for the future and emergent intelligent and autonomous robotic technologies. The invited talks will cover state-of-the-art challenges, developments, combined efforts and collaborations relevant to the topics of each session. Furthermore, the flow of the invited talks has been decided to ensure a good mix of academic and industrial points of view, as well as theoretical and practical approaches. This will encourage the participants to learn, ideate and conceptualize throughout. Questions will be collected digitally before and during the event. An online chat will be made available to continue with offline discussions. Notes from the event will be made publicly available after the event, as well as slides and any other produced materials.

Note that this will be the fifth in a series of workshops at major conferences. The previous workshops were at ICRA 2016, IROS 2017, ICRA 2018, and ICRA 2022.

8:30AM - 8:35AM   Introduction

8:35AM - 10:15AM   SESSION 1: What is "It"?   What do we mean by “it”? What do industry (practice) and academia (theory) understand by the system-under-test? Invited talks by 4 speakers, followed by an interactive discussion

10:15AM - 10:45AM - COFFEE BREAK

10:45AM - 12:30AM   SESSION 2: What does "Works" mean?   What do we mean by “works”? How are industry (practice) and academia (theory) defining tasks / what is to be verified differently? Invited talks by 4 speakers, followed by an interactive discussion

12:30PM - 1:30PM - LUNCH BREAK

1:30PM - 3:15PM   SESSION 3: What does "Really Well" mean?   What do we mean by “really well”? How do we define the metrics we use to evaluate our systems? Is “really well” good enough to abandon active supervision? Invited talks by 4 speakers, followed by an interactive discussion

3:15PM - 3:45PM - COFFEE BREAK

3:45PM - 5:20PM   SESSION 4: Understanding "It Works Really Well"   Verification in the context of robotics as a science - how reproducibility, replicability, generalizability, and evaluation come together in verification. Panel with 4 members

5:20PM - 5:30PM   Closing Remarks

Note that slides and notes will be made available after the workshop via links in the above agenda and on the TC-VAS official website.

Confirmed Speakers

The following speakers have confirmed their participation:

• Fabio Bonsignorio, Heron Robots and University of Zagreb
• Ralph Brewer, United States Army Research
• Mauricio Castillo-Effen, Lockheed Martin
• Georgios Fainekos, Toyota North America
• Daniel Koditscheck, University of Pennsylvania
• Hadas Kress-Gazit, Cornell University
• Changliu Liu, Carnegie Mellon University
• Claudio Menghi, Università degli studi di Bergamo
• David Novick, Sandia National Laboratories
• Signe Redfield, US Naval Research Laboratory and TC-VAS Co-Chair
• Joe T. Rexwinkle, United States Army Research
• Ben Schumeg, US Army DEVCOM Armaments Center
• Joseph Sifiakis, Université Grenoble-Alps
• Ben Werner, US Army DEVCOM Armaments Center

Participation

Prior to the Workshop, attendance will be requested with two considerations in mind:

• We have extended individual invitations to a large list of speakers with a wide range of backgrounds (age, expertise, geographical areas, career stages, gender), to promote an inclusive environment for discussion and exchange of ideas.
• We will advertise the workshop via global and regional mailing lists, e.g robotics-worldwide, the Autonomy and Verification Network UK’s mailing list, the US’ Verification of Autonomous Systems Working Group mailing list, and leveraging on the IEEE RAS TC’s mailing lists.

Awards for travel funds will be available to early career researchers and researchers from underrepresented communities - please contact workshop organizers to apply.

In terms of publicity and communications, we are pursuing the following courses of action:

• We plan to publish a special issue based on the proceedings of the workshop. .
• The available slides from the invited talks will be published openly on the TC’s official IEEE RAS website (Activities section).
• The discussion notes from the sessions and panel will be made available on the TC’s official RAS website (Activities section).
• Available slides and discussion notes will be posted on the Workshop webpage.

Registration

If this workshop is accepted, registration will be through the main conference registration process, at IROS 2023 Registration

IEEE RAS Technical Committee Support

The proposed workshop is supported by the IEEE RAS Technical Committee on Performance Evaluation & Benchmarking of Robotic and Automation Systems (PEBRAS) and the IEEE RAS Technical Committee on Verification of Autonomous Systems (VAS),

Workshop website hosted on robotistry.org