, )=next(a)!=a+1

, Second loop iteration: for i 1, p.0

, )=next(g)!=g+1

, Third loop iteration: for i 1, p.1

, )=next(a)!=a

, Fifth loop iteration: for i 0, p.2

, )=next(g)!=g

, Tentative list of candidate interventions

, )=next(g)!=g+1

. Abdessalem, Testing vision-based control systems using learnable evolutionary algorithms, Proceedings of the 40th International Conference on Software Engineering -ICSE '18, pp.1016-1026, 2018.

[. Alami, An Architecture for Autonomy, The International Journal of Robotics Research, vol.17, issue.4, pp.315-337, 1998.
URL : https://hal.archives-ouvertes.fr/hal-00123273

A. , Systems-Theoretic Safety Assessment of Robotic Telesurgical Systems, Computer Safety, Reliability, and Security, vol.9337, pp.213-227, 2015.

A. , A. , R. Arnold, J. ;. Kanade, T. Kittler et al., Testing Autonomous Robot Control Software Using Procedural Content Generation, Computer Safety, Reliability, and Security, vol.8153, pp.33-44, 2013.

[. Alexander, Deriving Safety Requirements for Autonomous Systems, 4th SEAS DTC Technical Conference, 2009.

. Ames, First steps toward formal controller synthesis for bipedal robots, Proceedings of the 18th International Conference on Hybrid Systems Computation and Control -HSCC '15, pp.209-218, 2015.

[. Aniculaesei, Towards the Verification of Safety-critical Autonomous Systems in Dynamic Environments, Electronic Proceedings in Theoretical Computer Science, vol.232, pp.79-90, 2016.

. Arora, Emergency maneuver library -ensuring safe navigation in partially known environments, 2015 IEEE International Conference on Robotics and Automation (ICRA), pp.6431-6438, 2015.

. Askarpour, SAFER-HRC: Safety Analysis Through Formal vERification in Human-Robot Collaboration, Computer Safety, Reliability, and Security, vol.9922, pp.283-295, 2016.

. Avizienis, Basic concepts and taxonomy of dependable and secure computing, IEEE Transactions on Dependable and Secure Computing, vol.1, issue.1, pp.11-33, 2004.

[. Basu, Modeling Heterogeneous Real-time Components in BIP, Fourth IEEE International Conference on Software Engineering and Formal Methods (SEFM'06), pp.3-12, 2006.
URL : https://hal.archives-ouvertes.fr/hal-00375298

[. Bensalem, Designing autonomous robots, IEEE Robotics & Automation Magazine, vol.16, issue.1, pp.67-77, 2009.

[. Blanquart, Software Safety Supervision On-board Autonomous Spacecraft, Proceedings of the 2nd European Congress Embedded Real Time Software (ERTS'04), p.9, 2004.
URL : https://hal.archives-ouvertes.fr/hal-02271033

[. Bloem, Shield Synthesis: Runtime Enforcement for Reactive Systems, Tools and Algorithms for the Construction and Analysis of Systems, Lecture Notes in Computer Science, vol.3, pp.2523-2528, 2001.

[. Bruyninckx, The BRICS component model: a model-based development paradigm for complex robotics software systems, Proceedings of the 28th Annual ACM Symposium on Applied Computing -SAC '13, p.1758, 2013.

. Böhm, P. Gruber-;-böhm, and T. Gruber, A Novel HAZOP Study Approach in the RAMS Analysis of a Therapeutic Robot for Disabled Children, Computer Safety, Reliability, and Security, pp.15-27, 2010.

[. Casimiro, A kernel-based architecture for safe cooperative vehicular functions, Proceedings of the 9th IEEE International Symposium on Industrial Embedded Systems (SIES 2014), pp.228-237, 2014.

[. Cavalli, New approaches for passive testing using an Extended Finite State Machine specification, Information and Software Technology, vol.45, issue.12, pp.837-852, 2003.

[. Cook, A survey of AUV and robot simulators for multi-vehicle operations, 2014 IEEE/OES Autonomous Underwater Vehicles (AUV), pp.1-8, 2014.

. Cpse-labs, CPSE Labs -Cyber-Physical Systems Engineering Labs, 2018.

[. Crestani, Enhancing fault tolerance of autonomous mobile robots, Robotics and Autonomous Systems, vol.68, pp.140-155, 2015.
URL : https://hal.archives-ouvertes.fr/lirmm-01241181

[. Delgado, A taxonomy and catalog of runtime software-fault monitoring tools, IEEE Transactions on Software Engineering, vol.30, issue.12, pp.859-872, 2004.

[. Desai, Combining Model Checking and Runtime Verification for Safe Robotics, Runtime Verification, pp.172-189, 2017.

F. Dhillon, B. Dhillon, and A. Fashandi, Safety and reliability assessment techniques in robotics, Robotica, vol.15, issue.6, pp.701-708, 1997.

[. Dixon, The Fridge Door is Open"-Temporal Verification of a Robotic Assistant's Behaviours, Advances in Autonomous Robotics Systems, vol.8717, pp.97-108, 2014.

[. Dogramadzi, Environmental Hazard Analysis -a Variant of Preliminary Hazard Analysis for Autonomous Mobile Robots, Journal of Intelligent & Robotic Systems, vol.76, issue.1, pp.73-117, 2014.

[. Durand, Fault tolerance enhancement using autonomy adaptation for autonomous mobile robots, 2010 Conference on Control and Fault-Tolerant Systems (SysTol), pp.24-29, 2010.
URL : https://hal.archives-ouvertes.fr/lirmm-00547856

[. Ertle, Action planning for autonomous systems with respect to safety aspects, 2010 IEEE International Conference on Systems, Man and Cybernetics, pp.2465-2472, 2010.

[. Falcone, What can you verify and enforce at runtime?, International Journal on Software Tools for Technology Transfer, vol.14, issue.3, pp.349-382, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00497350

[. Falcone, A Tutorial on Runtime Verification. Engineering dependable software systems, p.35, 2013.
URL : https://hal.archives-ouvertes.fr/hal-00853727

P. ;. Falzon, K. Falzon, G. J. Pace, D. Hutchison, T. Kanade et al., Combining Testing and Runtime Verification Techniques, Model-Based Methodologies for Pervasive and Embedded Software, vol.7706, pp.38-57, 2013.

[. Feth, A Conceptual Safety Supervisor Definition and Evaluation Framework for Autonomous Systems, Computer Safety, Reliability, and Security, vol.10488, pp.135-148, 2017.

. Fiacre, The Fiacre language and Frac compiler Home Page by LAAS/CNRS, 2018.

[. Fisher, Verifying autonomous systems, Communications of the ACM, vol.56, issue.9, p.84, 2013.

[. Foughali, Formal Verification of Complex Robotic Systems on Resource-Constrained Platforms, FormaliSE: 6th International Conference on Formal Methods in Software Engineering, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01778960

J. Fox and S. Das, Safe and sound -Artificial Intelligence in Hazardous Applications, 2000.

[. Gainer, CRutoN: Automatic Verification of a Robotic Assistant's Behaviours, Critical Systems: Formal Methods and Automated Verification, pp.119-133, 2017.

[. Goldberg, Runtime verification for autonomous spacecraft software, 2005 IEEE Aerospace Conference, pp.507-516, 2005.

[. Gribov, V. Voos-;-gribov, and H. Voos, A multilayer software architecture for safe autonomous robots, Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA), pp.1-8, 2014.

[. Gspandl, A dependable perception-decision-execution cycle for autonomous robots, 2012 IEEE International Conference on Robotics and Automation, pp.2992-2998, 2012.

J. Guiochet-;-guiochet, Hazard analysis of human-robot interactions with HAZOP-UML, Safety Science, vol.84, pp.225-237, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01271565

. Haddadin, Towards the Robotic Co-Worker, Robotics Research, Springer Tracts in Advanced Robotics, pp.261-282, 2011.

. Hazop-uml, Hazard Identification with HAZOP and UML, HAZOP-UML, 2018.

[. Horányi, Scenario-based automated evaluation of test traces of autonomous systems, SAFECOMP 2013-Workshop DECS (ERCIM/EWICS Workshop on Dependable Embedded and Cyberphysical Systems) of the 32nd International Conference on Computer Safety, Reliability and Security, p.page NA, 2013.

[. Huang, ROSRV: Runtime Verification for Robots, vol.8734, pp.247-254, 2014.

[. Huber, Formalising and Monitoring Traffic Rules for Autonomous Vehicles in Isabelle/HOL, vol.10510, pp.50-66, 2017.

, Effect of parallel planning on system reliability of real-time expert systems, IEEE Transactions on Reliability, vol.46, issue.1, pp.81-87, 1997.

, Safety of machinery -General principles for design -Risk assessment and risk reduction, p.12100, 2013.

, Safety aspects -Guidelines for their inclusion in standards, IEC Guide, vol.51, 2014.

[. Jiang, A Case Study on Runtime Monitoring of an Autonomous Research Vehicle (ARV) System, Runtime Verification, vol.41, pp.102-117, 2015.

K. Kane, A. Kane, and P. Koopman, Ride-through for Autonomous Vehicles, SAFECOMP 2013-Workshop CARS (2nd Workshop on Critical Automotive applications: Robustness & Safety) of the 32nd International Conference on Computer Safety, 2013.
URL : https://hal.archives-ouvertes.fr/hal-00848195

P. Klein-;-klein, The Safety-Bag Expert System in the Electronic Railway Interlocking System ELEKTRA, Operational Expert System Applications in Europe, pp.1-15, 1991.

, Knightscope Issues Field Incident Report, 2018.

, Knightscope Issues MIN42 Field Incident Report, Report. Accessed, 2017.

[. Kwiatkowska, , 2007.

, Formal Methods for Performance Evaluation: 7th International School on Formal Methods for the Design of Computer, Communication, and Software Systems, Stochastic Model Checking, pp.220-270, 2007.

[. Lesire, Mauve: a Componentbased Modeling Framework for Real-time Analysis of Robotic Applications, 7th full day Workshop on Software Development and Integration in Robotics (ICRA2012 -SDIR VII), 2012.
URL : https://hal.archives-ouvertes.fr/hal-01060327

M. Leucker and C. Schallhart, A brief account of runtime verification, The Journal of Logic and Algebraic Programming, vol.78, issue.5, pp.293-303, 2009.

[. Ligatti, Run-Time Enforcement of Nonsafety Policies, ACM Transactions on Information and System Security, vol.12, issue.3, pp.1-41, 2009.

[. Lopes, Supervisory control theory applied to swarm robotics, Swarm Intelligence, vol.10, issue.1, pp.65-97, 2016.

[. Lotz, Runtime monitoring of robotics software components: Increasing robustness of service robotic systems, 15th International Conference on Advanced Robotics (ICAR), pp.285-290, 2011.

[. Luckcuck, Formal Specification and Verification of Autonomous Robotic Systems: A Survey, 2018.

[. Lussier, Fault Tolerant Planning for Critical Robots, p.37, 2007.
URL : https://hal.archives-ouvertes.fr/hal-01292653

, Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07), pp.144-153

M. Machin, Synthèse de règles de sécurité pour des systèmes autonomes critiques, 2015.

[. Machin, Model-Checking and Game theory for Synthesis of Safety Rules, 2015 IEEE 16th International Symposium on High Assurance Systems Engineering, pp.36-43, 2015.
URL : https://hal.archives-ouvertes.fr/hal-01164981

[. Machin, SMOF: A Safety Monitoring Framework for Autonomous Systems, IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol.48, issue.5, pp.702-715, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01394139

[. Mallet, GenoM3: Building middleware-independent robotic components, 2010 IEEE International Conference on Robotics and Automation, pp.4627-4632, 2010.

[. Malm, Safety of Interactive Robotics-Learning from Accidents, International Journal of Social Robotics, vol.2, issue.3, pp.221-227, 2010.

[. Masson, Synthesis of Safety Rules for Active Monitoring: Application to an Airport Light Measurement Robot, 2017 First IEEE International Conference on Robotic Computing (IRC), pp.263-270, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01465734

[. Muscettola, IDEA: Planning at the Core of Autonomous Reactive Agents, AIPS Workshop on On-line Planning and Scheduling, 2002.

, Collision Between a Car Operating With Automated Vehicle Control Systems and a Tractor-Semitrailer Truck Near, 2016.

. Nusmv, NuSMV home page, NuSMV, 2016.

. O'brien, Automatic Verification of Autonomous Robot Missions, Simulation, Modeling, and Programming for Autonomous Robots, pp.462-473, 2014.

[. Pace, A Safety Integrated Architecture for an Autonomous Excavator, International Symposium on Automation and Robotics in Construction, 2000.

[. Pathak, Ensuring safety of policies learned by reinforcement: Reaching objects in the presence of obstacles with the iCub, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.170-175, 2013.

C. Pecheur, Verification and Validation of Autonomy Software at NASA, 2000.

O. Pettersson, Execution monitoring in robotics: A survey, Robotics and Autonomous Systems, vol.53, issue.2, pp.73-88, 2005.

[. Pike, Runtime Verification for Ultra-Critical Systems, vol.7186, pp.310-324, 2012.

[. Powell, Testing the Input Timing Robustness of Real-Time Control Software for Autonomous Systems, Ninth European Dependable Computing Conference, pp.73-83, 2012.

F. Py and F. Ingrand, Dependable execution control for autonomous robots, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), vol.2, pp.1136-1141, 2004.

[. Quigley, ROS : an open-source Robot Operating System, International Conference on Robotics and Automation (ICRA), Workshop on open source software, 2009.

P. J. Ramadge and W. M. Wonham, Supervisory control of a class of discrete event processes, SIAM journal on control and optimization, vol.25, issue.1, pp.206-230, 1987.

[. Roderick, The Ranger robotic satellite servicer and its autonomous software-based safety system, IEEE Intelligent Systems, vol.19, issue.5, pp.12-19, 2004.

, SAPHARI -Safe and Autonomous Physical Human-Aware Robot Interaction -Home, 2018.

[. Shoaei, Automatic generation of controllers for collision-free flexible manufacturing systems, 2010 IEEE International Conference on Automation Science and Engineering, pp.368-373, 2010.

S. , SMT-LIB The Satisfiability Modulo Theories Library, 2018.

. Sorin, Rulebased Dynamic Safety Monitoring for Mobile Robots, vol.7, pp.120-141, 2016.

. Sotiropoulos, Virtual Worlds for Testing Robot Navigation: A Study on the Difficulty Level, 12th European Dependable Computing Conference (EDCC), pp.153-160, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01328909

. Sotiropoulos, Can Robot Navigation Bugs Be Found in Simulation? An Exploratory Study, 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp.150-159, 2017.
URL : https://hal.archives-ouvertes.fr/hal-01534235

[. Stringfellow, Safety-Driven Design for Software-Intensive Aerospace and Automotive Systems, Proceedings of the IEEE, vol.98, pp.515-525, 2010.

[. Svensson, Safe Stop Trajectory Planning for Highly Automated Vehicles: An Optimal Control Problem Formulation, IEEE Intelligent Vehicles Symposium (IV), pp.517-522, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01945273

, The TINA toolbox Home Page -TIme petri Net Analyzerby LAAS/CNRS, 2018.

. Tomatis, Designing a secure and robust mobile interacting robot for the long term, 2003 IEEE International Conference on Robotics and Automation (Cat. No.03CH37422), vol.3, pp.4246-4251, 2003.

. Troubitsyna, E. Vistbakka-;-troubitsyna, and I. Vistbakka, Deriving and Formalising Safety and Security Requirements for Control Systems, Developments in Language Theory, vol.11088, pp.107-122, 2018.

. Tuleap, Tuleap ? Open Source Agile Project Management and Software Development tools, 2018.

[. Täubig, Guaranteeing functional safety: design for provability and computer-aided verification, vol.32, pp.303-331, 2012.

. Van-nunen, Towards a safety mechanism for platooning, 2016 IEEE Intelligent Vehicles Symposium (IV), pp.502-507, 2016.

[. Vistbakka, Multilayered Approach to Safe Navigation of Swarms of Drones, Computer Safety, Reliability, and Security, vol.11088, pp.112-125, 2018.

[. Volpe, The CLARAty architecture for robotic autonomy, IEEE Aerospace Conference Proceedings (Cat. No.01TH8542), vol.1, 2001.

[. Wagner, Balancing safety and availability for an electronic protection system, European Safety and Reliability Conference, 2008.

[. Woodman, Building safer robots: Safety driven control, The International Journal of Robotics Research, vol.31, issue.13, pp.1603-1626, 2012.

[. Zaman, An integrated model-based diagnosis and repair architecture for ROS-based robot systems, 2013 IEEE International Conference on Robotics and Automation, pp.482-489, 2013.

[. Zou, Safety Validation of Sense and Avoid Algorithms Using Simulation and Evolutionary Search, Computer Safety, Reliability, and Security, pp.33-48, 2014.