Available on InterRegs.NET

Our online subscription service, offering immediate access to our extensive library of global vehicle regulations, standards and legislation. A fully searchable, accurate, user-friendly resource for consolidated regulations that are updated quickly and frequently.

Tell me more | Already a subscriber

Available on SelectRegs.com

Our fast and easy means of purchasing up-to-date global vehicle and component standards and regulations on a pay-as-you-go basis. Pay securely by credit card and your documents are delivered directly and immediately to your computer as PDF files.

Tell me more | Go straight to site

ads, safety, testing, vehicle, scenarios, scenario, test, data, validation, ego, manufacturer, virtual, lane, road, system, requirements, assessment, performance, traffic, information, recommended, vehicles, driving, simulation, real, parameters, relevant, world, speed, track, model, provide, occurrences, reporting, conditions, management, description, functional, credibility, level, analysis, process, approach, documentation, toolchain, design, development, vmad, monitoring, document

ECE/TRANS/WP.29/2022/58 as amended by WP.29-187-08
NEW ASSESSMENT/TEST METHOD FOR AUTOMATED DRIVING
(NATM) GUIDELINES FOR VALIDATING AUTOMATED DRIVING
SYSTEM (ADS)
Submitted by the Working Party on
Automated/Autonomous and Connected Vehicles*
ENDORSED BY
THE WORLD FORUM FOR HARMONIZATION OF VEHICLE REGULATIONS WP.29
AT THE 187 SESSION IN JUNE 2022.

---

6. Validating ADS safety is a highly complex task which cannot be done comprehensively nor
effectively through one validation methodology alone. As a result, it is recommended to
adopt a multi-pillar approach for the validation of ADS, composed of a scenarios catalogue
and five validation methodologies (pillars).
(a)
(b)
(c)
(d)
(e)
(f)
Scenarios catalogue
Simulation/virtual testing,
Track testing
Real world testing
Audit/assessment
In-service monitoring and reporting
7. The following chapters of this guidance document explore each of these components of the
NATM in further detail and outline a number of recommendations and consideration when
using them to validate ADS safety. Further information on how the components of the NATM
guidelines (i.e., the scenarios catalogue and pillars) operate together, producing an efficient,
comprehensive, and cohesive process is discussed at the end of the document.
8. ADS technology is continuously evolving. Going forward, this document will be further
developed and regularly updated and informed by the outcomes of future research and
testing as well as through the work of WP.29 working groups.
9. In particular, updates to these guidelines will take into consideration the deliverables from
the informal working group on Functional Requirements for Automated Vehicles (FRAV),
which has been tasked by WP.29 to develop safety performance requirements, including
measurable/verifiable criteria, to assess ADS safety.
10. Subject to direction from GRVA and WP.29, once the guidelines have reached a sufficient
state of maturity it is anticipated that this document will be used to help inform the
development of regulatory requirements that meet the needs of both 1958 and 1998
Agreement parties (subject to approval by WP.29).
III.
DEFINITIONS
11. The introduction of ADS and related technologies has resulted in a proliferation of new
terms and concepts. To ensure consistency, a glossary of terms and definitions used in the
NATM guidelines are attached in Annex I. These terms, which are used throughout the
document, have been italicized for reference. This Glossary will be further developed and
updated on an ongoing basis. Where applicable, VMAD will ensure these terms are
consistent with those adopted by WP.29, GRVA, and other GRVA Informal Working Groups,
including definitions agreed upon by FRAV.

---

V. SCENARIOS CATALOGUE
14. At this relatively early stage in the development of AVs, much of the existing literature that
assesses the current state of AV development uses metrics such as miles/kilometres
travelled in real-world test situations with the absence of a collision, a legal infraction, or a
disengagement by the vehicle's ADS.
15. Simple metrics such as kilometres travelled without a collision, legal infraction, or
disengagement can be helpful for informing public dialogue about the general progress
being made to develop AVs. Such measurements on their own, however, do not provide
sufficient evidence to the international regulatory community that an AV will be able to safely
navigate the vast array of different situations a vehicle could reasonably be expected to
encounter.
16. Furthermore, validation through real world testing alone would be time and cost prohibitive,
potentially requiring an AV to drive billions of kilometres without incident to prove that it has
significantly better safety performance than a human driver. It would also not be feasible to
replicate this testing later.
17. With these considerations in mind, it is recommended that a scenarios-based approach be
used to systematically organize safety validation activities in an efficient, objective,
repeatable, and scalable manner.
18. Scenarios based validation consists of reproducing specific real-world situations that
exercise and challenge the capabilities of an ADS-equipped vehicle to operate safely.
19. Going forward, VMAD will establish a catalogue of scenarios that should be considered to
validate, using the NATM pillars, the functional safety requirements established by FRAV.
A. What is a Traffic Scenario?
20. A scenario is a description of one or more real-world driving situations that may occur during
a given trip . Scenarios can involve many elements, such as roadway layout, types of road
users, objects exhibiting static or diverse dynamic behaviours, and diverse environmental
conditions (among other factors).
B. Ensuring Adequate Scenario Coverage
21. It is recommended that the scenarios-based validation methods include adequate coverage
of relevant, critical, and complex scenarios to effectively validate an ADS. To note:
"Coverage" refers to the degree to which scenarios sufficiently incorporates real-world
driving situations in order to validate the relevant requirements made by FRAV IWG.
Sufficient coverage is essential to the overall effectiveness and credibility as a validation
approach.
22. When validating the safety of an ADS, it is recommended that each scenario selected to test
the ADS reflects the particular conditions (e.g., road configurations, direction of traffic in a
given lane, etc.) relevant to the ODD in which the ADS is designed to operate. Scenarios
should be relevant to the ADS feature being validated. For example, an ADS feature
intended only for highway use would not be subject to a scenario involving turns at
intersections.

---

27. It is recommended that the following structured approach for categorizing and describing
scenarios by different levels of abstraction according to three categories: functional, logical,
and concrete scenarios is used. See Figure 1 below.
(a)
(b)
(c)
Functional Scenario: Scenarios with the highest level of abstraction, outlining the core
concept of the scenario, such as a basic description of the ego vehicle's actions; the
interactions of the ego vehicle with other road users and objects; roadway geometry;
and other elements that compose the scenario (e.g. environmental conditions etc.).
This approach uses accessible language to describe the situation and its
corresponding elements.
Logical Scenario: Building off the elements identified within the functional scenario,
developers generate a logical scenario by selecting value ranges or probability
distributions for each element within a scenario (e.g., the possible width of a lane in
meters).
Concrete Scenarios: Concrete scenarios are established by selecting specific values
for each element. This step ensures that a specific test scenario is reproducible. In
addition, for each logical scenario with continuous ranges, any number of concrete
scenarios can be developed, helping to ensure a vehicle is exposed to a wide variety
of situations.
Figure 1
Examples of a Scenario Using Functional, Logical and Concrete Categorizations (Pegasus, 2018).

---

33. Open-loop virtual tests (also referred to as software or hardware reprocessing, shadow
mode, etc.) could be done through a variety of methods, such as the ADS interacting with a
virtual situation collected from the real world. In this case, the virtual objects' actions are
data-driven only and the information is not self-corrected based on feedback from the
output. Because the open-loop controller may vary due to external disturbances without the
ADS and/or the assessor being aware, the applicability of open-loop tests in the ADS
validation may be limited.
34. Closed-loop virtual tests include a feedback loop that continuously sends information from
the closed-loop controller to the ADS. Within these test systems, the digital objects in the
environment could react in different ways depending on the action of the system under test.
35. Selecting an open- or closed-loop test could depend on factors such as the objectives of the
virtual testing activity and the status of development of the system under test. For ADS
validation it is expected that mainly closed-loop virtual testing will be considered.
36. The flexibility of the pillar makes it a standard test method during a vehicle's design and the
ADS validation. For an ADS, given the impossibility to test the vehicle's behaviour in the real
world and in all possible situations as well as for any change in its driving logic, then virtual
testing becomes an indispensable tool to verify the capability of the automated system to
deal with a wide variety of possible traffic scenarios. In addition, virtual testing can be
extremely beneficial in replacing real world and proving ground testing if there are concerns
over safety-critical traffic scenarios. As a result, it is recommended that virtual testing be
used to test the ADS under safety critical scenarios that would be difficult and/or unsafe to
reproduce on test tracks or public roads.
37. Virtual tests used for ADS validation can achieve different objectives, depending on the
overall validation strategy and the accuracy of the underlying simulation models. When
assessing the safety of an ADS using virtual testing it is recommended that the following
strengths and weaknesses are considered:
(a)
(b)
(c)
(d)
Provide qualitative confidence in the safety of the full system.
Contribute directly to statistical confidence in the safety of the full system (caveats
apply).
Provide qualitative or statistical confidence in the performance of specific subsystems
or components.
Discover challenging scenarios that can be tested in the real world (e.g. real-world
tests and track tests described in Chapter 7 and 8 of this Document).
38. In contrast to all its potential benefits, a limitation of this approach is in its intrinsic limited
fidelity. As models provide a representation of the reality, the suitability of a model to
satisfactorily replace the real world for validating the safety of ADSs has to be carefully
assessed. Therefore, the validation of the simulation models used in virtual testing is
essential to determine the quality and reliability of the results compared to real-world
performance.

---

(f)
Virtual testing can play an important role in responding to concerns identified through
in-use monitoring of ADS performance. Virtual testing provides a quick and flexible
approach to analyse ADS performance based on real-world events. It allows
manufacturers to understand and verify the ADS behaviour and to understand why an
issue may have occurred. It may identify an untested scenario, or a set of untried
parameters. It may also identify the "scale" of any issue. If the ADS does demonstrate
unsafe behaviour it can help to assess modifications and ultimately to improve
performance. Where appropriate, the information and scenario descriptions can be
shared and integrated rapidly into virtual testing regimes worldwide.
42. Recognizing that specific functional safety requirements are still under development, virtual
testing using a validated simulation toolchain, shows particular promise for assessing the
following general safety requirements currently being considered:
(a)
(b)
(c)
The ADS should drive safely and manage safety critical situations. These are the
requirements where virtual testing can play the most prominent role. MIL/SIL, HIL and
VIL virtual testing can all be used to assess these requirements at different stages of
vehicle verification and validation.
The ADS should interact safely with the user. DIL virtual testing can be helpful to
support the assessment of this Category of safety requirement by analysing the
interaction between the driver and the ADS in a safe and controlled environment.
The ADS should safely manage failure modes and ADS should ensure a safe
operational state. The use of virtual testing in these two categories is also very
promising but would probably require further research work. SIL virtual testing could
include simulated failures and maintenance requests. HIL and VIL virtual testing could
be used to assess how the system would react to the occurrence of a real
malfunctioning induced to the real system.
VII. TRACK TESTING – PILLAR 2
43. Track testing occurs on a closed-access testing ground that uses real obstacles and obstacle
surrogates (e.g., vehicle crash targets, etc.) to assess the safety requirements of an ADS
(e.g., human factors, safety system). This testing approach allows for the physical vehicles to
be tested through realistic scenarios to evaluate either sub-systems or the fully assembled
system. These external inputs and conditions can be controlled or measured during a test.
44. Track testing is suitable for assessing the ADS capabilities in nominal scenarios and critical
scenarios. The same tests can be used to verify the performance of the vehicles regarding
human factors or fallback in these scenarios. However, operating on test tracks can be
resource intensive. For more background information on track testing, such as its strengths
and weaknesses, please review the NATM Master Document.
45. It is recommended that track testing be used to assess the performance of ADS in a number
of selected important nominal and critical scenarios, notably given that, unlike real-world
testing, track testing can accelerate exposure to known rare events or safety critical
scenarios, and in a more controlled and safer environment.
46. It is furthermore recommended to develop the track tests in line with the approach set out in
Annex V.

---

A. How the Pillar Interacts with Other Pillars
55. Data generated during real-world testing may be used as additional data to validate whether
portions of a virtual and/or track-testing environment were modelled properly by comparing
an ADS' performance within a simulation and track test with its performance in a real-world
environment when executing the same test scenario.
56. It can also be used to support the development of new traffic scenarios for track and virtual
testing, allowing for the identification of edge cases and other unanticipated hazardous
situations that could challenge the ADS.
57. The information gathered from real world testing may also support improvements in the
hazard and risk analysis and design of the ADS systems.
IX. AUDIT – PILLAR 4
58. The purpose of the audit pillar is to assess/demonstrate that the:
(a)
(b)
Manufacturer has the right processes to ensure operational and functional safety
during the vehicle lifecycle, and
Vehicle design is safe by design and this design is sufficiently validated before market
introduction.
58 bis. Therefore, this pillar is composed of two main components: one is the audit of the
manufacturer processes established through a safety management system, and the other
one consists in the safety assessment of the ADS design.
59. It is recommended that the manufacturer is required to demonstrate that:
(a)
(b)
(c)
Robust processes are in place to ensure safety throughout the vehicle lifecycle
(development phase, production, but also operation on the road and
decommissioning). This shall include taking the right measures to monitor the vehicle
in the field and to take the right action when necessary;
Hazard and risks relevant for the system have been identified and a consistent
safety-by-design concept has been put in place to mitigate these risks; and
The risk assessment and the safety-by-design concept have been validated through
testing by the manufacturer to show that the vehicle meets the safety requirements
before it is placed in the market. The vehicle should be free of unreasonable safety
risks to the broader transport ecosystem, in particular, the driver, passengers and
other road users.
60. On the basis of the evidence provided by the manufacturer and the targeted tests,
authorities will be able to audit and assess whether the processes, the risk assessment, the
design and the validation of the manufacturer are robust enough with regard functional and
operational safety.

---

65. Examples of processes and aspects that are recommended to be documented by the
manufacturer:
(a)
Safety governance
(i) Safety policies and principles (in line with the concept stated in ISO 21434,
Paragraph 5.4.1. and ISO 9001 Automotive 5.2, but from safety perspective)
(ii) Management commitment (in line with the concept stated in ISO 21434,
Paragraph 5.4.1. and ISO 9001 Automotive 5.1, but from safety perspective)
(iii)
Roles and responsibilities (ISO 26262-2, Paragraph 6.4.2., this relates to the
organizational as well as to the project dependent activities)
(b) Safety culture (ISO 26262-2, Paragraph 5.4.2.)
(c) Effective communications within the organization (ISO 26262-2, Paragraph 5.4.2.3.)
(d)
(e)
Information sharing outside of the organization (in line with the concept stated in
ISO 21434, Paragraph 5.4.5. and ISO 9001, but from safety perspective)
Quality management system (e.g., as per IATF 16949 or ISO 9001 or equivalent) to
support safety engineering, including change management, configuration
management, requirement management, tool management etc.
66. It is recommended that the design and development process is established and
documented including risk management, requirements management, requirements'
implementation, testing, failure tracking, remedy, and release
67. Examples of processes and aspects that are recommended to be documented to ensure the
robustness of the design and development phase:
(a)
(b)
A general description of the way in which the organization performs all the design and
development activities
Vehicle\system development, integration and implementation.
(i)
(ii)
Requirements management (e.g. Requirement capture and validation)
Validation strategies, including but not limited to
a.
Credibility assessment for virtual tool chain
b.
System Integration level
c.
Software level
d.
Hardware level
(iii)
Management of functional Safety and operational safety, including the
continuing evaluation and update of risk assessments and relationship with
In-Service Safety
(c)
Management of design changes and changes to design and development processes

---

74. It is recommended that manufacturers put in place suitable arrangements (e.g. contractual
arrangements, clear interfaces, quality management system) with suppliers to ensure that
the supplier safety management system complies with the requirements of guidance except
for vehicle related aspects like "operation" and "decommissioning"
75. Examples of processes and aspects that are recommended to be documented:
(a)
(b)
(c)
(d)
(e)
Organizational policy for supply chain
Incorporation of risks originating from supply chain
Evaluation of supplier SMS capability and corresponding audits
Processes to establish contracts, agreements for ensuring safety across the phases
of development, production and postproduction
Processes for distributed safety activities.
76. It is recommended that manufacturers have processes to monitor safety-relevant incidents/
crashes/collisions caused by the engaged ADS and a process to manage potential
safety-relevant gaps post-registration (closed loop of field monitoring) and to update the
vehicles.
77. The manufacturer should have processes to report critical incidents (e.g. collision with
another road users and potential safety-relevant gaps) to the relevant authority when critical
incidents occur.
2. Link with the In-service Monitoring/Reporting Pillar.
78. The manufacturers should set up process for the operational phase for confirmation of
compliance with the safety requirements in the field, early detection of new unknown
scenarios (in line with Safety Of the Intended Function (SOTIF) safety development goal to
minimize the unknown scenarios area), event investigation, to share learnings derived from
incidents and near-miss analysis to allow the whole community to learn from operational
feedback and to contribute to the continuous improvement of automotive safety
79. Example of guiding principles: Is there a document describing the appropriate procedure of
reporting incidents to the management? Is there evidence that the company is complying
with that procedure? Is there a document describing the appropriate procedure of
investigation and documentation of incidents? Is there evidence that the company is
complying with that procedure?
3. Expiration/Renewal of the SMS
80. It is recommended that documentation be regularly updated in line with any relevant
changes to the SMS processes. Any changes to SMS documentation should be
communicated as required to the relevant authority.

---

3. ADS Layout and Schematics
(a)
Inventory of Components
86. A list should be provided, collating all the units of the ADS and mentioning the other vehicle
systems which are needed to achieve the control function in question.
87. An outline schematic showing these units in combination should be provided, with both the
equipment distribution and the interconnections made clear.
88. It is recommended that the outline includes:
(a)
(b)
(c)
(d)
(e)
Perception and objects detection including mapping and positioning
Characterization of Decision-making
Remote supervision and remote monitoring by a remote supervision centre (if
applicable).
Information display/user interface
The data storage system (DSSAD).
(b)
Functions of the Units
89. The function of each unit of "The ADS" should be outlined and the signals linking it with
other units or with other vehicle systems should be shown. This may be provided by a
labelled block diagram or other schematic, or by a description aided by such a diagram.
90. It is recommended that interconnections within "The ADS" should be shown by a circuit
diagram for the electric transmission links, by a piping diagram for pneumatic or hydraulic
transmission equipment and by a simplified diagrammatic layout for mechanical linkages.
The transmission links both to and from other systems should also be shown.
91. There should be a clear correspondence between transmission links and the signals carried
between Units. Priorities of signals on multiplexed data paths should be stated wherever
priority may be an issue affecting performance or safety.
(c)
Identification of Units
92. Each unit should be clearly and unambiguously identifiable (e.g. by marking for hardware,
and by marking or software output for software content) to provide corresponding hardware
and documentation association. Where the software version can be changed without
requiring replacement of the marking or component, the software identification must be by
software output only.
93. It is recommended that where functions are combined within a single unit or indeed within a
single computer, but shown in multiple blocks in the block diagram for clarity and ease of
explanation, only a single hardware identification marking should be used. The
manufacturer should, by the use of this identification, affirm that the equipment supplied
conforms to the corresponding document.

---

101. If the chosen provision selects a second (back-up) means to realize the performance of the
dynamic driving task, it is recommended that the principles of the change-over mechanism,
the logic and level of redundancy and any built in back-up checking features be explained
and the resulting limits of back-up effectiveness defined.
102. If the chosen provision selects the removal of the automated driving function, it is
recommended that this is done in compliance with the relevant provisions of this regulation.
All the corresponding output control signals associated with this function should be inhibited.
103. The documentation should be supported, by an analysis which shows, in overall terms, how
the ADS will behave to mitigate or avoid hazards which can have a bearing on the safety of
the driver (if applicable), passengers and other road users. It should show how unknown
hazardous scenarios will be managed by the manufacturer in order to keep the residual
level or risk under control.
104. The chosen analytical approach(es) should be established by the manufacturer and made
available to the relevant authority before market introduction.
105. The auditor should perform an assessment of the application of the analytical approach(es):
(a)
(b)
(c)
(d)
Inspection of the safety approach at the concept (vehicle) level.
It is recommended that this approach be based on a Hazard/Risk analysis
appropriate to system safety.
Inspection of the safety approach at the ADS level including a top down (from
possible hazard to design) and bottom-up approach (from design to possible
hazards). The safety approach may be based on a Failure Mode and Effect Analysis
(FMEA), a Fault Tree Analysis (FTA) and a System-Theoretic Process Analysis
(STPA) or any similar process appropriate to system functional and operational
safety.
The documentation should demonstrate the validation/verification plans and results
including appropriate acceptance criteria. This should include validation testing
appropriate for validation, for example, Hardware in the Loop (HIL) testing, vehicle
on-road operational testing, testing with real end users, or any other testing
appropriate for validation/verification.
106. Results of validation and verification may be assessed by analysing coverage of the
different tests and setting coverage minimal thresholds for various metrics.

---

113. This documentation should also describe the measures in place to ensure the "The ADS" is
free from unreasonable risks for the driver (if applicable), vehicle occupants, and other road
users when the performance of "The ADS" is affected by environmental conditions e.g.
climatic, temperature, dust ingress, water ingress, ice packing.
(g)
Data Storage System
114. It is recommended that the documentation describe:
(a)
(b)
(c)
(d)
Storage location and crash survivability
Data recorded during vehicle operation and occurrences
Data security and protection against unauthorized access or use
Means and tools to carry out authorized access to data.
(h)
Cyber Security
115. The documentation should describe:
(a)
(b)
(c)
(d)
Cyber security and software update management,
Identification of risks, mitigation measures,
Secondary risks and assessment of residual risks,
Software update procedure and management put in place to comply with legislative
requirements.
(i)
Information Provisions to Users
116. It is recommended that the documentation describe:
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
System description and functional limitations,
Nominal operations,
Emergency operations,
Role of the user within the ADS's ODD,
Information related to the HMI's indications
Means to deactivate the automated driving mode (take-over),
Safety measures to be taken in the event of malfunctioning of the ADS,
Extent, timing and frequency of maintenance operations,
Means to enable a periodical technical inspection,
Documents and templates for maintenance, repair and periodical technical inspection,

---

125. Any changes to ADS safety design should be communicated as required to the relevant
authority.
X. IN-SERVICE MONITORING AND REPORTING – PILLAR 5
126. The In-Service Monitoring and Reporting pillar (ISMR) addresses the in-service safety of
automated vehicles after market introduction. In practice, the application of the other pillars
of the NATM guidelines will assess whether the ADS is reasonably safe for market
introduction; whereas the in-service monitoring and reporting will gather additional evidence
from the field operation to demonstrate that the ADS continues to be safe when operated on
the road. This pillar addresses the dynamic nature of road transportation to ensure attention
to and continuous improvement of road safety through the use of ADS.
127. The pillar consists in the collection of relevant data during AVs operation.
128. The obligation to have "real-time monitoring" (self-checks/on board diagnostics) of the
performance of ADS subsystems by the manufacturer is not part of this pillar but is part of
the safety requirements. However, some monitoring mechanisms on the performance of
ADS subsystems overtime could be part of the Objective 1 that has been described below in
"general guidance on ISMR implementation", and contribute to the predictive monitoring of
safety performance degradation.
129. The processes put in place by the manufacturer to manage safety during in use (e.g. to
manage changes in the traffic rules and in the infrastructure) fall outside this pillar and are
assessed with the audit pillar. This pillar focuses on the type of data to be monitored and
reported.
130. Whatever safety evaluation is done before market introduction, the actual level of safety will
only be confirmed once a sufficient number of vehicles are in the field and once they are
subjected to a sufficient range of traffic and environmental conditions. It is recommended
that a feedback loop (fleet monitoring) is in place to confirm the safety by design concept
and the validation carried out by the manufacturer before market introduction. The
operational experience feedback from in-use monitoring will allow ex-post evaluation of
regulatory requirements and validation methods, providing indications on gaps and needs
for review.
131. New scenarios and new risks might be introduced by AVs on the market. Therefore, the
In-Use Monitoring pillar can be used to identify new scenarios to support the development of
common scenario catalogue to cover these new safety risks.
132. Finally, in the early phase of market introduction of ADS, it is essential that the whole
community learns from crashes involving AVs in order to quickly respond to and develop
safety mitigation measures.

---

142. Unanticipated situations, risks and hazards might be identified during real-world ADS
operation, and this information could be used to develop new scenarios for the common
scenario catalogue.
143. In the early phase of market introduction of ADSs, it is essential that the whole community
learns from safety-critical situations involving AVs. It is important therefore that there is a
mechanism that allows information from the ISMR and recommendations from its analysis to
be shared with the ADS community. This will allow others to react and should lead to
developments that reduce or prevent that situation for other ADSs.
144. Collection, processing and dissemination of information related to ADS safety performance
from the ISMR will also facilitate the evaluation of the impact of ADS on the safety of the
road network.
Figure 2
ISMR Integration within the Multi-pillar Framework

---

152. The analysis techniques should comprise the following:
(a)
(b)
(c)
(d)
Routine measurements: a selection of parameters should be collected to characterise
each trip and to allow a comparative analysis. These measurements should aim at
identifying and monitoring emerging trends and tendencies before the trigger levels
associated with exceedances are reached. (e.g. vehicle performance monitoring)
Exceedance detection: a set of core events should be selected to cover the main
areas of interest for the ADS operation with aim at searching for deviations from
vehicle performance and limits. Typically, the main areas of interest are derived from
the assessment of the most significant risks before the market introduction. However,
they should be continuously reviewed to reflect the current operations. (e.g. speed
limits exceedance, near misses, harsh braking, etc.)
Occurrence analysis: recorded data should be able to characterize and investigate all
the occurrences listed in the Annex IV.
Statistics: Series of data should be collected to support the analysis process with
additional information. These data should provide information to generate rate and
trends. (e.g. driven km, operating hours).
153. The data monitoring programme should identify KPIs to assure that the monitoring is
performing at an optimal level, and address any issues which can affect the effectiveness of
the monitoring program (e.g. data corruption or loss, or result in delayed or degraded event
detection). Examples of KPIs for monitoring are trip collection rate – number of valid
trips/number of trips of ADS type or time between actual safety event/occurrence and first
detection by the in-service monitoring.
(a)
Vehicle Data Collection
154. There is regulatory work to introduce EDR and DSSAD requirements. Until those
requirements have been defined this section is only suggesting the data elements that may
be collected and uploaded by the manufacturer from ADS vehicles for aggregation and
processing in order to report performance metrics defined under the Reporting section.
(b)
Other Manufacturer-accessible Sources of Data Indicative of ADS Performance
155. Manufacturers may be expected to collect data relevant to typical operations such as dealer
reports, customer reports, etc.
3. In Service Reporting
156. The main purpose of occurrence reporting is the prevention of accidents and incidents and
not to attribute blame or liability.
(a)
149. Recommended Reporting by the Manufacturer
157. The manufacturer should report, as required by the Authority, on both critical and
non-critical occurrences. Two types of report on the in-service safety performance of the
ADS vehicle are expected.
(*)

---

168. The authority, where necessary, may verify the information provided and, if needed, may
make recommendations to the enforcement authority and/or to the ADS manufacturer to
remedy any detected conditions constituting an unreasonable risk to safety.
169. The authority may recommend temporary safety measures including immediately restricting
or suspending the relevant operations and actions to restore an acceptable level of safety, if
a serious safety risk is identified.
4. Collection and Storage of Information
170. It is recommended that a mandatory reporting system is established at national level by
means of a common national database and at international level by means of a Common
Central Repository.
171. Data quality and consistency should be ensured both at national and international level by
establishing checking processes.
(a)
National Level
172. In order to implement the ISMR framework, Contracting Parties are recommended to
designate one or more competent authorities to put in place a mechanism to collect,
evaluate, process and store occurrences reported in accordance with ISMR principles.
173. The safety authority/ies at national level should be in charge to collect, derive and share
safety recommendations and to exchange all safety-related information stored in the
national database with other competent authorities. The safety authority is also in charge of
issuing an annual report summarizing the level of ADS safety and providing a safety issues
assessment and the related action plan. The annual report should be submitted to WP29.
174. Short term and periodic reports should be stored within the common national database.
Safety recommendations should also be stored in the common national database and made
accessible to the relevant stakeholders.
175. Safety authorities should transfer safety recommendations and annual reports to the
Common Central Repository.
(b)
International Level
176. WP29 can provide the suitable international context for exchanges between Contracting
Parties and for defining the guiding principles on the ISMR framework implementation.
177. It is recommended that WP.29 establishes a proper management system of the Common
Central Repository, including accessibility and dissemination of information, data protection
where needed, data evaluation and release of an annual report. The technical protocols for
transferring all safety recommendations to the Common Central Repository should also be
established.
178. Clear guidance on the standardized approach to ISMR, including the harmonisation of the
data entry process, should be organized by WP.29 at international level by providing
guidelines, workshops and appropriate training.

---

191. Without prejudice to the applicable national law, it is recommended that Safety Authorities
refrain from instituting proceedings in respect of unpremeditated or inadvertent
infringements of the law which come to their attention only because they have been
reported under the ISMR occurrence-reporting scheme, except in cases of gross
negligence.
192. In accordance with the procedures defined in their national laws and practices, Safety
Authorities should ensure that employees who report incidents of which they may have
knowledge are not subjected to any prejudice by their employer.
8. Voluntary Reporting
193. Safety Authorities may put in place a system of voluntary reporting to collect and analyse
information on observed ADS behaviours which are not required to be reported under the
system of occurrences reporting set in the present Guidelines, but which are perceived by
the reporter as an actual or potential hazard.
XI.
NATM PILLARS/ELEMENT INTERACTION
162. The goal of the NATM guidelines document is to assess the safety of an ADS in a manner
that is as repeatable, objective and evidence based as possible, whilst remaining
technology neutral and flexible enough to foster ongoing innovation in the automotive
industry.
163. The overall purpose of the NATM is to assess, based on the safety requirements, whether
the ADS is able to cope with occurrences that may be encountered in the real world. In
particular, by looking at scenarios linked to road users' behaviour/environmental conditions
in Traffic scenarios as well as scenarios linked to driver behaviour (e.g. HMI) and ADS
failures.
164. As previously noted, the multi-pillar approach recognizes that the safety of an ADS cannot
be reliably assessed/validated using only one of the pillars. Each of the aforementioned
testing methodologies possesses its own strengths and limitations, such as differing levels
of environmental control, environmental fidelity, and scalability, which should be considered
accordingly.
165. It is important to note that a single assessment or test method may not be enough to assess
whether the ADS is able to cope with all occurrences that may be encountered in the real
world.
166. For instance, while real-world testing provides a high degree of environmental fidelity, a
scenario-based testing methodology using only real-world testing could be costly,
time-consuming, difficult to replicate, and pose safety risks. Consequently, track testing may
be more appropriate methods to run higher risk scenarios without exposing other road users
to potential harm. Further, test scenarios can also be more easily replicated in a closed
track environment compared to the real-world. That said, test track scenarios can be
potentially difficult to develop and implement, especially if there are numerous or complex
scenarios, involving a variety of scenario elements.
167. Consideration should be given to the fact that simulation/virtual testing, by contrast, can be
more scalable, cost-effective, safe, and efficient compared to track or real-world testing,
allowing a test administrator to safely and easily create a wide range of scenarios, including
complex scenarios, where a diverse range of elements are examined. However, simulations
may have lower fidelity than the other methodologies. Simulation software may also vary in
quality and tests could be difficult to replicate across different simulation platforms.

---

172. As previously noted, the NATM pillars not only include the three aforementioned test
methods but also an aggregated analysis (e.g., an audit/assessment/in service
monitoring/reporting pillar). Whereas the test methods will assess the safety of the ADS, the
audit/assessment pillar will serve to assess the safety of the ADS as well as the robustness
of organizational processes/strategies. Elements of the audit are:
(a)
(b)
(c)
(d)
(e)
Assessment of the robustness of safety management system.
Assessment of the (identified) hazards and risks for the system.
Assessment of the Verification strategy (e.g. verification plan and matrix) that
describe the validation strategy and the integrated use of the pillars to achieve the
adequate coverage.
Assessment of the level of compliance with requirements achieved through an
integrated use of all pillars, including consistency between the outcomes of one pillar
as input for another pillar (forward and backward) and adequate use of scenarios.
This level of compliance concerns both new vehicles as vehicles in use.
The audit/assessment phase also incorporate results from the Simulation, Track test
and Real-World tests carried out by the manufacturer.
173. Figure 3 provides a diagram that outlines how the pillars, scenarios, and safety
requirements (developed by FRAV) will interact. Further examination of each of these
elements follows in the subsequent sections of this Document.
Figure 3
Relationship between VMAD Pillars, Scenarios and FRAV Safety Requirements

---

ANNEX I
GLOSSARY OF TERMS AND DEFINITIONS
"Abstraction" is the process of selecting the essential aspects of a source system or referent system to
be represented in a model or simulation, while ignoring those aspects not relevant. Any modelling
abstraction carries with it the assumption that it should not significantly affect the intended uses of the
simulation tool.
"Automated Driving System (ADS)" means the vehicle hardware and software that are collectively
capable of performing the entire Dynamic Driving Task (DDT) on a sustained basis.
"ADS feature" means an application of an ADS designed specifically for use within an Operation Design
Domain (ODD).
"ADS function" means an application of ADS hardware and software designed to perform a specific
portion of the DDT.
"Closed Loop Testing" means a virtual environment that does take the actions of the element-in-the
loop into account. Simulated objects respond to the actions of the system (e.g. system interacting with a
traffic model).
"Concrete Scenarios": Concrete scenarios are established by selecting specific values for each
element. This step ensures that a specific test scenario is reproducible. In addition, for each logical
scenario with continuous ranges, any number of concrete scenarios can be developed, helping to ensure
a vehicle is exposed to a wide variety of situations.
"Complex Scenarios" means a traffic scenario containing one or more situations that involve a large
number of other road users, unlikely road infrastructure, or abnormal geographic/environmental
conditions.
"Critical Scenarios" means a traffic scenario containing a situation in which the ADS needs to perform
an emergency maneuver in order to avoid/mitigate a potential collision, or react to a system failure.
"Deterministic" is a term describing a system whose time evolution can be predicted exactly and a
given set of input stimuli will always produce the same output.
"Driver-In-the-Loop" (DIL) is typically conducted in a driving simulator used for testing the
human–automation interaction design. DIL has components for the driver to operate and communicate
with the virtual environment.
"Dynamic Driving Task (DDT)" means all of the real-time operational and tactical ADS functions
required to operate the ADS-equipped vehicle in on-road traffic.
● The DDT excludes strategic functions such as trip scheduling and selection of destinations and
waypoints.
● The DDT functions can be logically grouped under three main categories:
(a)
(b)
(c)
Sensing and Perception
Planning and Decision
Vehicle Control

---

"Critical Occurrence" means an occurrence in which the ADS is engaged at the time of the event and:
(a)
(b)
(c)
at least one person suffers an injury that requires medical attention as a result of being in the
vehicle or being involved in the event;
the ADS vehicle, other vehicles or stationary objects sustain physical damage that exceeds a
certain threshold;
any vehicle involved in the event experiences an airbag deployment
"Operational Design Domain (ODD)" means the operating conditions under which an ADS feature is
specifically designed to function.
"ODD exit" means:
(a)
(b)
the presence of one or more ODD conditions outside the limits defined for use of the ADS feature,
and/or
the absence of one or more conditions required to fulfil the ODD conditions of the ADS feature.
"Open Loop Testing" means a virtual environment that does not take the actions of the element-in-the
loop into account (e.g. system interacting with a recorded traffic situation).
"Probabilistic" is a term pertaining to non-deterministic events, the outcomes of which are described by
a measure of likelihood.
"Proving Ground or test-track" is a physical testing facility closed to the traffic where the performance
of an ADS can be investigated on the real vehicle. Traffic agents can be introduced via sensor
stimulation or via dummy devices positioned on the track.
"Sensor Stimulation" is a technique whereby artificially generated signals are provided to the element
under testing in order to trigger it to produce the result required for verification of the real world, training,
maintenance, or for research and development.
"Simulation" is the imitation of the operation of a real world process or system over time.
"Simulation toolchain" is a combination of simulation tools that are used to support the validation of an
ADS.
"Software-In-the-Loop" (SIL) is where the implementation of the developed model will be evaluated on
general-purpose computing systems. This step can use a complete software implementation very close
to the final one. SIL testing is used to describe a test methodology, where executable code such as
algorithms (or even an entire controller strategy), is tested within a modelling environment that can help
prove or test the software.
"Stochastic" means a process involving or containing a random variable or variables. Pertaining to
chance or probability.
"Test case specification" are the detailed specifications of what must be done by the tester to prepare
for the test.
"Test methods" is a structured approach to consistently derive knowledge about the ADS by means for
executing tests, e.g. virtual testing in simulated environments, physical, structured testing in controlled
test facility environments, and real world on-road conditions.

---

ANNEX II
FUNCTIONAL SCENARIOS FOR DIVIDED HIGHWAY APPLICATION
I. Introduction
Contents
II.
III.
IV.
Inputs to this Proposal
Building blocks of functional scenarios
Coverage
V. Symbols used in this document
VI.
A list of possible scenarios for L3 Highway Chauffeur ADS
A.
Nominal driving (Perform lane keeping)
1.
Nominal driving (Perform lane keeping)
B.
Interaction with other vehicles/objects
1.
Perform lane change
2.
Critical (Emergency) braking scenarios during lane keeping
C.
Detect and response to traffic rules and road furniture
D.
Country specific road geometry
E.
Unusual situation
VII.
References

---

IV.
COVERAGE
Collisions always occur with other vehicles/objects (assuming that they can operate properly when
there are no other vehicles/objects). The 24 functional scenarios in the figure described in
Section "2. Interaction with other vehicles" under Nominal Driving can cover all interactions
between other vehicles/objects and ego vehicle. These scenarios can cover collision with other
vehicles/objects appropriately.
As described above., factors not covered in the proposed functional scenarios (e.g. initial speed of
ego vehicle, size, initial position, initial speed, acceleration of other vehicles/objects), perception
factor (e.g. weather, brightness, blind spot, false positive factor, blinkers of other vehicles) and
vehicle stability factors (e.g. curve, slope, road surface μ, wind, etc.) can be described with
parameters in logical scenarios.
As previously mentioned, it is anticipated that future iterations of Annex II will also incorporate
scenarios with lower levels of abstraction (e.g. logical scenarios and potential approaches for
describing them). Functional scenarios should be added when agreement is reached between
SG1 and VMAD-IWG.
V. SYMBOLS USED IN THIS DOCUMENT
ICON
DESCRIPTION
Ego vehicle
Lead vehicle
Other vehicles part of the scenario
Impassable object on intended path
Passable object on intended path

---

Scenario family
Sub-scenario
Japan crash
scenarios
The
Netherlands
(TNO)
SAFE
scenario
library
China
functional
scenarios
Conflict Type
a. Speed limit
sign
X
X
3. Detect and response to
traffic rules and road
furniture
b. Signal
lights
c. Drive
through tunnel
d. Toll X
X
X
X
e.
Conventional
obstacles
X
X
4. Country specific road
geometry
a. Intercepter X
5. Unusual situation
a. Wrong way
driver
(oncoming)
X
X
Notes to the inputs from VMAD SG1 members:
● China (CATARC): This is a list cut from a general catalogue describing different ODDs, like
"General road", "City expressway" or "The highway" and their test items, like "speed limit
sign", "lane line", "toll station", etc. The functional scenarios proposed below in this
document are much more generic than the ones proposed by China, so they form a subset
of this list. For example, China proposal: "toll station" on the road or "conventional
obstacles" can be in line with "impassable object on intended path" from this scenario list.
● The Netherlands (TNO): a very thorough scenario catalogue containing much more
scenarios than needed for the highway use case. Terminology and descriptions worked out
fully. Scenarios can be created using a combination of tags from the different layers.
● Japan: crash scenarios, scenarios only containing interaction with other vehicles. They
describe different road geometries and possible other vehicle positions around ego. All other
parameters considered as features (acceleration - deceleration, lane change - lane keeping,
etc.).
● SAFE: a list of scenarios sometimes with very concrete examples, sometimes more generic
approach. There is a different scenario for passing by slowly moving vehicles in the adjacent
lane and a different one for passing by standing vehicles, but handles LV following as one
scenario.
● Conflict Type: a list of "conflict types" used i.a. by accident investigators to sort scenarios,
leading to accidents on road to different groups. These conflict types can be sorted into
conflicts with or without influence of other road user. Uses different symbols than other
documents for the description of a scenario or situation (mainly different kinds of arrows).
Separates left and right hand traffic. Contains 251 scenario types, structured in seven larger
types of conflicts, like: "longitudinal traffic" or "pedestrian crossing the road".
Note: "emphasized scenario parameters" and "tested parameters" in this paragraph are some
examples of parameters. Other parameters may be essential for the validation testing.

---

(b)
Manoeuvring a Bend (Right Curve and Left Curve)
(a)
(b)
(c)
Without LV
With LV
With other vehicles in adjacent lanes (moving or stopped)
General description:
Figure 2
Schematic Representation of Manoeuvring a Bend
The ego vehicle is driving on a curved road. The aim of this scenario is to test if the vehicle is able
to handle the road curvatures specified as part of the ODD [1,2,4].
Emphasized scenario parameters: ego speed demand (road rules), lane width, LV speed profile (if
present), layout and speed profile of other vehicles (if present).
Tested parameters: deviation from lane centre (nominal value and distribution), deviation from
desired speed, obeying to speed changes, temporal modifications, distance between ego and LV
(if present), distance to other vehicles etc.

---

1. Perform Lane Change
Note: LC scenarios are complicated by the fact that the ADS cannot be forced to make a lane
change. In addition, lane change functionality and principles shall be defined in a later stage
(like technical requirements, definitions, activation criteria, indication of lane change, etc.).
Lane changes can be grouped based on the number of vehicles in the target lane. If there is
enough space to execute the lane change, there is no need to cooperate with other vehicles. If the
target lane is occupied by other traffic participants, than the ego vehicle has to adapt to the other
participants and perform merging.
(a)
Ego Vehicle Performing Lane Change with Vehicle Behind
Figure 3
Schematic Representation of a Lane Change

---

2. Critical (Emergency) Braking Scenarios during Lane Keeping
Note: In this family of scenarios a couple critical functional scenarios are present. It can be noticed
in the input matrix of SG1 as well, these are scenarios that nearly every participant
highlighted in the input documents.
(a)
Impassable Object on Intended Path (Including Other Cars and VRUs)
General description:
Figure 5
Schematic Representation of an Impassable Object
The ego vehicle is driving on a road with an impassable object in the ego lane. The objective of the
ego vehicle is to continue driving straight. The ego vehicle needs to react [1,2]. Depending on the
velocity of the ego vehicle, the severity of the scenario is changing.
Emphasized scenario parameters: road layout (visibility of the object on the path), layout and
speed profile of other vehicles (if present), ego velocity.
Tested parameters: reaction of ego (lane change/braking), distance to object, lateral velocity of
ego (if changing lane) etc.

---

(c)
Lead Vehicle Braking
General description:
Figure 7
Schematic Representation of Lead Vehicle Braking
The ego vehicle is following a LV. The LV brakes, the ego vehicle has to adapt its speed in order
to stay at a safe distance from the lead vehicle [1-4].
Emphasized scenario parameters: ego velocity (road rules), LV speed profile (deceleration), layout
and speed profile of other vehicles (if present).
Tested parameters: distance between ego and LV, reaction to other vehicles in adjacent lanes,
etc.

---

(e)
Cut-in in Front of the Ego Vehicle
Figure 9
Schematic Representation of Cut-in
General description:
Another vehicle is driving in the same direction as the ego vehicle in an adjacent lane. The other
vehicle makes a lane change, such that is becomes the LV from the ego vehicle's perspective
[1-4]. Depending on the distance and lateral velocity of the LV, the severity of the cut-in
manoeuvre changes.
Emphasized scenario parameters: LV lateral speed, distance to LV, ego velocity, lane width,
layout and speed profile of other vehicles (if present).
Tested parameters: distance between ego and LV, distance to other vehicles, etc.

---

(g)
Detect and Respond to Swerving Vehicles
General description:
Figure 11
Schematic Representation of a Swerving Vehicle
Another vehicle is driving in the same direction as the ego vehicle in an adjacent lane. The other
vehicle swerves towards the ego vehicle's lane [1-3].
Emphasized scenario parameters: lateral speed of other vehicle, ego velocity, lane width, layout
and speed profile of other vehicles (if present).
Tested parameters: distance between ego and swerving vehicle, distance to other vehicles, etc.

---

(b)
Signal Lights
The test road consists of at least two lanes. The signal lights are set above the road, and the
signal lights of adjacent lanes are kept in green state.
Figure 13
Testing Scenario Diagram for Expressway Signal Lights

---

(e)
Conventional Obstacles
The test road is a long straight road containing at least two lanes, and the middle lane line is a
white dashed line. Within the lanes, conical traffic signs and traffic markings are placed according
to the traffic control requirements of the road maintenance operation. This is shown in Figure 16.
D. Country Specific Road Geometry
Figure 16
Diagram of a Conventional Obstacle Course.
Note: This scenario is only applicable for limited countries or regions. Therefore, application of this
scenario can be unnecessary depends on the target market of the ADS.
(a)
Interceptor
For the ego vehicle, junctions present a challenge due to the increased likelihood of conflicts with
other actors.
In this scenario, the ego vehicle traverses an intersection simultaneously with another car - the
interceptor. This scenario tests the ego vehicle's behaviour when on a collision course with
another car in an intersection, possibly with signs, signals, or traffic lights. The ego vehicle should
be able to safely manoeuvre through the intersection and avoid or mitigate a collision.
Environmental requirements: A junction with at least three ways. It may or may not be controlled
(i.e. have yield sign, traffic lights, etc.).
Ego vehicle behaviour: The ego vehicle traverses the junction in any direction (left, right or
straight).
Other actors' behaviour: Another car approaches the same junction, from a different direction and
traverses the junction such that its trajectory intersects with the ego vehicle's trajectory.

---

VII.
REFERENCES
[1] ALKS Regulation UN R157. Available online at
https://unece.org/transport/documents/2021/03/standards/un-regulation-no-157-automated-lanekeeping-systems-alks
[2] E. de Gelder, O. Op den Camp, N. de Boer, (The Netherlands): Scenario Categories for the
Assessment of Automated Vehicles, Version 1.7, January 21, 2020.
[3] SAFE (Foretellix) Highway and ADAS Traffic Scenario Library, Scenario Definitions at the
functional Level, Version 1.0, November 2020.
[4] Japan: Proposal of Traffic Scenarios for Highway Driving (Supplemental version for presentation),
December 2020.
[5] China (CATARC): Proposal about functional scenario from catarc, December 2020.
[6] EC-JRC. Speed profile for car-following tests. Available online at
https://wiki.unece.org/download/attachments/92013066/ACSF-25-
13%20%28EC%29%2020190121_TestSpecification_ALKS_JRC.pdf?api=v2
[7] IGLAD 2019 Codebook, Conflict Types, 2019.

---

II.
COMPONENTS OF THE CREDIBILITY ASSESSMENT FRAMEWORK
6. It is recommended that M&S be used for virtual testing if their credibility is established by
evaluating the its fitness for the intended purpose. It is recommended that credibility is
achieved by investigating and assessing five M&S properties:
(a)
(b)
(c)
(d)
(e)
Capability – what the M&S can do, and what are the risks associated;
Accuracy – how well M&S does reproduce the target data;
Correctness – how sound & robust are M&S data and algorithms;
Usability – what training and experience is needed and what quality of the process
applied to it.
Fit for Purpose – how suitable the M&S is for the ODD and ADS assessment.
Figure 1
Graphical Representation of the Relationships Between the Components of
the Credibility Assessment Framework

---

3. Team's Experience and Expertise.
12. Even though Experience and Expertise (E&E) are already covered in a general sense within
an organization, it is important to establish the basis for confidence on the specific
experience and expertise for M&S activities.
13. In fact, the credibility of M&S depends not only on the quality of the simulation models but
also on the E&E of the personnel involved in the validation and usage of the M&S. For
instance, a proper understanding of the limitations and validation domain will prevent from
possible misuse of M&S or from misinterpretation of its results.
14. In this perspective, important to establish the basis for the ADS manufacturer's confidence
on the experience and expertise of:
(a)
(b)
The teams that will validate the simulation toolchain and,
The teams that will use the validated simulation for the execution of virtual testing with
the purpose of validating the ADS.
15. Thus, if a team's E&E is good it increases the level of confidence and hence the credibility
of M&S and its outcomes by ensuring that the human factors behind the M&S are taken into
consideration and any possible human component risk is controlled, as expected, through
its Management System.
16. If the ADS manufacturer's toolchain incorporates or relies upon inputs from organizations or
products outside of the manufacturer's own team, it is recommended that the ADS
manufacturer will include an explanation of measures it has taken to manage and develop
confidence in the quality and integrity of those inputs.
17. The team's Experience and Expertise include two aspects:
(a)
Organizational Level:
18. The credibility is established by setting up processes and procedures to identify and
maintain the skills, knowledge, and experience to perform M&S activities. The following
processes should be established, maintained and documented:
(a)
(b)
Process to identify and evaluate the individual's competence and skills;
Process for training competent personnel to perform M&S-related duties
(b)
Team Level:
19. Once a toolchain has been finalized, its credibility is mainly dictated by the skills and
knowledge of the individual/team that will validate the M&S and will use it for the validation
of ADS. The credibility is established by documenting that these teams have received
adequate training to fulfil their duties.

---

5. Data/Output Pedigree
24. The data/output pedigree contains a record of the M&S outputs used for the ADS validation.
(a)
Description of the Data Generated by the M&S
(a)
(b)
(c)
The ADS manufacturer should provide information on any data and scenarios used for
virtual testing toolchain validation.
The ADS manufacturer should document the exported data and note important quality
characteristics e.g. using the correlation methodologies as defined Annex II.
The ADS manufacturer should trace M&S outputs to the corresponding simulation setup:
(i)
Effect of the Data Quality M&S Credibility
(a)
(b)
The M&S output data should be sufficient to ensure the correct execution of the
validation computation. The data should sufficiently reflect the ODD relevant to
the virtual assessment of the ADS.
The output data should allow consistency/sanity check of the virtual models via
possibly exploiting redundant information
(ii)
Managing Stochastic Models
(a)
(b)
Stochastic models should be characterized in terms of their variance
The use of a stochastic models should not prohibit the possibility of
deterministic re-execution
B. M&S Analysis and Description
25. The M&S analysis and description aim to define the whole toolchain and identify the
parameter space that can be assessed via virtual testing. It defines the scope and
limitations of the models and tools and the uncertainty sources that can affect its results.
1. General Description
(a)
(b)
ADS manufacturer should provide a description of the complete toolchain along with how
the simulation data will be used to support the ADS validation strategy.
The ADS manufacturer should provide a clear description of the test objective.

---

4. Criticality Assessment
26. The simulation models and the simulation tools used in the overall toolchain should be
investigated in terms of their impact in case of a safety error in the final product. The
proposed approach for criticality analysis is derived from ISO 26262, which requires
qualification for some of the tools used in the development process. In order to derive how
critical the simulated data is, the criticality assessment considers the following parameters:
(a) The consequences on human safety e.g. severity classes in ISO 26262.
(b)
The degree in which the simulated results influences the ADS.
27. The table below provides an example criticality assessment matrix to demonstrate this
analysis. ADS manufacturers may adjust this matrix to their particular use case.
Table 1
Criticality Assessment Matrix*
Influence
on ADS
Significant
N/A
Perform degraded
mode within reduced
system constraints
Moderate
Minor
Negligible
Strategic control
of the ADS by the
User
User interaction
with HMI
Determine its
location
Communicate and
interact with other
road users
User informed about
operational status
Create a collision
free and lawful
driving plan
Predict the future
behaviour of
other actors
Safe
management of
transitions of
control
N/A
Correctly execute
and actuate the
driving plan
Perceive relevant
static and
dynamic objects
in the proximity of
the ADS
Determine if
specified nominal
performance is
not achieved
Negligible Minor Moderate Significant
Decision consequence
28. From the perspective of the criticality assessment, the three possible cases for assessment
are:
(a)
(b)
(c)
Those models or tools that are clear candidates for following a full credibility
assessment;
Those models or tools that may or may not be candidates for following the full
credibility assessment at the discretion of the assessor;
Those models or tools that are not required to follow the credibility assessment.

---

3. Sensitivity Analysis
32. Sensitivity analysis aims at quantifying how model output values are affected by changes in
the model input values and thus identifying the parameters having the greatest impact on
the simulation model results. The sensitivity study also provides the opportunity to
determine the extent to which the simulation model satisfies the validation thresholds when
it is subjected to small variations of the parameters, thus it plays a fundamental role to
support the credibility of the simulation results.
(a)
(b)
(c)
The ADS manufacturer should provide supporting documentation demonstrating that
the most critical parameters influencing the simulation output have been identified by
means of sensitivity analysis techniques such as by perturbing the model's
parameters;
The ADS manufacturer should demonstrate that robust calibration procedures have
been adopted and that this has identified and calibrated the most critical parameters
leading to an increase in the credibility of the developed toolchain.
Ultimately, the sensitivity analysis results will also help to define the inputs and
parameters whose uncertainty characterization needs particular attention to
characterize the uncertainty of the simulation results.
4. Validation
33. The quantitative process of determining the degree to which a model or a simulation is an
accurate representation of the real world from the perspective of the intended uses of the
M&S. Examples of virtual toolchain validation are reported in Annex 3 - Appendix 3. It is
recommended that the following items be considered when assessing the validity of a model
or simulation:
(a)
Measures of Performance (Metrics)
(a)
(b)
The Measures of Performance are metrics that are used to compare the outputs of the
virtual testing tool with real world performances. The Measures of Performance are defined
during the M&S analysis.
Metrics for validation may include:
(i)
(ii)
(iii)
Discrete value analysis e.g. detection rate, firing rate;
Time evolution e.g. positions, speeds, acceleration;
Analysis of state changes e.g. distance/speed calculations, TTC calculation, brake
initiation.
(b)
Goodness of Fit Measures
(a)
(b)
The analytical frameworks used to compare real world and simulation metrics are generally
derived as Key Performance Indicators (KPIs) indicating the statistical comparability
between two sets of data.
The validation should show that these KPIs are met.

---

(h)
Uncertainty Characterisation
37. This Section is concerned with characterizing the expected variability of the virtual toolchain
results. The assessment should be made up of two phases. In a first phase the information
collected from the "M&S Analysis and Description" section and the "Data/Input Pedigree"
are used to characterise the uncertainty in the input data, in the model parameters and in
the modelling structure. Then, by propagating all of the uncertainties through the virtual
toolchain, the uncertainty of the model results is quantified. Depending on the uncertainty of
the model results, proper safety margins will need to be introduced by the ADS
manufacturer in the use of virtual testing as part of the ADS validation.
(i)
Characterization of the Uncertainty in the Input Data
38. The ADS manufacturer should demonstrate they have estimated the model's critical inputs
by means of robust techniques such as providing multiple repetitions for the assessment of
the quantity;
(ii)
Characterization of the Uncertainty in the Model Parameters (Following Calibration).
39. The ADS manufacturer should demonstrate that when a model's critical parameters cannot
be fully determined they are characterized by means of a distribution and/or confidence
intervals;
(iii)
Characterization of the Uncertainty in the M&S Structure
40. The ADS manufacturer should provide evidence that the modelling assumptions are given a
quantitative characterization by assessing the generated uncertainty (e.g. comparing the
output of different modelling approaches whenever possible).);
(iv)
Characterization of Aleatory vs. Epistemic Uncertainty
41. The ADS manufacturer should aim to distinguish between the aleatory component of the
uncertainty (which can only be estimated but not reduced) and the epistemic uncertainty
deriving from the lack of knowledge in the virtualization of the process.
III.
DOCUMENTATION STRUCTURE
42. This Section will define how the aforementioned information will be collected and organized
in the documentation provided by the ADS manufacturer to the relevant authority.
(a)
(b)
(c)
(d)
The ADS manufacturer should produce a document (a "simulation handbook")
structured using this outline to provide evidence for the topics presented;
The documentation should be delivered together with the corresponding release of
the toolchain and appropriate supporting data;
The ADS manufacturer should provide clear reference that allows tracing the
documentation to the corresponding parts of the toolchain and the data;
The documentation should be maintained throughout the whole lifecycle of the
toolchain utilization. The assessor may audit the ADS manufacturer through
assessment of their documentation and/or by conducting physical tests.

---

IV.
OCCURRENCES RELATED TO THE IDENTIFICATION OF NEW SAFETY-RELEVANT
SCENARIOS
Table 1
Occurrences Related to the Identification of New Safety-relevant Scenarios
Occurrence
1.a. Safety critical occurrences known to the ADS manufacturer or
OEM
Short-term
reporting
[1 Month]
X
(in case of
unreasonable
risk)
1.b. Occurrences related to ADS operation outside its ODD X X
1.c. ADS failure to achieve a minimal risk condition when necessary X X
1.d. Communication-related occurrences
1.e. Cybersecurity-related occurrences
1.f. Interaction with remote operator if applicable
2.a. Driver unavailability (where applicable) and other user-related
occurrences
2.b. Occurrences related to Transfer of Control failure
2.c. Prevention of takeover under unsafe conditions
3.a. Occurrences related ADS failure
3.b. Maintenance and repair problems
3.c. Occurrences related to unauthorized modifications
3.d. Modifications made by the ADS manufacturer or OEM to
address an identified and significant ADS safety issue
4. Occurrences related to the identification of new safety-relevant
scenarios
X
X
X
X
X
X
X
X
X
X
X
X
X
Periodic
Reporting
[1 Year]

---

A. The General Matrix for Physical Testing
8. The general matrix would provide a clear overview of the type or types of physical testing to
be used for assessing compliance with the applicable safety requirements set by FRAV.
9. The example in Table VII.1 illustrates the basic concept for the overview based on (a
selection of) the initial 40 safety topics drafted by FRAV. Please note that the example is
merely illustrative and should not be regarded as VMAD's position on the applicability of
each test method per safety topic.
10. Moreover, the example does not take into account any further development on the safety
topics by FRAV since the 18 VMAD meeting. The safety topics are furthermore expected
to be set out in more detail in the future, each topic containing one of more measurable
requirements.
11. It is envisaged that, once developed, these measurable requirements would be listed in the
left column of the table instead of the currently listed safety topics.
Table 1
Example of the General Matrix for Physical Testing
(FRAV) Safety Requirement
Track Testing
1. The ADS should perform the entire Dynamic Driving Task. Yes Yes
2. The ADS should control the longitudinal and lateral motion of the
vehicle.
(…)
Yes
Real World
Testing
7. The ADS should adapt its behaviour in line with safety risks. Yes If encountered
8. The ADS should adapt its behaviour to the surrounding traffic
conditions.
(…)
30. The ADS should safely manage short-duration ODD exits. Yes Yes
31. Pursuant to a collision, the ADS should stop the vehicle and
deactivate.
(…)
Yes
Yes
Yes
If encountered
12. 'If encountered', as used in the table above, would indicate that real world testing would not
seek to assess the respective safety requirement, generally due to the undesirability from a
safety perspective to assess such requirements on public roads. However, given that
random traffic situations are encountered during real world testing, such traffic situations
could organically occur and in this case, the performance with regards to the specific
requirement should be assessed. The testing safety on public roads should also be taken
into account, which the assessor or the driver should ensure and they should therefore take
over the driving task if needed.

---

Table 3
Example of a Test Matrix for Track Test

---

27. For both cases, VMAD's SG4 will further discuss what the most efficient and clear way
would be to signal such 'Assess if encountered' requirements. Suggestions made so far
include:
(a)
Handling such occurrences completely separate from the test matrix (e.g. only
provide guidance/instructions in the testing protocols);
(b) Including them in the test matrix itself, but signalling their conditionality (e.g. Row 2.2
in the example table. Note in particular the assessment specification for lane
changes, where both a required assessment and a conditional 'if encountered'
assessment are included);
(c)
Signalling the existing of 'If encountered' assessments (e.g. using '*'), but setting out
the conditional assessment specification as well as guidance/instructions in the
testing protocols. (Please note that this latter option has not been illustrated in the
example table).
28. Aspects related to routing (e.g. minimum duration, minimum frequency of a given traffic
situation encountered during testing, etc.) would be set out in the accompanying test
protocols.

---

ANNEX VI
AREAS FOR FUTURE WORK
SCENARIOS CATALOGUE — UPDATING THE CATALOGUE
1. The text below discusses future work that may be conducted to develop and maintain a
VMAD scenarios catalogue:
2. If scenarios not covered by the scenario catalogue are identified and deemed necessary,
they should be included in the scenario catalogue.
3. It is envisaged that a scenario catalogue will have tags for all scenarios corresponding to
their relevant ODD attributes (using a standardised ODD taxonomy) and behaviour
competencies.
4. Country specific scenarios should be respected and need to be covered in the scenario
catalogue in the long term.
5. The scenario catalogue is not necessarily exhaustive and authorities may need to consider
additional scenarios as necessary to support safety validation of an ADS feature. [Such a
decision could be based on the ODD and behaviour competencies of the ADS. For
example, if an AV is developed with an ODD which is not covered in the scenario catalogue,
it is essential to add new scenarios to the catalogue to ensure the scenarios used for testing
are a function of the ODD.]
PILLARS 2 AND 3 — TRACK AND REAL WORLD TESTING — CONSIDERATIONS AND NEXT
STEPS
6. The next step in the development of the test methods for track and real world testing
(described in Annex V) is populating the test matrices with safety requirements, traffic
scenarios/traffic situations and the assessment specifications. However, it will merely be the
first of several steps before the test matrix approach as such could be used as an
assessment method.
7. This Section therefore outlines the next steps that are required in order to operationalize the
test matrixes approach, together with some initial considerations.
A. Populating the Test Matrix
8. In order to be able to advance with the development process of the test matrixes testing
method, it is first necessary to populate the test matrixes with requirements, scenarios and
assessment specifications. This is because most, if not all of the subsequent steps depend
largely on the content of the matrix itself. For example, without knowing what will be
required to be tested and against which criteria, it would difficult, if not impossible, to
determine the length and scope of the real world testing aspect.
9. The test matrix would be populated with the requirements and assessment specifications to
be developed by FRAV, and for track testing the scenarios developed by VMAD's SG1 as
well. Given that FRAV and VMAD's SG1 are currently still in the process of developing
respectively the requirements and traffic scenarios, SG4's work on the matrixes themselves
will be largely on hold until the requirements and scenarios become available.

---

PILLAR 5 — IN-SERVICE MONITORING AND REPORTING — INFORMATION SHARING AMONG
SAFETY AUTHORITIES/CONTRACTING PARTIES
16. The final aim of the ISMR pillar is to improve ADS safety through dissemination of lessons
learned in the form of safety recommendations. If this information is shared promptly and
widely then it should have an impact at both a national and international level. Safety
authorities should have access to such recommendations as well as the reporting from
manufacturers, plus other relevant information (e.g., data from highway authorities, crash
investigations, research, national statistics, etc.). This should allow them to react to any
safety issues associated with ADS deployment.
17. A mechanism to share information across safety authorities at a global level is desirable and
could be coordinated by GRVA/VMAD, under WP.29 directions.