Automated Driving (NATM)

Name:Automated Driving (NATM)
Description:New Assessment/Test Method for Automated Driving (NATM) - Master Document.
Official Title:New Assessment/Test Method for Automated Driving (NATM) - Master Document.
Country:ECE - United Nations
Date of Issue:2021-04-09
Amendment Level:Original
Number of Pages:51
Vehicle Types:Agricultural Tractor, Bus, Car, Component, Heavy Truck, Light Truck
Subject Categories:Automated Driving Systems
Available on InterRegs.NET

Our online subscription service, offering immediate access to our extensive library of global vehicle regulations, standards and legislation. A fully searchable, accurate, user-friendly resource for consolidated regulations that are updated quickly and frequently.

Tell me more | Already a subscriber

Available on SelectRegs.com

Our fast and easy means of purchasing up-to-date global vehicle and component standards and regulations on a pay-as-you-go basis. Pay securely by credit card and your documents are delivered directly and immediately to your computer as PDF files.

Tell me more | Go straight to site

Keywords:

testing, scenarios, ads, safety, vehicle, scenario, virtual, test, lane, ego, road, vehicles, traffic, validation, data, simulation, track, speed, real-world, driving, system, requirements, performance, pillar, real, natm, parameters, tests, functional, assessment, document, conditions, elements, figure, assess, world, model, change, pillars, audit, methods, environment, present, environmental, automated, monitoring, vmad, development, developed, process

Text Extract:

All InterRegs documents are formatted as PDF files and contain the full text, tables, diagrams and illustrations of the original as issued by the national government authority. We do not re-word, summarise, cut or interpret the regulatory documents. They are consolidated, published in English, and updated on a regular basis. The following text extract indicates the scope of the document, but does not represent the actual PDF content.

ECE/TRANS/WP.29/2021/61
NEW ASSESSMENT/TEST METHOD FOR AUTOMATED DRIVING
(NATM) - MASTER DOCUMENT
Prepared by the Working Party on
Automated/Autonomous and Connected Vehicles (GRVA)
ADOPTED BY
THE WORLD FORUM FOR HARMONIZATION OF VEHICLE REGULATIONS WP.29
AT THE 184 SESSION IN JUNE 2021, RECOMMENDING THAT IT IS CONSIDERED BY GRs AND
IWGs AS A REFERENCE DOCUMENT WHEN DEVELOPING ACTIVITIES IN THE FIELD OF
AUTOMATION.

(b)
(c)
Take account of the developments of other subsidiary Working Parties (GRs) of
WP.29 and their IWGs and work in full cooperation with them; and,
Consider existing data, research and technical standards (e.g. SAE International,
ISO) available during the development of its action items.
II.
PURPOSE AND SCOPE
5. In order for the international community to maximize the potential safety benefits of ADS, a
safety validation framework that can be adopted by Contracting Parties of both the 1958 and
the 1998 UN vehicle regulations agreements must be established. The NATM developed by
VMAD aims to provide clear direction for validating the safety of an ADS in a manner that is
repeatable, objective and evidence-based, while remaining technology neutral and flexible
enough to foster ongoing innovation by the automotive industry.
6. This document consolidates the work accomplished by VMAD to date to develop the NATM.
It provides a clear overview of the NATM and its constituent pillars. This document also
serves to promote coordination between VMAD and the work of the GRVA Informal Working
Group on Functional Requirements for Automated Vehicles (FRAV). This coordination will
ensure that the NATM also addresses the validation of compliance of an ADS to common
safety requirements to be developed by FRAV.
7. Given the substantial technical work that is still needed to operationalize the NATM in
practice, this version of the Master Document provides a high-level framework for the
NATM, outlining:
(a)
(b)
Scope and general overviews of the scenario catalogue and each of the pillars
(simulation/virtual testing, test track, and real-world testing, audit/assessment and
in-use monitoring); and,
Overall process of the NATM (e.g., how the components of the NATM (i.e., the
scenarios catalogue and pillars) operate together, producing an efficient,
comprehensive, and cohesive process).
8. Going forward, this document will be further developed and regularly updated and informed
by the outcomes of future VMAD sessions.
9. As VMAD continues to develop the elements of the NATM and FRAV continues to develop
safety requirements for ADS, this document will be updated to incorporate this work.
Detailed technical documents will be outlined in an index of supporting reference materials,
located at the end of this document, as these are developed by VMAD.
10. Subject to direction from GRVA and WP.29, once the NATM has reached a state of maturity
to inform evaluation criteria (based on performance requirements specified by the IWG on
FRAV), it is anticipated that this document (and any supporting resources developed by
VMAD) will be used to help inform validation process guidelines and/or
regulations/requirements that align with the needs of both 1958 and 1998 Agreement parties
(subject to approval by WP.29).

(f)
In-service monitoring and reporting
19. It addresses the in-service safety of the ADS after its placing on the market. It relies on the
collection of fleet data in the field to assess whether the ADS continues to be safe when
operated on the road. This data collection can also be used to fuel the common scenario
database with new scenarios from the field and to allow the whole ADS community to learn
from major ADS accidents/incidents.
V. SCENARIOS CATALOGUE
A. Why should scenario-based testing be included in the NATM?
20. In order to maximize the potential safety of AVs, a robust safety validation framework shall
be established. Such a framework shall provide clear direction for assessing safety
requirements of AVs in a repeatable, objective, evidence-based and technology neutral
manner.
21. At this relatively early stage in the development of AVs, much of the existing literature that
assesses the current state of AV development uses metrics such as miles/kilometers
travelled in real-world test situations with the absence of a collision, a legal infraction, or a
disengagement by the vehicle's ADS.
22. Simple metrics such as kilometers travelled without a collision, legal infraction, or
disengagement can be helpful for informing public dialogue about the general progress
being made to develop AVs. Such measurements on their own however, do not provide
sufficient evidence to the international regulatory community that an AV will be able to safely
navigate the vast array of different situations a vehicle could reasonably be expected to
encounter.
23. In fact, some observers have suggested that an AV would have to drive billions of miles in
the real-world to experience an adequate number of situations without an incident to prove
that it has a significantly better safety performance than a human driver (Kalra & Paddock,
2016). Safety validation through such testing would not be cost and time effective, nor would
it be feasible to replicate the testing later on. As validation of AV in various traffic situations
is needed, therefore different traffic scenarios shall be considered.
24. A scenario-based approach helps to systematically organize safety validation activities in an
efficient, objective, repeatable, and scalable manner and is a critical part of the NATM for
ensuring a holistic and dense coverage of traffic situations.
25. Scenarios-based validation consists of reproducing specific real-world situations that
exercise and challenge the capabilities of an ADS-equipped vehicle to operate safely.
B. What is a traffic scenario?
26. A scenario is a description of one or more real-world driving situations that may occur during
a given trip. SG1 will design scenarios for use under the NATM pillars. A scenario can
involve many elements, such as roadway layout, types of road users, objects exhibiting
static or diverse dynamic behaviours, and diverse environmental conditions (among other
factors). (A trip is a traversal of an entire travel pathway by a vehicle from the point of origin
to a destination.)

D. Classifying scenarios
33. The amount of information that is included in a scenario can be extensive. For example, the
description of a scenario could contain information specifying a wide range of different
actions, characteristics and elements, such as objects (e.g., vehicles, pedestrians),
roadways, and environments, as well as pre-planned courses of action and major events
that should occur during the scenario. Therefore, it is critical that a standardized and
structured language for describing scenarios is established so that AV stakeholders
understand the intention of a scenario, each other's objectives, and the capabilities of an
ADS.
34. One approach that researchers have established for developing a standardized and
structured language for describing scenarios, which also incorporates different levels of
abstraction/detail, is classifying scenarios according to three categories: functional, logical,
and concrete scenarios.
(a)
(b)
(c)
(d)
Functional Scenario: Scenarios with the highest level of abstraction, outlining the core
concept of the scenario, such as a basic description of the ego vehicle's actions; the
interactions of the ego vehicle with other road users and objects; roadway geometry;
and other elements that compose the scenario (e.g. environmental conditions etc.).
This approach uses accessible language to describe the situation and its
corresponding elements.
Logical Scenario: Building off the elements identified within the functional scenario,
developers generate a logical scenario by selecting value ranges or probability
distributions for each element within a scenario (e.g., the possible width of a lane in
meters). The logical scenario description covers all elements and technical
requirements necessary to implement a system that solves these scenarios.
Concrete Scenarios: Concrete scenarios are established by selecting specific values
for each element. This step ensures that a specific test scenario is reproducible. In
addition, for each logical scenario with continuous ranges, any number of concrete
scenarios can be developed, helping to ensure a vehicle is exposed to a wide variety
of situations.
Refer to Figure 1 for examples of functional, logical and concrete scenarios.

(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)
(k)
(l)
(m)
(n)
(o)
"Deterministic" is a term describing a system whose time evolution can be predicted
exactly and a given set of input stimuli will always produce the same output.
"Driver-In-the-Loop (DIL)" is typically conducted in a driving simulator used for
testing the human-automation interaction design. DIL has components for the driver
to operate and communicate with the virtual environment.
"Hardware-In-the-Loop (HIL)" involves the final hardware of a specific vehicle
sub-system running the final software with input and output connected to a simulation
environment to perform virtual testing. HIL testing provides a way of replicating
sensors, actuators and mechanical components in a way that connects all the I/O of
the Electronic Control Units (ECU) being tested, long before the final system is
integrated.
"Model" is a description or representation of a system, entity, phenomenon, or
process.
"Model calibration" is the process of adjusting numerical or modelling parameters in
the model to improve agreement with a referent.
"Model Parameter" are numerical values used to support characterizing a system
functionality. A model parameter has a value that cannot be observed directly in the
real world but that must be inferred from data collected in the real world (in the model
calibration phase).
"Model-In-the-Loop (MIL)" is an approach which allows quick algorithmic
development without involving dedicated hardware. Usually, this level of development
involves high-level abstraction software frameworks running on general-purpose
computing systems.
"Open Loop Testing" means a virtual environment that does not take the actions of
the element-in-the loop into account (e.g. system interacting with a recorded traffic
situation).
"Probabilistic" is a term pertaining to non-deterministic events, the outcomes of
which are described by a measure of likelihood.
"Proving Ground or test-track" is a physical testing facility closed to the traffic
where the performance of an ADS can be investigated on the real vehicle. Traffic
agents can be introduced via sensor stimulation or via dummy devices positioned on
the track.
"Sensor Stimulation" is a technique whereby artificially generated signals are
provided to the element under testing in order to trigger it to produce the result
required for verification of the real world, training, maintenance, or for research and
development.
"Simulation" is the imitation of the operation of a real-world process or system over
time.
"Simulation model" is a model whose input variables vary over time.

39. Virtual testing includes replacing one or more physical elements characterized in a
scenario-based test by a simulation model. The goal of such virtualization is to resemble, to
a sufficient extent, the original physical elements. For automotive applications, virtual testing
can be used to reproduce the driving environment and the objects operating therein that
interact with either the entire system (e.g. a full vehicle with tires and ADS functions), a
subsystem (e.g. an actuator or a hardware controller), or a component (e.g. a sensor).
40. Through this approach, an assessor can get confidence about the ADS based on the virtual
tests and validation that was performed by the developer in an agile, controllable,
predictable, repeatable, and efficient manner.
41. The simulation toolchain used for virtual testing may result in the combination of different
approaches. In particular, tests can be performed:
(a)
(b)
entirely inside a computer (referred to as Model or Software in the Loop testing,
MIL/SIL), with the model of the elements involved (e.g. a simple representation of the
control logic of an ADS) interacting in a simulated environment; and/or
with a sensor, a subsystem, or the whole vehicle interacting with a virtual environment
(Hardware or Vehicle in the Loop testing, HIL/VIL). For VIL testing, the vehicle can
either be in:
(i)
(ii)
a laboratory where the vehicle would be standing still or moving on a chassis
dynamometer or powertrain test bed and be connected to the environment
model by wire or by direct stimulation of its sensors; or
a proving ground where the vehicle would be connected to an environment
model and would interact with virtual objects by physically moving on the
test-track.
(c)
With a subsystem interacting with a real driver (DIL testing).
42. The interaction between the system under the test and the environment can either be an
open- or closed-loop.
(a)
(b)
Open-loop virtual tests (also referred to as software or hardware reprocessing,
shadow mode, etc.) could be done through a variety of methods, such as the ADS
interacting with virtual situations collected from the real world. In this case, virtual
objects' actions are data-driven only and the information is not self-corrected based
on feedback from the output. Because the open-loop controller may vary due to
external disturbances without the ADS and/or the assessor being aware, the
applicability of open-loop tests in the ADS validation may be limited.
Closed-loop virtual tests includes a feedback loop that continuously sends information
from the closed-loop controller to the ADS. Within these test systems, the behaviour
of the digital objects could react in different ways depending on the action of the
system under test.
43. Selecting an open- or closed-loop test could depend on factors such as the objectives of the
virtual testing activity and the status of development of the system under test. For ADS
validation it is expected that mainly closed-loop virtual testing will be considered.

Table 1
Strengths and Weaknesses of the Virtual Testing Pillar
Strengths
● Controllability – Virtual testing affords an
unmatched ability to control many aspects of
a test.
● Agility – Virtual tests allows for system
changes to be revaluated immediately.
● Efficiency – In MIL and SIL, virtual tests can
be accelerated faster than real-time so that
many tests can be run concurrently in a
relatively short amount of time.
● Cost effectiveness at test execution – In
spite of the investments required to develop,
validate and maintain a virtual testing
toolchain, the running costs connected to its
use are considerably lower than those
required by physical testing.
● Wide scenario coverage – Compared to
other testing methods, virtual testing allows a
wider exploration of safety-critical scenarios.
By properly combining the experiments
parameters it can for example reduce the
space of the known unknowns and to the
extent possible that of the unknown
unknowns (including the effect of system
failures).
● Data gathering and analysis – Virtual testing
offers a convenient and error-free platform
for data gathering and analysis of the ADS
performance. Once Qualified, that data can
serve as a significant contribution for
assessing the risk from the ADS.
● Repeatability and replicability – Simulation
affords the re-execution of the same virtual
test without deviations due to stochastic
phenomena. Faults in the functioning of the
ADS can thus be identically replicated at any
moment.
Weaknesses
● Lower environmental fidelity/reliability – It is
difficult, and likely impossible for models to
completely reproduce the environment,
responses, as well as the behaviour of the
vehicle, other road users etc. in the real
world. Also the validation process cannot
prove the validity of the simulation across all
possible scenarios.
● Risk of over-reliance. Without proper
consideration of models' intrinsic limitations,
a risk exists to put too much emphasis on
virtual testing results without sufficient proof
of their validity by physical testing.
● Expensive software life-cycle. The
availability of a simulation model to execute
virtual testing requires covering certain
aspects of the software life-cycle which can
be costly and time-consuming

(c)
(d)
Virtual testing will be a key element in the audit assessment. Results of virtual testing
carried out both during vehicle development and in the verification and validation
phase will represent an important element to be subject to audit. Manufacturers will
need to provide evidence and documentation about how the virtual testing is carried
out and how the underlying simulation toolchain has been validated.
Real-world tests can aid in the generation of realistic simulation models and in
establishing their accuracy:
(i)
(ii)
Real-world data for vehicle and component model validation: vehicle data and
data measured via vehicle sensors are important sources for quantifying and
arguing model accuracy (e.g. vehicle dynamics or sensor models).
Real-world data for traffic modelling: the generation of novel scenarios requires
realistic road user behaviour for the simulation environment to remain
meaningful and representative.
(e)
Virtual testing can play an important role in responding to concerns identified through
in-use monitoring of ADS performance. Virtual testing provides speed and flexibility in
analysing real-world events to verify ADS performance against such events and, if
necessary, support modifications to improve performance. Scenario descriptions can
be shared and integrated rapidly into virtual testing regimes worldwide. The various
types of virtual testing, including HIL methods that come close to matching physical
testing, ensure robust and rapid responses.
G. Use of the pillar to assess ADS safety requirements
53. Virtual testing using a validated simulation toolchain can be used to assess the ADS'
compliance with the safety requirements. Considering the categories of safety requirements
currently being considered, virtual testing seems particularly relevant for assessing
requirements related to:
(a)
(b)
(c)
ADS should drive safely, and ADS should manage safety critical situations. These are
the requirements where virtual testing can play the most prominent role. MIL/SIL, HIL
and VIL virtual testing can all be used to assess these requirements at different
stages of vehicle verification and validation.
ADS should interact safely with the user. DIL virtual testing can be helpful to support
the assessment of this category of safety requirement by analysing the interaction
between the driver and the ADS in a safe and controlled environment.
ADS should safely manage failure modes and ADS should ensure a safe operational
state. The use of virtual testing in these two categories is also very promising but
would probably require further research work. SIL virtual testing could include
simulated failures and maintenance requests. HIL and VIL virtual testing could be
used to assess how the system would react to the occurrence of a real malfunctioning
induced to the real system.

Strengths
● Track testing can be used to validate the
quality of the simulation toolchain by
comparing an ADS' performance within a
simulation test with its performance on a test
track when executing the same scenario.
B. Why include this pillar in the NATM?
Weaknesses
● Representativeness even with its increased
fidelity. Whilst things like pedestrians can be
included, these won't typically be real people
due to safety reasons and the clutter or
real-world
environments
cannot
be
replicated.
57. As per Paragraph 7.3 as well as the strengths and limitation table, there are a number of
reasons for including track testing in the NATM. For instance, track testing can be used to
assess the performance of ADS in nominal and critical scenarios. Track testing can also
provide a higher level of environmental fidelity than simulation. Unlike real-world testing,
track testing can accelerate exposure to known rare events or safety critical scenarios.
C. Maturity of the pillar
58. Although track testing is a mature process which is used to assess safety requirements for
some existing technologies, testing of ADS vehicles is fairly new and may need to be further
refined. For instance, it may be difficult to develop specific ODD elements, such as rain, fog,
and snow to reliably test how an ADS interacts with these environmental elements.
D. How the pillar interacts with other pillars?
59. The information generated during the track-test could also be used to validate the virtual
tests by comparing an ADS' performance within a virtual test with its performance on a test
track when executing the same scenario. For instance, track testing can be used to validate
the quality/reliability of the virtual toolchain by comparing an ADS' performance within a
virtual test with its performance on a test track when executing the same scenarios.
VIII.
REAL-WORLD TESTING
A. Purpose
60. Real-world testing uses public roads to test the capabilities and compliance with safety
requirements (e.g., human factors, safety system) of a vehicle with an automated driving
system (ADS) in real-world traffic.
61. This testing method can expose the ADS to a wide variety of real-world conditions related to
an ODD. There are various approaches to real-world testing. For example, tests can be
done within a specific ODD (e.g., highway driving) with a safety driver who is
monitoring/ensuring the ADS is functioning safely.
62. Real world testing could be used to assess aspects of the ADS performance related to its
capability to drive in real traffic conditions, e.g. smooth driving, capability to deal with dense
traffic, interaction with other road users, maintaining flow of traffic, being considerate and
courteous to other vehicles.
63. Real world testing could also be used to assess part of the ADS performance at some ODD
boundaries (nominal and complex scenarios), i.e. is the system triggering transition
demands to the driver when it is supposed to (e.g. end of the ODD, weather conditions).
The same testing could be used to confirm the performances related to human factors under
these conditions.

B. Why include this pillar in the NATM?
68. Real world testing provides an opportunity to validate the safety of the ADS within its true
operating environment, as set out in greater detail in Paragraphs 8.3, 8.4, and 8.5.
C. Maturity of the pillar
69. Real-world testing is regularly conducted to assess the performance of human drivers.
However, testing of ADS performance may pose some new challenges for this test
methodology. Experiences could be drawn from other motor vehicle-related real-world
testing schemes, such as real driving emissions (RDE) testing and market surveillance.
D. How the pillar interacts with other pillars
70. Real-world testing may be used to validate if portions of a virtual and/or track-testing
environment were modelled properly by comparing an ADS' performance within a simulation
and track test with its performance on in a real-world environment when executing the same
test scenario.
71. It can also be used to identify new traffic scenarios for track and virtual testing, allowing for
the identification of edge cases and other unknown hazard vulnerabilities that could
challenge the ADS. The information gathered from real world testing can also be used in the
hazard and risk analysis and design of the ADS systems.
IX.
AUDIT
A. Purpose
72. The purpose of the audit pillar is to assess/demonstrate that the:
(a)
(b)
Manufacturer has the right processes to ensure operational and functional safety
during the vehicle lifecycle, and
Vehicle design is safe by design and this design is sufficiently validated before market
introduction. The validation should be confirmed by in use monitoring.
73. The manufacturer will be required to demonstrate that:
(a)
(b)
(c)
Robust processes are in place to ensure safety throughout the vehicle lifecycle
(development phase, production, but also operation on the road and
decommissioning). It shall include taking the right measures to monitor the vehicle in
the field and to take the right action when necessary;
Hazard and risks relevant for the system have been identified and a consistent
safety-by-design concept has been put in place to mitigate these risks; and
The risk assessment and the safety- by-design concept have been validated by the
manufacturer through testing showing before the vehicle is placed on the market that
the vehicle meets the safety requirements and in particular is free of unreasonable
safety risks to the broader transport ecosystem in particular the driver, passengers
and other road users.

X. IN-SERVICE MONITORING AND REPORTING
A. Purpose
81. The in-service monitoring and reporting pillar addresses the in-service safety of automated
vehicles after market introduction. In practice, the application of the other pillars of the
NATM will assess whether the ADS is reasonably safe for market introduction whereas the
in-service monitoring and reporting will gather additional evidence from the field operation to
demonstrate that that the ADS continues to be safe when operated on the road. This pillar
addresses the dynamic nature of road transportation to ensure attention to and continuous
improvement of road safety through the use of ADS.
82. The pillar consists in the collection of relevant data during AVs operation.
83. The three main purposes of in-service monitoring and reporting is to use retrospective
analysis of data from manufacturers and other relevant sources to:
(a)
(b)
(c)
demonstrate that the initial safety assessment (residual risk) in the audit phase before
the market introduction is confirmed in the field overtime ("safety confirmation").
to fuel the common scenario database with important new scenarios that may happen
with automated vehicles in the field ("scenario generation") and
to derive safety recommendations for the whole community by sharing learnings
derived from key safety accidents/incidents to allow the whole community to learn
from operational feedback, fostering continuous improvement of both technology and
legislation ("safety recommendations").
84. The obligation to have "real-time monitoring" (self-checks/on board diagnostics) of the
performance of ADs subsystems by the manufacturer is not part of this pillar but is part of
the safety requirements. However, some reporting mechanisms on the performance of ADS
subsystems overtime could be part of the first bullet above, and contribute to the predictive
monitoring of safety performance degradation.
85. The processes put in place by the manufacturer to manage safety during in use (e.g. to
manage changes in the traffic rules and in the infrastructure) fall outside this pillar and are
assessed with the audit pillar. This pillar focuses on the type of data to be monitored and
reported.
B. Why should this pillar be included in the NATM?
86. Whatever safety evaluation is done before market introduction, the actual level of safety will
only be confirmed once a sufficient number of vehicles is in the field and once they are
subjected to a sufficient range of traffic and environmental conditions. It is therefore
essential that a feedback loop (fleet monitoring) is in place to confirm the safety by design
concept and the validation carried out by the manufacturer before market introduction. The
operational experience feedback from in-use monitoring will allow ex-post evaluation of
regulatory requirements and validation methods, providing indications on gaps and needs
for review.
87. New scenarios and new risks might be introduced by AVs on the market. Therefore, the
In-Use Monitoring pillar could be used to generate new scenarios in the common scenario
database to cover these new safety risks.

96. The development of new traffic scenarios on the basis of traffic data has already started
from the manufacturers' side, through post-processing of recorded data elements and
images (tools for complete automatic scenario generation are not available yet).
97. Operational Accident/incident analysis is a well-established practice in some vehicle safety
regimes, e.g. through the analysis of data from event data recorders (EDRs) from
conventional vehicles which are collecting relevant information in certain crash situations .
No standard data elements are currently defined for ADS crash/near-missed investigation:
entities engaging in testing or deployment are encouraged to voluntarily collect data
associated with crashes . Because it is first time the concept of in-service-monitoring is
introduced into the automotive safety sector and vehicles are usually used by normal
citizens (different from air or rail sector), feasibility, such as how to collect data, which data
(e.g. including or not if the ADs caused the circumstances that resulted in near-miss-crash),
would be important view points and it should be well discussed.
98. Mechanisms for operational feedback to improve common knowledge are already in place
for decades in other transport sectors (see ECCAIRS portal,
http://eccairsportal.jrc.ec.europa.eu/). Existing systems for reporting of safety concerns in
the automotive sector have also been developed over decades of experience. A first step
would be to investigate the suitability of such tools for ADs too. However, the main effort
would still remain in defining common reporting criteria and developing a common
repository. According to mechanisms already in place in other sectors (e.g. see Figure 2),
in-service data recorded related to safety-relevant events (i.e. accidents, near-miss events,
abnormal functioning etc.) are processed by manufacturers/operators and then an accident
report (what happened) by manufacturers/operators is delivered to the National authority.
National authorities are then responsible to perform the accident analysis (why did it
happened), derive safety recommendations (how could this be avoided), and evaluate the
possible impact on existing legislation. National information is then recorded into:
(a)
(b)
Central Repository of Occurrences; and,
a Central Repository of Safety Recommendations. Access to data recorded into the
Central Repository is subject to strict rules and mainly limited to competent
Authorities. Safety recommendations are shared internationally according to the
guiding principle that transport safety is of global concern and its improvement should
not be limited by geographical or organizational borders. Privacy is ensured at all
levels. Another option could be for the measured data to be directly communicated to
the authorities, who will then be in charge of collection, storage and post-processing
of the information.

103. A single assessment or test method may not be enough to assess whether the ADS is able
to cope with all occurrences that may be encountered in the real world.
104. For instance, while real-world testing provides a high degree of environmental fidelity, a
scenario-based testing methodology using only real-world testing could be costly,
time-consuming, difficult to replicate, and pose safety risks. Consequently, track testing may
be more appropriate methods to run higher risk scenarios without exposing other road users
to potential harm. Further, test scenarios can also be more easily replicated in a closed
track environment compared to the real-world. That said, test track scenarios can be
potentially difficult to develop and implement, especially if there are numerous or complex
scenarios, involving a variety of scenario elements.
105. Simulation/virtual testing, by contrast, can be more scalable, cost-effective, safe, and
efficient compared to track or real-world testing, allowing a test administrator to safely and
easily create a wide range of scenarios including complex scenarios where a diverse range
of elements are examined. However, simulations may have lower fidelity than the other
methodologies. Simulation software may also vary in quality and tests could be difficult to
replicate across different simulation platforms.
106. In-service monitoring and reporting can confirm the pre-deployment safety assessment and
fill the gaps between safety validation through virtual/physical testing and real-life
conditions. Evaluation of in-service performance will also serve to update the scenario
database with new scenarios deriving from increasing deployment of driving automation.
Finally, the feedback from operational experience can support ex-post evaluation of
regulatory requirements.
107. In addition to the respective strengths and weakness of each test pillar, the nature of the
safety requirements being assessed will also inform what pillars are used.
108. For instance: the most appropriate method to assess an ADS's overall system safety prior to
market introduction may be the audit pillar, using a systematic approach to perform a risk
analysis. The audit could include information such as safety by design confirmed validation
outputs as well as analysis of data collected in the field by the manufacturer.
109. Virtual testing may be more suitable when there is a need to vary test parameters and a
large number of tests need to be carried out to support efficient scenario coverage (e.g., for
path planning and control, or assessing perception quality with pre-recorded sensor data).
110. Track tests may be best suited for when the performance of an ADS can be assessed in a
discrete number of physical tests, and the assessment would benefit from higher levels of
fidelity (e.g., for HMI or fall back, critical traffic situations).
111. Real-world testing may be more suitable where the scenario may not be precisely
represented virtually or on a test track (e.g., interactions with other road-users and
perception quality may be assessed through real world evaluation).
112. In-service monitoring and reporting of field data represent the best way to confirm the safety
performance of an ADS in the field after market introduction over a wide variety of real
driving traffic and environmental conditions.

Figure 3
Relationship between VMAD Pillars, Scenarios and FRAV Safety Requirements
XII.
INTEGRATION
117. This section on the VMAD activities on NATM and the FRAV related activities will be further
developed, regularly updated, and informed by the outcomes of future VMAD and FRAV
sessions. The purpose of this section is to establish the links between the different pillars of
the NATM and the safety requirements developed by FRAV, as outlined in Common Safety
Requirements for Autonomous Vehicles document developed by FRAV (FRAV-02-05,
https://wiki.unece.org/download/attachments/92013878/FRAV-02-05-Rev.2.docx?api=v2).
118. As the safety requirements and technical aspects of each of the pillars are further
developed, each of these sections will be updated to include additional detail. To provide
further context, this section will also include examples of how the NATM pillars can be
applied to certain functional capabilities of an ADS (e.g., highway driving) based on the
established safety requirements.

ANNEX II
FUNCTIONAL SCENARIOS FOR DIVIDED HIGHWAY APPLICATION
Contents
I. Introduction
II.
III.
IV.
Inputs to this proposal:
Building blocks of functional scenarios
Coverage
V. Symbols used in this document:
VI.
A list of possible scenarios for L3 Highway Chauffeur ADS
A.
Nominal driving (Perform lane keeping)
1.
Nominal driving (Perform lane keeping)
B.
Interaction with other vehicles/objects
1.
Perform lane change
2.
Critical (Emergency) braking scenarios during lane keeping
C.
Detect and response to traffic rules and road furniture
D.
Country specific road geometry
E.
Unusual situation
VII.
References

IV.
COVERAGE
Since collisions always occur with other vehicles/objects (assuming that they can operate properly
when there are no other vehicles/objects), and 24 functional scenarios in the figure described in
"2. Interaction with other vehicles" can cover all interactions between other vehicles/objects and
ego vehicle, the scenarios can cover collision with other vehicles/objects appropriately.
As described in Paragraph 3., factors not covered in the proposed functional scenarios (e.g. initial
speed of ego vehicle, size, initial position, initial speed, acceleration of other vehicles/objects),
perception factor (e.g. weather, brightness, blind spot, false positive factor, blinkers of other
vehicles) and vehicle stability factors (e.g. curve, slope, road surface μ, wind, etc.) can be
described with parameters in logical scenarios.
Functional scenarios should be added anytime if SG1 and IWG on VMAD discussed and agreed.
V. SYMBOLS USED IN THIS DOCUMENT

Scenario family
Sub-scenario
Japan
crash
scenarios
The
Netherlands
(TNO)
SAFE
scenario
library
China
functional
scenarios
Conflict
Type
a. Speed limit
sign
X
X
b. Signal lights X X
C. Detect and response to traffic
rules and road furniture
c. Drive through
tunnel
X
d. Toll X
e. Conventional
obstacles
D. Country specific road geometry a. Interceptor X
X
X
E. Unusual situation
a. Wrong way
driver (oncoming)
X
X
Notes to the inputs from VMAD SG1 members:
● China (CATARC): This is a list cut from a general catalogue describing different ODDs, like
"General road", "City expressway" or "The highway" and their test items, like "speed limit
sign", "lane line", "toll station", etc. The functional scenarios proposed below in this
document are much more generic than the ones proposed by China, so they form a subset
of this list. For example China proposal: "toll station" on the road or "conventional obstacles"
can be in line with "impassable object on intended path" from this scenario list.
● The Netherlands (TNO): a very thorough scenario catalogue containing much more
scenarios than needed for the highway use case. Terminology and descriptions worked out
fully. Scenarios can be created using a combination of tags from the different layers.
● Japan: crash scenarios, scenarios only containing interaction with other vehicles. They
describe different road geometries and possible other vehicle positions around ego. All other
parameters considered as features (acceleration – deceleration, lane change – lane
keeping, etc.).
● SAFE: a list of scenarios sometimes with very concrete examples, sometimes more generic
approach. There is a different scenario for passing by slowly moving vehicles in the adjacent
lane and a different one for passing by standing vehicles, but handles Lead Vehicle (LV)
following as one scenario.
● Conflict Type: a list of "conflict types" used i.a. by accident investigators to sort scenarios,
leading to accidents on road to different groups. These conflict types can be sorted into
conflicts with or without influence of other road user. Uses different symbols than other
documents for the description of a scenario or situation (mainly different kinds of arrows).
Separates left and right hand traffic. Contains 251 scenario types, structured in seven larger
types of conflicts, like: "longitudinal traffic" or "pedestrian crossing the road".
Note: "emphasized scenario parameters" and "tested parameters" in this paragraph are some
examples of parameters. Other parameters may be essential for the validation testing.

(b)
Manoeuvring a bend (right curve and left curve)
(a)
(b)
(c)
Without LV
With LV
With other vehicles in adjacent lanes (moving or stopped)
Figure 2
Schematic Representation of Manoeuvring a Bend
General description:
The ego vehicle is driving on a curved road. The aim of this scenario is to test if the
vehicle is able to handle the road curvatures specified as part of the ODD [1], [2], [4].
Emphasized scenario parameters: ego speed demand (road rules), lane width, LV
speed profile (if present), layout and speed profile of other vehicles (if present).
Tested parameters: deviation from lane centre (nominal value and distribution),
deviation from desired speed, obeying to speed changes, temporal modifications,
distance between ego and LV (if present), distance to other vehicles, etc.

(a)
Ego vehicle performing lane change with vehicle behind
Figure 3
Schematic Representation of a Lane Change
General description
In an adjacent lane, another vehicle is driving in the same direction as the ego
vehicle. The intention of the ego vehicle is, to perform a lane change to the lane in
which the other player is driving [1], [3].
Emphasized scenario parameters: time of lane change, ego speed demand (road
rules), lane width, LV speed profile (if present), layout and speed profile of other
vehicles (if present).
Tested parameters: deviation from lane centres (nominal value, overshoot), time of
lane change (lateral velocity of ego), distance between ego and LV (if present),
distance to other vehicles, etc.
(b)
Merging at highway entry
No description provided.
(c)
Merging at lane end
No description provided.

2. Critical (Emergency) braking scenarios during lane keeping
Note: In this family of scenarios a couple critical functional scenarios are present. It can be
noticed in the input matrix of SG1 as well, these are scenarios that nearly every participant
highlighted in the input documents.
(a)
Impassable object on intended path (Including other cars and Vulnerable Road Users
(VRUs))
Figure 5
Schematic Representation of an Impassable Object
General description:
The ego vehicle is driving on a road with an impassable object in the ego lane. The
objective of the ego vehicle is to continue driving straight. The ego vehicle needs to
react [1], [2]. Depending on the velocity of the ego vehicle, the severity of the scenario
is changing.
Emphasized scenario parameters: road layout (visibility of the object on the path),
layout and speed profile of other vehicles (if present), ego velocity.
Tested parameters: reaction of ego (lane change/braking), distance to object, lateral
velocity of ego (if changing lane), etc.

(c)
Lead vehicle braking
Figure 7
Schematic Representation of Lead Vehicle Braking
General description:
The ego vehicle is following a LV. The LV brakes, the ego vehicle has to adapt its
speed in order to stay at a safe distance from the lead vehicle [1], [2], [3], [4].
Emphasized scenario parameters: ego velocity (road rules), LV speed profile
(deceleration), layout and speed profile of other vehicles (if present).
Tested parameters: distance between ego and LV, reaction to other vehicles in
adjacent lanes, etc.

(e)
Cut-in in front of the ego vehicle
General description:
Figure 9
Schematic Representation of Cut-in
Another vehicle is driving in the same direction as the ego vehicle in an adjacent lane.
The other vehicle makes a lane change, such that is becomes the LV from the ego
vehicle's perspective [1-4]. Depending on the distance and lateral velocity of the LV,
the severity of the cut-in manoeuvre changes.
Emphasized scenario parameters: LV lateral speed, distance to LV, ego velocity, lane
width, layout and speed profile of other vehicles (if present).
Tested parameters: distance between ego and LV, distance to other vehicles, etc.

(g)
Detect and respond to swerving vehicles
Figure 11
Schematic Representation of a Swerving Vehicle
General description:
Another vehicle is driving in the same direction as the ego vehicle in an adjacent lane.
The other vehicle swerves towards the ego vehicle's lane [1], [2], [3].
Emphasized scenario parameters: lateral speed of other vehicle, ego velocity, lane
width, layout and speed profile of other vehicles (if present).
Tested parameters: distance between ego and swerving vehicle, distance to other
vehicles…

(b)
Signal lights
The test road consists of at least two lanes. The signal lights are set above the road, and
the signal lights of adjacent lanes are kept in green state.
Figure 13
Testing Scenario Diagram for Expressway Signal Lights
(c)
Drive through tunnel
General description:
Figure 14
Schematic Representation of Driving Through Tunnel
The ego vehicle is driving through a tunnel (lack of GPS signals and natural light) [4]. The
vehicle needs to adapt to the quickly changing light parameters and lack of global
positioning. Depending on the speed of the ego vehicle, the difference between the light
conditions outside and inside the tunnel and the length of the tunnel, the difficulty of the
scenario is changing.
Emphasized scenario parameters: ego velocity, light conditions.
Tested parameters: ego lateral and longitudinal velocity, deviation to lane centre, etc.

D. Country specific road geometry
Note: This scenario is only applicable for limited countries or regions. Therefore, application of this
scenario can be unnecessary depends on the target market of the ADS.
(a)
Interceptor
For the DUT, junctions present a challenge due to the increased likelihood of conflicts with
other actors.
In this scenario, the DUT traverses an intersection simultaneously with another car - the
interceptor. This scenario tests the DUT's behaviour when on a collision course with another
car in an intersection, possibly with signs, signals, or traffic lights. The DUT should be able
to safely manoeuvre through the intersection and avoid or mitigate a collision.
Environmental requirements: A junction with at least three ways. It may or may not be
controlled (i.e. have yield sign, traffic lights, etc.).
DUT behaviour: The DUT traverses the junction in any direction (left, right or straight).
Other actors' behaviour: Another car approaches the same junction, from a different
direction and traverses the junction such that its trajectory intersects with the DUT's
trajectory.
Figure 17
Interceptor Scenario

VII.
REFERENCES
1. UN Regulation No. 157 (Automated Lane Keeping System), Available online at
https://undocs.org/ECE/TRANS/WP.29/2020/81
2. E. de Gelder, O. Op den Camp, N. de Boer, (The Netherlands): Scenario Categories for the
Assessment of Automated Vehicles, Version 1.7, January 21, 2020.
3. SAFE (Foretellix) Highway and ADAS Traffic Scenario Library, Scenario Definitions at the
functional Level, Version 1.0, November 2020.
4. Japan: Proposal of Traffic Scenarios for Highway Driving (Supplemental version for
presentation), December 2020.
5. China (CATARC): Proposal about functional scenario from CATARC, December 2020.
6. EC-JRC. Speed profile for car-following tests. Available online at
https://wiki.unece.org/download/attachments/92013066/ACSF-25-
13%20%28EC%29%2020190121_TestSpecification_ALKS_JRC.pdf?api=v2
7. IGLAD 2019 Codebook, Conflict Types, 2019.
New Assessment/Test Method for Automated Driving (NATM) - Master Document.