Software/SystemVerification and Validation

Transcription

ÉCOLE DE TECHNOLOGIE
SUPÉRIEURE
MGA-855
Certification des systèmes embarqués
d’aéronefs
Maîtrise en génie : Concentration en génie aérospatial
It’s Only Software!
1
SUPÉRIEURE
MGA-855
Certification des systèmes embarqués
d’aéronefs
Maîtrise en génie : Concentration en génie aérospatial
Chapitre 2.2
Software/System
Verification and Validation (V and V)
présenté par :
Maxence Vandevivere
[email protected]
Professeur responsable : René Jr. Landry
Poste : 8506
Porte : 2950
Email : [email protected]
Site web : www.etsmtl.ca/rlandry
2
2.2 V&V
SUPÉRIEURE
Overview
 2.2 V & V
•
•
•
•
•
•
•
•
•
•
•
•
•
•
2.2.1
2.2.2
2.2.3
2.3.4
2.3.5
2.3.6
2.3.7
2.3.8
2.3.9
2.3.10
2.3.11
2.3.12
2.3.13
2.3.13
Requirements Validation & Verification
Good Requirements?
System Validation & Verification
Valid System Requirements?
Case Studies
Ariane 5
Hubble
Hubble Space Telescope
Mars Climate Orbiter
Saab Gripen Software Flows (PIO)
Lockheed F-22 PIO
Phasers On Stun…
Therac-25
Convair XFY-1 Pogo
 Summary
Certification des systèmes embarqués d’aéronefs
MGA-855: Chapitre 2
3
2.2 V&V
SUPÉRIEURE
Caution
This module is designed to show the application of the
certification principles contained in the others chapters. As
such, it touches upon many aspects of the certification process,
but the material is not complete, comprehensive, or necessarily
current with the latest regulations and guidance materials. For
these reasons, the analysis contained in the following slides
should not be used as the basis of a certification program for
the system under investigation.
MGA-855: Chapitre 2
4
2.2 V&V
SUPÉRIEURE
2.2.1 Requirements V&V
• Requirements V&V is a subset of other required qualities
of requirements, which include:
–
–
–
–
–
–
–
–
Verifiable
Correct (i.e. Valid)
Complete
Unique
Consistent
Traceable
Shall be positive statements
etc…
MGA-855: Chapitre 2
5
2.2 V&V
SUPÉRIEURE
2.2.1 Requirements V&V (cont’d)
• Verifiable requirements:
–
–
–
–
Usually quantitative
Contain “shall” statements
Typically contain a single requirement statement per requirement
Should be testable (manually, or via analysis)
• Valid requirements:
– In addition to above, valid requirements contribute positively to meeting
a system’s intended function
MGA-855: Chapitre 2
6
2.2 V&V
SUPÉRIEURE
• Rules for good requirements:
1. Unambiguous : Requirements shall be written in a way that allows only
one interpretation.
2. Complete: Enough information shall be included or referenced to allow
software design and coding (see Section 4.5.2 above).
3. Verifiable: It shall be possible to verify proper implementation of each
requirement by testing, analysis, or inspection.
4. Consistent: There shall be no conflicts within the requirements.
5. Modifiable (maintainable): Requirements shall be subdivided to a low
enough level that small changes can be made by changing one or two
statements rather than reorganizing sections.
6. Traceable: Requirements shall be traceable from the System
requirements, to software requirements, to design requirements, to test
case and testing results.
MGA-855: Chapitre 2
7
2.2 V&V
SUPÉRIEURE
2.2.2 Good Requirements?
• A good set of requirements?
Figure 2.2.1 – A pecial gun
MGA-855: Chapitre 2
8
2.2 V&V
SUPÉRIEURE
• Examples of a poor requirements:
–“The operator interface should be
easy to use and have a low
workload”
–“The software should be robust and
fast.”
• Class exercise: take five minutes to reword these
requirements to be verifiable and valid.
MGA-855: Chapitre 2
9
2.2 V&V
SUPÉRIEURE
2.2.3 System V&V
• Verification is:
– A quality control process
– used to evaluate whether a product, service, or system complies with
regulations, specifications, or conditions imposed at the start of a
development phase.
– can be in development, scale-up, or production.
– often an internal process.
• Validation is:
– A quality assurance process
– Establishes evidence that provides a high degree of assurance that a
product, service, or system accomplishes its intended [requirements]
function.
– often involves acceptance of fitness for purpose with end users and
other product stakeholders.
– Validation can entail Operational Testing and Evaluation, “OT&E.”
MGA-855: Chapitre 2
10
2.2 V&V
SUPÉRIEURE
2.2.3 System V&V (cont’d)
• It is sometimes said:
–
–
Validation can be expressed by the query "Are you building the right thing?"
Verification can be expressed by the question "Are you building it right?“
• "Building the right thing" refers back to the user's needs,
• “Building it right" checks that the
specifications/requirements are correctly implemented by
the system.
• Sometimes it is required to have written requirements for
both (V & V), as well as formal procedures or protocols
for determining compliance.
http://www.critech.com/vv.htm, http://en.wikipedia.org/wiki/Verification_and_validation
MGA-855: Chapitre 2
11
2.2 V&V
SUPÉRIEURE
2.2.4 Valid System Requirement?
Figure 2.2.2
MGA-855: Chapitre 2
12
2.2 V&V
SUPÉRIEURE
2.2.5 Case Studies
 Ariane 5 (Ported software, ported flaw, V&V issues)
(http://www.youtube.com/watch?v=EMVBLg2MrLs)
 Hubble Space Telescope (bad verification)
 Mars Climate Orbiter (imperial vs. metric units!)
 Saab Gripen (software flaws x2, same pilot in both crashes)
(http://www.youtube.com/watch?v=4iToQ2FykoI
http://www.youtube.com/watch?v=k6yVU_yYtEc&feature=related)
 Lockheed F-22 (http://www.youtube.com/watch?v=faB5bIdksi8)
 Therac-25, or “Set Phasers on Stun” (robustness case, bad
verification)
 Convair XFY-1 Pogo (system requirements validity)
MGA-855: Chapitre 2
13
2.2 V&V
SUPÉRIEURE
2.2.6 Ariane 5
• An expendable launch system, launched under european
space agency (ESA) authority in approx 1995.
• Intent was to launch payloads into low-earth orbit
• Reused software from Ariane 4
• Ariane 5 design flight path was much different, beyond
range of Ariane 4 software
MGA-855: Chapitre 2
14
2.2 V&V
SUPÉRIEURE
2.2.6 Ariane 5 (cont’d)
Figure 2.2.3 and 2.2.4 – Ariane 5
MGA-855: Chapitre 2
15
2.2 V&V
SUPÉRIEURE
2.2.6 Ariane 5 (cont’d)
• Software bug, integer overflow from ported software
• Ariane 5 had quicker acceleration, which caused the
primary and backup guidance computers to crash.
• For effiency of the code, the software error handler was
disabled for this particular error
• Result was the rocket nozzles were sent garbage
steering commands
• Pre-flight tests had never been performed on realignment code for Ariane 5
• Loss of $370M USD
MGA-855: Chapitre 2
16
2.2 V&V
SUPÉRIEURE
2.2.7 Hubble
• Within weeks of the launch of the telescope, the returned
images showed that there was a serious problem with
the optical system… (http://en.wikipedia.org/wiki/Hubble_Space_Telescope#Flawed_mirror)
Figure 2.2.5 and 2.2.6 - Hubble
MGA-855: Chapitre 2
17
2.2 V&V
SUPÉRIEURE
2.2.7 Hubble
(cont’d)
• Analysis of initial images showed a problem with the
primary mirror
• Mirror had been ground to the wrong shape (although it
was probably the most precisely mirror ever made!)
• Although error was about 2200 nanometers (2.2 microns)
the difference was catastrophic
• It introduced severe spherical aberration (a flaw in which
light reflecting off the edge of a mirror focuses on a
different point from the light reflecting off its center.)
MGA-855: Chapitre 2
18
2.2 V&V
SUPÉRIEURE
2.2.8 Hubble Space Telescope
• No aberration
• Spherical aberration
Figure 2.2.7 – Example of aberration
MGA-855: Chapitre 2
19
2.2 V&V
SUPÉRIEURE
2.2.8 Hubble Space Telescope (cont’d)
• A commission headed by Lew Allen, (JPL director) was created to
figure out what happened
• They found that the main null corrector (used to measure the exact
shape of the mirror), had been incorrectly assembled.
• During the polishing of the mirror, Perkin-Elmer (the manufacterer)
had analyzed its surface with two other null correctors
• Both other null correctors correctly indicated that the mirror was
suffering from spherical aberration.
• These test results were ignored - it is believed that the two null
correctors were less accurate than the primary device that was
reporting that the mirror was perfectly figured (bad measurement).
(http://hubblesite.org/hubble_discoveries/10th/vault/allabout.shtml,
http://topics.nytimes.com/top/news/science/topics/hubble_space_telescope/index.html,
http://en.wikipedia.org/wiki/Hubble_Space_Telescope#Flawed_mirror)
MGA-855: Chapitre 2
20
2.2 V&V
SUPÉRIEURE
2.2.9 Mars Climate Orbiter
MGA-855: Chapitre 2
21
2.2 V&V
SUPÉRIEURE
2.2.9 Mars Climate Orbiter (cont’d)
Figure 2.2.8
MGA-855: Chapitre 2
22
2.2 V&V
SUPÉRIEURE
• Launched December 11, 1998
• Intent was to study Mars:
–
–
–
–
climate,
atmosphere,
surface changes
And act as the communications relay for the Mars lander
• September 23, 1999 communication was lost due to
“navigation error”.
MGA-855: Chapitre 2
23
2.2 V&V
SUPÉRIEURE
• November 10, 1999, the “Mishap Investigation Board”
released a report with their findings:
–
–
–
–
–
A trajectory correction manuever was computed, but incorrectly
Intent was to place spacecraft in optimal position in orbit
Altitude was actually much lower than intended
Orbiter disintegrated because altitude was too low (unsustainable)
Flight system computer was written to calculate thruster performance in
Newtons, while ground crew was using Imperial Pound-force.
• “This is an end-to-end process problem”, Tom Gavin,
JPL adminstrator. (CNN, Sept 30, 1999, http://articles.cnn.com/1999-0930/tech/9909_30_mars.metric.02_1_climate-orbiter-spacecraft-team-metric-system?_s=PM:TECH)
MGA-855: Chapitre 2
24
2.2 V&V
SUPÉRIEURE
2.2.10 SaabGripen Software Flaws (PIO)
Figure 2.2.9 – Saab JAS 39 Gripen
MGA-855: Chapitre 2
25
2.2 V&V
SUPÉRIEURE
2.2.10 Saab Gripen Software Flaws (PIO)
(cont’d)
• First roll out on 26 April 1987
• Concerns about the aircraft FBW (fly by wire) system and
stability and control
• Feb 2, 1989, crash of prototype – cause of the crash was
PIO (pilot induced oscillation)
• Saab made software improvements to fix the PIO
• Aug 18, 1999, another crash due to more PIO
• Issue was high amplification of quick and large inputs
from pilot controls
• Same pilot in both crashes – he only suffered a broken
arm on the first crash
MGA-855: Chapitre 2
26
2.2 V&V
SUPÉRIEURE
2.2.11 Lockheed F-22 PIO
• A Lockheed single seat fighter that uses stealth
technology
• Design began in approx 1986
• In production for roughly 15 years
• April 1992, an F-22 crashed at Edwards because of
software that did not prevent PIO
MGA-855: Chapitre 2
27
2.2 V&V
SUPÉRIEURE
2.2.12 Phasers On Stun…
• Therac-25 (http://www.onlineethics.org/Resources/19049/therac25.aspx,
http://en.wikipedia.org/wiki/Therac-25)
• Machine that provided radiation therapy and X-Rays
• When in “therapy” mode to treat cancer patients, the
machine was in “low-power” mode
• When in X-Ray mode to take X-Rays, the machine used
a much higher power beam, with filters and a beam
spreader
• Accident occurred when the high-power beam was used
without beam spreader, which subjected patient to 100x
the dose of radiation
MGA-855: Chapitre 2
28
2.2 V&V
SUPÉRIEURE
2.2.13 Therac-25
Figure 2.2.10 – Typical Therac-25 Facility
MGA-855: Chapitre 2
29
2.2 V&V
SUPÉRIEURE
2.2.13 Therac-25 (cont’d)
MGA-855: Chapitre 2
30
2.2 V&V
SUPÉRIEURE
• This was due to several software errors
• Due to bad design and development practices, like:
– No independent review
– No safety consideration for failure modes
– No testing with hardware and software until unit was finally assembled at
hospital
MGA-855: Chapitre 2
31
2.2 V&V
SUPÉRIEURE
• A sample of the bugs found:
– When “non-standard” (i.e.: unexpected) keystrokes were entered, the
software could enter this fatal mode
– No hardware interlocks that would have stopped the high-energy beam
from operating without the spreader in place
– Software was reused from earlier mode which had hardware interlocks,
but the new hardware did not
– No way to ensure operator and software were “in sync”
– Software used a flag to ensure safety checks were done, however the
flag was incremented instead of set, which led to overflows which
bypassed safety checks
MGA-855: Chapitre 2
32
2.2 V&V
SUPÉRIEURE
2.2.14 Convair XFY-1 Pogo
• I wasn’t kidding with the earlier picture…
Figure 2.2.11 – Convair XFY-1 Pogo
MGA-855: Chapitre 2
33
2.2 V&V
SUPÉRIEURE
2.2.14 Convair XFY-1 Pogo (cont’d)
• 1951: US Navy awarded Lochkeed and Convair to
develop an experimental aircraft
• Requirements were:
–
–
–
–
Fleet defense
VTOL (vertical take off and landing)
Agility of a fighter
Flexibility of a helicopter
MGA-855: Chapitre 2
34
2.2 V&V
SUPÉRIEURE
• Requirements conflict at a high level – let’s look at some
of the resulting (“band-aid”) design:
– Nicknamed “pogo” after a pogo stick and the XFY-1’s tendency to
bounce on landing on its set of four shock-absorbing landing gear
– Difficult to land – pilot had to look behind over their shoulder to land the
aircraft
– Eight bladed contra-rotating propeller
– Even more difficult to transition from vertical to horizontal flight
– Pilot sat in a seat that rotated depending on type of flight (vertical or
horizontal)
– Pilot entry was a 25ft ladder
– Emergency exit for pilot was a 25ft rope (during ground ops)
MGA-855: Chapitre 2
35
2.2 V&V
SUPÉRIEURE
• 70 flights before transition from vertical to horizontal was attempted
• Project abandoned after 40 hours of flight time
• Test pilot was awarded the Harmon Trophy for being able to take off
and land the aircraft
• “The Achilles Heel of the Convair XFY-1 VTOL aircraft was the
vertical landing phase. The pilot just could not judge rate-of-descent
accurately. This was attributed partly to the fact that he had to look
over his shoulder throughout the descent. In addition, XFY-1
throttle-induced lateral-directional handling qualities were poor and
forced the pilot to work very hard at landing the aircraft even in lowwind conditions.” (http://blog.seattlepi.com/americanaerospace/2010/11/01/convair-xfy-1-pogo)
• “Landing the XFY-1 was too demanding for the average squadron
pilot.” (The Illustrated Directory of Fighters, Mike Spick)
MGA-855: Chapitre 2
36
2.2 V&V
SUPÉRIEURE
Summary
• Both verification and validation are critical processes
• Both are often confused and misunderstood
• Failure to do either in some way will almost certainly cost
lives or money, or both.
• If these processes do not exist, are not working correctly,
are misunderstood, expect product issues.
• “Learn from the mistakes of others. You can’t live long
enough to make them all yourself.” Eleanor Roosevelt (1884 – 1962)
MGA-855: Chapitre 2
37
2.2 V&V
SUPÉRIEURE
Questions?
MGA-855: Chapitre 2
38
2.2 V&V
SUPÉRIEURE
•
•
•
•
Images References
Figure 2.2.1 – http://techepics.com/files/gangsta.jpg
Figure 2.2.10 – http://www.iro.umontreal.ca/~pift1025/bigjava/Ch10/ch10.html
Figure 2.2.11 – http://blog.seattlepi.com/americanaerospace/2010/11/01/convair-xfy1-pogo/
Others images and pictures are in the public domain.
MGA-855: Chapitre 2
39

Software/SystemVerification and Validation

Transcription

Documents pareils

Sciences de la nature - maternelle

Preface - CEUR

Webquest British schools questionnaire

Water Resources in China and the Effect of Climate Change on them

[Descartes e il Seicento 5] Jean-Robert Armogathe, Vincent Carraud

4228 Holland Dr Des Moines, IA 50310 | Home for Sale | Real

Additional Web Links

The Study Drama Postcard

Imagine you are working for the Scottish Tourist Board and you are

INTRODUCTION TO THE BI-FUEL TECHNOLOGY AND IT`S

It`s Only Software (and Hardware)!