Verification, Validation and Accreditation of Simulation Models and

Transcription

Verification, Validation and Accreditation
of Simulation Models and Applications
Axel Lehmann
Institut für Technische Informatik
ITIS e.V.
Fakultät für Informatik
Institut für Technik Intelligenter Systeme
Universität der Bundeswehr
München
an der Universität der Bundeswehr
München
Email: [email protected]
http://www.unibw.de/inf4/personen/prof/al/index_html
NATO Advanced Research Workshop
Velingrad, Bulgaria, Oct. 24, 2006
Trends in Engineering and Manufacturing

Axel Lehmann
Rapid technological innovations
information and communication technologies
computer-assisted tools (e.g. CAD, CAM, . . . )
Increasing systems complexity & lifetime
embedded systems
networked components / systems
distributed systems
Increasing productivity & cost-benefit relations
Decreasing cycle times for system’s innovations
Major Challenges
Axel Lehmann

“Mastering” of system(s) complexity
over lifetime w.r.t. multiple aspects/ goals?
(safety, reliability, performance,…)

Approach:
Goal-oriented, efficient, effective and dependable models!

“Mastering” of model(s) complexity !?
⇒ STRATEGIC approach: “Devide and Conquer”!
⇒ MODEL-ENGINEERING approach!!
M&S-Approach: “Divide and Conquer”
Multi-Phase M&S process(es)
(intermediate products)
Axel Lehmann
M&S „components“
verification and validation
reusability;
distributed execution
correctness;
suitability
improving
EFFICENCY
increasing
CREDIBILITY
Outline
1. M&S: A Multiple-Phase Development & Application
2. Improving M&S-Efficiency by Component-Based M&S
3. Increasing Credibility through M&S-Verification &
Validation
4. Conclusion
Axel Lehmann
1. M&S – A Multiple-Phase Development and
Application Process

Example: Effectivness and efficiency of a „Booking System“
(e.g. client-server architecture)

Analyses goals e.g.:
→ processing time per transaction
→ client/server utilization
→ queueing

How to approach this (complex) problem?
⇒ “Divide and Conquer”:
⇒ A Multiple-Phase M&S Process
Axel Lehmann
M&S – Development Team (Skills)
Customer
User
Domain Expert
Modeller
Project Manager
(Contractor)
Problem
Definition
User
Knowledge
System
Analysis
Domain
Knowledge
Model
Formalization
Modeling
Expertise
Implementation
Experimental
Design and
Analysis
Experimentation
(HW-)SWExpertise
Programmer
Axel Lehmann
Phases & Products in the M&SDevelopment Process
Input:
Examination Aim
System Observations
Conceptual req.&constr.
Phase:
Intermediate Products:
Problem
Definition
Structured
Problem
Description
,
System
Analysis
Communicative
Conceptual Model
Project
Objectives
Model
Documentation
“Hauptobjekt” mit
Modellattributen
Systemgrenze
Objekt
Objekt
Int
era
Modeling Method
Formal req.&constr.
Solution Techniques
Technical req.&constr.
Model Input Data
Experimental req.&constr.
Axel Lehmann
Model
Formalization
Formal
Model
Implementation
Executable
Model
Experimentation
Model
Results
n
ktio
Objekt
Objekt
“Umweltobjekt“
Example: „Booking System“

Conceptual Model:
I/O device 1
T1
•
•
•
•
•
•
CPU
Tn
Clients
Axel Lehmann
I/O device n
Server

Formal Model:
peripheral devices
Axel Lehmann

Executable model: Performance measures
Little´s law:
k =λ +t
;
with: response time t, queueing time w, service rate µ
1
t = w + µ ;
State probability p(k):

k = Σ k • p (k)
Utilization ρ ( m service stations):
ρ
=
arrival rate
service rate
Axel Lehmann
=
λ
mμ
<1
2. Model “Component”-Based Approach:
(e.g. regarding the example „Booking System“)

Axel Lehmann
Problem Description:
→ pragmatism/ goal specification
Conceptual Model:
→ structural & functional description of “components”
→ different levels of abstraction
Formal Model:
→ formal specification of “components”
(↔ selected modeling paradigm(s))
→ hierarchical modeling approach
(↔ decomposition into submodels/ “components”)
Executable Model(s):
e.g.
→ analytic solution
→ simulation
reusable SW- “components“
Phases & Products in the M&SDevelopment Process
Input:
Examination Aim
System Observations
Conceptual req.&constr.
Phase:
Intermediate Products:
Problem
Definition
Structured
Problem
Description
,
System
Analysis
M&S„components“:
Project
Objectives
Communicative
Conceptual Model
Model
Documentation
Modellattributen
Systemgrenze
Objekt
Objekt
Int
Model
Formalization
Formal
Model
Solution Techniques
Technical req.&constr.
Implementation
Executable
Model
Model Input Data
Experimental req.&constr.
Experimentation
Model
Results
Modeling Method
Formal req.&constr.
Axel Lehmann
n
ito
erak
Objekt
Objekt
“Umweltobjekt“
X
X X
X
X
Model („Component“) Specification Levels

Model
repository
Model Federation Level:
„Black Boxes“

Model Level:
Autonomous, interoperable models

Submodel/ Object Level:
Submodels/ Object structures

Function Level:
Library of submodels
& communication
infrastructure
Library of
objects/methods
(for interaction)
Program Library
Coded basic functions/ algorithms
⎧l (t ) − f si ⋅ rs ⋅ ( ti +1 − ti ) + z s (ti , ti +1 ) : wenn l s (ti +1 ) > 0
∀s.l s (ti+1 ) = ⎨ s i
: sonst
⎩0
mit 0 < i ≤ n und 0 < s ≤ 6
Axel Lehmann
3. Increasing M&S-Credibility
M&SCredibility
Perceived
suitability
Perceived
correctness
Verification
Validation
Suitability
Correctness
(requirements, constraints)
(specification Æ Implementation)
Capability
Fidelity
Accuracy
Consistency
Completeness
(Æ Formalisms, specifications,
transformation rules)
Axel Lehmann
Influences:
V&V Framework
Risk
System
knowledge
Model knowledge
V&V
Results:
Axel Lehmann
Credibility
indicators
Multi-phase M&S-process →
intermediate products
Problem
definition
System
analysis
Structured problem
description
,
Conceptual
model
Project
Objectives
Model
Documentation
Modellattributen
Systemgrenze
Objekt
Objekt
Int
Axel Lehmann
ktion
er a
Objekt
Objekt
“Umweltobjekt“
Formalization
Formal
model
Implementation
Executable
model
Experimentation
Simulation
results
Interpretation
Consistency of transformation
(pairwise comparison)
Internal consistency
& completeness
Our Approach: „The V&V Triangle“
Model
Results
V&V Phase
nt
e
pm
o
l
ve
Formal
e
D
Model
l
e
d
o
Conceptual
M
Intermediate Results
Executable
Model
Model
“UmweltModellattributen
objekt“
Systemgrenze
Objekt
Objekt
“Umwelt“ Umwel tob jekt “
“Ha upt ob e
j kt ” m it
Mod el at tr ib ut en
Sy st em g r e nze
O bj ekt
O bj ekt
ob e
j kt “
O bj ekt
Sy st em g r e nze
O bj ekt
O bj ekt
on
O bj ekt
Ok, we have got the
following problem...
?
?
3.3
3.4
?
Ok, we have got the
?
4.3
“ Umwel tob e
j kt “
“Ha upt ob jekt ” mit
M od ell at tr ib ut en
Sy stem g r e nze
O bj ekt
O bj ekt
O bj ekt
O bj ekt
?
4.4
4.5
?
n
“ Ha upt ob e
j kt ” m ti
Mod ell at tr b
i ut en
“ Umwel t-
Ok, we have got the
O bj ekt
ti
o
? 2.3
ti
? 1.2
t erak
I
n
Ok, we have got the
i on
rakt
Ok, we have got the
O bj ekt
ak
te
In
1.1
O bj ekt
O bj ekt
O bj ekt
4.2
nter
Mod ell at tr b
i ut en
O bj ekt
O bj ekt
“ Ha upt ob jekt ” m it
Mod ell at rt b
i ut en
Sy st em g r e nze
?
I
“ Ha upt ob e
j kt ” m ti
Sy st em g r e nze
3.2
“ Umwel -t
ob jekt “
n
?
? 2.2
O bj ekt
?
ti
o
O bj ekt
on
O bj ekt
3.1
er ak
I nt
i
rakt
2.1
“ Umwel tob e
j kt “
Sponsor Needs
Axel Lehmann
Objekt
I te
n
Structured
Problem
Description
ktio n
e ra
I nt
Objekt
?
?
4.1
Ok, we have got the
?
5.1
5.2
5.3
5.4
5.5
5.6
Model Development & V&V
V&V Phase
Intermediate Results
p
o
l
Formal
ve
n
e
m
t
Executable
Model
e Model
D
l
e
d
o
Model
MConceptual
“UmweltModellattributen
objekt“ Systemgrenze
Objekt
Objekt
Sponsor Needs
Ok, we have got the
Axel Lehmann
n
t
kio
ra
nt e
I
Structured
Problem
Description
Objekt
Model
Requirements
Validation
Static
Model
Structure
Verification &
Validation
P(EP)
P(EA)
Objekt
Model
Results
Formal
Model
Verification &
Validation
P(EF)
Static
Code & HW
Verification &
Validation
P(EI)
Dynamic
Simulation
Model
Behavior
Verification &
Validation
P(EE)
V&V - Process „Tailoring“
Model
Results
nt
e
m
p
o
l
e
v
e
D
l
de
o
M
Executable
Model
Formal
Model
Conceptual
Model
I
t
“Umweltobjekt“ SystemgrenzeModellattributen
Objekt
Objekt
“Umweltnte
kt o
a
Objekt
Objekt
“Ha upt ob e
j kt ” m ti
Mod el at tr b
i ut en
“Umwel tob e
j kt “
Sy stem g r e nze
O bj ekt
O bj ekt
rak
ti on
1.1
Ok, we have got the
? 2.3
“ Ha upt ob jekt ” m ti
Mod ell at tr ib ut en
“Umwel tob e
j kt “
Sy st em g r e nze
O bj ekt
O bj ekt
ti
O bj ekt
Ok, we have got the
O bj ekt
on
? 1.2
t erak
I
n
Ok, we have got the
O bj ekt
Ok, we have got the
O bj ekt
O bj ekt
?
?
3.3
? 3.4
O bj ekt
O bj ekt
Ok, we have got the
?
4.3
“Umwel tob jekt “
“ Ha upt ob e
j kt ” mit
Mod ell at tr b
i ut en
Sy st em g r e nze
O bj ekt
O bj ekt
O bj ekt
O bj ekt
?
4.4
4.5
?
n
te
In
O bj ekt
Sy st em g r e nze
ti
o
O bj ekt
O bj ekt
ob e
j kt “
er ak
nt
Sy st em g r e nze
“ Ha upt ob e
j kt ” m ti
M od ell at tr ib ut en
“Umwel t-
3.2
?
4.2
I
on
?
“Ha upt ob e
j kt ” m ti
Mod el at tr ib ut en
“ Umwel t-
?
ti n
o
O bj ekt
er ak
I nt
kti
ra
O bj ekt
? 2.2
ob jekt “
Sponsor Needs
Axel Lehmann
2.1
I te
n
Structured
Problem
Description
?
3.1
?
4.1
Ok, we have got the
?
5.1
5.2
5.3
5.4
5.5
5.6
Our Contribution to International
Efforts on V&V:

⇒
Axel Lehmann
ITOP:
International Test Operations Procedure
(Working Group of Experts WGE 7.2: V&V)
NMSG:
Nato Modeling and Simulation Group
(019 Task Group 016: “V&V of Federations”)
SISO PDG: SISO Product Development Group:
(V&V of Federations)
V&V – CONVENTION
(October 20-24, Munich-Germany)
http://www.itis-ev.de/megameeting
4. Conclusions

Summary:
How to improve M&S-credibility and efficiency for
increasingly complex simulation applications?

Axel Lehmann
Strategic approach: „Divide and Conquer“ !
→ M&S as a Multi-Phase Engineering Process:
¾ Efficient M&S-design by component-based modeling
¾ Credibility assurance by risk-based V&V
Focus of our research efforts
Axel Lehmann

Model engineering process:
¾ Re-useability of model components
¾ Tailored model documentation
¾ (Risk analysis) ← tailored V&V (&A)

M&S performance & reliability:
¾ Parallel and distributed M&S
¾ (Collaborative modeling)
Thank you very much
for your interest
and attention!
I
t
Axel Lehmann
M&S – Source of Knowledge and Expertise
UserKnowledge
Problem
Definition
Domain
Knowledge
System
Analysis
Model
Formalization
Modeling
Expertise
Implementation
(HW-)SWExpertise
Experimentation
Experimental
Design and
Analysis
Axel Lehmann
Example:
„Booking System“
Formal Model (Version 1): queuing net
Axel Lehmann

Verification, Validation and Accreditation of Simulation Models and

Transcription

Documents pareils

JAVA D`ALFRA