combustion

HPC and combustion
Thierry POINSOT
IMFT, Université de Toulouse, CNRS and CERFACS
Contributions from G. Staffelbach, L. Gicquel, B. Cuenot, O. Vermorel, F. Duchaine,
E. Riber (CERFACS), L. Selle (IMFT), V. Moureau (CORIA), S. Mendez (UM2)
1
OUTLINE
• COMBUSTION RESEARCH AND HPC
• PERFORMING HPC COMBUSTION SIMULATION ON TIER0/
TIER1 SYSTEMS FOR INDUSTRY:
! HOW CAN THIS BE DONE ? WHAT DID THE FRENCH
COMMUNITY DO ?
! WHAT ARE THE WEAKEST PARTS OF OUR
ORGANIZATION TO DO THIS ?
2
COMBUSTION OVERVIEW
Just remember two equations:
ENERGY ON EARTH TODAY =
COMBUSTION
3
ENERGY ON EARTH = COMBUSTION
COMBUSTION IS PRODUCING MORE THAN 90 PERCENT OF THE
ENERGY TODAY. THIS WILL DECREASE... BUT NOT TOMORROW
4
COMBUSTION OVERVIEW
Just remember two equations:
ENERGY ON EARTH TODAY =
COMBUSTION
ENERGY ON EARTH TOMORROW =
COMBUSTION
5
CLIMATE CHANGE AND
ENERGY MARKET: 2010/2030
•COMBUSTION SCIENCE MUST ALLOW THIS
WITHOUT INCREASING EMISSIONS,
WASTING FOSSIL FUELS OR MAKING
CLIMATE CHANGE WORSE (!...)
OPTIMIZATION AND
SIMULATION NEEDED
6
SIMULATION OF COMBUSTION:
MULTISCALE -MULTIPHYSICS
Heat transfer, radiation, flow, turbulence,
chemistry, fatigue, vibration, acoustics
Engines!
Contrails
7
COMBUSTION: MULTISCALE MULTIPHYSICS
Within the combustion chamber:
nanoseconds and nanometers
1 cm
1 cm
Fields of density in a H2-O2 engine
8
COMBUSTION: MULTISCALE MULTIPHYSICS
Outside the engine: miles and days !
2 miles
9
WHICH EQUATIONS ?
•The Navier Stokes equations (5 + N
unknowns: density, velocities and energy, N
species). Partial differential equations ->
non local, intense communication required
•Kinetics : N = 10 to 300 species reacting
through 3000 reactions (everything local)
•Heat transfer through the walls, radiation,
noise, soot
•All these flows are turbulent
10
WHAT DO WE COMPUTE ?
•FINITE VOLUME CODES USING DOMAIN
DECOMPOSITION AND MPI.
•Typically 100 Mcells with 100 variables at
each cell (3 velocities, density, energy + 5
to 90 species) over 1000000 time steps. ->
10^16 unknowns
11
WHAT ABOUT PARALLELISM ?
•THE NAVIER STOKES EQUATIONS ARE
‘EMBARRASINGLY DIFFICULT’ TO
PARALLELIZE -> THIS PROBLEM HAS
BEEN IDENTIFIED AND IS REMODELING
OUR COMMUNITY IN LARGER
COLLABORATIVE TEAMS.
•THIS IS NOT A ‘ONE-PROFESSOR ONECODE’ SHOW ANY MORE BECAUSE THE
CODES ARE USED FOR INDUSTRY
APPLICATIONS ON A DAILY BASIS
12
ok
!"##$!!%%
http://success.coria-cfd.fr
A joint initiative of French labs for the promotion of SUper-Computing for the modeling of
Combustion, mixing and complex fluids in rEal SyStems.
The SUCCESS scientific group
SUCCESS was created in 2012 to help the promotion of super-computing in the
area of Computational Fluid Dynamics (CFD) for complex geometries. It is
coordinated by the CORIA lab and is composed of 8 French public labs
Our objectives
Distribute in the labs research HPC codes
for CFD in complex geometries
Ensure the training of users
Manage the development roadmap
Share databases of high-resolution
simulations
Promote super-computing
Some facts
8 French public labs
Around 120 researchers and students
2 PRACE proposals accepted over the
recent years
Several prizes related to !"##$!!
codes: Bull-Joseph Fourier prize, IBM
faculty award, ...
The codes
A massively-parallel finite-volume and finiteelement 3D code for the simulation of
compressible turbulent reactive and two-phase
flows.
A massively-parallel finite-volume 3D code for
the simulation of turbulent reactive and twophase flows at low-Mach number.
The labs
CNRS labs: CORIA, EM2C, I3M, LEGI,
IMFT, LMA
EPIC labs: CERFACS, IFP-EN
13
AVBP Strong scaling examples
65536
(1)
ANL INTREPID, Bluegene P
PRACE/TGCC, CURIE, BullX
57344
(2)
(3)
GENCI/CINES, JADE, SGI Altix ICE
(4)
PRACE/JSC, JUQUEEN, Bluegene Q
equivalent performance
49152
INCITE/ARNL, INTREPID, Bluegene P
HLRS/PRACE, HERMIT, CRAY XE6
Ideal
40960
(5)
(5)
1 billion cells (BG/Q)
(1)
(2)
(3)
(4)
(5)
32768
93M Tetrahedra case - 1 step Chemistry - 2 tasks per node
200M Tetrahedra case - 2 step Chemistry
29M Tetrahedra case - 7 step Chemistry
75M Tetrahedra case - No chemistry - 64 tasks per nod
75M Tetrahedra case - No chemistry - 4 tasks per node
24576
16384
8192
0
0
8192
16384
24576
32768
cores
40960
49152
57344
65536
14
DNS analysis of a Re = 40,000 swirl burner
V. Moureau, P. Domingo, L.Vervisch, D.Veynante
YALES2 scale-up on Babel @ IDRIS (Blue Gene/P)
Up to 12288 cores and 2.6 billion tetrahedrons
linear
YALES2
12288
12288
10240
10240
Scale-up
2.6B tets
8192
8192
2008
6144
6144
4096
2048
0
4096
329M tets
878M tets
2048
41M tets
14M tets
0
2048
4096
6144
8192
10240
12288
0
Number of cores
Juelich Workshop 2010
YALES2 solver
(CORIA Rouen)
ORGANIZATION OF THE FRENCH
NUMERICAL COMBUSTION COMMUNITY
SNECMA
SNECMA DMS
TURBOMECA
RHODIA
ALSTOM
AIR LIQUIDE
HERAKLES
AIRBUS
SIEMENS
ANSALDO
16
G6S'564'58+*Q
!"#$%&$'()'(*+,
-./!0%&/'(+1,
2"3-%&-+1456)+%7658*,
!/792%&"65*+8))+,
$:"%&"(18;<,
$:%281=<'>+1
$?2@$2
"0A/!A
.@2/0
:18>B%76(
-'=+
"'=+)*
CERFACS
Institut Français du Pétrole
!"#$%&'()*()+,#-./#0)-1(23'(4567(89&:;<
@++=*
-'=+
:*+5*Q
C@2-"0%&D8))65';<+,
$:/E."2-0%&E'5=+*,
C@2-"0%&D+51'1,
C7C%&E'5=+6(F,
/2@0:G$
7C0
0GC$."
0@C0GA.
C!2"2@C
#2//0/!
0!/%G!H:!A2
I0J%A2%#/0@-2
K
C$0@#./A
=>?=@!
A4+,)--'B
17
-'LM(4+5%;'LM618+*%N
-'LM(O1P%;+14+5*Q%-20R%E65;+)'16R%I2@-!
=>@!C!D@=>E(@F(GFCH(G=@I(@I!(JD4(@FF1
GFCH=>E(6F@I(=>(7C4?!(4>K(=>?=@!(7CFL!?@D
!"#$%&'()$*+,-.*/$01#2)&34$#5$67*6!+$85$9:;<
MN
THIS ORGANIZATION ALLOWS TO DO OTHER
THINGS IN OTHER FIELDS:
S. Mendez, F. Nicoud
YALES 2
Université de Montpellier
19
TWO TYPICAL PRACE EXAMPLES
• PRECCINSTA: A laboratory burner using a Turbomeca
injection system. Developed in EC project PRECCINSTA and
used as the classical benchmark for LES codes in the world.
Most important problems in the GT industry can be
reproduced on the PRECCINSTA burner. CH4 + Air at 1 bar.
• MASCOTTE: a high pressure experiment corresponding to
rocket combustion (60 to 100 bars). Very powerful flames:
H2/O2. Very extreme conditions (flames of the order or 10
microns)
20
PRECCINTSA (2005): THE FIRST LES OF SWIRLED BURNERS
3 millions cells
21
Roux, Lartigue, Poinsot, Meier and Bérat Comb and Flame 141, 2005
COLD FLOW - PRECCINSTA
22
Comparison of mean velocity fields
All LES are compared with measurements DLR (LDA):
• Velocity profiles are compared at five stations along the burner.
• Comparison for axial, tangential and radial velocities (mean and RMS)
X= 1,5 mm
W
X= 5 mm
X= 15 mm
X= 25 mm
X= 35 mm
FROM
SWIRLER
23
TO OUTLET
U
Distance from axis [mm]
RMS velocity profiles: Stanford code (red), CERFACS code (black)
Uxp Profiles: Red solid: CDP - Black solid: TTGC - Black dotted: TTGC_SSS - Circles: Exp.
and DLR experiments (symbols)
40
40
40
40
40
20
20
20
20
20
0
0
0
0
0
-20
-20
-20
-20
-20
-40
-40
-40
-40
-40
0
5
10
15
x=1.5 mm
24
0
5
10
15
x=5 mm
0
5
10
15
x=15 mm
0
5
10
15
x=25 mm
0
5
10
15
x=35 mm
PRECCINSTA:
•First computed with LES in 2004: 1
Million cells (AVBP)
•Repeated in 2007 with 10 Mcells (AVBP)
•In 2009, repeated with 100 Mcells, 500
Mcells (YALES)
•In 2010 and 2011, on PRACE machines:
JUGENE: 2 billion and CURIE 12 billion
cells (YALES)
25
=84>%?84%'#5$#@$(#1'&)4$>485A$3B)$C$&183)18#5$85$
D,+**67/!.$EF1$G#>1)%>H$*I,6.J
3T
DLR PRECCINSTA BURNER, Experiments: W. Meier et al., Combust. Flame, 150(1/2):2–26, 2007
More details in:
Roux et al, Combustion and Flame (2005)
Moureau et al, Journal of Computational Physics (2007) (2 papers)
Galpin et al, Combustion and Flame (2008)
Moureau et al, Combustion and Flame (2011)
Franzelli et al, Combustion and Flame (2011)
3U
IS THIS ENOUGH ?:
•At 20 billion cells, we are almost reaching
what we need in terms of resolution for the
large flow structures but near walls and
within the flame front, this is not enough
• This is a single burner at atmospheric
pressure with gaseous fuel (no liquid
phase). Real combustors have 16 to 24
burners, working at 20 to 100 bars, with
liquid fuels.
•We still miss a 10^6 factor in power
28
ROCKET COMBUSTION:
• PRACE Project by EM2C and CERFACS
• ROCKET COMBUSTION IS A VERY SPECIFIC AND NARROW
FIELD: IT IS ALSO DIFFICULT SINCE FLUIDS IN THESE
ENGINES ARE IN SUPERCRITICAL CONDITIONS (100 bars)
• CERFACS AND EM2C HAVE DEVELOPED SUPERCRITICAL
CAPABILITIES IN AVBP IN THE LAST FIVE YEARS
• PRACE WORK PERFORMED IN 2012: ACOUSTIC
EXCITATION OF SUPERCRITICAL FLAMES IN THE
CONFIGURATION MASCOTTE OF ONERA
29
!
!
T. Schmitt, H. Layal, M. Boileau, S. Ducruix, S.Candel (EM2C),
A. Ruiz, G. Staffelbach, B. Cuenot and T. Poinsot (CERFACS)
Large-Eddy Simulation of high-frequency instabilities under
transcritical conditions
Objective: Observe and understand the flame behavior in transcritical flows of
an experimental setup submitted to artificial acoustic transverse perturbation.
8.5M hours at TGCC
!
CEA - CERFACS
30
WITH FORCING:
CEA - CERFACS
31
WHAT IS GOING ON IN THE USA IN THE
FUELD OF COMBUSTION AND HPC ?
In the last 15 years: the ASCI projects. Most ASCI
projects in the USA were actually combustion
projects: gas turbines, scramjets, rockets, fires,
galaxies...
Now: the CODESIGN Centers.
Example: Sandia CODESIGN Center. Main idea:
«Compute combustors before you build them»
32
FOR THE FUTURE IN EUROPE
• EESI and PRACE have opened the path for a fast HPC
evolution in Europe.
• Machines are already here (Tier1 and Tier0).
• The main questions today are:
(1) the codes: see GIS SUCCESS lead by CNRS
(2) the money: from industry and the EC
(3) the users !
34
Money: the ERC (European Research Council)
INTECOCIS advanced grant at Institut de
Mécanique des Fluides and CERFACS 2013-2018
(intecocis.inp-toulouse.fr)
• Five year, 2.5 Meuros project on HPC tools for combustion
instabilities. Coordinator: IMFT
• Ten researchers on numerical combustion, 5 years
• Collaboration with GENCI, SAFRAN, ANSALDO, SIEMENS
35
USERS / FORMATION: ‘ignored’ problem ?
The last ten years using AVBP at CERFACS and the
last two years supporting an explosive
development of AVBP and YALES at SAFRAN have
shown that building an efficient HPC CFD team in
combustion required multiple experts:
CERFACS
Combustion experts
- know the physics
- write the models
- use the code but
do not address HPC
HPC experts
- develop the code
- port it on new
machines
- do the first demos
36
CERFACS
Combustion experts
- know the physics
- write the models
- use the code but
do not address HPC
Students:
- know nothing
Industry experts:
- know combustors
- know ‘old style’
CFD and learn HPC
HPC experts
- develop the code
- port it on PRACE/
INCITE machines
- do the first demos
Combustion labs:
- know theory and
experiments
- dont know the
code details or HPC
Sous traitants:
- know a little bit on CFD
and the combustors
37
CERFACS today
Interface / formation
- interfaces to
access the code
- formation to learn
the basics
Students:
- know nothing
Combustion experts
- know the physics
- write the models
- use the code but
do not address HPC
Industry experts:
- know the engine
- know ‘old style’
CFD and learn HPC
HPC experts
- develop the code
- port it on PRACE
machines
- do the first demos
Combustion labs:
- know theory and
experiments
- dont know the
code details or HPC
Sous traitants:
- know a little bit on
CFD and the engine
38
INTERFACES:
DISTRIBUTING HPC CODES TO INDUSTRY
Mmmh...
C’est
bientôt
fini?
La taille de ce tube est le vrai
point critique aujourd’hui
A . Dauptain
Copyright
39
DISTRIBUTING LES CODES:
0
0
• Codes CANNOT be used in industry without additional
interfaces (it works with labs but not with industry).
• These interfaces must integrate a very large set of
information coming from industrial needs
• The time required to write these interfaces is VERY
large: the codes MUST be modified and simplified to
adjust to the interfaces
40
INTERFACES: THE C3S EXAMPLE
• In the last 4 years, CERFACS has
developed an interface called
C3S for AVBP users in labs and
industry
0
0
• C3S installed and working at CERFACS, IMFT, EM2C,
IFPEN, SNECMA, Villaroche, SNECMA Vernon,
TURBOMECA Bordes, etc
41
EXAMPLE :
42
BUT THIS IS NOT ENOUGH
!CREATING ONE INTERFACE FOR ONE CODE IS NOT
ENOUGH:
• Keeping up with AVBP developments in C3S is
difficult:-> must coordinate code and interface
evolutions. The interface MUST evolve with the
code (or vice versa ?)
• If you write a piece of code, you should also, at the
same time, write the interface
!In 2011: a new initiative -> C3Sm (A. Dauptain)
43
C3SM
• Is an engine... which writes interfaces:
!developed by CERFACS
!used by all french groups developing AVBP or any
other code linked to AVBP, to automatically
produce and update the interface and the
documentation: YALES, PRISSMA (radiation),
AVTP (heat transfer), N3S (RANS), Coolant
(SAFRAN dedicated software for cooling)...
44
!"#$%&'&(&)*#+,-"+./'%-0)-/0'-&"1#2345
234
5
2*#$%&'&(&)*#%*"+#$&//-:)*#1
•
#)0#(%;0'-&"#+,895#$0%#)*/#+;<*)&$$*.%/#+*#(&+*
5&'*.%
895
•2&"'*".#657
#)0#(%;0'-&"#+*##=&'*.%/#895#$*%>&%=0"'/#*'#(&=$0'-:)*/
?<0"'#1#2&"'%-:.'-&"#=&"&)-'@-A.*
B;<*)&$$*.%/#+*#
(&+*/#/(-*"'-CA.*/
2%;0'*.%/#
+,-"'*%>0(*/
!"#$%
&'%
!'-)-/0'*.%/#+*#(&+*/#
/(-*"'-CA.*/
7*#(%;0'*.%#+,-"'*%>0(*/#*/'#0.#=-)-*.#+.#+-/$&/-'->D#-)#*/'#
-"(&"'&.%"0:)*D#*'#(%;*#+*#),0''*"'*E
?$%F/#1#2&"'%-:.'-&"#%;$0%'-*
2%;0'*.%/#
+,-"'*%>0(*/
&()*"&%
B;<*)&$$*.%/#+*#
(&+*/#/(-*"'-CA.*/
234
5
!'-)-/0'*.%/#+*#(&+*/#
/(-*"'-CA.*/
7*/#+;<*)&$$*.%/#*'#.'-)-/0'*.%/#$*.<*"'#>0-%*#;<&).*%#)*.%#
&.'-)#(@*G#*.HD#/0"/#$0//*%#$0%#),-"'*%=;+-0-%*#+.#(%;0'*.%#
+,-"'*%>0(*E
WHAT ABOUT FORMATION ?
• PhDs are of course a good solution:
!EM2C, CORIA, IMFT, IFP, CERFACS produce more
than 20 PhD per year. They go to industry and they
know HPC.
!But this is not fast enough
• Systematic courses on HPC/CFD/Combustion are
required: in 2011, formation cycles at CERFACS
48
!"#$%&'()*#$+,-./*0-%*1(%2$*3..4*5&#")(6-7*"/&72*
89:;*
<-.$*'-"=)&72*"/&72*>=$7;81?*
!"#$%%&&&'()*+,(-'+*%.*,/0/01%
%%%%(+.0606?@'*+<.';A<+%
B6% ,"!!C% DEFGHDI% 060J.(*;% .% (+.0606?%
/+&?+.=%<&+%'&=/7(*+%;'0*6'*%(.+?*J6?%
;(7K*6(;C% L>M;C% *6?06**+;% .6K%
+*;*.+'>*+;C% 06% .'.K*=0'% .6K% .//10*K%
+*;*.+'>A%%
N>*% (+.0606?% ;*;;0&6;% .+*% (.7?>(% )8%
DEFGHDI%+*;*.+'>*+;%.6K%*6?06**+;A%%
L+.'J'.1% (+.0606?% ;*;;0&6;% K*K0'.(*K% (&%
(>*% '&K*;% K*O*1&/*K% &+% '&#K*O*1&/*K%
)8% DEFGHDI% .+*% /.+(% &<% (>*% /+&?+.=A%
DEFGHDI% 0;% .7(>&+0P*K% (&% ?0O*% (+.0606?%
;*;;0&6;% Q0(>06% (>*% B6K0O0K7.1% +0?>(% (&%
(+.0606?%RMBGSA%
!"#!$%&'(&)*+%,"!!%
,#-%./+01%,"!,%
%!2#!3%&'(&)*+%,"!!%
!4#,"%5.67.+8%,"!,%
!9#!:%576*%,"!,%
8.@(7'$.*#$/,*2$7$%(6-7*"/&72*<3!A8BC*
,-#,2%5.67.+8%,"!,%
D(+(*(//&#&)(6-7*"/&72*>=$7;81?*
9"%5.67.+8#!;(%<*)+7.+8%
,"!,%
;%-2%(##&72*&7*E-%+%(7*FG***
!9#!$%=.+'>%,"!,%
<-#="+$%*(%',&+$'+"%$/*(7.*-=6#&/(6-7*
,,%=.+'>%,"!,%
;(%())$)*=%-2%(##&72*+--)/*
,9#%,$%./+01%,"!,%
D-H7/'()&72*')&#(+$*/'$7(%&-/*H&+,*+,$*H$(+,$%I+4=&72*
J(/$.*#$+,-.-)-24*D5<1K?*
,:#,-%./+01%,"!,%
5-)"6-7*#$+,-./*0-%*-=6#&/(6-7*=%-J)$#/*
3#!!%=.8%,"!,%
E"7.(#$7+()/*+-*"7.$%/+(7.*(7.*(7()4L$*,&2,*M.$)&+4*
<ED*/&#")(6-7/*
,!#,:%=.8%,"!,%
<-.$*'-"=)&72*"/&72*>85K5*
%9"#9!%=.8#!;(%576*%,"!,%
!"#$%&'()*#$+,-./*0-%*+,$*$)$'+%-#(27$6'*=%-J)$#*
"/&72*<35<*
-#4%%576*%,"!,%
5-)"6-7*#$+,-./*0-%*)&7$(%*/4/+$#/*
!!#!,%576*%,"!,%
*K7+%-."'6-7*+-*.(+(*(//&#&)(6-7*
!4#!3%576*%,"!,%
5-)"6-7/*-0*=(%6()*.&N$%$76()*$O"(6-7/P*(==)&'(6-7/*
+-*+,$*H(@$*=%-=(2(6-7*=%-J)$#*
,:%#%,-%%576*%,"!,%
8.@(7'$.*()2-%&+,#/*0-%*.(+(*(//&#&)(6-7*
,2#,3%576*%,"!,%
49
50
www.cerfacs.fr/elearning/combustion/
51
CONCLUSIONS
• DANS 10 ANS, LES GROUPES QUI AURONT CONTINUE A
DEVELOPPER DES CODES DE PETITE TAILLE EN COMBUSTION,
AURONT BEAUCOUP DE MAL A PUBLIER MAIS L’EFFORT POUR
CONSTRUIRE DES GRANDS CODES EST ENORME
• SUCCESS: REPONSE ‘LOGICIEL’ DES COMBUSTIONNISTES
FRANCAIS. COUVRE LES LABOS ET LES INDUSTRIELS
• L’ARRIVEE DE GENCI A PERMIS A LA COMMUNAUTE
‘COMBUSTION NUMERIQUE’ DE DISPOSER DES MOYENS
NÉCESSAIRES POUR DEMONTRER SON SAVOIR FAIRE
• PRACE ET INCITE COMPLETENT CE DISPOSITIF
• LA COMBUSTION EST UN EXEMPLE OU CES OUTILS HPC SONT
UTILISES AUJOURD’HUI PAR L’INDUSTRIE QUI SUPPORTE LEUR
DEVELOPPEMENT DE FACON TRES FORTE
52