NEMO Monte Carlo Application on the Grid

Consorzio COMETA - Progetto PI2S2
UNIONE EUROPEA
NEMO Monte Carlo Application
on the Grid
R. Calcagno
for the NEMO Collaboration
Grid Open Day at Lns
Catania 2008 January 23
www.consorzio-cometa.it
The NEMO project
R&D for the design and construction of a km3 high energy neutrino detector
Km3 detector
Shore station
Electro-optical cable
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
2
Neutrino astronomy: detector concept
atmospheric
muon
~5000 PMT
Cherenkov
light
neutrino
Detection of Cherenkov
light in transparent natural
media (water or ice)
muon
neutrino
depth
>3000m
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
3
Neutrino detection: signal and background
Search for muon tracks from
neutrino interactions
Earth is used as shield to cut off
the downgoing atmospheric µ
background
Upgoing tracks can
unambigously be attributed to
neutrinos
The predicted neutrino fluxes are
weak and it is necessary to take
into account the expected
background noise.
One of the main sources of
background are the atmospheric
muon generated by the
interaction of cosmic rays with
the atmosphere.
Simulation of this background
needs a huge number of
MonteCarlo events.
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
4
NEMO proposed km3 telescope
km3 size square array of detection
structures ( “towers“ )
secondary JB
“tower”
main Junction Box
main EO cable
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
5
Code Chain
Code chain for atmospheric muon simulation are:






Corsika
Music
Km3
Modk40
Femsim
Reco
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
6
Simulation
• Corsika code simulates the interaction of cosmic rays with the
atmosphere and the propagation of the generated showers.
Muons are propagated up to the sea level.
•
Music simulates the muon propagation in the sea water.
• Km3 simulates the light emission, the propagation in water,and
the detection by the PMTs. In order to simulate the light radiation
is necessary a parameterization which depends on the muon
energy lost in the trace segment and on the optical water
properties. PMT hits are simulated.
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
7
Simulation
• Modk40 adds hits due to optical background that in salt
water is due 40K
• FemSim simulates the working operation of the electronic
devices
• Reco performs the reconstruction of the muon track
direction
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
8
Simulation Codes Structure
Input
CORSIKA
Simulation codes
Output
Input
MUSIC
Output
Input
KM3
Output
Input
MOD40
Output
Input
FEMSIM
Output
Input
Simulation codes are performed
in sequential way: the output of
the first code is the input of the
second one, the output for the
second one is the input of the
third one and so on
Moreover, many files are
necessary to the code chain to
run (detector geometry file,
input parameter files, tables …)
RECO
Output
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
9
Grid Application
Input
CORSIKA
Output
Input
MUSIC
Output
Input
KM3
Output
Input
MOD40
Background Monte Carlo
simulation needs a wide
statistic and big processing
time, therefore GRID is the
ideal tool to simulate this
huge amount of events.
Output
Input
FEMSIM
Output
Input
RECO
Output
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
10
Grid Application
Intput
CORSIKA
Intput
MUSIC
Intput
KM3
Intput
Output
Output
Intput
MOD40
Intput
FEMSIM
Intput
RECO
Output
MOD40
FEMSIM
Intput
RECO
MUSIC
Output
...
Intput
KM3
Output
Intput
MOD40
Output
Intput
FEMSIM
Output
Output
Output
Intput
Intput
Output
Output
Intput
MUSIC
KM3
CORSIKA
Output
Output
Output
Intput
Intput
Output
Output
Intput
CORSIKA
Intput
Output
RECO
Output
The developed application is a trivial parallel execution of
simulation codes on the Grid.
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
11
Implementation
• The first of Code Chain is Corsika
QGSDAT01
Inputs
EGSDAT4_3.
corsika_flat.exe
EGSDAT4_.15
CORSIKA
EGSDAT4_.4
Input file size < 10MB
esegui_100.sh
EGSDAT4_3.
EGSDAT4_.05
inputs_v2_100
EGSDAT4_1.
EGSDAT4_25.
Output
NUCNUCCS
...
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
12
Upload on SE
SE
 A virtual directory is created
on the storage element (SE)
CORSIKA
CORSIKA
CORSIKA
WN
 The upload of Corsika file is
performed on a SE and is
recorded on the LFC
Catalog
 The submitted job transfers
the input files from the SE
to the WN
 The output file is
downloaded
Output
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
13
Upload on SE: User
Catalog File
SE
SE
UI
SE
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
14
Input for Music
Output
( ~200 MB zipped file )
music-eloss-seawater.dat
music-double-diff-seawater.dat
MUSIC
TestSite.dat
music-double-diff-water.dat
trasf_out_wn_100.sh
music-cross-sections-seawater.dat
Input file size > 10MB
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
15
I/O Upload on the SE
INPUT >10M
SE
OUTPUT
CORSIKA
...
 The output Corsika and
the Input file are uploaded
on a SE and are recorded
in the LFC Catalog
 The job is submitted
 The output file is
downloaded
...
MUSIC
CORSIKA
WN
WN
Output
...
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
16
Replicate file
• A file can be stored on multiple SE.
SE
SE
INPUT >10M
INPUT
(OUTPUT_
_CORSIKA)
SE
SE
lcg-rep –-vo VO –d addressSE pathLFN
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
17
Job Description Language (JDL)
The jdl is a high-level, user-oriented language for describing jobs and
aggregates of jobs such as Direct Acyclic Graphs (DAG)
A jdl file describes a job that can be run on the Grid
UI
Input
Sandbox
Mass
Storage
RB
Output
Sandbox
SE
CE
Job execution
WN
WN
WN
WN
WN
WN
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
18
InputSandbox
The input file is transferred
from the local UI to the WN
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
19
OutputSandbox
The output for downloading
from the WN to the UI
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
20
Requirements
Job requirements on resourse (CE)
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
21
Job List Match
• This list shows the Computing Element that matching
our job requirements.
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
22
Polling
submit
status
no success
sleep 90 s
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
23
Polling
submit
status
success
output
no success
sleep 900 s
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
24
Input File
• The input for Corsika will be different for every chain performed in
parallel way, changing the SEED parameter.
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
25
Step
User chooses:
• the number of job to execute
• a seed value as reference
To set up:
• n files .jdl to submit on Corsika
• n files .jdl to submit on Music
• n files .jdl to submit on Km3
• … … …
• n files .jdl to submit on Reco
To observe
the sequence
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
26
Trivial Parallel Execution
unipa-ce-01.pa.pi2s2.it:2119
WN
WN
RB
CE
MUSIC-1
WN
WN
WN
WN
unime-ce-01.me.pi2s2.it:2119
infnlns-ce-01.ct.pi2s2.it:2119
CORSIKA-1
WN
WN
Every jdl file
works in independent
way the one from the
other on the destination
WN chosen by the CE
WN
CE
WN
CE
WN
WN
WN
WN
WN
WN
CORSIKA-2
WN
WN
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
27
GRID implementation
Output
jdl 1
IN File 1
EXE File
WN
Output
jdl 1
...
jdl 2
IN File 1
EXE File
WN
Output
jdl 2
IN File 1
EXE File
WN
Output
jdl 2
...
...
...
...
...
WN
Output
jdl N
IN File 1
EXE File
WN
Output
jdl N
...
WN
...
IN File 1
EXE File
...
Expert
jdl 1
jdl N
IN File 1
EXE File
CORSIKA
MUSIC
...
KM3
submit status get-output execute
submit status get-output execute
CONTROLLER
CONTROLLER
The codes need the use of libraries (CERN libraries, NAG libraries…)
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
28
Why Grid ?
• Many CPU
• To ask specific requirements
• Many SE
• Redundance of the systems
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
29
Any Questions ?
Grid Open Day at Laboratori Nazionali del Sud, 2008 January 23
30