5-CABOT_2005

www.enea.it
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
26 April 2005 Johon Cabot University
GRID Computing: esperienze realizzative ed elementi architetturali.
GRID COMPUTING
Real Case GRID At ENEA
Ing. S. Migliori
ENEA INFO [Servizio Centralizzato Informatica e Reti], Roma
Outline
• ENEA-GRID Infrastructure and Resources
• Case studies: numerical simulations and support to
experimental facilities
• Connection with other GRID projects
• GRID paradigm & ENEA-GRID
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Ian Foster
The Grid: The Web on Steroids
http://
Web: Uniform
access to HTML
documents
http://
Software
catalogs
Grid: Flexible,
high-perf access
to all significant
resources
Computers
Sensor
nets
Colleagues
Data archives
On-demand creation of powerful virtual computing systems
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
TELEMICROSCOPY & GRID - BASED COMPUTING
REMOTE ACCESS FOR DATA ACQUISITION AND ANALYSIS
DATA ACQUISITION
DATA ANALYSIS
ADVANCED
COMPUTER
GRAPHICS
By
Mark Ellisman
NETWORK
Cell Centered Data Base
“CCDB”
QuickTi me™ and a YU V420 codec decompressor are needed to see thi s pi ctur e.
IMAGING
INSTRUMENTS
GRID computing Frascati 14/10/2004
COMPUTATIONAL
RESOURCES
MULTISCALE
DATABASES
[email protected] www.telegrid.enea.it
Architecture --- > Function
Application
User
Collective
Software
catalogs
?
Computers
Resource
Connectivity
Colleagues
Sen
sor
net
s
Data archives
Fabric
Le Funzioni / Esigenze comandano sulla tecnologia
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Ian Foster
Network Exponentials
• Network vs. computer
performance
– Computer speed doubles
every 18 months
– Network speed doubles
every 9 months
– Difference = order of
magnitude per 5 years
• 1986 to 2000
– Computers: x 500
– Networks: x 340,000
• 2001 to 2010
GRID computing Frascati 14/10/2004
[email protected]
Moore’s Law vs. storage improvements vs. optical improvements.
Graphwww.telegrid.enea.it
from Scientific American (Jan2001) by Cleo Vilett, source Vined Khoslan, Kleiner, Caufield and Perkins.
– Computers: x 60
Jac. Dongarra SC2002
Performance Extrapolation
10 PFlop/s
1 PFlop/s
100 TFlop/s
10 TFlop/s Sum
1 TFlop/s
100 GFlop/s
N=1
TFlop/s
To enter
the list
10 GFlop/s
1 GFlop/s
N=500
gi
u9
gi 3
u94
gi
u9
gi 5
u9
gi 6
u9
gi 7
u9
gi 8
u9
gi 9
u0
gi 0
u01
gi
u0
gi 2
u0
gi 3
u0
gi 4
u0
gi 5
u0
gi 6
u0
gi 7
u0
gi 8
u0
gi 9
u10
100 MFlop/s
PFlop/s
computer
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Ian Foster
Selected Major Grid Projects
URL & Sponsors
Name
Access Grid
www.mcs.anl.gov/FL/
g accessgrid; DOE, NSF
BlueGridNew g IBM
DISCOM
g www.cs.sandia.gov/
discom
DOE Defense Programs
Focus
Create & deploy group collaboration systems
using commodity technologies
Grid testbed linking IBM laboratories
Create operational Grid providing access to
resources at three U.S. DOE weapons
laboratories
DOE Science
sciencegrid.org
g DOE Office of Science
Grid
Create operational Grid providing access to
resources & applications at U.S. DOE science
laboratories & partner universities
Earth System
Grid (ESG)
earthsystemgrid.org
DOE Office of Science
Delivery and analysis of large climate model
datasets for the climate research community
eu-datagrid.org
Create & apply an operational grid for
applications in high energy physics,
environmental science, bioinformatics
New
European
Union (EU)
DataGrid
g
g
European Union
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Ian Foster
Selected Major Grid Projects
Name
TeraGrid
URL/Sponsor
g teragrid.org
New
NSF
Focus
U.S. science infrastructure linking four major
resource sites at 40 Gb/s
UK Grid Support g grid-support.ac.uk
New
Center
U.K. eScience
Support center for Grid projects within the
U.K.
Unicore
Technologies for remote access to
supercomputers
BMBFT
Also many technology R&D projects:
e.g., Condor, NetSolve, Ninf, NWS
See also www.gridforum.org
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA
Italian Agency for New Tecnologies,
Energy and Environment.
12 research centers in Italy
Centro
di Brindisi
A Central Computer and Network
Service (INFO)
6 Computer Centres
Casaccia, Frascati, Bologna,Trisaia,
Portici, Brindisi
Multiplatform resources for serial &
parallel computation and graphical
post-processing.
Others computer resources in ENEA:
departments & individuals
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA Network connection
ENEA computational resources are distributed over
WAN, connected by GARR, the Italian Academic &
Research Network
ENEA-GARR
9 PoP, 4-32 Mbps
Brindisi
Bologna
Casaccia
Frascati
Portici
Trisaia
Palermo
Pisa
Roma Sede
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID
ENEA-GRID has been developed since 1998 and now it
is an integrated infrastructure:
• offering a production quality, service oriented system
• covering most of the computational resources
belonging to ENEA INFO Scientific Computation
Service, connected in a Wide Area Network.
• providing a unified user environment and an
homogenous access method for all ENEA researchers
irrespective of their location
• implementing tools to facilitate the integration of
department and individual resources and support to
experimental facilities
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID Services
Services for research activity in ENEA
• Multiplatform Parallel systems:
• AIX, Linux (Alpha,x86,AMD64), IRIX, [Unicos (Cray)]
• Graphical simulation & code result post-processing:
• SGI + 3D immersive facilities
• Software resources:
• Commercial Codes: Fluent, Gambit, Abacus, Catia, Ansys
• Research codes: mcpn/x, eranos, fluka,…
• Elaboration environments: IDL, Matlab, Mathematica, SAS
• Windows applications
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID Computational resources
OS
#cpu
Gflops
154
1000
Linux x86 32/64
96
200
Frascati(28), Casaccia(54),
Portici(4),Trisaia(4), Brindisi(4)
Linux Alpha
80
100
Casaccia
IRIX
26
40
Frascati(8), Casaccia(4),
Portici(1),Trisaia(8), Brindisi(1),
Bologna(5)
Solaris
8
10
Trisaia(4), Casaccia(2), Bologna(2)
Windows
18
50
Frascati(6), Portici(4),Trisaia(4),
Brindisi(4)
Mac OS X
21
80
Frascati(1), Trisaia(13)
AIX
Frascati(128), Bologna(8), Portici(18)
Portici, Brindisi & Trisaia resources increased in the framework of
TELEGRID Project
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
GEANT
boquad.bologna.enea.it
dafne.casaccia.enea.it
power3.frascati.enea.it
Portici
Kleos.portici.enea.it
new
infocal.trisia.enea.it
Brindisi
Citrix
Nfuse
……...brindisi.enea.it
Citrix
Metaframe
WEB(ICA)
GRID computing Frascati 14/10/2004
ICA
[email protected] www.telegrid.enea.it
ENEA GRID
WEB
ICA
Application Portal (Metaframe/Nfuse)
Monitoring, Automatic recovery
Accaunting
Quality of service
Graphical User Interface
User programs and commercial codes
LSF multi-cluster as integrator
LSF
XTERM
RMS
Load Leveler
AFS Geographical cross platform & File System
AIX
SGI
GRID computing Frascati 14/10/2004
SUN
HP
Win
Linux
Linux
Alpha
[email protected] www.telegrid.enea.it
MAC
Software Conceptual Structure
WEB
NEW
ICA
User programs & commercial code
Graphic User Interface
Telnet
WEB
WEB
LSF
ICA
ICA
User programs & commercial code
Graphic User Interface
LSF multi-cluster as integrator
Cluster 5° ( Portici)
NEW
Graphic User Interface
LSF
Load Leveler
File System
Telnet
LSF
AFS
Cluster 6° ( Brindisi)
WEB
Cluster 2° (Nord Rome)
ICA
WEB
ICA
WEB
User programs & commercial code
Graphic User Interface
LSF multi-cluster as integrator
Load Leveler
Telnet
LSF
AFS
User programs & commercial code
Graphic User Interface
LSF multi-cluster as integrator
Load Leveler
Telnet
LSF
AFS
ICA
User programs & commercial code
Graphic User Interface
LSF multi-cluster as integrator
Load Leveler
Telnet
LSF
AFS
Cluster 1° ( Bologna)
Cluster 3° (Frascati)
Cluster 4° (Trisaia )
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Gestione generale di una specifica richiesta al sistema
Cluster
Configuration
(STATICO)
Stato delle
Risorse
(TEMPO REALE)
Politica
delle code
(STATICO)
•Accoda la richiesta
Interfaccia
Grafica
(RICHIESTA)
LSF
•Assegna la risorsa
•Lancia il comando sul
calcolatore selezionato
Data Base Server
File Server
AFS
DATI
(SW/Prog.)
GRID computing Frascati 14/10/2004
•Risolve la piattaforma
•Verifica i diritti di accesso
•Mantiene la coerenza dei dati
Client AFS
Cache
locale
[email protected]
www.telegrid.enea.it
ENEA GRID Graphics and 3D resources
(1)
SGI systems are coupled with
3D immersive facilities together
with 3D modelling and
rendering software as
CAVELIB, DV-Mockup, AVS
C.R. Casaccia
C.R. Frascati
GRID computing Frascati 14/10/2004
C.R. Bologna
[email protected] www.telegrid.enea.it
ENEA GRID Graphics and 3D resources
(2)
SGI systems are coupled with
3D immersive facilities together
with 3D modelling and
rendering software as
CAVELIB, DV-Mockup, AVS
C.R. Portici
C.R. Brindisi
GRID computing Frascati 14/10/2004
C.R. Trisaia
[email protected] www.telegrid.enea.it
ENEA GRID ARCHITECTURE
A choice of mature components for reliability and easiness of
support and maintenance:
Distributed File System: AFS
Job and resources manager: LSF Multicluster
Unified GUI access: Java and Citrix Technologies
Quality monitoring system: Patrol
Licence Servers
Integration with department and individual resources
Distributed File System: AFS for software and data
Licence pool sharing
Possible integration with other institutions
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
GRID usage, examples and some case studies
650 registered users
2 TB stored software/data out of 4 TB available in enea.it AFS cell
Examples and cases studies
●
Running FLUENT on ENEA GRID
●
Support to department experimental research:
• Biotechnology
• Electron microscopy
• Nuclear fusion research
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
GEANT
boquad.bologna.enea.it
dafne.casaccia.enea.it
power3.frascati.enea.it
Portici
Kleos.portici.enea.it
new
infocal.trisia.enea.it
Brindisi
Citrix
Nfuse
……...brindisi.enea.it
Citrix
Metaframe
WEB(ICA)
GRID computing Frascati 14/10/2004
ICA
[email protected] www.telegrid.enea.it
CITRIX / NFUSE
ENEA GRID makes use of Citrix Metaframe to publish an
application providing a unified GUI interface all the available
resources
Application components:
● Java (GUI)
● shell scripts
● EnginFrame
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
FLUENT (CFD) code batch submission
A FLUENT case can be
run from a dedicated GUI
for submission
LSF Options
-o filename.%J - output file
-w “done(idjob)”
- start after idjob
-u e-mail userid - output by E-mail
-b begintime
- job start time
….
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Monitoring job status by xlsbatch (LSF)
xlsbatch information
Jobid
Users
Job status
Queues
Submission host
Run host
Submission time
Submission command
…..
Job status:
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Monitoring host status by xlsmon (LSF)
Status of sp3-1 host using
xlsmon utility
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
3D CFD Simulation on Aircraft Axial Turbin Stage with Wake &
Vortex Shedding Analysis on Turbine Blades [FLUENT]
Large Eddy Simulation
(LES) provides high
resonable results on wake
analysis and turbine
Pressure traces on stator trailing edge
stage performance.
195000
190000
pressure (Pa)
Max, min and time average Cp for the vane
0.5
0
185000
180000
175000
Cp
-0.5
Temperature contours on
Stator blades & Cp
experimental data comparison
-1
170000
-1.5
165000
isolated stator
-2
-2.5
-0.01
0
0.01
0.02
0.03
stage stator
0.04
x (m)
Secondary flow prevision and
3D Path Lines analysis
Mesh: 1 million cells
CPU Time: 6000 hours
Elapsed Time:
525 hours on
12 IBM SP3 platforms
GRID computing Frascati 14/10/2004
Temperature & wakes
interactions
[email protected] www.telegrid.enea.it
Mesh: 600,000 cells
CPU Time: 960 hours
Elapsed Time: 130 hours
on 8 Linux platforms
CFD Analysis of a fluid dumper [FLUENT]
Secondary flux
Primary Flux
Vortex stagnation
Pressure jump (Pa)
Pressure losses
Serie1
Punto di progetto (apertura
valvola)
3000000
2500000
2000000
1500000
1000000
500000
0
0
0.2
0.4
0.6
0.8
1
1.2
velocity (m/s)
Caratteristica parabolica dello smorzatore con valvola
aperta fino a 3 mm
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Blast Wave Analysis for VEGA Programm
DMA-Uniroma1 – AviogroupRome 2003
4.00E+04
3.50E+04
3.00E+04
2.50E+04
2.00E+04
1.50E+04
1.00E+04
5.00E+03
0.00E+00
0.00E+00
2.00E+01
4.00E+01
6.00E+01
p-patm da 0 a 1.55 s.
50
p-patm
40
Real P-Patm
30
linea-1
20
linea-2
linea-3
10
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5
time (s)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
La simulazione con Cover:
il campo di Temperatura
Grid: 1,5 million cells
CPU Time: 5000hours
Elapsed Time:
672 hours on
6 SGI platforms
(onyx2ced )
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Uno sguardo verso il futuro: la parametrizzazione via Web
V
100.0
Profilo
NACA xxx

4.0
STEPS
Definire la geometria e le condizioni al contorno.
Generare la griglia di calcolo necessaria
Eseguire il calcolo
Risultati del Post-processing (su Web o home/directory)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
FLUENT performance on different platforms in ENEA GRID
#CPU
GHz
Sp3-1
16
.375
Sp4-1
32
1.1
Ostro
16
1.3
Pace
8
1.3
C..03
4
3
bw
8
1.8
8
0.4
AIX
Linux
IRIX
Onyx.
Scaling with #CPU & platform type
GRID computing Frascati 14/10/2004
Cpu time normalized to
the case: Sp4-1, 2 CPUs
[email protected] www.telegrid.enea.it
ENEA GRID and experimental facilities
DNA Sequence
system
( ABI Prism 3700)
Trisaia
DB_2
ENEA GRID
Controlled Nuclear Fusion:
FTU
Frascati Tokamak Upgrade
Video Acquisition
CPUS
DB_1
WEB
ICA
SSH
DB_3
Electronic Microscope
(Brindisi)
300 Kev (sept. 2004)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID: FTU video acquisition data
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Conceptual structure for TEM
Progetto UTS Materiali
Electronic Microscope
(Brindisi)
300 KeV (sept. 2004)
TEM Room
Remote User
Remote Controlled
Web-cam
Mtaframe
Special Driver for TEM
Local
Operation
GRID computing Frascati 14/10/2004
Remote
Local
Operation
AFS
Local Data
AFS Data Integration
> 4 TByte
Local GRID Cluster
[email protected] www.telegrid.enea.it
ENEA GRID/DATAGRID Test case: GOME data analysis
Output File
LIV2
List of path
Input File
LIV1
IDL “GOME”
Program
.jpg
Image
Input File
LIV1
List of
path
Input File
LIV1
Case-1
Case_2
ENEA-Geographic
SUBMISSION PROCEDURE
List of path
Input File
LIV1
IDL Program
“GOME”
GRID computing Frascati 14/10/2004
User afs/home
WEB
Input File
LIV1
Case n
Output File
LIV2
.jpg Image
Input Files
LIV1
[email protected] www.telegrid.enea.it
ENEA GRID/DATAGRID : GOME run cases in ENEA
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID/DATAGRID: GOME results on Web
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA IGNITOR/Pellet Collaboration Activity
ANSALDO
(GE)
CRIOTEC
(TO)
ORNL
(USA)
ENEA GRID
ENEA
(ISPRA)
• DATA sharing
• Applications in ENEA GRID
• Sharing Working Sessions (Citrix)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Collaborazioni
 DMA (
Prof. ENRICO Scubba)
 Fluidodinamica e sviluppo di metodologie
dell’utilizzo di grid …
 Istituto Matemetica Castelnuovo
 (In fase di perfezionamento)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA
Main activities & resources for EGEE
•Integration of EGEE-GRID to ENEA-GRID;
•100 CPUs at 20% of time
ENEA-GRID
GRID.IT
WP13 Chem-Grid
(MAT)
DataGrid
at
EGEE
ESA\ESRIN
GRID
GRID computing Frascati 14/10/2004
S-PACI
[email protected] www.telegrid.enea.it
GRID
CITRIX: ENEA GRID Web Access
http://www.afs.enea.it
Windows services
GRID computing Frascati 14/10/2004
Unix/Linux Services
[email protected] www.telegrid.enea.it
ENEA GRID & GRID PARADIGM
“GRID concept: coordinated resource sharing and problem
solving in dynamic, multi-institutional virtual organizations”
Foster & Kesselman, The Anatomy of the Grid, 2001
ENEA GRID infrastructure has been designed to provide a multisite
resource sharing inside ENEA, both to optimize the utilization of
Central Computer Service (ENEA INFO) resources and to integrate
the other resources available inside the organization.
VO concept can not be fully developed inside an unique
organization but similarities can be found with Project Groups
sharing specific resources and ENEA GRID components can
provide support to this (AFS groups, LSF resources).
There are cases where ENEA GRID infrastructure has been used in
support to the collaboration with other institutions (+ other GRID
projects)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID & GRID PARADIGM
ENEA GRID architecture answers to many of the challenges posed
by the “GRID Problem” : “unique authentication, authorization,
resource access and resource discovery”, Foster & Kesselman 2001
Software
catalogs
Computers
Sensor
nets
Colleagues
Data archives
by Ian Foster
The choice of mature components (LSF & Citrix - proprietary, AFS opensource project since 2000) reduce the generality of ENEA GRID
as a GRID model but improves greatly the reliability and the easiness
of update & management: production quality infrastructure.
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID EXPERIENCE
Some consideration can be drawn from the analysis of the
operation of ENEA GRID infrastructure in the last years.
User acceptance of GRID working mode requires a new
cultural approach to the access to computational
resources and depends strongly on the service quality:
●
“Value” of the accessed resources
●
Efficient networking and system faults monitoring
●
Support of user customization for GRID environment
The administration of a distributed & integrated resource
system is a technical, cultural and organization challenge
in a diversified structure as ENEA.
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ITALIAN NATIONAL AGENCY
FOR NEW TECNOLOGY, ENERGY
AND THE ENVIRONMENT
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID improvements in progress
Increasing resources: AIX +32 SP nodes/Frascati, Cluster
16 AMD 64bit/Casaccia, Cluster 8 G5/Trisaia, 2 CPU
AIX/Brindisi (TELEGRID Project).
Extending Windows services.
AFS optimization.
Studies in collaboration with CASPUR to manage the
replication of large amount of data over WAN with better
performances than AFS.
Participation in CASPUR storage technology tracking
activity (SAN systems and file systems testing and
evaluation, AFS implementation for parallel systems [IPP
Garching]=> DEISA Grid Project)
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
Connection with other GRID Projects
Each GRID project is characterized by its own architectural
design and middleware.
The focus of the partecipation of ENEA INFO in other GRID
projects is GRID interoperability
Access to ENEA GRID through other GRID models should
be compatible with ENEA GRID architecture: gateway
implementation.
Current GRID projects:
●
●
EGEE [Datagrid]
FIRB/Grid.it : WP13, Computational Chemistry
(CHEMGRID), Perugia Univ., ENEA Casaccia, et al.
ENEA activity starting July 2004 at Frascati
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA GRID and DATAGRID PROJECT
ENEA has partecipated (unfunded) in DATAGRID project
(ended 3/2004) through the collaboration with ESA, ESRIN
Laboratory in Frascati.
A gateway bewteen DATAGRID infrastructure and ENEAGRID has been set-up at ESRIN enabling the submission of
jobs from DATAGRID to ENEA-GRID
The gateway has been tested running the analysis of the
data from the GOME satellite (Ozone level) using IDL codes
which can be easily run in a multiplatform environment.
The gateway provided a limited integration between the
different GRID environment: EGEE should a attain a full
integration
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it
ENEA-GRID and EGEE Project
EGEE (Enabling GRID for e-science in Europe) is a project
in the 6th EU framework program, with the mission to
deliver production level GRID services [www.eu-egee.org].
ENEA is one of the funded partners in the project, inside
the Italian partecipation which is coordinated by INFN.
EGEE project is starting (April 2004); a kick-off Meeting
has been hold in Cork, April 2004.
The installation of a small testing environment (7 Linux
boxes) with standard EGEE middleware is currently in
progress at ENEA Frascati.
The development of a gateway to ENEA-GRID will follow.
GRID computing Frascati 14/10/2004
[email protected] www.telegrid.enea.it